Understanding the Guix approach when language package managers are around

Understanding the Guix approach when language package managers are around

[Daniel Sockwell]

Hi all,

(I also asked this question in r/guix; sorry if you're seeing it
twice.)

I'm trying to understand how Guix "wants" me to install software
that's in language package managers, such as npm, Leiningen, etc.
My current approach to installing some non-packages software
gives me the feeling that I'm fighting against Guix -- usually a
clear sign that I'm missing something.

Here's a specific example (though note that I'm asking about overall
approaches; I'm not currently seeking help debugging this specific
install).  I use a keyboard with open-source firmware and would like
to install the configuration utility, Chrysalis
(https://github.com/keyboardio/Chrysalis) for that firmware.

My first step, of course, was to check if Guix packages Chrysalis
(it's licensed under GPLv3, so we could); but we don't package it.
From the project `README`, I see that Chrysalis is distributed as an
AppImage, so on any other distro I'd use that.  However, I know that
AppImages don't cooperate well with Guix, so that's out.  The `README`
also shows that Chrysalis is a JavaScript program (actually an
Electron app), so the normal/non-Guix installation would be to run
`npm install` and go from there.

I can think of two ways to proceed:

  1. Install Node/npm and attempt to install Chrysalis just like I
  would on any other distro

  2. Attempt to package Chrysalis via Guix (probably using the
  `node-build-system`) and then install it normally

But both of these approaches give me the feeling that I'm Doing It
Wrong™.  With the first, I immediately hit errors about libraries not
being where Node expected.  I might be able to work around those (a
guide on the mailing list seems promising
https://lists.gnu.org/archive/html/help-guix/2018-06/msg00052.html).
But, even if I did, I'd be left with a package that is entirely
outside Guix's functional model -- which seems like a hint that this
isn't the Guix approach.

But looking into option 2 (packaging Chrysalis) doesn't seem promising
either.  My understanding is that packaging Chrysalis involves listing
its dependencies as `inputs`.  And that could be hard. Chrysalis has
~40 dependencies most of which aren't packaged for Guix.  And many of
those dependencies have their own transitive dependencies -- the full
graph seems to include 1,534 programs (for comparison, Guix currently
packages 51 `node-*` programs).  Given that I'm fairly new to Guix, I
very much doubt I should be trying to package a bunch of software.
Plus, even if I did, I doubt Guix would want to host an order of
magnitude more Node packages.  So, again, I get the sense that going
this route would be fighting against Guix instead of working with it.

So that's my question.  Are one of those approaches correct and I'm
just confused on the details?  Or is there some other approach to
installing this GPLv3 software that's the "real" Guix approach?  Or is
this sort of multi-dependency JavaScript app just a weak point for
Guix at the moment?

Thanks in advance for any insights you can provide.

Best regards,
Daniel

--
Daniel Sockwell / codesections
website:  www.codesections.com

Re: Understanding the Guix approach when language package managers are around

[Thompson, David]

Hi Daniel,

On Fri, Sep 16, 2022 at 10:48 AM Daniel Sockwell
<daniel@codesections.com> wrote:
>
> Hi all,
>
> (I also asked this question in r/guix; sorry if you're seeing it
> twice.)
>
> I'm trying to understand how Guix "wants" me to install software
> that's in language package managers, such as npm, Leiningen, etc.
> My current approach to installing some non-packages software
> gives me the feeling that I'm fighting against Guix -- usually a
> clear sign that I'm missing something.
>
> Here's a specific example (though note that I'm asking about overall
> approaches; I'm not currently seeking help debugging this specific
> install).  I use a keyboard with open-source firmware and would like
> to install the configuration utility, Chrysalis
> (https://github.com/keyboardio/Chrysalis) for that firmware.
>
> My first step, of course, was to check if Guix packages Chrysalis
> (it's licensed under GPLv3, so we could); but we don't package it.
> From the project `README`, I see that Chrysalis is distributed as an
> AppImage, so on any other distro I'd use that.  However, I know that
> AppImages don't cooperate well with Guix, so that's out.  The `README`
> also shows that Chrysalis is a JavaScript program (actually an
> Electron app), so the normal/non-Guix installation would be to run
> `npm install` and go from there.
>
> I can think of two ways to proceed:
>
>   1. Install Node/npm and attempt to install Chrysalis just like I
>   would on any other distro

This is the way I would recommend, if you can get it to work.

>   2. Attempt to package Chrysalis via Guix (probably using the
>   `node-build-system`) and then install it normally

As you explain below, here be dragons. :(

> But both of these approaches give me the feeling that I'm Doing It
> Wrong™.  With the first, I immediately hit errors about libraries not
> being where Node expected.  I might be able to work around those (a
> guide on the mailing list seems promising
> https://lists.gnu.org/archive/html/help-guix/2018-06/msg00052.html).
> But, even if I did, I'd be left with a package that is entirely
> outside Guix's functional model -- which seems like a hint that this
> isn't the Guix approach.

Yes, ideally, everything could be managed with Guix.  However, with
node, this is the best that you can do right now.

> But looking into option 2 (packaging Chrysalis) doesn't seem promising
> either.  My understanding is that packaging Chrysalis involves listing
> its dependencies as `inputs`.  And that could be hard. Chrysalis has
> ~40 dependencies most of which aren't packaged for Guix.  And many of
> those dependencies have their own transitive dependencies -- the full
> graph seems to include 1,534 programs (for comparison, Guix currently
> packages 51 `node-*` programs).  Given that I'm fairly new to Guix, I
> very much doubt I should be trying to package a bunch of software.
> Plus, even if I did, I doubt Guix would want to host an order of
> magnitude more Node packages.  So, again, I get the sense that going
> this route would be fighting against Guix instead of working with it.

Unfortunately, you're right.  Many have tried to untangle the mess,
but no one has succeeded.  Here's a blog post from 2015 that
illustrates some of the problems:

https://dustycloud.org/blog/javascript-packaging-dystopia/

Unfortunately, the problems are still the same in 2022.  Huge
dependency trees with likely multiple versions of *the same library*
in it, circular dependencies when "dev dependencies" (aka the stuff
you need to build from source) are taken into account, missing
licenses, binary blobs with no source, proprietary code, etc.  It's a
real mess.

> So that's my question.  Are one of those approaches correct and I'm
> just confused on the details?  Or is there some other approach to
> installing this GPLv3 software that's the "real" Guix approach?  Or is
> this sort of multi-dependency JavaScript app just a weak point for
> Guix at the moment?

Guix's highly reproducible, source-based approach is very much at odds
with the npm model of distribution which is effectively the polar
opposite.  If you can sort out your issues with making the npm tool
work on a Guix system, I would just go that route for lack of a
feasible alternative.

Sorry for the bad news!  Good luck!

- Dave

Re: Understanding the Guix approach when language package managers are around

[Dominic Martinez]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]


"Daniel Sockwell" <daniel@codesections.com> writes:

> I can think of two ways to proceed:
>
>   1. Install Node/npm and attempt to install Chrysalis just like I
>   would on any other distro
>
>   2. Attempt to package Chrysalis via Guix (probably using the
>   `node-build-system`) and then install it normally

You've pretty much hit the nail on the head; Guix's dependency model is
much stricter than any language package manager, so unless you put the
(potentially significant) effort to package something in Guix, you'll
likely have to use the language tool directly. The strength of Guix is
that it provides a single, consistent interface for packages, but it's
significantly at odds with the Javascript/Python ecosystems that evolved
around "download and run this mysterious blob".

For other languages guix import can help, but right now Guix does not
have a consistent way to package Javascript, which is why so few
packages are currently available.

I have a few fallbacks for when I can't work on a project in Guix:
1. Use Nix. Nix is more...liberal in their packaging, and so you still
lose out on the benefits of having packages defined from source, but you
can at least create a consistent working environment.
2. Use Docker. A docker container with a volume linked to your code is
almost always seamless.
3. The upcoming --emulate-fhs option. This isn't merged yet, but
you can build Guix with the following patches so that downloaded
binaries will see system libraries: https://issues.guix.gnu.org/56677

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Understanding the Guix approach when language package managers are around

[Daniel Sockwell]

David and Dominic,

Thanks to you both for your very helpful (though slightly
disappointing) replies.

September 16, 2022 11:36 AM, "Thompson, David" <dthompson2@worcester.edu> wrote:
> Unfortunately, you're right. Many have tried to untangle the mess,
> but no one has succeeded. Here's a blog post from 2015 that
> illustrates some of the problems:
> 
> https://dustycloud.org/blog/javascript-packaging-dystopia
> 
> Unfortunately, the problems are still the same in 2022.

That's too bad.  I'm glad to no longer be working primarily
in JS but there's a lot of great free software written in 
the language and missing out on easy access to it is a shame.

> [A JS project will often have huge] dependency trees with 
> likely multiple versions of *the same library* in it

But isn't that something that Guix is really *good* at handling?
My understanding is that one of Guix's selling points is that it
allows package A to depend on both B and C even if B and C depend
on different versions of D (aka, the diamond-dependency problem).
So maybe Guix would be a better JavaScript package manager than
NPM is!

(Or am I confused about what you meant or how Guix works?)


September 16, 2022 12:57 PM, "Dominic Martinez" <dom@dominicm.dev> wrote:
> [Guix] provides a single, consistent interface for packages, but it's
> significantly at odds with the Javascript/Python ecosystems that evolved
> around "download and run this mysterious blob".
> 
> For other languages guix import can help, but right now Guix does not
> have a consistent way to package Javascript, which is why so few
> packages are currently available.

Thanks, that makes sense.

> I have a few fallbacks for when I can't work on a project in Guix:
> ...
> 2. Use Docker. A docker container with a volume linked to your code is
> almost always seamless.

Ah, that's a good option.  I experimented with some of Guix's --container
options but didn't consider using docker directly.  I take it that's what
you do?  Do you use docker compose and/or do anything to make your docker
containers more functional/fit better with Guix? 

> 3. The upcoming --emulate-fhs option. This isn't merged yet, but
> you can build Guix with the following patches so that downloaded
> binaries will see system libraries: https://issues.guix.gnu.org/56677

Ooh, that looks *very* cool and might be exactly what I need!  It might
not solve the problem for JavaScript packages, where NPM really is 
diametrically opposed to the Guix approach.  But I spend most of my
time writing Raku these days, and Raku's approach to package management
is quite a lot like Guix's -- it even installs programs to immutable
directories named based on a hash, see https://docs.raku.org/language/compilation
So exposing just a few system libraries to Raku programs could be
perfect for getting the two package managers to work well together!

Thanks again to you both!

Best,
Daniel

Re: Understanding the Guix approach when language package managers are around

[Philip McGrath]

Hi,

On 9/16/22 09:15, Daniel Sockwell wrote:
> But looking into option 2 (packaging Chrysalis) doesn't seem promising
> either.  My understanding is that packaging Chrysalis involves listing
> its dependencies as `inputs`.  And that could be hard. Chrysalis has
> ~40 dependencies most of which aren't packaged for Guix.  And many of
> those dependencies have their own transitive dependencies -- the full
> graph seems to include 1,534 programs (for comparison, Guix currently
> packages 51 `node-*` programs).  Given that I'm fairly new to Guix, I
> very much doubt I should be trying to package a bunch of software.
> Plus, even if I did, I doubt Guix would want to host an order of
> magnitude more Node packages.  So, again, I get the sense that going
> this route would be fighting against Guix instead of working with it.
> 
> So that's my question.  Are one of those approaches correct and I'm
> just confused on the details?  Or is there some other approach to
> installing this GPLv3 software that's the "real" Guix approach?  Or is
> this sort of multi-dependency JavaScript app just a weak point for
> Guix at the moment?
> 
> Thanks in advance for any insights you can provide.
> 
> Best regards,
> Daniel
> 

I think Guix would *love* to distribute an order of magnitude more Node 
packages, if only we could figure out how to do so well.

As others have said, using NPM directly is probably the pragmatic thing 
to do for now.

For option 2, though, there is a `wip-node-importer` branch in the Guix 
repository which I have found useful. There has been discussion on the 
lists about merging it, most recently, I think, in e.g. 
<https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00073.html> 
and 
<https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00076.html>. I 
continue to think we should merge it!

I had several thoughts on improving JavaScript things while working on 
<https://issues.guix.gnu.org/51838> (be warned it's a massive thread), 
but I haven't needed to do much JavaScript since, so I'm a little fuzzy 
on some of the details. I know one idea was that we look closely at 
PNPM, an alternative to NPM that has a content-addressed store and other 
features that seem relevant to Guix, and try to implement something 
similar in Guile, which might also let us unbundle NPM from Node.js. 
Some pointers along those lines:

- https://pnpm.io/symlinked-node-modules-structure
- https://pnpm.io/how-peers-are-resolved
- https://pnpm.io/blog/2020/05/27/flat-node-modules-is-not-the-only-way
- 
https://medium.com/pnpm/pnpms-strictness-helps-to-avoid-silly-bugs-9a15fb306308
- https://pnpm.io/workspaces

-Philip

Re: Understanding the Guix approach when language package managers are around

[Dominic Martinez]

[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]


"Daniel Sockwell" <daniel@codesections.com> writes:

> But isn't that something that Guix is really *good* at handling?
> My understanding is that one of Guix's selling points is that it
> allows package A to depend on both B and C even if B and C depend
> on different versions of D (aka, the diamond-dependency problem).
> So maybe Guix would be a better JavaScript package manager than
> NPM is!
>
> (Or am I confused about what you meant or how Guix works?)

Guix is great at solving the diamond dependency problem, but exactly
representing that dependency structure implies an exponential number of
package definitions. Usually packagers will try to find the minimal
number of package versions that satisfy the requirements of their
dependents, but when having multiple major versions of packages in your
dependency tree is practically a given, it's a lot of duplication.

Also consider that by default, module resolution is tied to the folder
structure of 'node_modules', so as Philip said we'd have to re-implement
PNPM's logic:
https://pnpm.io/motivation
https://npm.github.io/how-npm-works-docs/npm3/non-determinism.html
https://npm.github.io/how-npm-works-docs/npm3/duplication.html

>> I have a few fallbacks for when I can't work on a project in Guix:
>> ...
>> 2. Use Docker. A docker container with a volume linked to your code is
>> almost always seamless.
>
> Ah, that's a good option.  I experimented with some of Guix's --container
> options but didn't consider using docker directly.  I take it that's what
> you do?  Do you use docker compose and/or do anything to make your docker
> containers more functional/fit better with Guix?

If I'm just trying to have a more standard Linux environment I usually
don't feel the need to use docker compose. You can just have a script
that builds your environment, and another that launches the container in
a directory:

$ docker build -t <tag> - < <Dockerfile> # Build the image
$ docker run -it --rm -v ${1}:/src <tag> # Run the image, bringing in a directory

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]