Adding extra package importers

Adding extra package importers

[pinoaffe]

Hi guix,

Recently, I made an effort to get the guix-npm binary importer working
in combination with a relatively "modern" guix - my changes are rather
hacky and I think I may have broken a thing or two, but i got it to work
reasonably well in the REPL.
(many thanks to Jelle Licht and possibly others who initially wrote the importer)

Considering that such a binary importer is probably not well-suited for
inclusion in guix-proper, I would like to make it available through an
additional channel, mostly for my own use but maybe also for others.

However, the commandline interface for ~guix import~ operates based on a
list of importers in (guix scripts import) in guix proper - is there a
way to replace this functionality so that additional guix channels could
"extend" the list of importers?

Additionally, does anyone have an idea as to how to deal with the fact
that a single recursive import will often add several versions of the
same package? The only thing I could come up with would be to always
append the full package version to variable names, but that seems like a
somewhat ugly fix.

My current version of the code can be seen at
https://git.pixie.town/pinoaffe/guix-npm-binary

Kind regards,
pinoaffe

Re: Adding extra package importers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

pinoaffe schreef op zo 19-09-2021 om 15:01 [+0200]:
> Hi guix,
> 
> Recently, I made an effort to get the guix-npm binary importer working
> in combination with a relatively "modern" guix - my changes are rather
> hacky and I think I may have broken a thing or two, but i got it to work
> reasonably well in the REPL.
> (many thanks to Jelle Licht and possibly others who initially wrote the importer)
> 
> Considering that such a binary importer is probably not well-suited for
> inclusion in guix-proper, I would like to make it available through an
> additional channel, mostly for my own use but maybe also for others.
> 
> However, the commandline interface for ~guix import~ operates based on a
> list of importers in (guix scripts import) in guix proper - is there a
> way to replace this functionality so that additional guix channels could
> "extend" the list of importers?

(guix scripts import) could scan the subdirectories guix/scripts/import of
%load-path for importers.  See 'importer-modules' and '%updaters' in
(guix upstream) for how to do this.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Merging the “binary” NPM importer?

[Ludovic Courtès]

Hello!

pinoaffe <pinoaffe@airmail.cc> skribis:

> Recently, I made an effort to get the guix-npm binary importer working
> in combination with a relatively "modern" guix - my changes are rather
> hacky and I think I may have broken a thing or two, but i got it to work
> reasonably well in the REPL.
> (many thanks to Jelle Licht and possibly others who initially wrote the importer)
>
> Considering that such a binary importer is probably not well-suited for
> inclusion in guix-proper, I would like to make it available through an
> additional channel, mostly for my own use but maybe also for others.

We’ll have to check what maintainers think, but I’ve come to think that
we could include it in Guix proper, while of course properly documenting
its shortcomings.

We would probably not include packages made with it in Guix proper, but
it can still be useful to users who want a way to manage free JS code
with Guix.  And on top of that, it already exists.  :-)

Thoughts?

Ludo’.

Re: Merging the “binary” NPM importer?

[pinoaffe]

Hi!

Ludovic Courtès writes:
> We’ll have to check what maintainers think, but I’ve come to think that
> we could include it in Guix proper, while of course properly documenting
> its shortcomings.
>
> We would probably not include packages made with it in Guix proper, but
> it can still be useful to users who want a way to manage free JS code
> with Guix.  And on top of that, it already exists.  :-)

Having it in Guix proper would be great!

I do however think that quite a couple of improvements would be
necessary before this can be merged

kind regards,
pinoaffe

Re: Merging the “binary” NPM importer?

[Christine Lemmer-Webber]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> pinoaffe <pinoaffe@airmail.cc> skribis:
>
>> Recently, I made an effort to get the guix-npm binary importer working
>> in combination with a relatively "modern" guix - my changes are rather
>> hacky and I think I may have broken a thing or two, but i got it to work
>> reasonably well in the REPL.
>> (many thanks to Jelle Licht and possibly others who initially wrote the importer)
>>
>> Considering that such a binary importer is probably not well-suited for
>> inclusion in guix-proper, I would like to make it available through an
>> additional channel, mostly for my own use but maybe also for others.
>
> We’ll have to check what maintainers think, but I’ve come to think that
> we could include it in Guix proper, while of course properly documenting
> its shortcomings.
>
> We would probably not include packages made with it in Guix proper, but
> it can still be useful to users who want a way to manage free JS code
> with Guix.  And on top of that, it already exists.  :-)
>
> Thoughts?
>
> Ludo’.

I think it's a good idea to include it.  It's likely we'd see some
channels appear which start incorporating such things, then.  But of
course we would not want those packages in Guix proper.  But it could be
a "path in".

I seem to remember my time in Debian being that ending up in "contrib"
(or worse, non-free) felt like, okay, now Debian users have an easier
way to use this, but it doesn't feel good.  This seemed to result in a
lot of pressure to end up in Debian main, and lots of packages seem to
have made changes to become eligible to do that.  I think "getting in
Guix proper" likewise can be a positive incentive.

Re: Merging the “binary” NPM importer?

[Jelle Licht]

Hello there,

pinoaffe <pinoaffe@airmail.cc> writes:

> Hi!
<snip>
> I do however think that quite a couple of improvements would be
> necessary before this can be merged

The major ones I can think of are;
- Be rebased on current master (or perhaps core-updates-fixed?)
- tests
- documentation

Could you perhaps list some additional blockers?  We can always refine
and improve on an acceptable upstreamed version, since it will be a
once-and-done tool instead of some stable API.

Thanks,
- Jelle

Re: Merging the “binary” NPM importer?

[pinoaffe]

Hi!

Jelle Licht writes:
> pinoaffe <pinoaffe@airmail.cc> writes:
>> I do however think that quite a couple of improvements would be
>> necessary before this can be merged
> The major ones I can think of are;
> - Be rebased on current master (or perhaps core-updates-fixed?)
> - tests
> - documentation
>
> Could you perhaps list some additional blockers?  We can always refine
> and improve on an acceptable upstreamed version, since it will be a
> once-and-done tool instead of some stable API.

I think one would also need to deal with the fact that a recursive
import is likely to include several versions of the same package,

and I think rebasing might end up requiring a bit of rewriting since it
appears to me that some of the interfaces used in the guix-npm-binary
importer have changed since it was written

Kind regards,
pinoaffe

Re: Merging the “binary” NPM importer?

[Philip McGrath]

Hi,

I've been using the NPM binary importer recently, and I'm also in favor 
of merging it. Aside from helping me get practical work done, it's also 
pointed me toward things in the Node package and node-build-system that 
may need to be tweaked (e.g. https://issues.guix.gnu.org/49946#71). So, 
like Christine, I hope that letting people use Guix to work on Node/NPM 
stuff would ultimately help the cause of making NPM packages work in 
Guix the right way.

On 9/26/21 5:34 PM, pinoaffe wrote:
> Jelle Licht writes:
>> Could you perhaps list some additional blockers?  We can always refine
>> and improve on an acceptable upstreamed version, since it will be a
>> once-and-done tool instead of some stable API.
> 
> I think one would also need to deal with the fact that a recursive
> import is likely to include several versions of the same package,

This did seem to happen with a large package I imported, but the 
multiple versions coexisting seemed to work ok. Ultimately, Guix will 
somehow need to deal with the messy versioning of the NPM world, and it 
would be nice to keep the duplication to a minimum, but that seems like 
it might be a challenge. Since the recursive imports do currently work, 
I don't think it needs to be a blocker.

> and I think rebasing might end up requiring a bit of rewriting since it
> appears to me that some of the interfaces used in the guix-npm-binary
> importer have changed since it was written

Here's the version I've been using, rebased on August 3: 
<https://gitlab.com/philip1/guix-patches/-/commit/7b8732bbfcacd52b2ea823358cff842a891b51b2>

One issue I did notice is that the export `package-json->guix-package` 
seems to be undefined.

-Philip

Re: Merging the “binary” NPM importer?

[pinoaffe]

Hi, 

Philip McGrath writes:
> This did seem to happen with a large package I imported, but the
> multiple versions coexisting seemed to work ok.
Oh okay, it didn't really seem to work in my tree - yet another argument
for merging it, in order to unify it :)

> Ultimately, Guix will somehow need to deal with the messy versioning
> of the NPM world, and it would be nice to keep the duplication to a
> minimum, but that seems like it might be a challenge.
I think it would be great if at some point we could somehow encode
version constraints for dependencies in guix packages, as even if one
were to properly package a bunch of npm packages, figuring out whether
one can safely upgrade one of them without breaking any of the
dependents would be quite a hassle (or would need to involve running a
lot of builds to see whether they fail).

Kind regards,
pinoaffe

Re: Merging the “binary” NPM importer?

[Katherine Cox-Buday]

pinoaffe <pinoaffe@airmail.cc> writes:

> Hi, 
>
> Philip McGrath writes:
>> This did seem to happen with a large package I imported, but the
>> multiple versions coexisting seemed to work ok.
> Oh okay, it didn't really seem to work in my tree - yet another argument
> for merging it, in order to unify it :)

+1, if this works. These packages aren't meant to end up in Guix anyway, so if there's a lot of duplicates, it only harms the local store, or a channel.

Maybe someday Guix can untangle the JS ecosystem knot, but in the meantime this unblocks folks.

>
>> Ultimately, Guix will somehow need to deal with the messy versioning
>> of the NPM world, and it would be nice to keep the duplication to a
>> minimum, but that seems like it might be a challenge.
> I think it would be great if at some point we could somehow encode
> version constraints for dependencies in guix packages, as even if one
> were to properly package a bunch of npm packages, figuring out whether
> one can safely upgrade one of them without breaking any of the
> dependents would be quite a hassle (or would need to involve running a
> lot of builds to see whether they fail).

This sounds neat. And maybe pre-defined rules, e.g. semantic versioning?

Good luck!

-- 
Katherine

Re: Merging the “binary” NPM importer?

[Ludovic Courtès]

Hello!

(Cc: guix-maintainers to check whether everyone is on-board with the
idea before going further.)

Jelle Licht <jlicht@fsfe.org> skribis:

> The major ones I can think of are;
> - Be rebased on current master (or perhaps core-updates-fixed?)
> - tests
> - documentation

Yeah, these three things would be needed to get it in ‘master’.  A quick
review would also be nice, mostly to catch stylistic issue, code
duplication, and to validate the testing strategy.

Would the two of you willing to help with this?

> Could you perhaps list some additional blockers?  We can always refine
> and improve on an acceptable upstreamed version, since it will be a
> once-and-done tool instead of some stable API.

Once the basics are in place, you could always improve the tool later on
as you see fit.

Thanks,
Ludo’.

Re: Merging the “binary” NPM importer?

[Maxim Cournoyer]

Hey,

Christine Lemmer-Webber <cwebber@dustycloud.org> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello!
>>
>> pinoaffe <pinoaffe@airmail.cc> skribis:
>>
>>> Recently, I made an effort to get the guix-npm binary importer working
>>> in combination with a relatively "modern" guix - my changes are rather
>>> hacky and I think I may have broken a thing or two, but i got it to work
>>> reasonably well in the REPL.
>>> (many thanks to Jelle Licht and possibly others who initially wrote the importer)
>>>
>>> Considering that such a binary importer is probably not well-suited for
>>> inclusion in guix-proper, I would like to make it available through an
>>> additional channel, mostly for my own use but maybe also for others.
>>
>> We’ll have to check what maintainers think, but I’ve come to think that
>> we could include it in Guix proper, while of course properly documenting
>> its shortcomings.
>>
>> We would probably not include packages made with it in Guix proper, but
>> it can still be useful to users who want a way to manage free JS code
>> with Guix.  And on top of that, it already exists.  :-)
>>
>> Thoughts?

I'm not too keen on having an importer which produces packages that
can't be included in Guix proper -- it seems a double standard to me.
I'd personally prefer to have such tool maintained outside of Guix
proper.

My 2 cents.

Maxim

Re: Merging the “binary” NPM importer?

[Katherine Cox-Buday]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I'm not too keen on having an importer which produces packages that
> can't be included in Guix proper -- it seems a double standard to me.
> I'd personally prefer to have such tool maintained outside of Guix
> proper.
>
> My 2 cents.

I disagree with this because the benefit of Guix can extend beyond the packages it provides. It is a computing environment and not the sum of its parts.

I don't always have full control over my computing environments. There are certain contexts, say my job, that require that I run binaries for which I don't have source code. Guix having the flexibility to support me in these contexts allows me to keep using, contributing, and supporting Guix. I.e., there's a network effect.

In my opinion, Guix should be the gentle current towards its free ecosystem. But it should also acknowledge the wider world in which it exists, and be supportive of its users in those contexts too. I don't think this in any way diminishes the strong principles it adheres to.

What do you think?

-- 
Katherine

Re: Merging the “binary” NPM importer?

[zimoun]

Hi,

On Fri, 8 Oct 2021 at 16:17, Katherine Cox-Buday
<cox.katherine.e@gmail.com> wrote:
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> > I'm not too keen on having an importer which produces packages that
> > can't be included in Guix proper -- it seems a double standard to me.
> > I'd personally prefer to have such tool maintained outside of Guix
> > proper.
>
> I disagree with this because the benefit of Guix can extend beyond the packages it provides. It is a computing environment and not the sum of its parts.

Instead of a strong inclusion, one could imagine this importer as a
regular package using GUIX_EXTENSION_PATH.  For instance, it could be:

   guix install guix-import-npm

then something like:

  guix import-npm <options>

This just works, modulo some polishing and doc. :-)  For an example,
see the package gwl.  An example using another strategy is using a
channel, for instance
<https://framagit.org/tyreunom/guix-home-manager>.

I have not checked if this "extension" system could work for
subcommands as the importers, i.e., having something like:

   guix import npm

where this 'npm' would not be part of Guix proper (not in
guix/import/) but instead would be a package extending Guix.

This seems road to take, IMHO.

WDYT?


> I don't always have full control over my computing environments. There are certain contexts, say my job, that require that I run binaries for which I don't have source code. Guix having the flexibility to support me in these contexts allows me to keep using, contributing, and supporting Guix. I.e., there's a network effect.
>
> In my opinion, Guix should be the gentle current towards its free ecosystem. But it should also acknowledge the wider world in which it exists, and be supportive of its users in those contexts too. I don't think this in any way diminishes the strong principles it adheres to.

I agree.  However, instead of include all directly in Guix proper, I
think it is better to make experiments outside Guix proper and in the
same keep the Guix tooling at hand.  And from my understanding, it
means extend Guix. :-)  If it shows it is worth, then let merge to
Guix.

Cheers,
simon

Re: Merging the “binary” NPM importer?

[pinoaffe]

Hi,

zimoun writes:
> Instead of a strong inclusion, one could imagine this importer as a
> regular package using GUIX_EXTENSION_PATH.  For instance, it could be:
>
>    guix install guix-import-npm
>
> then something like:
>
>   guix import-npm <options>
>
> This just works, modulo some polishing and doc. :-)  For an example,
> see the package gwl.  An example using another strategy is using a
> channel, for instance
> <https://framagit.org/tyreunom/guix-home-manager>.
>
> I have not checked if this "extension" system could work for
> subcommands as the importers, i.e., having something like:
>
>    guix import npm
>
> where this 'npm' would not be part of Guix proper (not in
> guix/import/) but instead would be a package extending Guix.

This would currently not work, but I've submitted a patch that would
make it work (with some performance concerns), and I think it should be
possible to resolve or at least significantly lessen those concerns.

However, I think inclusion in guix proper would encourage usage and make
maintenance easier (since maintaining two tightly coupled but separate
codebases is more work than maintaining a single codebase with the same
functionality)

While I would not be opposed to a package- or channel-based approach
(I initially set out to make a guix-npm-binary channel afterall),
merging it into guix-proper would definitely have my preference.

Kind regards,
pinoaffe

Re: Merging the “binary” NPM importer?

[Timothy Sample]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I'm not too keen on having an importer which produces packages that
> can't be included in Guix proper -- it seems a double standard to me.
> I'd personally prefer to have such tool maintained outside of Guix
> proper.

The reason I called it the “NPM Binary Importer” was to scare people
away, because its results are rather limited.  Maybe I went too far,
though, and people are too scared!  :)

It’s more accurately the “Half-finished NPM Importer”.  It produces
packages that are missing two important parts: the dependencies needed
to build the package; and the upstream source code location.  Getting
the dependencies is quite hard, since NPM doesn’t differentiate between
build tools, testing frameworks, linters, scripts for publishing, etc.
This results in a ton of circular dependencies since the publishing
scripts use linters, and the linters use build tools, and the build
tools use publishing scripts, etc., etc.  We would need some sort of
block list to get around this (e.g., we should just skip over stuff like
ESLint).  Getting the source code can be tricky, too.  Normally, NPM
packages have a “repository” field that points to the source code
repository.  However, in some cases, packages use a monorepo setup where
many packages are produced from one repository.  This complicates the
correspondence between source code repositories and NPM packages.

What the “binary” importer does is ignore all the “development”
dependencies, use the NPM tarball as the package source, and set the
‘#:build?’ argument to false.  This is a hack to get the half-finished
importer to produce working (even if unpleasant) packages.

I write all this to make it clear that the fact the importer produces
“binary” packages is not the result of a “who cares about free software
or source code?” attitude!  :)  It just seemed like a reasonable
checkpoint between nothing and a Guix-ready NPM importer.

I think keeping the half-finished importer alive by including it in Guix
might be a good strategy.  It might help us work – collectively –
towards the goal of providing a clean, freedom respecting collection of
JavaScript packages in Guix.

Of course, I do respect and understand your concerns.  After all, I
called it the “binary” importer to be as scary as possible!  In the end,
it’s up to you and the other maintainers – hopefully my explanations are
helpful.


-- Tim

Re: Merging the “binary” NPM importer?

[Maxim Cournoyer]

Hello Timothy,

Timothy Sample <samplet@ngyro.com> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I'm not too keen on having an importer which produces packages that
>> can't be included in Guix proper -- it seems a double standard to me.
>> I'd personally prefer to have such tool maintained outside of Guix
>> proper.
>
> The reason I called it the “NPM Binary Importer” was to scare people
> away, because its results are rather limited.  Maybe I went too far,
> though, and people are too scared!  :)
>
> It’s more accurately the “Half-finished NPM Importer”.  It produces
> packages that are missing two important parts: the dependencies needed
> to build the package; and the upstream source code location.  Getting
> the dependencies is quite hard, since NPM doesn’t differentiate between
> build tools, testing frameworks, linters, scripts for publishing, etc.
> This results in a ton of circular dependencies since the publishing
> scripts use linters, and the linters use build tools, and the build
> tools use publishing scripts, etc., etc.  We would need some sort of
> block list to get around this (e.g., we should just skip over stuff like
> ESLint).  Getting the source code can be tricky, too.  Normally, NPM
> packages have a “repository” field that points to the source code
> repository.  However, in some cases, packages use a monorepo setup where
> many packages are produced from one repository.  This complicates the
> correspondence between source code repositories and NPM packages.
>
> What the “binary” importer does is ignore all the “development”
> dependencies, use the NPM tarball as the package source, and set the
> ‘#:build?’ argument to false.  This is a hack to get the half-finished
> importer to produce working (even if unpleasant) packages.
>
> I write all this to make it clear that the fact the importer produces
> “binary” packages is not the result of a “who cares about free software
> or source code?” attitude!  :)  It just seemed like a reasonable
> checkpoint between nothing and a Guix-ready NPM importer.
>
> I think keeping the half-finished importer alive by including it in Guix
> might be a good strategy.  It might help us work – collectively –
> towards the goal of providing a clean, freedom respecting collection of
> JavaScript packages in Guix.
>
> Of course, I do respect and understand your concerns.  After all, I
> called it the “binary” importer to be as scary as possible!  In the end,
> it’s up to you and the other maintainers – hopefully my explanations are
> helpful.

Thanks for sharing all this background information!  It helped a lot to
refine my current understanding of where this importer came from and
where it's heading; I must confess I hadn't reviewed the importer
details and the idea I had of it was mostly attributable to its scary
name :-).

In light of your message though, I think it may be fine to have the
importer in Guix as long as it is labeled as a technology preview, with
its current shortcomings documented :-).  This may make it more
accessible and allow to refine it faster.

Thanks again,

Maxim

Re: Merging the “binary” NPM importer?

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> In light of your message though, I think it may be fine to have the
> importer in Guix as long as it is labeled as a technology preview, with
> its current shortcomings documented :-).  This may make it more
> accessible and allow to refine it faster.

Yes, I agree that the shortcomings should appear prominently in
documentation, in particular the fact that the output does not meet (or
not always) the standards for inclusion in Guix proper.

It’s an unusual situation, but it seems that “modern” development
practices make it hard or impossible to meet our standards in the first
place; yet, we’re missing out on a whole range of free software packages
by not doing anything.  Offering the tool while not compromising on our
standards seems like a reasonable middle ground.

(It’s probably more than a “technology preview” though, in the sense
that the code itself does what it’s meant to do and is (hopefully :-))
well-tested.)

So whoever is interested in it can prepare patches!  :-)

Ludo’.

Opining on "modern" development practices (was Re: Merging the “binary” NPM importer?)

[Katherine Cox-Buday]

Ludovic Courtès <ludo@gnu.org> writes:

> It’s an unusual situation, but it seems that “modern” development
> practices make it hard or impossible to meet our standards in the first
> place; yet, we’re missing out on a whole range of free software packages
> by not doing anything.  Offering the tool while not compromising on our
> standards seems like a reasonable middle ground.

I think this is yet another example of the "worse is better"[1] debate, seemingly still ongoing in the world, thirty years later.

I don't have much practical to say on the subject, but a few things have often occurred to me which someone may find useful or interesting to ruminate on:

1. The premise of the "worse is better" philosophy seems to me to have been
   proven true. Development tools and environments which are easier to get,
   start using, and distribute, proliferate. And these communities produce the
   most software. As you pointed out, some of the software itself is free and
   useful.

2. Sometimes these ecosystems (e.g. Javascript) are so volatile, bad things fall
   out. It is difficult to stay abreast of changes, there are security issues
   (e.g. tainting a very common dependency, bootstrapping issues, etc),
   maintenance issues, and lots of wasted effort rewriting things. Still, a
   large percentage of developers' time and energy goes into that ecosystem
   because of point one, and they create useful things.

3. Sometimes these ecosystems are so volatile, good things fall out. Through the
   lens of experience, solutions and tools are created which address the hard
   won lessons.

4. This seems to be how nature and evolution work.

Me? I like well-ordered things that have been thoughtfully produced. But I think about number four a lot.

[1] - https://en.wikipedia.org/wiki/Worse_is_better

-- 
Katherine

Re: Opining on "modern" development practices (was Re: Merging the “binary” NPM importer?)

[Ludovic Courtès]

Hi Katherine,

Katherine Cox-Buday <cox.katherine.e@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> It’s an unusual situation, but it seems that “modern” development
>> practices make it hard or impossible to meet our standards in the first
>> place; yet, we’re missing out on a whole range of free software packages
>> by not doing anything.  Offering the tool while not compromising on our
>> standards seems like a reasonable middle ground.
>
> I think this is yet another example of the "worse is better"[1] debate, seemingly still ongoing in the world, thirty years later.
>
> I don't have much practical to say on the subject, but a few things have often occurred to me which someone may find useful or interesting to ruminate on:
>
> 1. The premise of the "worse is better" philosophy seems to me to have been
>    proven true. Development tools and environments which are easier to get,
>    start using, and distribute, proliferate. And these communities produce the
>    most software. As you pointed out, some of the software itself is free and
>    useful.
>
> 2. Sometimes these ecosystems (e.g. Javascript) are so volatile, bad things fall
>    out. It is difficult to stay abreast of changes, there are security issues
>    (e.g. tainting a very common dependency, bootstrapping issues, etc),
>    maintenance issues, and lots of wasted effort rewriting things. Still, a
>    large percentage of developers' time and energy goes into that ecosystem
>    because of point one, and they create useful things.
>
> 3. Sometimes these ecosystems are so volatile, good things fall out. Through the
>    lens of experience, solutions and tools are created which address the hard
>    won lessons.
>
> 4. This seems to be how nature and evolution work.
>
> Me? I like well-ordered things that have been thoughtfully produced. But I think about number four a lot.

I do too.  :-)

My early free software experience was that of a project managed in
typical MIT style: the Hurd; I learned a lot from that.

In Guix, I think we’ve always tried from Day 1 to do the Right Thing,
but also from Day 1, we’ve always tried not to go too far and to “cut
corners” when doing the Right Thing would have jeopardized practicality.

The package simplification work that landed this summer in
‘core-updates’ is an example of a case where the Right Thing was delayed
for several years because it just wasn’t attainable in a timely fashion
back then.

Merging the “binary” npm importer IMO is one way to acknowledge that
there’s a very concrete use case, that we’re unable to address it the
Right Way, but that we offer a middle ground for users.

Ludo’.

Re: Opining on "modern" development practices (was Re: Merging the “binary” NPM importer?)

[Katherine Cox-Buday]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Katherine,
>
> Katherine Cox-Buday <cox.katherine.e@gmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> It’s an unusual situation, but it seems that “modern” development
>>> practices make it hard or impossible to meet our standards in the first
>>> place; yet, we’re missing out on a whole range of free software packages
>>> by not doing anything.  Offering the tool while not compromising on our
>>> standards seems like a reasonable middle ground.
>>
>> I think this is yet another example of the "worse is better"[1] debate,
>> seemingly still ongoing in the world, thirty years later.
>>
>> I don't have much practical to say on the subject, but a few things have often
>> occurred to me which someone may find useful or interesting to ruminate on:
>>
>> 1. The premise of the "worse is better" philosophy seems to me to have been
>>    proven true. Development tools and environments which are easier to get,
>>    start using, and distribute, proliferate. And these communities produce the
>>    most software. As you pointed out, some of the software itself is free and
>>    useful.
>>
>> 2. Sometimes these ecosystems (e.g. Javascript) are so volatile, bad things fall
>>    out. It is difficult to stay abreast of changes, there are security issues
>>    (e.g. tainting a very common dependency, bootstrapping issues, etc),
>>    maintenance issues, and lots of wasted effort rewriting things. Still, a
>>    large percentage of developers' time and energy goes into that ecosystem
>>    because of point one, and they create useful things.
>>
>> 3. Sometimes these ecosystems are so volatile, good things fall out. Through the
>>    lens of experience, solutions and tools are created which address the hard
>>    won lessons.
>>
>> 4. This seems to be how nature and evolution work.
>>
>> Me? I like well-ordered things that have been thoughtfully produced. But I think about number four a lot.
>
> I do too.  :-)

Think about #4, or like well-ordered things?

> My early free software experience was that of a project managed in
> typical MIT style: the Hurd; I learned a lot from that.

Have you written about this anywhere? I'd love to hear your opinions.

> In Guix, I think we’ve always tried from Day 1 to do the Right Thing,
> but also from Day 1, we’ve always tried not to go too far and to “cut
> corners” when doing the Right Thing would have jeopardized practicality.
>
> The package simplification work that landed this summer in
> ‘core-updates’ is an example of a case where the Right Thing was delayed
> for several years because it just wasn’t attainable in a timely fashion
> back then.

I have been trying to follow the project more closely for the past few months, but I have completely missed this simplification work. Is there somewhere I can read about it? Scroll back through guix-patches maybe?

> Merging the “binary” npm importer IMO is one way to acknowledge that
> there’s a very concrete use case, that we’re unable to address it the
> Right Way, but that we offer a middle ground for users.

Thanks for these comments, and the maintainers' stewardship of Guix. I believe taking the middle way is usually the correct path, and I think Guix has benefited from it, and will continue to do so.

-- 
Katherine

Re: Opining on "modern" development practices (was Re: Merging the “binary” NPM importer?)

[Ludovic Courtès]

Hi,

Katherine Cox-Buday <cox.katherine.e@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>> 4. This seems to be how nature and evolution work.
>>>
>>> Me? I like well-ordered things that have been thoughtfully produced. But I think about number four a lot.
>>
>> I do too.  :-)
>
> Think about #4, or like well-ordered things?

Both.

>> My early free software experience was that of a project managed in
>> typical MIT style: the Hurd; I learned a lot from that.
>
> Have you written about this anywhere? I'd love to hear your opinions.

Nope!  You’ve probably noticed that the Hurd isn’t known as a lively and
successful project, and then there are things like the Critique¹ or work
on porting the Hurd to the L4 microkernel which, while showing the
“right” direction research-wise, have also had IMO the effect of
suggesting that it’s never going to be good enough.

I think a project needs to be aware of its shortcomings, but it also
needs achievable milestones, and it needs to refrain from elitism.

¹ http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf

>> In Guix, I think we’ve always tried from Day 1 to do the Right Thing,
>> but also from Day 1, we’ve always tried not to go too far and to “cut
>> corners” when doing the Right Thing would have jeopardized practicality.
>>
>> The package simplification work that landed this summer in
>> ‘core-updates’ is an example of a case where the Right Thing was delayed
>> for several years because it just wasn’t attainable in a timely fashion
>> back then.
>
> I have been trying to follow the project more closely for the past few months, but I have completely missed this simplification work. Is there somewhere I can read about it? Scroll back through guix-patches maybe?

See <https://issues.guix.gnu.org/49169>.

Thanks,
Ludo’.

Re: Opining on "modern" development practices (was Re: Merging the “binary” NPM importer?)

[Katherine Cox-Buday]

Ludovic Courtès <ludo@gnu.org> writes:

> Nope!  You’ve probably noticed that the Hurd isn’t known as a lively and
> successful project, and then there are things like the Critique¹ or work
> on porting the Hurd to the L4 microkernel which, while showing the
> “right” direction research-wise, have also had IMO the effect of
> suggesting that it’s never going to be good enough.
>
> I think a project needs to be aware of its shortcomings, but it also
> needs achievable milestones, and it needs to refrain from elitism.
>
> ¹ http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf

Yes, I'm generally aware of the challenges the Hurd (and other micro kernels) have had taking off. I'm not clear on whether this is because monolithic kernels are intrinsically better (is this yet another example of worse is better?), or because of an organizational expression of worse is better.

I happened to be watching John Carmack speak last night. I haven't watched many of his talks, but he seems like a consummate pragmatist. I've always admired people like this because it's so opposite of my own idealistic tendencies. One thing he said in particular seems relevant:

https://youtu.be/BnSUk0je6oo?t=883
"The metaverse is a honeypot trap for astronaut architects."

I.e. he was arguing against trying to build (read: architect) the metaverse because it's a great way to either not get anything useful done, or to get the wrong thing done.

I guess I've been circling around this philosophy for awhile, but I think I'm going to let it cross over into being my working theory of how to build useful software (among other things in life): worse is better, and collect as many forcing factors as you can, as early as you can. I wonder how I can apply this to Guix.

> See <https://issues.guix.gnu.org/49169>.

Thank you for the link!

-- 
Katherine