On merging the npm importer

On merging the npm importer

[Jan Nieuwenhuizen]

Hi,

We have a working importer for npm packages written by Jelle that I have
been using for about half a year.  It can use some improvements and
that's why I think we should merge it.

Have alook at my npm branch here, rebased on master

    https://gitlab.com/janneke/guix

I added a patch with several fixes for the importer and and build
system.  So far, so good.

There's a problem however with the --recursive option and the build
system.  To quote Jelle[1]

   To start of with something that did not work out as well as I had
   hoped, getting a popular build system (e.g. Gulp, Grunt, Broccoli and
   others) packaged.  As mentioned in my earlier mails, the list of
   transitive dependencies of any of these suffer from at least the
   following:

   - It is a list with more than 4000 packages on it
   - It is a list with at some point the package itself on it

Most nontrivial npm packages use a build system, and all build systems
have circular development dependencies.  Not all development
dependencies are always required to build a package, but some certainly
are nd there's no way to tell which is which, afaik.

That's why I added a --binary option to the importer: it will not
try to use the build system and instead mimick `what npm does.'  This
does provide, however, an amazing reproducibility feature to the
dependency woes that npm hackers are familar with.

I suggest to not add any npm package to Guix that is the result of using
the --binary option and to build a base of full-source/sanitized npm
packages.

Greetings, janneke

[1] https://lists.gnu.org/archive/html/guix-devel/2016-08/msg01567.html

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: On merging the npm importer

[Christopher Allan Webber]

Jan Nieuwenhuizen writes:

> Hi,

Hi Jan!

> We have a working importer for npm packages written by Jelle that I have
> been using for about half a year.  It can use some improvements and
> that's why I think we should merge it.
>
> Have alook at my npm branch here, rebased on master
>
>     https://gitlab.com/janneke/guix

Would like to review soon, though I'll say that I think unless there are
serious problems, we should probably merge it.  Avoiding bitrot is prety
important, and at the very least I don't think it will hurt to have it
merged.

> I added a patch with several fixes for the importer and and build
> system.  So far, so good.
>
> There's a problem however with the --recursive option and the build
> system.  To quote Jelle[1]
>
>    To start of with something that did not work out as well as I had
>    hoped, getting a popular build system (e.g. Gulp, Grunt, Broccoli and
>    others) packaged.  As mentioned in my earlier mails, the list of
>    transitive dependencies of any of these suffer from at least the
>    following:
>
>    - It is a list with more than 4000 packages on it
>    - It is a list with at some point the package itself on it
>
> Most nontrivial npm packages use a build system, and all build systems
> have circular development dependencies.  Not all development
> dependencies are always required to build a package, but some certainly
> are nd there's no way to tell which is which, afaik.
>
> That's why I added a --binary option to the importer: it will not
> try to use the build system and instead mimick `what npm does.'  This
> does provide, however, an amazing reproducibility feature to the
> dependency woes that npm hackers are familar with.
>
> I suggest to not add any npm package to Guix that is the result of using
> the --binary option and to build a base of full-source/sanitized npm
> packages.

Cool... makes sense to me to have this as something we don't use for
Guix packages, but which might make Guix more useful for people who have
to use npm in the awkward "real world" that is the current state of npm.

> Greetings, janneke
>
> [1] https://lists.gnu.org/archive/html/guix-devel/2016-08/msg01567.html

Re: On merging the npm importer

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 3057 bytes --]

2017-03-29 19:39 GMT+02:00 Christopher Allan Webber <cwebber@dustycloud.org>
:

> Jan Nieuwenhuizen writes:
>
> > Hi,
>
> Hi Jan!
>

Hello Jan and Christopher!

>
> > We have a working importer for npm packages written by Jelle that I have
> > been using for about half a year.  It can use some improvements and
> > that's why I think we should merge it.
> >
> > Have alook at my npm branch here, rebased on master
> >
> >     https://gitlab.com/janneke/guix
>
> Would like to review soon, though I'll say that I think unless there are
> serious problems, we should probably merge it.  Avoiding bitrot is prety
> important, and at the very least I don't think it will hurt to have it
> merged.
>

> > I added a patch with several fixes for the importer and and build
> > system.  So far, so good.
> >
> > There's a problem however with the --recursive option and the build
> > system.  To quote Jelle[1]
> >
> >    To start of with something that did not work out as well as I had
> >    hoped, getting a popular build system (e.g. Gulp, Grunt, Broccoli and
> >    others) packaged.  As mentioned in my earlier mails, the list of
> >    transitive dependencies of any of these suffer from at least the
> >    following:
> >
> >    - It is a list with more than 4000 packages on it
> >    - It is a list with at some point the package itself on it
> >
> > Most nontrivial npm packages use a build system, and all build systems
> > have circular development dependencies.  Not all development
> > dependencies are always required to build a package, but some certainly
> > are nd there's no way to tell which is which, afaik.
> >
> > That's why I added a --binary option to the importer: it will not
> > try to use the build system and instead mimick `what npm does.'  This
> > does provide, however, an amazing reproducibility feature to the
> > dependency woes that npm hackers are familar with.
> >
> > I suggest to not add any npm package to Guix that is the result of using
> > the --binary option and to build a base of full-source/sanitized npm
> > packages.
>
> Cool... makes sense to me to have this as something we don't use for
> Guix packages, but which might make Guix more useful for people who have
> to use npm in the awkward "real world" that is the current state of npm.
>

As one of these people living in the "real world", this is exactly how I
have been using the importer up till now.
I like and agree with most of your changes as they make the code much more
robust in the face of inevitable failure.

Nonetheless, one could say that we should not make it too easy to
inadvertently create package specifications for 'binaries'.

One tiny improvement might be to use `spdx-string->license` from (guix
import utils), instead of duplicating this effort in the npm importer.

How would you propose we get to reviewing your code? Would you care to send
some patches, or should we bother you via gitlab a bit more?

>
> > Greetings, janneke
> >
> > [1] https://lists.gnu.org/archive/html/guix-devel/2016-08/msg01567.html
>
>
Regards,

jlicht

[-- Attachment #2: Type: text/html, Size: 4565 bytes --]

Re: On merging the npm importer

[Ludovic Courtès]

Hello!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Jan Nieuwenhuizen writes:

[...]

>> We have a working importer for npm packages written by Jelle that I have
>> been using for about half a year.  It can use some improvements and
>> that's why I think we should merge it.
>>
>> Have alook at my npm branch here, rebased on master
>>
>>     https://gitlab.com/janneke/guix
>
> Would like to review soon, though I'll say that I think unless there are
> serious problems, we should probably merge it.  Avoiding bitrot is prety
> important, and at the very least I don't think it will hurt to have it
> merged.

Agreed.  If there are tests and doc and it’s all “reasonable”, which I
think it is, I second that!

>> I suggest to not add any npm package to Guix that is the result of using
>> the --binary option and to build a base of full-source/sanitized npm
>> packages.
>
> Cool... makes sense to me to have this as something we don't use for
> Guix packages, but which might make Guix more useful for people who have
> to use npm in the awkward "real world" that is the current state of npm.

How terrible must it be to live in the “real world”!  :-)

Ludo’.

Re: On merging the npm importer

[Jan Nieuwenhuizen]

Jelle Licht writes:

Hi Jelle!

> As one of these people living in the "real world", this is exactly how
> I have been using the importer up till now.  I like and agree with
> most of your changes as they make the code much more robust in the
> face of inevitable failure.

Great!

> Nonetheless, one could say that we should not make it too easy to
> inadvertently create package specifications for 'binaries'.

Is the --binary flag not obvious enough?  Do you have a suggestion?

> One tiny improvement might be to use `spdx-string->license` from (guix
> import utils), instead of duplicating this effort in the npm importer.

That sounds like an improvement, would you like to help with that?

> How would you propose we get to reviewing your code? Would you care to
> send some patches, or should we bother you via gitlab a bit more?

I think review should be done here, with patches.  I thought it might be
just a bit too early for that and was hoping others [you] would want to
make some changes first...and maybe pulling my git would be handier
then.

I'm happy to send the patches here, whatever is convenient.

Greetings, janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl