npm (mitigation)

npm (mitigation)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

I read that Jelle and Jan used their own branch in order to have npm based
software to be installed in their GuixSD environments, as binary blobs

Can I ask you for instructions about how to do that exactly ?

I might need to work a litle on some web siites in the future

Which branch exactly did you use, and how, exactly ?

Thanks

[-- Attachment #2: Type: text/html, Size: 428 bytes --]

Re: npm (mitigation)

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Hi Catonano,

I would be be happy to help you with this, but tbh, I am not comfortable
discussing this in-depth on guix-devel, as this seems antithetical to Guix'
goals.
All I will say here is that you need to adapt the npm importer to use the
sources from the npm registry instead of resolving to any 'upstream' urls.
I believe Jan's importer was already able to do this last time I checked,
so you might really only need to checkout their branch and rebase on
current master.

Regardless, the biggest issue that remains is still that npm-land is mired
in cyclical dependencies and a fun-but-not-actually unique dependency
resolving scheme.
I am currently working on a guile version of what Sander did for Nix for
importing entire npm dependency trees, but this will likely lead to lots of
programmatically
defined packages instead of the guix approach of mostly-manually defining
each package. It might therefore be a good candidate for a guix channel, if
that is still
being worked on.

Good luck!

- Jelle

2017-07-14 10:52 GMT+02:00 Catonano <catonano@gmail.com>:

> I read that Jelle and Jan used their own branch in order to have npm based
> software to be installed in their GuixSD environments, as binary blobs
>
> Can I ask you for instructions about how to do that exactly ?
>
> I might need to work a litle on some web siites in the future
>
> Which branch exactly did you use, and how, exactly ?
>
> Thanks
>

[-- Attachment #2: Type: text/html, Size: 1950 bytes --]

Re: npm (mitigation)

[Jan Nieuwenhuizen]

Catonano writes:

> I read that Jelle and Jan used their own branch in order to have npm
> based software to be installed in their GuixSD environments, as binary
> blobs

Jelle wrote a nice and clean npm importer, no binary anything.

Npm packages have the can be source or binary.  Depending on how you
want to look at it you can make change this source/binary disctinction
less white/black and turn it into a gray-scale.

As we are talking about javascript, in some cases source and binary
packages are identical.  In other cases, the binary packages come with
preprocessed documentation and lack the sources.  Other binary packages
include minimized javascript and even further into the darkness some
binary packages do not include the non-minimized javascript.  Then some
binary packages come with pre-compiled binaries and the worst are binary
package that do not come with the C/C++ sources that were used to
compile these binaries.

In all these cases the binary packages can be built from their source
package.  Here is where it starts to get nasty.  Building a package from
source can only be done if you have all its dependencies already
installed.  In theory that should not be a problem.

It appears that the npm ecosystem has manouvered itself into a place
where bootstrapping seems impossible: it turns out that any serious
package (notably all npm build system packages) have over 1000
dependencies, often with cyclic dependencies or even missing packages.

To break this boostrap loop on Guix I have added a couple of patches
onto Jelle's npm importer branch that implement a --binary flag.  This
allows binary npm packages to be installed in Guix and serve as a basis
to build other npm packages from source.

Apparently have no problem*) adding binary blobs for gcc, haskell (...?)
to Guix.  Similarly we could consider adding a/some binary blobs for npm
buildsystem packages to Guix and use those for a basis to build
source-only packages.

> Can I ask you for instructions about how to do that exactly ?

   git clone https://gitlab.com/janneke/guix.git

The branch `npm' is rebased on version-0.13.0, have a look at

   guix import npm --help

and look in gnu/packages/npm.scm for instructions.

Greetings,
janneke

*) Actually I do not like this very much and that's why some of us seek
   to remove the need for our bootstrap binaries with our stage0 and Mes
   projects.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: npm (mitigation)

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote:
> Regardless, the biggest issue that remains is still that npm-land is mired
> in cyclical dependencies and a fun-but-not-actually unique dependency
> resolving scheme.

I still think the largest issue is trying to determine if a given
package and its entire [cyclic cluster] subgraph is Free.  That's a lot
of manual verification to be had (to verify any automated
checks).  npm's package.json does include a `license' field, but that is
metadata with no legal significance, and afaik _defaults_ to "MIT"
(implying Expat), even if there's actually no license information in the
repository.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: npm (mitigation)

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

2017-07-15 5:34 GMT+02:00 Mike Gerwitz <mtg@gnu.org>:

> On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote:
> > Regardless, the biggest issue that remains is still that npm-land is
> mired
> > in cyclical dependencies and a fun-but-not-actually unique dependency
> > resolving scheme.
>
> I still think the largest issue is trying to determine if a given
> package and its entire [cyclic cluster] subgraph is Free.  That's a lot
> of manual verification to be had (to verify any automated
> checks).  npm's package.json does include a `license' field, but that is
> metadata with no legal significance, and afaik _defaults_ to "MIT"
> (implying Expat), even if there's actually no license information in the
> repository.


And that is exactly why this probably won't end up in Guix proper, at least
for the
foreseeable future. And also the reason that the entire npm situation is so
sad.

The default MIT/Expat only applies to people who generate their package
metadata
via npm init by just pressing enter; IANAL, but directly referring to a
valid and
common SPDX identifier is not that different from including some file under
the name
of LICENSE/COPYING.

It is true that lots of npm projects do not include copyright and/or
license headers in
 each source file, but this is also true for lots of other free software.

-
Jelle

[-- Attachment #2: Type: text/html, Size: 1852 bytes --]

Re: npm (mitigation)

[Adonay Felipe Nogueira]

Hm... If this is indeed true, then it's indeed a concern.

I wonder if we can check that out...

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: npm (mitigation)

[Adonay Felipe Nogueira]

Note that it's not required to include license notices (not to be
confused with the license itself) in the header (or as metadata) of each
source file. It's a recommendation, such that if someone gets only that
specific file, then that person can easily know and have proof that such
file is under that license.

For this specific/current paragraph, I also want to point out that these
are own ways of checking the license notices of works, and I'm not a
lawyer. If there is no license notice in a given file, then the sibling
or parent README files (or similar in non-English languages) dictate
where to find the license notices, or provide the license notices
there. Failing this, then one must check for the source file with the
same name of the project or some name similar to "main" (assumed to be
in English, but this name can also change according to the language).

Files similar to COPYING, LICENSE, or [License name, or abbreviation
indicating exceptions and if "or later"/"+" applies] files (or similar
ones for each of those in non-English languages) are not that useful if
there is no match for what was described in the last paragraph.

Annoying fact: I just read the package.json npm specification syntax
documentation and it says to follow the "SPDX" specification when
specifying the licenses. However, the current "SPDX" specification no
longer seems to accept licenses with "+" ("or later"), there is no way
to tell that there is a proxy involved (such as what can be specified
when one adapts a CC BY-SA 4.0 work into another one under GNU GPL 3+,
according to [[https://www.gnu.org/licenses/license-list.html#ccbysa]]),
there is no exception for website management systems and associated
templates (such as what can be found in
[[https://www.gnu.org/licenses/gpl-faq.html#WMS]]), and there is
exception for source files written in JavaScript
([[https://www.gnu.org/software/librejs/free-your-javascript.html]], in
section 3.2.1 of this reference).

Re: npm (mitigation)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1193 bytes --]

2017-07-14 13:57 GMT+02:00 Jelle Licht <jlicht@fsfe.org>:

> Hi Catonano,
>
> I would be be happy to help you with this, but tbh, I am not comfortable
> discussing this in-depth on guix-devel, as this seems antithetical to Guix'
> goals.
>

Ok, at least we made this clear.
I'll keep the list off the hook, should I need to discuss tis further, in
the future.

Just for the record, I don't mean to conter thhe Guix aim. I consider this
as a temporary mitigation. Were "temporary" is the key word

Regardless, the biggest issue that remains is still that npm-land is mired
> in cyclical dependencies and a fun-but-not-actually unique dependency
> resolving scheme.
> I am currently working on a guile version of what Sander did for Nix for
> importing entire npm dependency trees, but this will likely lead to lots of
> programmatically
> defined packages instead of the guix approach of mostly-manually defining
> each package.
>

What ? Who is Sander ans what did the do for Nix ?

How is importing "entire npm dependencies trees" different than importing
one package at a time and building the dependency tree gradually ?

I have no problem with programmatically defined pacages, anyway ;-)

[-- Attachment #2: Type: text/html, Size: 1905 bytes --]

Re: npm (mitigation)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 2617 bytes --]

2017-07-14 19:11 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:

> Catonano writes:
>
> > I read that Jelle and Jan used their own branch in order to have npm
> > based software to be installed in their GuixSD environments, as binary
> > blobs
>
> Jelle wrote a nice and clean npm importer, no binary anything.
>
> Npm packages have the can be source or binary.  Depending on how you
> want to look at it you can make change this source/binary disctinction
> less white/black and turn it into a gray-scale.
>
> As we are talking about javascript, in some cases source and binary
> packages are identical.  In other cases, the binary packages come with
> preprocessed documentation and lack the sources.  Other binary packages
> include minimized javascript and even further into the darkness some
> binary packages do not include the non-minimized javascript.  Then some
> binary packages come with pre-compiled binaries and the worst are binary
> package that do not come with the C/C++ sources that were used to
> compile these binaries.
>
> In all these cases the binary packages can be built from their source
> package.  Here is where it starts to get nasty.  Building a package from
> source can only be done if you have all its dependencies already
> installed.  In theory that should not be a problem.
>
> It appears that the npm ecosystem has manouvered itself into a place
> where bootstrapping seems impossible: it turns out that any serious
> package (notably all npm build system packages) have over 1000
> dependencies, often with cyclic dependencies or even missing packages.
>
> To break this boostrap loop on Guix I have added a couple of patches
> onto Jelle's npm importer branch that implement a --binary flag.  This
> allows binary npm packages to be installed in Guix and serve as a basis
> to build other npm packages from source.
>
> Apparently have no problem*) adding binary blobs for gcc, haskell (...?)
> to Guix.  Similarly we could consider adding a/some binary blobs for npm
> buildsystem packages to Guix and use those for a basis to build
> source-only packages.
>
> > Can I ask you for instructions about how to do that exactly ?
>
>    git clone https://gitlab.com/janneke/guix.git
>
> The branch `npm' is rebased on version-0.13.0, have a look at
>
>    guix import npm --help
>
> and look in gnu/packages/npm.scm for instructions.
>
> Greetings,
> janneke
>
> *) Actually I do not like this very much and that's why some of us seek
>    to remove the need for our bootstrap binaries with our stage0 and Mes
>    projects.
>

Thanks.
Both for the wrap up and for the instructions.

[-- Attachment #2: Type: text/html, Size: 3298 bytes --]

Re: npm (mitigation)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Mike,

2017-07-15 5:34 GMT+02:00 Mike Gerwitz <mtg@gnu.org>:

> On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote:
> > Regardless, the biggest issue that remains is still that npm-land is
> mired
> > in cyclical dependencies and a fun-but-not-actually unique dependency
> > resolving scheme.
>
> I still think the largest issue is trying to determine if a given
> package and its entire [cyclic cluster] subgraph is Free.  That's a lot
> of manual verification to be had (to verify any automated
> checks).  npm's package.json does include a `license' field, but that is
> metadata with no legal significance, and afaik _defaults_ to "MIT"
> (implying Expat), even if there's actually no license information in the
> repository.


 in my idea I would have build a database withh conditions for being non
free forr every npm package.

So we could have queried the database for questions like: is there any non
free or non buildable package in the dependency tree of, say, the current
Jquery ?

So we could have focused on such problems before embarking in a demanding
packaging process and then get struck by said problem along the way (withh
the risk of loosing the work already done)

You might remember my post of a few months back about an attempt of mine to
crawl thhe npm registry and storing data found there.

I used amz3's wrap around Wiredtiger and that was probably not the best
choice as I run into some maturity problems (maturity both of the framewrok
and my own maturity).

And then I slacked a bit

I also posted more recently about a research team that published a SPARQL
endpoint containing data about the npm packages

I thought it would be important but the feedback I collected was not
exactly warm

So I thought there must be some fundamental flaw in my way of thinking
about a more data centric way of dealing with this

Now I'm not sure what Jelle is talking about but any approach that cold be
shared among at least 2 persons would be a progress, in my opinion.

Jelle, please, say something more about whaht you're doing !

[-- Attachment #2: Type: text/html, Size: 2662 bytes --]

Re: npm (mitigation)

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Mon, Jul 17, 2017 at 11:45:29 +0200, Catonano wrote:
>  in my idea I would have build a database withh conditions for being non
> free forr every npm package.
>
> So we could have queried the database for questions like: is there any non
> free or non buildable package in the dependency tree of, say, the current
> Jquery ?

Being able to query the graph for non-free dependencies is good,
yes.  My concern is developing a (reasonably) fool-proof system for
detecting those packages that doesn't require manual verification, which
would be extremely costly, outside of a reasonable randomly-chosen set.

I'm not saying it's impossible; it's just difficult with such wildly
varying standards and carelessness with regards to licensing that is
prominent in the JS community.

But we have to start somewhere, so anything you can come up with would
be good. :)

> You might remember my post of a few months back about an attempt of mine to
> crawl thhe npm registry and storing data found there.

I do---I'm sorry if there are details that I missed or should know; I
haven't been able to follow this too closely.  I can be a bit of a
parrot sometimes with certain issues. :x

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: npm (mitigation)

[Ricardo Wurmus]

Catonano <catonano@gmail.com> writes:

>> I am currently working on a guile version of what Sander did for Nix for
>> importing entire npm dependency trees, but this will likely lead to lots of
>> programmatically
>> defined packages instead of the guix approach of mostly-manually defining
>> each package.
>>
>
> What ? Who is Sander ans what did the do for Nix ?

Jelle is referring to the work by Sander van der Burg, who presented
“Deploying NPM packages with the Nix package manager” at FOSDEM 2017.

Here’s the video:

    https://video.fosdem.org/2017/K.4.601/deploying_npm_packages_with_nix.vp8.webm

-- 
Ricardo
  
  GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
  https://elephly.net

Re: npm (mitigation)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

2017-07-24 17:39 GMT+02:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Catonano <catonano@gmail.com> writes:
>
> >> I am currently working on a guile version of what Sander did for Nix for
> >> importing entire npm dependency trees, but this will likely lead to
> lots of
> >> programmatically
> >> defined packages instead of the guix approach of mostly-manually
> defining
> >> each package.
> >>
> >
> > What ? Who is Sander ans what did the do for Nix ?
>
> Jelle is referring to the work by Sander van der Burg, who presented
> “Deploying NPM packages with the Nix package manager” at FOSDEM 2017.
>
> Here’s the video:
>
>     https://video.fosdem.org/2017/K.4.601/deploying_npm_
> packages_with_nix.vp8.webm
>
>
Yes, thanks to the both of you

Also his blog posts are enlightening but I must confess that I'm not sure I
can wrap my mind around the issue

It's larger and more nuanced than it looks like

I'll read some more about it in the next days

Thanks Jelle for the pointers

[-- Attachment #2: Type: text/html, Size: 1758 bytes --]