* NPM importer @ 2018-11-11 10:41 swedebugia 2018-11-11 15:37 ` Julien Lepiller 2018-11-11 17:10 ` NPM importer Ludovic Courtès 0 siblings, 2 replies; 31+ messages in thread From: swedebugia @ 2018-11-11 10:41 UTC (permalink / raw) To: guix-devel Hi Has anybody worked on that? -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-11 10:41 NPM importer swedebugia @ 2018-11-11 15:37 ` Julien Lepiller 2018-11-19 23:29 ` swedebugia 2018-11-30 16:13 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 2018-11-11 17:10 ` NPM importer Ludovic Courtès 1 sibling, 2 replies; 31+ messages in thread From: Julien Lepiller @ 2018-11-11 15:37 UTC (permalink / raw) To: guix-devel, swedebugia I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles). Le 11 novembre 2018 11:41:34 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit : >Hi > >Has anybody worked on that? ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-11 15:37 ` Julien Lepiller @ 2018-11-19 23:29 ` swedebugia 2018-11-20 7:50 ` Julien Lepiller 2018-11-30 16:13 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 1 sibling, 1 reply; 31+ messages in thread From: swedebugia @ 2018-11-19 23:29 UTC (permalink / raw) To: Julien Lepiller, guix-devel On 2018-11-11 16:37, Julien Lepiller wrote: > I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm > > There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles). Ok. Is there a reason to have this out-of-tree? Could you push it to a wip-npm branch in guix? Some npm packages has no or almost no dependencies. E.g. https://github.com/substack/minimist/blob/master/package.json These we could start with if the build-system works and incrementally improve the build-system and the complexity of added packages. Thoughts? -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-19 23:29 ` swedebugia @ 2018-11-20 7:50 ` Julien Lepiller 2018-11-20 19:58 ` swedebugia 2018-11-24 13:42 ` NPM importer swedebugia 0 siblings, 2 replies; 31+ messages in thread From: Julien Lepiller @ 2018-11-20 7:50 UTC (permalink / raw) To: swedebugia, guix-devel Well, minimist is not a small package. I have a small script somewhere that browse the registry to find recursively the list of dependencies of a package. I tried to run it on minimist and I stopped it after a few thousand different packages were found. That's also why we can't make a recursive importer: it would almost never end. I think my plan was to build these packages without testing them at all, focusing on packaging test frameworks that will be needed to test other packages. See this script that builds a graph of dependencies, ignoring devDependencies (so none of the packages listed by this script is going to be tested): https://framagit.org/tyreunom/guix/snippets/2534 Not testing these packages is maybe less problematic than not testing other kinds of packages because we are packaging the same thing as developpers test, in a similar environment. I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches. Le 20 novembre 2018 00:29:02 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit : >On 2018-11-11 16:37, Julien Lepiller wrote: >> I improved a bit over jlicht's work here, but there still a few >tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm >> >> There is an importer and a build system as well as a few packages. >One of tge issue is that the importer is not recursive, so it doesn't >get the specified version, and the packages aren't tested because the >tests depend on cylic dependencies (and sometimes very big circles). > >Ok. >Is there a reason to have this out-of-tree? >Could you push it to a wip-npm branch in guix? > >Some npm packages has no or almost no dependencies. >E.g. https://github.com/substack/minimist/blob/master/package.json > >These we could start with if the build-system works and incrementally >improve the build-system and the complexity of added packages. > >Thoughts? ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 7:50 ` Julien Lepiller @ 2018-11-20 19:58 ` swedebugia 2018-11-20 21:12 ` swedebugia 2018-11-24 13:42 ` NPM importer swedebugia 1 sibling, 1 reply; 31+ messages in thread From: swedebugia @ 2018-11-20 19:58 UTC (permalink / raw) To: Julien Lepiller, guix-devel, Jelle Licht Hi :) On 2018-11-20 08:50, Julien Lepiller wrote: > Well, minimist is not a small package. I have a small script somewhere that browse the registry to find recursively the list of dependencies of a package. I tried to run it on minimist and I stopped it after a few thousand different packages were found. That's also why we can't make a recursive importer: it would almost never end. > > I think my plan was to build these packages without testing them at all, focusing on packaging test frameworks that will be needed to test other packages. ok. > > See this script that builds a graph of dependencies, ignoring devDependencies (so none of the packages listed by this script is going to be tested): https://framagit.org/tyreunom/guix/snippets/2534 Unfortunately I did not understand how to use the script :/ I got this after wgetting the snippet: scheme@(guile-user)> (load "npm-explorer.scm") digraph dependencies { overlap=false; splines=true; "mocha" -> "supports-color"; "supports-color" -> "has-flag"; "mocha" -> "glob";s "glob" -> "inherits"; "glob" -> "fs.realpath"; "glob" -> "minimatch"; "minimatch" -> "brace-expansion"; "brace-expansion" -> "concat-map"; "brace-expansion" -> "balanced-match"; "glob" -> "path-is-absolute"; "glob" -> "once";ges "once" -> "wrappy";s ^Conce [color=red]; "glob" -> "inflight"; "inflight" -> "once"; "inflight" -> "wrappy"; "mocha" -> "mkdirp"; "mkdirp" -> "minimist"; "mocha" -> "commander"; "mocha" -> "he";ages "mocha" -> "diff";es "mocha" -> "minimatch"; "mocha" -> "escape-string-regexp"; "mocha" -> "growl";s "mocha" -> "debug";s "debug" -> "ms";ages "mocha" -> "browser-stdout"; } $1 = #t Can you explain how to use it? > > Not testing these packages is maybe less problematic than not testing other kinds of packages because we are packaging the same thing as developpers test, in a similar environment. It sounds like a good start. This mess is not a situation we created, we do our best to shoehorn these packages into a reproducible environment that the js-devs does not really care about or gave up on long ago. Even if we succeed to build every major Node web app in a few years it is not certain that the js-devs or the users are going to use them. But we can hope and enjoy them ourselves :D > > I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches. That would be nice. I did not see it yet in the list http://git.savannah.gnu.org/cgit/guix.git/refs/heads BTW: Did you see that Jelle did two commits after your fork? https://github.com/jellelicht/guix/commits/wip-node-build-system -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 19:58 ` swedebugia @ 2018-11-20 21:12 ` swedebugia 2018-11-20 22:35 ` Julien Lepiller 2018-11-21 1:41 ` Mike Gerwitz 0 siblings, 2 replies; 31+ messages in thread From: swedebugia @ 2018-11-20 21:12 UTC (permalink / raw) To: Julien Lepiller, guix-devel, Jelle Licht [-- Attachment #1: Type: text/plain, Size: 1247 bytes --] Hi On 2018-11-20 20:58, swedebugia wrote: > On 2018-11-20 08:50, Julien Lepiller wrote: snip >> See this script that builds a graph of dependencies, ignoring >> devDependencies (so none of the packages listed by this script is >> going to be tested): https://framagit.org/tyreunom/guix/snippets/2534 What is devDependencies? Required to build the npm package? I understood the script now I think. :) I replaced "mocha" with name in the call in the bottom. Then did: $guile scheme@(guile-user)> (define name "ssb-patchwork") scheme@(guile-user)> (load "npm-explorer.scm") ... As you said it runs for(ever) (in my case 5 minutes) listing dependencies of dependencies of dependencies of dependencies of dependencies of dependencies ... 1100 last lines of the output is attached. :p One thing is for sure. There are a LOT of npm packages. "Over 477,000 packages are available on the main npm registry." according to WP. :O I wonder how many are free software? 90%? 50%? I hope we can automate this some way. npm.scm will be a VERY VERY long file. Maybe we should rethink about how to best store all these variables... Just 1 npm application (ssb-patchwork) will add at least 1000 package definitions to it. -- Cheers Swedebugia [-- Attachment #2: screen-npm-dep-ssb-patchwork.log.bz2 --] [-- Type: application/x-bzip, Size: 6621 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 21:12 ` swedebugia @ 2018-11-20 22:35 ` Julien Lepiller 2018-11-21 15:36 ` swedebugia 2018-11-21 1:41 ` Mike Gerwitz 1 sibling, 1 reply; 31+ messages in thread From: Julien Lepiller @ 2018-11-20 22:35 UTC (permalink / raw) To: swedebugia, guix-devel Le Tue, 20 Nov 2018 22:12:18 +0100, swedebugia <swedebugia@riseup.net> a écrit : > Hi > > On 2018-11-20 20:58, swedebugia wrote: > > On 2018-11-20 08:50, Julien Lepiller wrote: > > snip > > >> See this script that builds a graph of dependencies, ignoring > >> devDependencies (so none of the packages listed by this script is > >> going to be tested): > >> https://framagit.org/tyreunom/guix/snippets/2534 > > What is devDependencies? Required to build the npm package? the package.json file declares two kinds (I think I've seen a third kind, but I'm not sure, nor am I a js developper) of dependencies: dependencies and devDependencies that map to our inputs and native-inputs. devDependencies are tools used during the build or development, but they are not needed at runtime. Sometimes, they are required at build time (for instance the coffeescript compiler is a devDependencies of some packages, but we need it to build the package), usually they are required at test time (for instance mocha, tap, or tape are test frameworks), otherwise, they can be documentation tools, and sometimes really not useful for us (we probably don't want to run linters or release-related tools). > > I understood the script now I think. :) > > I replaced "mocha" with name in the call in the bottom. > > Then did: > $guile > scheme@(guile-user)> (define name "ssb-patchwork") > scheme@(guile-user)> (load "npm-explorer.scm") > ... The way I wrote it, you were supposed to change the "mocha" with the package you're interested in, and run the script with guile like so: guile npm-explorer.scm > mocha.dot and then: fdp -Tpng mocha.dot > mocha.png (fdp is part of graphviz, but produces nicer graphs than dot in my opinion) > > As you said it runs for(ever) (in my case 5 minutes) listing > dependencies of dependencies of dependencies of dependencies of > dependencies of dependencies ... > > 1100 last lines of the output is attached. :p > > One thing is for sure. There are a LOT of npm packages. "Over 477,000 > packages are available on the main npm registry." according to WP. :O That's probably the number of packages on which depend any node packages through its dependencies and devDependencies... I suspect you can always find a path between any two packages in this graph... in either direction. > > I wonder how many are free software? 90%? 50%? > > I hope we can automate this some way. It looks very difficult, but if you have any ideas, please tell us! > > npm.scm will be a VERY VERY long file. Maybe we should rethink about > how to best store all these variables... Very long files are frowned upon, because they have very long compilation times and require a lot of memory. We'll have to find a way to split the file before it grows too much. Maybe we will do something similar than python packages? We have python.scm, python-web.scm... > > Just 1 npm application (ssb-patchwork) will add at least 1000 package > definitions to it. > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 22:35 ` Julien Lepiller @ 2018-11-21 15:36 ` swedebugia 0 siblings, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-21 15:36 UTC (permalink / raw) To: Julien Lepiller, guix-devel On 2018-11-20 23:35, Julien Lepiller wrote: > Le Tue, 20 Nov 2018 22:12:18 +0100, > swedebugia <swedebugia@riseup.net> a écrit : > >> Hi >> >> On 2018-11-20 20:58, swedebugia wrote: >>> On 2018-11-20 08:50, Julien Lepiller wrote: >> >> snip >> >>>> See this script that builds a graph of dependencies, ignoring >>>> devDependencies (so none of the packages listed by this script is >>>> going to be tested): >>>> https://framagit.org/tyreunom/guix/snippets/2534 >> >> What is devDependencies? Required to build the npm package? > > the package.json file declares two kinds (I think I've seen a third > kind, but I'm not sure, nor am I a js developper) of dependencies: > dependencies and devDependencies that map to our inputs and > native-inputs. devDependencies are tools used during the build or > development, but they are not needed at runtime. Sometimes, they are > required at build time (for instance the coffeescript compiler is a > devDependencies of some packages, but we need it to build the package), > usually they are required at test time (for instance mocha, tap, or > tape are test frameworks), otherwise, they can be documentation tools, > and sometimes really not useful for us (we probably don't want to run > linters or release-related tools). Ok, that sounds resonable. Maybe we can blacklist all these linters and release tools during import and import everything else as dependencies? snip > The way I wrote it, you were supposed to change the "mocha" with the > package you're interested in, and run the script with guile like so: > > guile npm-explorer.scm > mocha.dot > > and then: > > fdp -Tpng mocha.dot > mocha.png > > (fdp is part of graphviz, but produces nicer graphs than dot in my > opinion) Thanks! snip >> >> npm.scm will be a VERY VERY long file. Maybe we should rethink about >> how to best store all these variables... > > Very long files are frowned upon, because they have very long > compilation times and require a lot of memory. We'll have to find a way > to split the file before it grows too much. Maybe we will do something > similar than python packages? We have python.scm, python-web.scm... yeah good idea. but with half a miljon packages and maybe max a 1000 definitons in each we will still have ~500 npm files. By then I suggest we put them in a separate dir like gnu/packages/npm/web.scm etc. -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 21:12 ` swedebugia 2018-11-20 22:35 ` Julien Lepiller @ 2018-11-21 1:41 ` Mike Gerwitz 2018-11-21 22:01 ` Brett Gilio 1 sibling, 1 reply; 31+ messages in thread From: Mike Gerwitz @ 2018-11-21 1:41 UTC (permalink / raw) To: swedebugia; +Cc: guix-devel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Tue, Nov 20, 2018 at 22:12:18 +0100, swedebugia wrote: > I wonder how many are free software? 90%? 50%? > > I hope we can automate this some way. The JavaScript community has poor licensing practices, and the culture is somewhat hostile to the ideals of the free software movement (they focus on permissive licensing to empower non-free software developers using those libraries). The package.json has a license field, but package.json is often auto-generated and I think is MIT Expat by default. It is metadata---I can't imagine it carries any legal weight by itself. Consequently, we'd have to fall back on COPYING or LICENSE files (of various sorts) in the projects. Even then, a project may contain things under various licenses. Further, since there tend to be many really small packages, if _any_ one of those is missing proper license information, then anything that depends on it will be non-free. Since npm doesn't ensure that its packages are actually free, the odds of there being some sort of licensing issue---just by sheer number---are probably higher than we would like them to be. I'm not suggesting malice; it may be accidental, or maybe someone knows nothing about licensing and simply never attached a license to begin with (making it non-free by default).[0] There's also the risk of any of these projects using incompatible licenses. Both GitLab and GitHub detect licenses on projects. I forget the name of the software they use to do that (and it may not be the same for both of them), and it's probably not perfect, but something like that may help with automation. [0]: https://blog.github.com/2015-03-09-open-source-license-usage-on-github-com/ (as of 2015) - -- Mike Gerwitz Free Software Hacker+Activist | GNU Maintainer & Volunteer GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05 https://mikegerwitz.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJb9Le7AAoJEIyRe39dxRuijsQP/19DjT/G4/806rbtECrZAOeB DhsWG6fm4rFkv9VbdVgKFASF2qIYSmq8p3yiVHwaNNoV0HcUurK/Gz44880S9EPO EnHoN7naBYEzv1JRkGFdiHh6eEeUsGPdt4RB1/bunjY4K3EOfeSVQWWeXlDjI4GK KKsLqNto00ESiwkDW2FOvElnUwfkFa1i+mtc48PbNtoy0VPie9kid51NDQLw4fZG 4VTU+kExeBHriq1AsNREWOQB2Ctg09ydkrzTvzlg4kucnFp8S3ohvbLoSoPil0YH 09hihN/sSwtws2BkMjgB2cO/y46MZeRsvazDD6ZZNyz3PqVZYsPa+5F0eFp7MlpV a56lsND20IEtmITUZsaIx+W4F71iLJPIVWMN0NZA/rMgEntWSgexSx3w9RFZNYWH m3UAyACIV9M9WDlTN/OJ/Q98clJ8G3AcZ3M5hKMyvtupctMWxZmDhY3dEGGeavvv AKIXJ0Qau0mi9HFvT4eusfyih01kCFGddMcUCkY7QY6yuDxg1hHglM75t6YHvfwP 4lZlCywwC6UMazpv0QO6KMOhq7pwEVsXvn9agZR9gd5d18H+6EMv0AogNrh2a48u HFtRNSMx4woFxXUGd2P50/Zy8hJFyijMbLyksLkxfr7HcTfJ+hUGb1e94xuEdZzr xR/YjftLIOSzNUV9y20y =F0Yh -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-21 1:41 ` Mike Gerwitz @ 2018-11-21 22:01 ` Brett Gilio 2018-11-21 23:22 ` swedebugia 0 siblings, 1 reply; 31+ messages in thread From: Brett Gilio @ 2018-11-21 22:01 UTC (permalink / raw) To: Mike Gerwitz; +Cc: guix-devel Mike Gerwitz writes: > The JavaScript community has poor licensing practices, and the culture > is somewhat hostile to the ideals of the free software movement (they > focus on permissive licensing to empower non-free software developers > using those libraries). To say the least. It will take a good deal of implementing a license checker on the importer, as well as human verification to ensure that we are maintaining a high ethical standard. Brett Gilio ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-21 22:01 ` Brett Gilio @ 2018-11-21 23:22 ` swedebugia 2018-11-22 1:02 ` swedebugia 0 siblings, 1 reply; 31+ messages in thread From: swedebugia @ 2018-11-21 23:22 UTC (permalink / raw) To: Brett Gilio, Mike Gerwitz; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 2589 bytes --] On 2018-11-21 23:01, Brett Gilio wrote: > > Mike Gerwitz writes: > >> The JavaScript community has poor licensing practices, and the culture >> is somewhat hostile to the ideals of the free software movement (they >> focus on permissive licensing to empower non-free software developers >> using those libraries). > > To say the least. It will take a good deal of implementing a license > checker on the importer, as well as human verification to ensure that we > are maintaining a high ethical standard. We might want to use the same approach as licensee: "The solution Licensee automates the process of reading LICENSE files and compares their contents to known licenses using a several strategies (which we call "Matchers"). It attempts to determine a project's license in the following order: If the license file has an explicit copyright notice, and nothing more (e.g., Copyright (c) 2015 Ben Balter), we'll assume the author intends to retain all rights, and thus the project isn't licensed. If the license is an exact match to a known license. If we strip away whitespace and copyright notice, we might get lucky, and direct string comparison in Ruby is cheap. If we still can't match the license, we use a fancy math thing called the Sørensen–Dice coefficient, which is really good at calculating the similarity between two strings. By calculating the percent changed from the known license to the license file, you can tell, e.g., that a given license is 95% similar to the MIT license, that 5% likely representing legally insignificant changes to the license text." https://github.com/benbalter/licensee We could perhaps also semi-automate the generation of emails to authors of the offending npm packages with unclear packages. Say only 1% of 470.000 has unclear license, that equals 4700 emails to authors. :) In a hypothetical scenario with import of 20 npm packages a day it will take us 477.000/20 = 23850 days = 65 years to import them all. In a hypothetical scenario with import of 500 npm packages a day it will take us 477.000/500 = 954 days = 2,6 years to import them all. This is based on the assumption that all are free software, but that is probably not the case. BTW ssb-patchwork had over 400 dependants in 10+ levels and the dotfile is attached. The rendered png is crazy looking. Reminds me of the holy spaghetti monster. A graph of all npm packages and top packages is also available: https://exploring-data.com/info/npm-packages-dependencies/ -- Cheers Swedebugia [-- Attachment #2: ssb-patchwork.dot.bz2 --] [-- Type: application/x-bzip, Size: 6041 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-21 23:22 ` swedebugia @ 2018-11-22 1:02 ` swedebugia 2018-11-22 5:43 ` Brett Gilio ` (3 more replies) 0 siblings, 4 replies; 31+ messages in thread From: swedebugia @ 2018-11-22 1:02 UTC (permalink / raw) To: Brett Gilio, Mike Gerwitz; +Cc: guix-devel Hi On 2018-11-22 00:22, swedebugia wrote: snip > A graph of all npm packages and top packages is also available: > https://exploring-data.com/info/npm-packages-dependencies/ While investigating the top libraries* packages with most depends in npm I found the following: Lib Dep DevDep RecdevDep Dependants license underscore 0 12 2400+ 18000+ mit async 1 37 2696 26069 mit rec=recursive See also the issue I created here: https://github.com/caolan/async/issues/1600 asking which of asyncs 37 devdeps are needed to build or build+test async. And similar here: https://github.com/jashkenas/underscore/issues/2790 -- Cheers Swedebugia * (according to the link above) ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-22 1:02 ` swedebugia @ 2018-11-22 5:43 ` Brett Gilio 2018-11-22 11:27 ` import libjs-*.deb from Debian? (was Re: NPM importer) Giovanni Biscuolo 2018-11-22 8:36 ` NPM importer Julien Lepiller ` (2 subsequent siblings) 3 siblings, 1 reply; 31+ messages in thread From: Brett Gilio @ 2018-11-22 5:43 UTC (permalink / raw) To: swedebugia; +Cc: guix-devel swedebugia writes: > Hi > > On 2018-11-22 00:22, swedebugia wrote: > snip > >> A graph of all npm packages and top packages is also available: >> https://exploring-data.com/info/npm-packages-dependencies/ > > While investigating the top libraries* packages with most depends in > npm I found the following: > > Lib Dep DevDep RecdevDep Dependants license > underscore 0 12 2400+ 18000+ mit > async 1 37 2696 26069 mit > > rec=recursive > > See also the issue I created here: > https://github.com/caolan/async/issues/1600 asking which of asyncs 37 > devdeps are needed to build or build+test async. > > And similar here: https://github.com/jashkenas/underscore/issues/2790 Holy cow. Thank you for taking the initiative on this, swedebugia. I tend to agree with your licensee approach, but I think this will require an undoubted good deal of man power before we are able to sufficiently move forward on npm imports with a decent velocity. I know that the largest issue of this as usual will be the JavaScript trap, but there will have to be some compromises on the part of people wanting to develop for Node on Guix when their libraries are missing due to non-free or ambiguous licensing. Not the least of which being the issues surrounding Electron on the Chromium licenses, which seems to be where a lot of energy is lately on the Node development side. I also do not want Guix to detract too much for "more important issues", not to say this isn't important. Just perhaps it isn't as much of a priority simply because that community and ours are not so cohesive at the current moment. I guess one could argue that we should be wanting to bring them to us, but I also know how disuasive "a lack of convenience" can be to those who are not as freedom and ethicality conscious as the rest of us. Brett Gilio ^ permalink raw reply [flat|nested] 31+ messages in thread
* import libjs-*.deb from Debian? (was Re: NPM importer) 2018-11-22 5:43 ` Brett Gilio @ 2018-11-22 11:27 ` Giovanni Biscuolo 2018-11-30 3:23 ` Ricardo Wurmus 0 siblings, 1 reply; 31+ messages in thread From: Giovanni Biscuolo @ 2018-11-22 11:27 UTC (permalink / raw) To: Brett Gilio, swedebugia; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 2075 bytes --] Hi Guix! sorry if I'm going to reinvent some wheels but I've not found previous discussions nor documentation about what I'm writing what about reusing work already done from the Debian JavaScript Maintainers team [1]? they set up some interesting policies we could adapt to Guix needs and pusblish somewhere (where?) I mean pages like: - https://wiki.debian.org/Javascript/Policy - https://wiki.debian.org/Javascript/Nodejs they have an npm2deb importer https://wiki.debian.org/Javascript/Nodejs/Npm2Deb ... we could import what they import :-) I see in Apr 2016 a brief discussion on a .deb importer was done https://lists.gnu.org/archive/html/guix-devel/2016-04/msg01053.html what is the updated situation today? Brett Gilio <brettg@posteo.net> writes: [...] > Thank you for taking the initiative on this, swedebugia. I tend to agree > with your licensee approach, but I think this will require an undoubted > good deal of man power before we are able to sufficiently move forward > on npm imports with a decent velocity. could a .deb importer help us to manage this *very big* workload, sharing this burden with Debian JavaScript maintainers? I found this project as a starting point https://gitlab.digitalcourage.de/htgoebel/guix-import-debian [...] > I guess one could argue that we should be wanting to bring them to us, > but I also know how disuasive "a lack of convenience" can be to those > who are not as freedom and ethicality conscious as the rest of us. should we ignore/deprecate npm packages with licensing issues and work with the ethically conscious like Debian JavaScript Maintainers? having a fully free software _and_ reproducible [3] nodejs/javascript dev environment would benefit a **huge** amount of web developers... they will be so happy to be guixified! :-) [...] ciao Giovanni [1] https://wiki.debian.org/Javascript [2] https://packages.debian.org/stretch/libjs-jquery [3] and bootstrappable? -- Giovanni Biscuolo Xelera IT Infrastructures [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: import libjs-*.deb from Debian? (was Re: NPM importer) 2018-11-22 11:27 ` import libjs-*.deb from Debian? (was Re: NPM importer) Giovanni Biscuolo @ 2018-11-30 3:23 ` Ricardo Wurmus 0 siblings, 0 replies; 31+ messages in thread From: Ricardo Wurmus @ 2018-11-30 3:23 UTC (permalink / raw) To: Giovanni Biscuolo; +Cc: guix-devel Hi Giovanni, > what about reusing work already done from the Debian JavaScript > Maintainers team [1]? […] > should we ignore/deprecate npm packages with licensing issues and work > with the ethically conscious like Debian JavaScript Maintainers? I think this would be a good thing to try. Work on npm can of course still continue, but it could be useful to have a couple of easier JavaScript library packages sooner. Would you like to adapt one or more of the libjs* Debian packages for Guix to see if this is a reasonable way forward? -- Ricardo ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-22 1:02 ` swedebugia 2018-11-22 5:43 ` Brett Gilio @ 2018-11-22 8:36 ` Julien Lepiller 2018-11-24 13:47 ` swedebugia 2018-11-23 19:50 ` swedebugia 2018-11-30 14:17 ` Packaging async and underscore (Was: Re: NPM importer) swedebugia 3 siblings, 1 reply; 31+ messages in thread From: Julien Lepiller @ 2018-11-22 8:36 UTC (permalink / raw) To: guix-devel, swedebugia, Brett Gilio, Mike Gerwitz Thanks! It made me wonder if we could better approximate the set of needed dependencies by looking at package tags. I'll try to improve my script in that spirit, and share it again here. I think once we have a clearer view of what we want, we should focus on finding the best way forward: what packages do we need first, etc. I think build tools are obviously more important than test tools, which should have a greater priority than the rest of packages. Le 22 novembre 2018 02:02:52 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit : >Hi > >On 2018-11-22 00:22, swedebugia wrote: >snip > >> A graph of all npm packages and top packages is also available: >> https://exploring-data.com/info/npm-packages-dependencies/ > >While investigating the top libraries* packages with most depends in >npm >I found the following: > >Lib Dep DevDep RecdevDep Dependants license >underscore 0 12 2400+ 18000+ mit >async 1 37 2696 26069 mit > >rec=recursive > >See also the issue I created here: >https://github.com/caolan/async/issues/1600 asking which of asyncs 37 >devdeps are needed to build or build+test async. > >And similar here: https://github.com/jashkenas/underscore/issues/2790 ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-22 8:36 ` NPM importer Julien Lepiller @ 2018-11-24 13:47 ` swedebugia 0 siblings, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-24 13:47 UTC (permalink / raw) To: Julien Lepiller, guix-devel, Brett Gilio, Mike Gerwitz On 2018-11-22 09:36, Julien Lepiller wrote: > Thanks! It made me wonder if we could better approximate the set of needed dependencies by looking at package tags. I'll try to improve my script in that spirit, and share it again here. Sounds good. I would like to work on implementing a way to blacklist dependencies during guix import npm. I look forward to see your snippet. I suggest we work in tree on master going forward and I think we should submit both the node-build-system and import script to master as they seem to work fine. Both apply cleanly. Getting this in before 1.0 is possible as I see it. We can improve the importer with blacklisting and recursive capacities in a wip-branch. -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-22 1:02 ` swedebugia 2018-11-22 5:43 ` Brett Gilio 2018-11-22 8:36 ` NPM importer Julien Lepiller @ 2018-11-23 19:50 ` swedebugia 2018-11-30 3:17 ` Ricardo Wurmus 2018-11-30 14:17 ` Packaging async and underscore (Was: Re: NPM importer) swedebugia 3 siblings, 1 reply; 31+ messages in thread From: swedebugia @ 2018-11-23 19:50 UTC (permalink / raw) To: Brett Gilio, Mike Gerwitz; +Cc: guix-devel On 2018-11-22 02:02, swedebugia wrote: > Hi > > On 2018-11-22 00:22, swedebugia wrote: > snip > >> A graph of all npm packages and top packages is also available: >> https://exploring-data.com/info/npm-packages-dependencies/ This graph is in-degree and does not consider versions. It is useful only to see the relative popularity of libraries. It does not reveal the out-degree which is what we are interested in: recursive dependencies (given x what does it take to build x? build and test x?) Also it leaves out any information about licensing. It would be nice with a graph of which npm packages has which licenses with licenses as vertices/nodes and packages as edges. snip > > See also the issue I created here: snip > And similar here: https://github.com/jashkenas/underscore/issues/2790 Got an answer which is good news for building and bad news for testing: "devDependencies": { "coveralls": "^2.11.2", // NOT NEEDED: Posts coverage reports to converalls.io "docco": "*", // NOT NEEDED: Builds docs "eslint": "1.10.x", // Needed for linting (not sure if you think of linting as part of testing) "gzip-size-cli": "^1.0.0", // NOT NEEDED "karma": "^0.13.13", // Needed for testing "karma-qunit": "~2.0.1", // Needed for testing "karma-sauce-launcher": "^1.2.0", // Needed for testing "nyc": "^2.1.3", // Needed for testing "pretty-bytes-cli": "^1.0.0", // NOT NEEDED "qunit-cli": "~0.2.0", // Needed for testing "qunit": "^2.6.0", // Needed for testing "uglify-js": "3.3.21" // Needing for building }, In summary direct devdeps: 1 needed for building, 6 needed for testing. uglify-js is a minifier used in a lot of projects and the only one needed for this specific project. For packages like this we could probably patch the package to skip the minifying step and let the user choose which minifier (if any) to use. In this particular case the minifier compacts the code from 57KB -> 17.6KB. The most important thing I think is to be able to start packaging this mess so cutting an arm and a leg in the beginning (= skip minification and testing is warranted in my view). In this case the only thing left to do to package underscore is to: 1) name it: e.g. node-underscore (like debian) 2) decide whether to delete the minified upstream or not in the guix package. Hooray! -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-23 19:50 ` swedebugia @ 2018-11-30 3:17 ` Ricardo Wurmus 0 siblings, 0 replies; 31+ messages in thread From: Ricardo Wurmus @ 2018-11-30 3:17 UTC (permalink / raw) To: swedebugia; +Cc: guix-devel swedebugia <swedebugia@riseup.net> writes: > uglify-js is a minifier used in a lot of projects and the only one > needed for this specific project. > > For packages like this we could probably patch the package to skip the > minifying step and let the user choose which minifier (if any) to use. > > In this particular case the minifier compacts the code from 57KB -> 17.6KB. We already have a JS uglifier package (written in Common Lisp), so this should be doable. -- Ricardo ^ permalink raw reply [flat|nested] 31+ messages in thread
* Packaging async and underscore (Was: Re: NPM importer) 2018-11-22 1:02 ` swedebugia ` (2 preceding siblings ...) 2018-11-23 19:50 ` swedebugia @ 2018-11-30 14:17 ` swedebugia 2018-11-30 16:08 ` Packaging async and underscore Julien Lepiller 3 siblings, 1 reply; 31+ messages in thread From: swedebugia @ 2018-11-30 14:17 UTC (permalink / raw) To: guix-devel [-- Attachment #1: Type: text/plain, Size: 811 bytes --] On 2018-11-22 02:02, swedebugia wrote: > Hi > > On 2018-11-22 00:22, swedebugia wrote: > snip > snip > While investigating the top libraries* packages with most depends in npm > I found the following: > > Lib Dep DevDep RecdevDep Dependants license > underscore 0 12 2400+ 18000+ mit > async 1 37 2696 26069 mit I succeded in packaging the latest underscore. I discovered a bug in the build system and fixed it to be able to build it. (see the diff attached) Debian packaged a 4 year old async 0.8.0 that has 0 dep and only uglify-js as devdep (which we already have packaged). From 0.9.2 lodash was made a devdep. -- Cheers Swedebugia [-- Attachment #2: diff node-build-system --] [-- Type: text/plain, Size: 4007 bytes --] modified guix/build/node-build-system.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson <davet@gnu.org> ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org> +;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;; @@ -108,9 +109,13 @@ is an npm global install." (#f #f)))) (mkdir-p tgt-dir) (copy-recursively "." (string-append tgt-dir "/node_modules/" modulename)) - ;; Remove references to dependencies - (delete-file-recursively - (string-append tgt-dir "/node_modules/" modulename "/node_modules")) + ;; Remove references to bundled dependencies + (let ((tgt-node-modules (string-append tgt-dir + "/node_modules/" modulename "/node_modules"))) + ;; Check if tgt-node-modules exist and remove them if #t + (when (file-is-directory? tgt-node-modules) + (delete-file-recursively + tgt-node-modules))) (when global? (cond ((string? bin-conf) ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Packaging async and underscore 2018-11-30 14:17 ` Packaging async and underscore (Was: Re: NPM importer) swedebugia @ 2018-11-30 16:08 ` Julien Lepiller 2018-11-30 16:44 ` swedebugia 0 siblings, 1 reply; 31+ messages in thread From: Julien Lepiller @ 2018-11-30 16:08 UTC (permalink / raw) To: guix-devel Le 2018-11-30 15:17, swedebugia a écrit : > On 2018-11-22 02:02, swedebugia wrote: >> Hi >> >> On 2018-11-22 00:22, swedebugia wrote: >> snip >> > snip > >> While investigating the top libraries* packages with most depends in >> npm I found the following: >> >> Lib Dep DevDep RecdevDep Dependants license >> underscore 0 12 2400+ 18000+ mit >> async 1 37 2696 26069 mit > > I succeded in packaging the latest underscore. I discovered a bug in > the build system and fixed it to be able to build it. (see the diff > attached) > > Debian packaged a 4 year old async 0.8.0 that has 0 dep and only > uglify-js as devdep (which we already have packaged). From 0.9.2 > lodash was made a devdep. Thanks, I'll merge that with my version of the patches, and send them both for review. I think they are already very good, unless you think we need more time to make sure the build system is perfect. Could you explain why this is necessary? In what case does this directory doesn't exist? I also wonder if it can be the case that node_modules is a file or a symlink that isn't catched by your check... ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Packaging async and underscore 2018-11-30 16:08 ` Packaging async and underscore Julien Lepiller @ 2018-11-30 16:44 ` swedebugia 0 siblings, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-30 16:44 UTC (permalink / raw) To: guix-devel, Julien Lepiller [-- Attachment #1: Type: text/plain, Size: 840 bytes --] Hi :) On 2018-11-30 17:08, Julien Lepiller wrote: > Le 2018-11-30 15:17, swedebugia a écrit : snip > > Thanks, I'll merge that with my version of the patches, and send them > both for review. I think they are already very good, unless you think we > need more time to make sure the build system is perfect. Sounds good to me :) We can improve it further in-tree. > Could you explain why this is necessary? In what case does this > directory doesn't exist? I also wonder if it can be the case that > node_modules is a file or a symlink that isn't catched by your check... The following patch triggers the error. I build it again today and saw that my fix seemed not to work anymore for some reason I do not understand :-/ See this error log: https://paste.debian.net/1053871/. -- Cheers Swedebugia [-- Attachment #2: 0001-gnu-Add-node-underscore.patch --] [-- Type: text/x-patch, Size: 2959 bytes --] From cdfbaa7c882a810756ec34ad3a4f885960c8a970 Mon Sep 17 00:00:00 2001 From: swedebugia <swedebugia@riseup.net> Date: Fri, 30 Nov 2018 17:30:50 +0100 Subject: [PATCH] gnu: Add node-underscore * gnu/packages/node.scm(node-underscore): New variable --- gnu/packages/node.scm | 47 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm index 70bbb1779..580071fb1 100644 --- a/gnu/packages/node.scm +++ b/gnu/packages/node.scm @@ -22,6 +22,10 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. +;; The node-build-system allows two ways to install node: +;; Locally and globally. Default is "local". +;; See https://nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation/ + (define-module (gnu packages node) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages) @@ -38,6 +42,7 @@ #:use-module (gnu packages icu4c) #:use-module (gnu packages libevent) #:use-module (gnu packages linux) + #:use-module (gnu packages lisp) #:use-module (gnu packages perl) #:use-module (gnu packages pkg-config) #:use-module (gnu packages python) @@ -547,3 +552,45 @@ commonjs, browser, AMD, electron, etc.") (home-page "https://github.com/kumavis/browser-stdout") (license license:expat))) + +(define-public node-underscore + (package + (name "node-underscore") + (version "1.9.1") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/jashkenas/underscore") + (commit "ae037f7c41323807ae6f1533c45512e6d31a1574"))) + (file-name (string-append name "-" version)) + (sha256 + (base32 + "1f75wrln5kv5ihkbb9zwhyjqd9imwil801abhv36w09dkkabpjy5")))) + (build-system node-build-system) + (native-inputs `(("uglify-js" ,uglify-js))) + (arguments + `(#:tests? #f + ;; FIXME: These test dependencies are missing: + ;; "karma": "^0.13.13", + ;; "karma-qunit": "~2.0.1", + ;; "karma-sauce-launcher": "^1.2.0", + ;; "nyc": "^2.1.3", + ;; "qunit-cli": "~0.2.0", + ;; "qunit": "^2.6.0", + #:phases ;No configure script + (modify-phases %standard-phases + (delete 'configure) + (add-before 'build 'patch-files + (lambda* (#:key inputs #:allow-other-keys) + ;; Fix wrong path to uglify-js + (substitute* "package.json" + (("uglifyjs") "uglify-js"))))))) + (synopsis + "JavaScript's functional programming helper library.") + (description + "Underscore.js is a utility-belt library for JavaScript that provides +support for the usual functional suspects (each, map, reduce, filter...) +providing over 100 functions.") + (home-page "http://underscorejs.org") + (license license:expat))) -- 2.19.1 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-20 7:50 ` Julien Lepiller 2018-11-20 19:58 ` swedebugia @ 2018-11-24 13:42 ` swedebugia 1 sibling, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-24 13:42 UTC (permalink / raw) To: Julien Lepiller, guix-devel Hi On 2018-11-20 08:50, Julien Lepiller wrote: snip > I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches. I don't see it in guix yet. Could you push it? I corrected a bug in node-build-system and added node-underscore. See mangled changes below (mangled by hard-wrap). I will create commit and create 2 patches when I have somewhere to share it: modified gnu/packages/node.scm @@ -22,6 +22,10 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. +;; The node-build-system allows two ways to install node: +;; Locally and globally. Default is "local". +;; See https://nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation/ + (define-module (gnu packages node) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages) @@ -38,6 +42,7 @@ #:use-module (gnu packages icu4c) #:use-module (gnu packages libevent) #:use-module (gnu packages linux) + #:use-module (gnu packages lisp) #:use-module (gnu packages perl) #:use-module (gnu packages pkg-config) #:use-module (gnu packages python) @@ -547,3 +552,45 @@ commonjs, browser, AMD, electron, etc.") (home-page "https://github.com/kumavis/browser-stdout") (license license:expat))) + +(define-public node-underscore + (package + (name "node-underscore") + (version "1.9.1") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/jashkenas/underscore") + (commit "ae037f7c41323807ae6f1533c45512e6d31a1574"))) + (file-name (string-append name "-" version)) + (sha256 + (base32 + "1f75wrln5kv5ihkbb9zwhyjqd9imwil801abhv36w09dkkabpjy5")))) + (build-system node-build-system) + (native-inputs `(("uglify-js" ,uglify-js))) + (arguments + `(#:tests? #f + ;; FIXME: These test dependencies are missing: + ;; "karma": "^0.13.13", + ;; "karma-qunit": "~2.0.1", + ;; "karma-sauce-launcher": "^1.2.0", + ;; "nyc": "^2.1.3", + ;; "qunit-cli": "~0.2.0", + ;; "qunit": "^2.6.0", + #:phases ;No configure script + (modify-phases %standard-phases + (delete 'configure) + (add-before 'build 'patch-files + (lambda* (#:key inputs #:allow-other-keys) + ;; Fix wrong path to uglify-js + (substitute* "package.json" + (("uglifyjs") "uglify-js"))))))) + (synopsis + "JavaScript's functional programming helper library.") + (description + "Underscore.js is a utility-belt library for JavaScript that provides +support for the usual functional suspects (each, map, reduce, filter...) +providing over 100 functions.") + (home-page "http://underscorejs.org") + (license license:expat))) modified guix/build/node-build-system.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson <davet@gnu.org> ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org> +;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;; @@ -109,8 +110,12 @@ is an npm global install." (mkdir-p tgt-dir) (copy-recursively "." (string-append tgt-dir "/node_modules/" modulename)) ;; Remove references to dependencies - (delete-file-recursively - (string-append tgt-dir "/node_modules/" modulename "/node_modules")) + ;; FIXME: clarify why this is done. + (let ((tgt-node-modules (string-append tgt-dir "/node_modules/" modulename "/node_modules"))) + ;; Check if tgt-node-modules exist and remove them if #t + (when (file-is-directory? tgt-node-modules) + (delete-file-recursively + tgt-node-modules))) (when global? (cond ((string? bin-conf) -- Cheers Swedebugia ^ permalink raw reply [flat|nested] 31+ messages in thread
* Improved NPM importer with blacklist (Was: Re: NPM importer) 2018-11-11 15:37 ` Julien Lepiller 2018-11-19 23:29 ` swedebugia @ 2018-11-30 16:13 ` swedebugia 2018-11-30 16:24 ` Improved NPM importer with blacklist Julien Lepiller 2018-11-30 23:27 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 1 sibling, 2 replies; 31+ messages in thread From: swedebugia @ 2018-11-30 16:13 UTC (permalink / raw) To: Julien Lepiller, guix-devel [-- Attachment #1: Type: text/plain, Size: 2505 bytes --] Hi :) On 2018-11-11 16:37, Julien Lepiller wrote: > I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm > > There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles). There is a stub of a recursive importer, but it does not seem to work. -- I improved on Julliens work and added a blacklister. After much sweat it now works and the blacklist is populated with a lot of unneeded development dependencies and complex packages. I added the version to all imported npm-packages and to the inputs as it does not make sense to reference the packages without a version given all the cyclic dependencies. Thanks for all the help! TODO: * make npm-recursive-import work by not fetching blacklisted packages * implement keyword blacklisting based on the descriptions * match not just the whole string of blacklisted packages: e.g. match also "rollup-plugin" when "rollup" is in the blacklist. * get the tarballs from npm-registry instead as they are never missing (githubs sometimes are) and likely reproducible. * Output a (define-public <guixname> (inherit <guixname>-<version>)) for all imported npm-packages. * Make it possible to specify a specific version to import (and perhaps the latest of all minor versions of a package :D). (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way up to "2.6.1" which is the current beast. This would mean that we in total import about 477.000 packages times the number of minor releases (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will definitely need to speed up guile. My guess is that we will have to import at least 1,5 versions for every npm package to mitigate cyclic dependencies (this means 477.000*1,5 = 715.500 npm-package-versions). * Make it easy to analyze a given npm-package to see when deps/devdeps were added. In the case async, propose we import 0.9.0 first which is the last version without lodash as devdep. From 1.0.0 more devdeps were added. (source: https://registry.npmjs.org/async) Perhaps some kind of tree output for these complex packages with versions as branches and dependencies as subbranches would be nice? -- See the files I changed attached. -- Cheers Swedebugia [-- Attachment #2: npm.scm.bz2 --] [-- Type: application/x-bzip, Size: 5882 bytes --] [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: npm-blacklist.scm --] [-- Type: text/x-scheme; name="npm-blacklist.scm", Size: 6251 bytes --] ;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;; ;;; GNU Guix is free software; you can redistribute it and/or modify it ;;; under the terms of the GNU General Public License as published by ;;; the Free Software Foundation; either version 3 of the License, or (at ;;; your option) any later version. ;;; ;;; GNU Guix is distributed in the hope that it will be useful, but ;;; WITHOUT ANY WARRANTY; without even the implied warranty of ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;;; GNU General Public License for more details. ;;; ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (define-module (guix import npm-blacklist) #:export (blacklist keyword-blacklist)) ;; Many of these are already in debian. (define blacklist-benchmark '( "matcha" "benchmark" )) (define blacklist-complex ;; Complex packages we should not begin with '( ;; From debian wiki /Javascript: ;; Packages used to transform code written for nodejs to run in ;; the browser: "babel" "@babel" "babel-core" "webpack" "rollup" "rollup-plugin-node-resolve" "browserify" ;This can be replaced by one ;of the above. ;; jquery dependant "async" "ember" ;A JavaScript framework for creating ambitious web applications "broccoli" ;Browser compilation library "nsp" ;security related ;; Other complex packages not yet in debian: "electron" "statsd" "vega" "grunt-release" "lineman" "lineman-angular" "lerna" "openlayers" "openpgp" "yarnpkg" "wekan" "etherpad-lite" "meteor" )) (define blacklist-deprecated '( ;; From debian wiki: "keybase" )) (define blacklist-doc '( ;; From underscore: "docco" ;; From mocha: "jsdoc" "markdown-toc")) (define blacklist-lint '( ;; From underscore: "eslint" ;; From mocha: "eslint-config-prettier" "eslint-config-semistandard" "eslint-config-standard" "eslint-plugin-import" "eslint-plugin-node" "eslint-plugin-prettier" "eslint-plugin-promise" "eslint-plugin-standard" "lint-staged" "markdownlint-cli" "prettier-eslint-cli" ;;From yargs "standard" ;linter & automatic code fixer "standard-version" ;Automate versioning and CHANGELOG generation ;;From tap "jscs" ;JavaScript Code Style checker ;; From nsp "eslint-config-nodesecurity" ;; From "@ljharb/eslint-config" ;; From eslint-config "editorconfig-tools" ;A tool to verify ;; From requirejs "jshint" ;; from path-is-absolute "xo" ;; from glob "tick" ;; video.js "tsml" ;; callsites "tsd-check" ;; promptly "@commitlint/config-conventional" "eslint-config-moxy" "@commitlint/cli" ;; async "jshint-stylish" ;; fs-extra "standard-markdown" ;; is "safe-publish-latest" )) (define blacklist-grunt '( ;; From tap: "grunt" ;The JavaScript Task Runner - is in ;debian "grunt-contrib-concat" "grunt-contrib-uglify" ;; From jquery "grunt-compare-size" )) (define blacklist-management '( ;; From parser5: "lerna" ;; CI "vows")) (define blacklist-other '( ;; From mocha: "husky" "nps" "rimraf" ;complex also ;;async "colors" ;cli "cli-table" )) (define blacklist-test-assertion '( ;; mocha "chai" ;Chai is an assertion library. "unexpected" )) (define blacklist-test-browser '( ;; mocha "karma" "karma-qunit" "karma-sauce-launcher" ;; async "karma-browserify" )) (define blacklist-test-coverage '( ;; mocha "coveralls" ;is in debian "covert" ;code coverage command "nyc" )) (define blacklist-test-general ;; sinon and tape are the most common '( ;; From underscore: "qunit-cli" "qunit" ;; From yargs "mocha" ;is in debian "yargs-test-extends" ;; From commander "sinon" ;Test spies, stubs and mocks ;; From nsp "code" ;assertion, fork of chai ;; From minimist "tape" ;tap-producing test harness ;; universalify "colortape" "airtap" "tap" ;Test-Anything-Protocol ;library ;; klaw "tap-spec" "lab" ;Node.js test framework ;; From nsp "mock-fs" ;; From many :) "nyc" ;istanbul coverage, testing ;; from is-wsl "proxyquire" ;; mocha "coffe-script" "cross-swawn" ;; secure-random "terst" ;; through2 "stream-spigot" ;; is "make-generator-function" ;; From github search "test" javascript, sort by stars "forking-tap" "bogota" "faucet" "gremlins" "jest" "redux-mock-store" "webdriverio" "ava" "enzyme" "testem" "protractor" "testcafe" "test262" "storybook" "jasmine" "volkswagen" "nightwatch" "supertest" "istanbul" "sizzy" "zombie" "PhantomCSS" )) (define blacklist ;; Combine blacklists into one. (append blacklist-benchmark blacklist-complex blacklist-deprecated blacklist-doc blacklist-lint blacklist-grunt blacklist-management blacklist-other blacklist-test-assertion blacklist-test-browser blacklist-test-coverage blacklist-test-general )) (define keyword-blacklist ;; Blacklist based on description or keywords '("assert" "assertion" "check" "test" "testing" "mock" "coverage" "lint" "linter" "security")) ;; Debugging ;; (format (current-error-port) "Blacklisted: ~a \n" blacklist) ;; (force-output) ;; (format (current-error-port) "Total: ~a packages blacklisted \n" (length blacklist)) ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Improved NPM importer with blacklist 2018-11-30 16:13 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia @ 2018-11-30 16:24 ` Julien Lepiller 2018-11-30 17:20 ` swedebugia 2018-11-30 23:27 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 1 sibling, 1 reply; 31+ messages in thread From: Julien Lepiller @ 2018-11-30 16:24 UTC (permalink / raw) To: swedebugia; +Cc: guix-devel Le 2018-11-30 17:13, swedebugia a écrit : > Hi :) > > On 2018-11-11 16:37, Julien Lepiller wrote: >> I improved a bit over jlicht's work here, but there still a few tgings >> we want co work on: https://framagit.org/tyreunom/guix/tree/npm >> >> There is an importer and a build system as well as a few packages. One >> of tge issue is that the importer is not recursive, so it doesn't get >> the specified version, and the packages aren't tested because the >> tests depend on cylic dependencies (and sometimes very big circles). > > There is a stub of a recursive importer, but it does not seem to work. > > -- > > I improved on Julliens work and added a blacklister. After much sweat > it now works and the blacklist is populated with a lot of unneeded > development dependencies and complex packages. > > I added the version to all imported npm-packages and to the inputs as > it does not make sense to reference the packages without a version > given all the cyclic dependencies. > > Thanks for all the help! Hi, I never used the recursive importer, so I didn't know it wasn't very good. I wonder if we really need to import every version of the packages. That doesn't seem very practical. There are a few cyclic dependencies issues in Java packages too, and they are dealt with in a case-by-case basis. Most often, we made a degraded version of one of the packages, the second can use to build itself, then we rebuild the first with the second package. Sometimes, we also have to adapt some of our packages for the newer versions of the dependencies we have. If we didn't, we'd have a lot of versions of every package, and most of them would be outdated, probably buggy or contain security holes. I'd prefer using the latest versions of dependencies, and contribute patches back to upstream, so they can use the latest and greatest too :) That's obviously a lot more work, but that's also probably a saner way of doing things. > > TODO: > * make npm-recursive-import work by not fetching blacklisted packages Let's be careful though: we don't want to fetch blacklisted packages when they are devDependencies, but we still want them if they are runtime dependencies. > > * implement keyword blacklisting based on the descriptions We can probably use tags instead of the description : '("test" "testing" "check" "doc" "coverage" "unit") seem like a good approximation of what we want to blacklist. > > * match not just the whole string of blacklisted packages: > e.g. match also "rollup-plugin" when "rollup" is in the blacklist. > > * get the tarballs from npm-registry instead as they are never missing > (githubs sometimes are) and likely reproducible. Are they actual source tarballs, or are they somewhat different than the source used to build the "binary" npm package? With maven (for java) for instance, some sources are hosted, but they aren't supposed to be used to build the package, they're only here for the debugger. > > * Output a (define-public <guixname> (inherit <guixname>-<version>)) > for > all imported npm-packages. I don't think that's a good idea: if we have multiple versions of a package, we'll have multiple <guixname> packages... > > * Make it possible to specify a specific version to import (and perhaps > the latest of all minor versions of a package :D). > (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way > up to "2.6.1" which is the current beast. This would mean that we in > total import about 477.000 packages times the number of minor releases > (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will > definitely need to speed up guile. My guess is that we will have to > import at least 1,5 versions for every npm package to mitigate cyclic > dependencies (this means 477.000*1,5 = 715.500 npm-package-versions). Again, I'm more in favor of patching them, rather than importing more versions. Do we still have as many cyclic deps with the blacklist? > > * Make it easy to analyze a given npm-package to see when deps/devdeps > were added. In the case async, propose we import 0.9.0 first which is > the last version without lodash as devdep. From 1.0.0 more devdeps > were added. (source: https://registry.npmjs.org/async) > > Perhaps some kind of tree output for these complex packages with > versions as branches and dependencies as subbranches would be nice? > -- > > See the files I changed attached. Thanks for your work! ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Improved NPM importer with blacklist 2018-11-30 16:24 ` Improved NPM importer with blacklist Julien Lepiller @ 2018-11-30 17:20 ` swedebugia 0 siblings, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-30 17:20 UTC (permalink / raw) To: Julien Lepiller; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 4946 bytes --] On 2018-11-30 17:24, Julien Lepiller wrote: > Le 2018-11-30 17:13, swedebugia a écrit : snip > Hi, > > I never used the recursive importer, so I didn't know it wasn't very good. > > I wonder if we really need to import every version of the packages. That > doesn't seem very practical. There are a few cyclic dependencies issues > in Java packages too, and they are dealt with in a case-by-case basis. > Most often, we made a degraded version of one of the packages, the > second can use to build itself, then we rebuild the first with the > second package. Sounds good. > Sometimes, we also have to adapt some of our packages for the newer > versions of the dependencies we have. If we didn't, we'd have a lot of > versions of every package, and most of them would be outdated, probably > buggy or contain security holes. I'd prefer using the latest versions of > dependencies, and contribute patches back to upstream, so they can use > the latest and greatest too :) > > That's obviously a lot more work, but that's also probably a saner way > of doing things. Agreed, this seems better. With a good tree browser we can probably avoid importing more than 2-5 versions of the worst packages. I collected a few cyclic devdeps. See attached. (these definitions is of little value as the versions of deps and devdeps are discarded) >> TODO: >> * make npm-recursive-import work by not fetching blacklisted packages > > Let's be careful though: we don't want to fetch blacklisted packages > when they are devDependencies, but we still want them if they are > runtime dependencies. Totally agree. This is exactly why I only implemented blacklisting of native-inputs. >> * implement keyword blacklisting based on the descriptions > > We can probably use tags instead of the description : '("test" "testing" > "check" "doc" "coverage" "unit") seem like a good approximation of what > we want to blacklist. Fewer that half the npm packages have tags to my knowledge. We can do both though :D > >> >> * match not just the whole string of blacklisted packages: >> e.g. match also "rollup-plugin" when "rollup" is in the blacklist. >> >> * get the tarballs from npm-registry instead as they are never missing >> (githubs sometimes are) and likely reproducible. > > Are they actual source tarballs, or are they somewhat different than the > source used to build the "binary" npm package? With maven (for java) for > instance, some sources are hosted, but they aren't supposed to be used > to build the package, they're only here for the debugger. Fortunately it seems it is the full source. :D See https://registry.npmjs.org/underscore/-/underscore-1.9.1.tgz https://registry.npmjs.org/nodeunit/-/nodeunit-0.5.5.tgz https://registry.npmjs.org/async/-/async-0.9.0.tgz > >> >> * Output a (define-public <guixname> (inherit <guixname>-<version>)) for >> all imported npm-packages. > > I don't think that's a good idea: if we have multiple versions of a > package, we'll have multiple <guixname> packages... Ok, got it. I thought the define-publics would collide, but I guess not. > >> >> * Make it possible to specify a specific version to import (and perhaps >> the latest of all minor versions of a package :D). >> (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way >> up to "2.6.1" which is the current beast. This would mean that we in >> total import about 477.000 packages times the number of minor releases >> (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will >> definitely need to speed up guile. My guess is that we will have to >> import at least 1,5 versions for every npm package to mitigate cyclic >> dependencies (this means 477.000*1,5 = 715.500 npm-package-versions). > > Again, I'm more in favor of patching them, rather than importing more > versions. Do we still have as many cyclic deps with the blacklist? No, the blacklist makes a BIG difference (but only to the cycdevdeps. The deps still introduce just as many cycles. These can be avoided by carefully choosing a version just before the cycdep was added :) (or by patching but I know nothing about JS so I leave that to others) >> * Make it easy to analyze a given npm-package to see when deps/devdeps >> were added. In the case async, propose we import 0.9.0 first which is >> the last version without lodash as devdep. From 1.0.0 more devdeps >> were added. (source: https://registry.npmjs.org/async) >> >> Perhaps some kind of tree output for these complex packages with >> versions as branches and dependencies as subbranches would be nice? I will try parsing the registry to output something intelligently to help the user choose which version to import. > Thanks for your work! Thanks for sharing so we can improve this together :) -- Cheers Swedebugia [-- Attachment #2: node-cyclic.scm --] [-- Type: text/x-scheme, Size: 5635 bytes --] (define-public node-rimraf (package (name "node-rimraf") (version "2.6.2") (source (origin (method url-fetch) (uri "https://github.com/isaacs/rimraf/archive/v2.6.2/rimraf-v2.6.2.tar.gz") (sha256 (base32 "0bmssxz3s30nhq5f8ldssf6s8ga5w0aarn71wjsmvqb1j15b2r6d")))) (build-system node-build-system) (inputs `(("node-glob" ,node-glob))) (native-inputs `( ;; tests("node-tap" ,node-tap) ("node-mkdirp" ,node-mkdirp))) (synopsis "A deep deletion module for node (like `rm -rf`)") (description "A deep deletion module for node (like `rm -rf`)") (home-page "https://github.com/isaacs/rimraf#readme") (license license:isc))) (define-public node-glob (package (name "node-glob") (version "7.1.3") (source (origin (method url-fetch) (uri "https://github.com/isaacs/node-glob/archive/v7.1.3/node-glob-v7.1.3.tar.gz") (sha256 (base32 "0qcymwljbm947gvfn7g7871dnwv5s0jq0r8c8ih9xgrfcynfw3hx")))) (build-system node-build-system) (inputs `(("node-inflight" ,node-inflight) ("node-once" ,node-once) ("node-path-is-absolute" ,node-path-is-absolute) ("node-minimatch" ,node-minimatch) ("node-fs.realpath" ,node-fs.realpath) ("node-inherits" ,node-inherits))) (native-inputs `( ;; benchm ("node-tick" ,node-tick) ;; tests ("node-tap" ,node-tap) ("node-rimraf" ,node-rimraf) ("node-mkdirp" ,node-mkdirp))) (synopsis "a little globber") (description "a little globber") (home-page "https://github.com/isaacs/node-glob#readme") (license license:isc))) (define-public node-jasmine-core (package (name "node-jasmine-core") (version "3.3.0") (source (origin (method url-fetch) (uri "https://github.com/jasmine/jasmine/archive/v3.3.0/jasmine-v3.3.0.tar.gz") (sha256 (base32 "1rg4p487hf8mlxcj99wywzwp7jp3s4d114n4j12r3mkh8qyi8nck")))) (build-system node-build-system) (inputs `()) (native-inputs `( ;; ("node-grunt" ,node-grunt) ;; ("node-grunt-contrib-compass" ;; ,node-grunt-contrib-compass) ("node-jsdom" ,node-jsdom) ("node-shelljs" ,node-shelljs) ("node-jasmine" ,node-jasmine) ;; ("node-load-grunt-tasks" ,node-load-grunt-tasks) ;; ("node-grunt-contrib-compress" ;; ,node-grunt-contrib-compress) ;; ("node-grunt-contrib-concat" ;; ,node-grunt-contrib-concat) ;; ("node-grunt-cli" ,node-grunt-cli) ("node-temp" ,node-temp) ("node-glob" ,node-glob) ;; ;; ("node-grunt-contrib-jshint" ;; ,node-grunt-contrib-jshint) )) (synopsis "Official packaging of Jasmine's core files for use by Node.js projects.") (description "Official packaging of Jasmine's core files for use by Node.js projects.") (home-page "https://jasmine.github.io") (license license:expat))) (define-public node-jasmine (package (name "node-jasmine") (version "3.3.0") (source (origin (method url-fetch) (uri "https://github.com/jasmine/jasmine-npm/archive/v3.3.0/jasmine-npm-v3.3.0.tar.gz") (sha256 (base32 "1b6mgxmxv71bpr4fg75azfyh1v0m469prb7srg990fkf7i5bszw9")))) (build-system node-build-system) (inputs `(("node-jasmine-core" ,node-jasmine-core) ("node-glob" ,node-glob))) (native-inputs `(("node-grunt" ,node-grunt) ("node-shelljs" ,node-shelljs) ("node-grunt-cli" ,node-grunt-cli) ("node-grunt-contrib-jshint" ,node-grunt-contrib-jshint))) (synopsis "Command line jasmine") (description "Command line jasmine") (home-page "http://jasmine.github.io/") (license license:expat))) (define-public node-domhandler (package (name "node-domhandler") (version "2.4.2") (source (origin (method url-fetch) (uri "https://github.com/fb55/DomHandler/archive/v2.4.2/DomHandler-v2.4.2.tar.gz") (sha256 (base32 "16hi0vapmavw9g9s321b4c9nvwfg06cclj7pjnvjzk0imnzxjngp")))) (build-system node-build-system) (inputs `(("node-domelementtype" ,node-domelementtype))) (native-inputs `(("node-htmlparser2" ,node-htmlparser2) ("node-jshint" ,node-jshint) ("node-mocha" ,node-mocha))) (synopsis "handler for htmlparser2 that turns pages into a dom") (description "handler for htmlparser2 that turns pages into a dom") (home-page "https://github.com/fb55/DomHandler#readme") (license #f))) (define-public node-htmlparser2 (package (name "node-htmlparser2") (version "3.10.0") (source (origin (method url-fetch) (uri "https://github.com/fb55/htmlparser2/archive/v3.10.0/htmlparser2-v3.10.0.tar.gz") (sha256 (base32 "1qvsv4aixmgnh4h7q726wapg7qnk7srw4z9nmy71jc5r2krimnvn")))) (build-system node-build-system) (inputs `(("node-readable-stream" ,node-readable-stream) ("node-domhandler" ,node-domhandler) ("node-domelementtype" ,node-domelementtype) ("node-inherits" ,node-inherits) ("node-domutils" ,node-domutils) ("node-entities" ,node-entities))) (native-inputs `(("node-eslint" ,node-eslint) ("node-coveralls" ,node-coveralls) ("node-istanbul" ,node-istanbul) ("node-mocha-lcov-reporter" ,node-mocha-lcov-reporter) ("node-mocha" ,node-mocha))) (synopsis "Fast & forgiving HTML/XML/RSS parser") (description "Fast & forgiving HTML/XML/RSS parser") (home-page "https://github.com/fb55/htmlparser2#readme") (license license:expat))) ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Improved NPM importer with blacklist (Was: Re: NPM importer) 2018-11-30 16:13 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 2018-11-30 16:24 ` Improved NPM importer with blacklist Julien Lepiller @ 2018-11-30 23:27 ` swedebugia 1 sibling, 0 replies; 31+ messages in thread From: swedebugia @ 2018-11-30 23:27 UTC (permalink / raw) To: guix-devel [-- Attachment #1: Type: text/plain, Size: 710 bytes --] Hi I programmed the whole day and learned a lot. :D On 2018-11-30 17:13, swedebugia wrote: snip > TODO: > * make npm-recursive-import work by not fetching blacklisted packages See the attached patches :) > > * implement keyword blacklisting based on the descriptions > > * match not just the whole string of blacklisted packages: > e.g. match also "rollup-plugin" when "rollup" is in the blacklist. Done. See patch > * get the tarballs from npm-registry instead as they are never missing > (githubs sometimes are) and likely reproducible. Done, but not tested yet. Also added a npm-uri (similar to pypi-uri) see patch. Happy hacking! -- Cheers Swedebugia [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-guix-Add-a-npm-uri-helper-method.patch --] [-- Type: text/x-patch; name="0001-guix-Add-a-npm-uri-helper-method.patch", Size: 1708 bytes --] From 9ad29ede5b26a9052518adb422bc66398c38c9b4 Mon Sep 17 00:00:00 2001 From: swedebugia <swedebugia@riseup.net> Date: Fri, 30 Nov 2018 20:52:42 +0100 Subject: [PATCH 1/2] guix: Add a "npm-uri" helper method. * guix/build-system/node.scm (npm-uri): New method. --- guix/build-system/node.scm | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm index b6d362904..ce0d9f84c 100644 --- a/guix/build-system/node.scm +++ b/guix/build-system/node.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org> +;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;; @@ -30,7 +31,22 @@ #:export (npm-meta-uri %node-build-system-modules node-build - node-build-system)) + node-build-system + npm-uri)) + +;; Commentary: +;; +;; Standard build procedure for Node packages using 'packages.json' and +;; Makefiles. This is implemented as an extension of 'gnu-build-system'. +;; +;; Code: + +(define* (npm-uri name version #:optional (extension ".tgz")) + "Return a URI string for the NPM package hosted on the NPM Registry +corresponding to NAME and VERSION. EXTENSION is the file name extension, such +as '.tar.gz'. E.g. https://registry.npmjs.org/async/-/async-1.8.0.tgz" + (string-append "https://registry.npmjs.org/" name "/-/" name "-" version + extension)) (define (npm-meta-uri name) "Return a URI string for the metadata of node module NAME found in the npm -- 2.19.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-guix-Add-blacklisting-to-recursive-importer.patch --] [-- Type: text/x-patch; name="0002-guix-Add-blacklisting-to-recursive-importer.patch", Size: 6473 bytes --] From 54515dbb9e30e3cdf0ac1835c465f157302aefd1 Mon Sep 17 00:00:00 2001 From: swedebugia <swedebugia@riseup.net> Date: Sat, 1 Dec 2018 00:15:08 +0100 Subject: [PATCH 2/2] guix: Add blacklisting to recursive importer * guix/import/npm.scm (blacklisted?, npm->guix-package): Remove blacklisted packages from the list passed to the recursive importer. * guix/scripts/import/npm.scm (show-help, %options): Document and improve "recursive" option. * doc/guix.texi (Running guix import): Document the blacklist function. Inspired by ac906cb7bb2ec77821ddec291db4857cc812599d by Ricardo --- doc/guix.texi | 33 ++++++++++++++++++++++++++++++++- guix/import/npm.scm | 16 ++++++++++++---- guix/scripts/import/npm.scm | 10 ++++++++-- 3 files changed, 52 insertions(+), 7 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 8f694a20e..ccb3c45f1 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -7258,7 +7258,7 @@ guix import npm minimist @end example The importer implements an experimental recursive subsystem, allowing -for a significant of npm packages to be imported. The command below +for a significant number of npm packages to be imported. The command below imports metadata for the @code{optimist} npm package, as well as its dependencies. @@ -7266,6 +7266,37 @@ dependencies. guix import npm -r optimist @end example +The importer currently has does not warn when a cyclic dependency has been +encountered. It will run until the buffer overruns which could take a long +time. You should generally see output every couple of seconds, if not a cyclic +dependency has probably been encountered. These are best avoided by first +packaging an earlier version of one of the packages without the offending +input to avoid the cycle. + +@quotation Note +We have implemented an experimental blacklist functionality, enabled by +default, to help you avoid importing a lot of garbage development +dependencies. You can customize this blacklist by editing +@file{guix/import/npm-blacklist.scm} only when running @command{guix import +npm} from a git checkout, see @pxref{Running Guix Before It Is Installed}. + +These packages and every package that begins with these 103 names are +currently blacklisted: (matcha benchmark babel @babel webpack rollup +browserify async ember broccoli nsp uglifyjs2 uglify-js electron statsd vega +grunt-release lineman lerna openlayers openpgp yarnpkg wekan etherpad-lite +meteor keybase docco jsdoc markdown eslint lint markdownlint prettier standard +jscs @ljharb/eslint-config editorconfig-tools jshint xo tick tsml +tsd-check @commitlint safe grunt lerna vows husky nps rimraf colors cli +source-map chai unexpected karma karma-qunit karma-sauce-launcher +karma-browserify coveralls covert nyc qunit mocha yargs sinon code tape +colortape airtap tap tap-spec lab mock nyc proxyquire coffe-script cross-swawn +terst stream-spigot make-generator-function forking-tap bogota faucet gremlins +jest redux-mock-store webdriverio ava enzyme testem protractor testcafe +test262 storybook jasmine volkswagen nightwatch supertest istanbul sizzy +zombie PhantomCSS) +This expands to thousands of packages. +@end quotation + @item elpa @cindex elpa Import metadata from an Emacs Lisp Package Archive (ELPA) package diff --git a/guix/import/npm.scm b/guix/import/npm.scm index 2efe81fa9..1d46fd384 100644 --- a/guix/import/npm.scm +++ b/guix/import/npm.scm @@ -350,8 +350,12 @@ a git checkout." ,(guix-hash-url temp-dir))))))))))) (define (blacklisted? name) - "Check if the pair name-version is blacklisted. RETURN #t if yes, else #f." - (member name blacklist)) + "Check if the string name is blacklisted. RETURN #t if yes, else #f." + ;; Split the string to enable ut so blacklist scoped packages like + ;; @babel/core and packages like eslint-popup without having to type in + ;; every single combination. + (if (member (car (string-split name (char-set #\- #\/))) blacklist) + #t #f)) (define (sanitize-npm-version version) "Return version without prefixed ^." @@ -518,8 +522,12 @@ npm list of dependencies DEPENDENCIES." (npm-dependencies (append (extract-npm-dependencies dependencies) - ;; TODO blacklist! - (extract-npm-dependencies dev-dependencies))) + ;; Remove the blacklisted devdeps to avoid big dependency + ;; cycles and all the linters, cli-tools, benchmarking, + ;; etc. JS-devs adds to their development environment. + (remove + blacklisted? + (extract-npm-dependencies dev-dependencies)))) (description (assoc-ref package "description")) (home-page (assoc-ref package "homepage")) (license (extract-license curr)) diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm index 1f4ed777c..c540c3574 100644 --- a/guix/scripts/import/npm.scm +++ b/guix/scripts/import/npm.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson <davet@gnu.org> +;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;; @@ -25,6 +26,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37) + #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-npm)) @@ -41,7 +43,9 @@ Import and convert the npm package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) - (display (G_ " + (display (G_ " + -r, --recursive import packages recursively")) + (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information)) @@ -88,7 +92,9 @@ `(define-public ,(string->symbol name) ,pkg)) (G_ #f)) - (recursive-import package-name)) + (reverse + (stream->list + (npm-recursive-import package-name)))) ;; Single import (let ((sexp (npm->guix-package package-name))) (unless sexp -- 2.19.1 ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-11 10:41 NPM importer swedebugia 2018-11-11 15:37 ` Julien Lepiller @ 2018-11-11 17:10 ` Ludovic Courtès 2018-11-21 16:37 ` Giovanni Biscuolo 1 sibling, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2018-11-11 17:10 UTC (permalink / raw) To: swedebugia; +Cc: guix-devel Hello, swedebugia <swedebugia@riseup.net> skribis: > Has anybody worked on that? Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d). But don’t hold your breath: as Chris Webber explained, the npm situation is very hard to address sanely: http://dustycloud.org/blog/javascript-packaging-dystopia/ Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-11 17:10 ` NPM importer Ludovic Courtès @ 2018-11-21 16:37 ` Giovanni Biscuolo 2018-11-21 17:15 ` Julien Lepiller 0 siblings, 1 reply; 31+ messages in thread From: Giovanni Biscuolo @ 2018-11-21 16:37 UTC (permalink / raw) To: Ludovic Courtès, swedebugia; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 809 bytes --] ludo@gnu.org (Ludovic Courtès) writes: [...] > Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d). But > don’t hold your breath: as Chris Webber explained, the npm situation is > very hard to address sanely: > > http://dustycloud.org/blog/javascript-packaging-dystopia/ (semi OT: today Debian ships a recent jquery 3.2.1) I'm not an expert in js (nor guix) packaging so I'm not able to judge this: https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/ is yarn a viable solution to the NPM packaging problems? can we achieve reproducible builds ala guix with a yarn importer and some amount of yarn packages downloading/automation and offline mirroring? Ciao Giovanni P.S.: why -- Giovanni Biscuolo Xelera IT Infrastructures [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-21 16:37 ` Giovanni Biscuolo @ 2018-11-21 17:15 ` Julien Lepiller 2018-11-22 9:29 ` Giovanni Biscuolo 0 siblings, 1 reply; 31+ messages in thread From: Julien Lepiller @ 2018-11-21 17:15 UTC (permalink / raw) To: Giovanni Biscuolo; +Cc: guix-devel Le 2018-11-21 17:37, Giovanni Biscuolo a écrit : > ludo@gnu.org (Ludovic Courtès) writes: > > [...] > >> Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d). But >> don’t hold your breath: as Chris Webber explained, the npm situation >> is >> very hard to address sanely: >> >> http://dustycloud.org/blog/javascript-packaging-dystopia/ > > (semi OT: today Debian ships a recent jquery 3.2.1) > > I'm not an expert in js (nor guix) packaging so I'm not able to judge > this: > > https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/ > > is yarn a viable solution to the NPM packaging problems? > > can we achieve reproducible builds ala guix with a yarn importer and > some amount of yarn packages downloading/automation and offline > mirroring? How different is it to build an npm package and a yarn package? Could you elaborate a bit on your idea? We can already build packages with our wip node-build-system, as long as we have build- and run-time dependencies available. The real hard parts are: sometimes build-tools depend on what they build, there is just too many dependencies and some packages don't declare a license properly. For instance, grunt is a build tool for node packages; it has 179 dependencies at runtime (including recursive dependencies). All of them need to be built before grunt can be run. What's the chance that none of them require grunt? I haven't taken the time to look at these dependencies, so maybe I'm pessimistic with no good reason. Another instance is application-config-path that declares its license only in the Makefile, in the form of "License: MIT". Do we consider this free software? Now if yarn has some build recipes and has taken the time to make this whole mess more manageable, I'm all for a yarn importer. Otherwise, it's just another source of package information, which is fine, but npm seems to do the job already. > > Ciao > Giovanni > > P.S.: why > > -- > Giovanni Biscuolo > > Xelera IT Infrastructures ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: NPM importer 2018-11-21 17:15 ` Julien Lepiller @ 2018-11-22 9:29 ` Giovanni Biscuolo 0 siblings, 0 replies; 31+ messages in thread From: Giovanni Biscuolo @ 2018-11-22 9:29 UTC (permalink / raw) To: Julien Lepiller; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 1177 bytes --] Hi Guix, sorry: reading at today messages from swdebugia and your comments below I realize that mine was just noise from a packaging-ignorant :-S Julien Lepiller <julien@lepiller.eu> writes: [...] >> https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/ >> >> is yarn a viable solution to the NPM packaging problems? [...] > How different is it to build an npm package and a yarn package? no difference > Could you elaborate a bit on your idea? I'm not able: I was mislead by the title of this article I cited https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/ "reproducuble builds" and "npm/yarn" seems like an oxymoron to me, now :-) > We can already build packages with our wip node-build-system, as long as > we have build- and run-time dependencies available. The real hard parts > are: sometimes build-tools depend on what they build, there is just too > many dependencies and some packages don't declare a license properly. ACK, now I finally understand the problems [...] thany you for your patience ciao Giovanni -- Giovanni Biscuolo Xelera IT Infrastructures [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2018-11-30 23:22 UTC | newest] Thread overview: 31+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-11-11 10:41 NPM importer swedebugia 2018-11-11 15:37 ` Julien Lepiller 2018-11-19 23:29 ` swedebugia 2018-11-20 7:50 ` Julien Lepiller 2018-11-20 19:58 ` swedebugia 2018-11-20 21:12 ` swedebugia 2018-11-20 22:35 ` Julien Lepiller 2018-11-21 15:36 ` swedebugia 2018-11-21 1:41 ` Mike Gerwitz 2018-11-21 22:01 ` Brett Gilio 2018-11-21 23:22 ` swedebugia 2018-11-22 1:02 ` swedebugia 2018-11-22 5:43 ` Brett Gilio 2018-11-22 11:27 ` import libjs-*.deb from Debian? (was Re: NPM importer) Giovanni Biscuolo 2018-11-30 3:23 ` Ricardo Wurmus 2018-11-22 8:36 ` NPM importer Julien Lepiller 2018-11-24 13:47 ` swedebugia 2018-11-23 19:50 ` swedebugia 2018-11-30 3:17 ` Ricardo Wurmus 2018-11-30 14:17 ` Packaging async and underscore (Was: Re: NPM importer) swedebugia 2018-11-30 16:08 ` Packaging async and underscore Julien Lepiller 2018-11-30 16:44 ` swedebugia 2018-11-24 13:42 ` NPM importer swedebugia 2018-11-30 16:13 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 2018-11-30 16:24 ` Improved NPM importer with blacklist Julien Lepiller 2018-11-30 17:20 ` swedebugia 2018-11-30 23:27 ` Improved NPM importer with blacklist (Was: Re: NPM importer) swedebugia 2018-11-11 17:10 ` NPM importer Ludovic Courtès 2018-11-21 16:37 ` Giovanni Biscuolo 2018-11-21 17:15 ` Julien Lepiller 2018-11-22 9:29 ` Giovanni Biscuolo
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.