From mboxrd@z Thu Jan 1 00:00:00 1970 From: Julien Lepiller Subject: Re: Improved NPM importer with blacklist Date: Fri, 30 Nov 2018 17:24:57 +0100 Message-ID: <0cfa10d0f59225c3897d4fc004722ee2@lepiller.eu> References: <70F182DB-C157-4763-A4C6-89985545661C@lepiller.eu> <12fdf913-eb03-b898-f9ff-8dd455935975@riseup.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSlbG-0000mw-0p for guix-devel@gnu.org; Fri, 30 Nov 2018 11:25:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gSlbB-0006oc-T4 for guix-devel@gnu.org; Fri, 30 Nov 2018 11:25:06 -0500 Received: from lepiller.eu ([2a00:5884:8208::1]:36934) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gSlbB-0006nU-IV for guix-devel@gnu.org; Fri, 30 Nov 2018 11:25:01 -0500 In-Reply-To: <12fdf913-eb03-b898-f9ff-8dd455935975@riseup.net> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: swedebugia Cc: guix-devel@gnu.org Le 2018-11-30 17:13, swedebugia a écrit : > Hi :) > > On 2018-11-11 16:37, Julien Lepiller wrote: >> I improved a bit over jlicht's work here, but there still a few tgings >> we want co work on: https://framagit.org/tyreunom/guix/tree/npm >> >> There is an importer and a build system as well as a few packages. One >> of tge issue is that the importer is not recursive, so it doesn't get >> the specified version, and the packages aren't tested because the >> tests depend on cylic dependencies (and sometimes very big circles). > > There is a stub of a recursive importer, but it does not seem to work. > > -- > > I improved on Julliens work and added a blacklister. After much sweat > it now works and the blacklist is populated with a lot of unneeded > development dependencies and complex packages. > > I added the version to all imported npm-packages and to the inputs as > it does not make sense to reference the packages without a version > given all the cyclic dependencies. > > Thanks for all the help! Hi, I never used the recursive importer, so I didn't know it wasn't very good. I wonder if we really need to import every version of the packages. That doesn't seem very practical. There are a few cyclic dependencies issues in Java packages too, and they are dealt with in a case-by-case basis. Most often, we made a degraded version of one of the packages, the second can use to build itself, then we rebuild the first with the second package. Sometimes, we also have to adapt some of our packages for the newer versions of the dependencies we have. If we didn't, we'd have a lot of versions of every package, and most of them would be outdated, probably buggy or contain security holes. I'd prefer using the latest versions of dependencies, and contribute patches back to upstream, so they can use the latest and greatest too :) That's obviously a lot more work, but that's also probably a saner way of doing things. > > TODO: > * make npm-recursive-import work by not fetching blacklisted packages Let's be careful though: we don't want to fetch blacklisted packages when they are devDependencies, but we still want them if they are runtime dependencies. > > * implement keyword blacklisting based on the descriptions We can probably use tags instead of the description : '("test" "testing" "check" "doc" "coverage" "unit") seem like a good approximation of what we want to blacklist. > > * match not just the whole string of blacklisted packages: > e.g. match also "rollup-plugin" when "rollup" is in the blacklist. > > * get the tarballs from npm-registry instead as they are never missing > (githubs sometimes are) and likely reproducible. Are they actual source tarballs, or are they somewhat different than the source used to build the "binary" npm package? With maven (for java) for instance, some sources are hosted, but they aren't supposed to be used to build the package, they're only here for the debugger. > > * Output a (define-public (inherit -)) > for > all imported npm-packages. I don't think that's a good idea: if we have multiple versions of a package, we'll have multiple packages... > > * Make it possible to specify a specific version to import (and perhaps > the latest of all minor versions of a package :D). > (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way > up to "2.6.1" which is the current beast. This would mean that we in > total import about 477.000 packages times the number of minor releases > (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will > definitely need to speed up guile. My guess is that we will have to > import at least 1,5 versions for every npm package to mitigate cyclic > dependencies (this means 477.000*1,5 = 715.500 npm-package-versions). Again, I'm more in favor of patching them, rather than importing more versions. Do we still have as many cyclic deps with the blacklist? > > * Make it easy to analyze a given npm-package to see when deps/devdeps > were added. In the case async, propose we import 0.9.0 first which is > the last version without lodash as devdep. From 1.0.0 more devdeps > were added. (source: https://registry.npmjs.org/async) > > Perhaps some kind of tree output for these complex packages with > versions as branches and dependencies as subbranches would be nice? > -- > > See the files I changed attached. Thanks for your work!