Re: Help wanted with recursive npm import returning #f

[swedebugia@riseup.net]

On 2018-12-04 23:07, Jelle Licht wrote:
> Hi swedebugia,
> 
> Super swell to see you take an interest in this! :D.

I'm so glad to be part of this community and that I finally understand
more about how to implement my ideas. :)

> 
> Some points;
> 
> It seems you wrote your own sanitize-npm-version, but this is not (at
> all) how npm
> deals with versions; I implore you to have a look at
> https://semver.org/ again to see
> what all the silly squigles mean :).

Thanks for the link. I did not mean for it to do anything other that
provide us with a triplet "?.?.?" to append to input-names.

> 
> A general question on your blacklist approach; I like and appreciate
> this attempt 
> 
> at 'breaking the cycle', but could we not just as well define some
> dummy packages?
> As far as the importer is concerned, a dummy package would still be
> seen as 
> 
> "dependency resolved, my work here is done", or am I missing an
> advantage of 
> 
> your approach?

I had not thought about that! Sounds like a good idea. :)

We are going to need quite a few hundred of these dummy packages though.


The advantage of blacklist is to let the import user self determine what
to add and not to the native inputs.

Actually in this version I completely removed the generation of native
inputs and thus also the blacklisting functionallity.

Maybe we can have both dummy-pkg and blacklist. Preferable we should
implement the ability to add a blacklist.scm on the command line like a
manifest. 

> 
> Op di 4 dec. 2018 om 21:44 schreef <swedebugia@riseup.net>:
> 
>> Hi
>>
>> Introduction
>> ------------
>> Inspired by Ricardos commit here I rewrote most of the npm importer.
>>
>> Added memoization, receive, stream->list, values and rewrote the
>> tarball
>> fetcher to use only npm-uri and tarballs from the registry.
>> Additionally
>> I implemented handling of scoped packages (e.g. @babel/core).
> 
>> It contains less lines of code than Jelles importer.
> 
> And hopefully less places for bugs to hide in.

:D It modeled it very very close to the pypi-importer. So they are very
similar now.

> 
>> The single import works and is a lot faster and more reliable than
>> before when fuzzy matching on github was used. See it in action:
>> http://paste.debian.net/1054384/
> 
> I think it is a step in the _wrong_ direction to depend in major ways
> on the npm 
> registry, besides for meta-information where needed. Nonetheless, the
> fuzzy 
> 
> matching was really brittle as you say, and could have been a lot
> faster indeed. 

We could have 2 importers, one for npm-registry and one for npm-github.
As you say we dont know how long the registry will work. All the 200
packages I have looked is on GH.

> 
>> Caveats:
>> 1) we don't know if the registry-tarballs are reproducible.
> 
> Back in the day, they most definitely were not. Seeing as npm-land
> does not put
> an emphasis on this at all, I think it is unwise to assume that any
> reproducible
> features they offer today will still be available in the future.
> 
>> 2) filename is the same as the upstream tarball -> we should convert
>> it
>> to guix-name.
>> 3) we have to download the tarball because sha256 is not among the
>> hashes computed by npm. (I emailed npm@npmjs.org to ask for them to
>> compute it for all their tarballs :) )
> 
> The result of the importer will probably be a package that we will be 
> 
> building in the near future, right?
> 
>> Help wanted
>> -----------
>>
>> There is a bug which only affects the recursive importer. I tried
>> hard
>> finding it but I'm in way over my head and my guile-foo does not
>> seem to
>> cut it with this one. :)
> 
>> For recursive output it downloads but outputs #f in the end instead
>> of
>> the sexps. See example output: http://paste.debian.net/1054383/
>>
>> Trying to debug via the REPL I met this:
>> scheme@(guile-user) [1]> (npm-recursive-import "async")
>> $3 = #<stream ? ...>
>>
>> Any ideas?
> 
> I think I also ran into this. Could you please make your work
> available on a
> public repo somewhere? It would be easier to look at your changes and
> play
> around with it that way.

Good idea. Is this ok? https://gitlab.com/swedebugia/guix/commits/npm

It was my first push ever! :)

-- 
Cheers 
Swedebugia