Cyclic npm dependencies

Cyclic npm dependencies

[swedebugia]

Hi

We need to wonder about what to do with cyclic dependencies.

In the case below node-rimraf has node-glob in input and node-glob has 
node-rimraf in NATIVE-input.

I have no idea how to solve this. Both packages are by the same author.

Same problem with jasmine and jasmine-core

~/guix-tree/pre-inst-env guix build -K node-rimraf
allocate_stack failed: Cannot allocate memory
Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind 
handler.
allocate_stack failed: Cannot allocate memory
Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind 
handler.
allocate_stack failed: Cannot allocate memory
Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind 
handler.


-- 
Cheers Swedebugia

Re: Cyclic npm dependencies

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

Hey swedebugia,

I will still send a more elaborate reply to the general npm-importer
thread later this week, but we can assume that generally these
recursive dependencies can be untangled by looking at the different
versions of the dependencies.

So in your example, I imagine an input chain like:
node-glob 0.1  -> node-rimraf 0.1 -> node-glob 0.2 -> node-rimraf 0.2 ->
.... -> node-glob 1.0 -> node-rimraf 1.0

While *extremely* annoying to untangle, this is definitely doable.
Problems arise if this chain does not actually exist, which basically
means that we have to hunt down commits [1] which are steps in these
chains. Another complication is the versioning scheme used by many npm
packages, the semver [2] + ranges notation [3]. This makes this kind of
'versioning archeology' even harder to do.

For the case where this chain does exist, I have been working on a
semi-npm-compatible semver parser for guile [4], which I was hoping to
integrate in the npm importer or a standalone tool to assist people
wanting to untangle these dependency chains. The goal would be to
reconstruct the needed versions to package by parsing data in the
package.json files of historic versions of these packages.

HTH,

Jelle

[1]: ... or even more granular, parts of commits!
[2]: https://semver.org/
[3]: https://docs.npmjs.com/misc/semver
[4]: https://git.fsfe.org/jlicht/guile-semver


Op za 24 nov. 2018 om 15:10 schreef swedebugia <swedebugia@riseup.net>:

> Hi
>
> We need to wonder about what to do with cyclic dependencies.
>
> In the case below node-rimraf has node-glob in input and node-glob has
> node-rimraf in NATIVE-input.
>
> I have no idea how to solve this. Both packages are by the same author.
>
> Same problem with jasmine and jasmine-core
>
> ~/guix-tree/pre-inst-env guix build -K node-rimraf
> allocate_stack failed: Cannot allocate memory
> Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind
> handler.
> allocate_stack failed: Cannot allocate memory
> Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind
> handler.
> allocate_stack failed: Cannot allocate memory
> Warning: Unwind-only `stack-overflow' exception; skipping pre-unwind
> handler.
>
>
> --
> Cheers Swedebugia
>
>

[-- Attachment #2: Type: text/html, Size: 2905 bytes --]

Re: Cyclic npm dependencies

[Pjotr Prins]

On Sat, Nov 24, 2018 at 03:41:35PM +0000, Jelle Licht wrote:
>    Hey swedebugia,
>    I will still send a more elaborate reply to the general npm-importer
>    thread later this week, but we can assume that generally these
>    recursive dependencies can be untangled by looking at the different
>    versions of the dependencies.
>    So in your example, I imagine an input chain like:
>    node-glob 0.1  -> node-rimraf 0.1 -> node-glob 0.2 -> node-rimraf 0.2
>    -> .... -> node-glob 1.0 -> node-rimraf 1.0
>    While *extremely* annoying to untangle, this is definitely doable.

Appears to me that it would suffice to pick the latest version. In
case there is no clear version info just pick whatever is there.

In general this should work. Any unit tests should show breakage.

Circular dependencies are (unfortunately) getting more common. Not
only in npm, but in all ad hoc package managers. I think their
assumption is too that you pick the latest.

Pj.

Re: Cyclic npm dependencies

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 1841 bytes --]

Op za 24 nov. 2018 om 16:38 schreef Pjotr Prins <pjotr.public12@thebird.nl>:

> On Sat, Nov 24, 2018 at 03:41:35PM +0000, Jelle Licht wrote:
> >    Hey swedebugia,
> >    I will still send a more elaborate reply to the general npm-importer
> >    thread later this week, but we can assume that generally these
> >    recursive dependencies can be untangled by looking at the different
> >    versions of the dependencies.
> >    So in your example, I imagine an input chain like:
> >    node-glob 0.1  -> node-rimraf 0.1 -> node-glob 0.2 -> node-rimraf 0.2
> >    -> .... -> node-glob 1.0 -> node-rimraf 1.0
> >    While *extremely* annoying to untangle, this is definitely doable.
>
> Appears to me that it would suffice to pick the latest version. In
>
What do you mean? In my specific example, you would need to package
and build each version in succession in order to actually be able to
use recent versions of either of these packages. IOW, you can not
choose any; you need to choose each.

case there is no clear version info just pick whatever is there.
>
> In general this should work. Any unit tests should show breakage.
>

If only we could run unit tests for most packages. Most test
frameworks have huge list of transitive dependencies, although not
nearly as bad as the build tooling.


>
> Circular dependencies are (unfortunately) getting more common. Not
> only in npm, but in all ad hoc package managers. I think their
> assumption is too that you pick the latest.
>
> I agree that we should expose the latest-and-greatest (and secure)
version of most packages, but we would still need to build older
intermediate releases in order to have reproducible builds from source
in a lot of cases. Whether we should expose these bootstrap packages
as well is another issue. Am I perhaps missing the point you are making?


Pj.
>
>

[-- Attachment #2: Type: text/html, Size: 2666 bytes --]

Re: Cyclic npm dependencies

[Pjotr Prins]

On Sat, Nov 24, 2018 at 11:29:08PM +0000, Jelle Licht wrote:
>      >    So in your example, I imagine an input chain like:
>      >    node-glob 0.1  -> node-rimraf 0.1 -> node-glob 0.2 ->
>      node-rimraf 0.2
>      >    -> .... -> node-glob 1.0 -> node-rimraf 1.0
>      >    While *extremely* annoying to untangle, this is definitely
>      doable.
>      Appears to me that it would suffice to pick the latest version. In
> 
>    What do you mean? In my specific example, you would need to package
>    and build each version in succession in order to actually be able to
>    use recent versions of either of these packages. IOW, you can not
>    choose any; you need to choose each.

If that is the case there is no way around it. All I am saying you can
start with an 'optimistic' importer and if that fails go back to
resolving the circular depency. I.e., opt for the simple resolution
first.

Arguably a dependency is not circular if versions differ. But that may
be a semantic argument ;)

Pj.

Re: Cyclic npm dependencies

[swedebugia]

On 2018-11-24 16:41, Jelle Licht wrote:
> Hey swedebugia,
> 
> I will still send a more elaborate reply to the general npm-importer
> thread later this week, but we can assume that generally these
> recursive dependencies can be untangled by looking at the different
> versions of the dependencies.
> 
> So in your example, I imagine an input chain like:
> node-glob 0.1  -> node-rimraf 0.1 -> node-glob 0.2 -> node-rimraf 0.2 -> 
> .... -> node-glob 1.0 -> node-rimraf 1.0

Thank you for showing me a way forward. I did not think of this :p

> 
> While *extremely* annoying to untangle, this is definitely doable.
> Problems arise if this chain does not actually exist, which basically
> means that we have to hunt down commits [1] which are steps in these
> chains. Another complication is the versioning scheme used by many npm
> packages, the semver [2] + ranges notation [3]. This makes this kind of
> 'versioning archeology' even harder to do.
> 
> For the case where this chain does exist, I have been working on a
> semi-npm-compatible semver parser for guile [4], which I was hoping to
> integrate in the npm importer or a standalone tool to assist people
> wanting to untangle these dependency chains. The goal would be to
> reconstruct the needed versions to package by parsing data in the
> package.json files of historic versions of these packages.

Sounds good with a dedicated tool.

For now I concluded:
jquery, browserify, async are off limits because of one or more cycdeps.

Right now I am pursuing rollup and leaflet.

I put all cycdeps aside and intend to publish here in a thread for brave 
hackers to work on. ;-)

-- 
Cheers Swedebugia