python-distlib vs python-distlib/next

python-distlib vs python-distlib/next

[Ricardo Wurmus]

Hi,

commit ce6efff6eca0ed88cb9538803f5d1252c91a3b5e updated 
virtualenv.  As part of this change python-distlib was replaced 
with python-distlib/next.

This broke the *installation* (not the build) of other packages 
such as python-pygenometracks.  The reason is that the closure of 
python-pygenometracks would include both python-distlib and 
python-distlib/next.  Guix would then prevent the installation of 
python-pygenometracks into user profiles:

> guix package: error: profile contains conflicting entries for 
> python-distlib
> guix package: error:   first entry: python-distlib@0.3.1 
> /gnu/store/mypg42bass5n61liwyq7llrwla4w8bny-python-distlib-0.3.1
> guix package: error:    ... propagated from 
> python-virtualenv@20.2.1
> guix package: error:    ... propagated from python-tox@3.20.0
> guix package: error:    ... propagated from 
> python-argh@0.26.2-1.dcd3253
> guix package: error:    ... propagated from 
> python-gffutils@0.9-1.4034c54
> guix package: error:    ... propagated from 
> python-pygenometracks@3.3
> guix package: error:   second entry: python-distlib@0.3.0 
> /gnu/store/fwya18dazamh2cjdkgh5wpxi7lqn4fcm-python-distlib-0.3.0
> guix package: error:    ... propagated from 
> python-sortedcontainers@2.1.0
> guix package: error:    ... propagated from 
> python-intervaltree@3.0.2
> guix package: error:    ... propagated from 
> python-pygenometracks@3.3

We can’t easily detect these problems because they only strike at 
installation time.  This means that we may not do partial upgrades 
in the Python world due to the quirks of propagation.  There may 
be other packages that are broken in a similar fashion right now.

How can we make this more robust and avoid problems like this in 
the future?

(I worked around this problem for pygenometracks by removing 
python-tox from the propagated inputs of python-argh.)

-- 
Ricardo

Re: python-distlib vs python-distlib/next

[Leo Prikler]

Hi,

Am Montag, den 21.06.2021, 12:32 +0200 schrieb Ricardo Wurmus:
> Hi,
> 
> commit ce6efff6eca0ed88cb9538803f5d1252c91a3b5e updated 
> virtualenv.  As part of this change python-distlib was replaced 
> with python-distlib/next.
> 
> This broke the *installation* (not the build) of other packages 
> such as python-pygenometracks.  The reason is that the closure of 
> python-pygenometracks would include both python-distlib and 
> python-distlib/next.  Guix would then prevent the installation of 
> python-pygenometracks into user profiles:
> 
> > guix package: error: profile contains conflicting entries for 
> > python-distlib
> > guix package: error:   first entry: python-distlib@0.3.1 
> > /gnu/store/mypg42bass5n61liwyq7llrwla4w8bny-python-distlib-0.3.1
> > guix package: error:    ... propagated from 
> > python-virtualenv@20.2.1
> > guix package: error:    ... propagated from python-tox@3.20.0
> > guix package: error:    ... propagated from 
> > python-argh@0.26.2-1.dcd3253
> > guix package: error:    ... propagated from 
> > python-gffutils@0.9-1.4034c54
> > guix package: error:    ... propagated from 
> > python-pygenometracks@3.3
> > guix package: error:   second entry: python-distlib@0.3.0 
> > /gnu/store/fwya18dazamh2cjdkgh5wpxi7lqn4fcm-python-distlib-0.3.0
> > guix package: error:    ... propagated from 
> > python-sortedcontainers@2.1.0
> > guix package: error:    ... propagated from 
> > python-intervaltree@3.0.2
> > guix package: error:    ... propagated from 
> > python-pygenometracks@3.3
> 
> We can’t easily detect these problems because they only strike at 
> installation time.  This means that we may not do partial upgrades 
> in the Python world due to the quirks of propagation.  There may 
> be other packages that are broken in a similar fashion right now.
> 
> How can we make this more robust and avoid problems like this in 
> the future?
I think we can detect this and similar breakages through the use of
graph algorithms.  Start with all versions of your input package as
nodes and no edges.  For all packages P, add packages P' that
immediately propagate P as nodes to G and PP' as edge.  Repeat this
until closure is reached.

Your closure should now have as many components as you had versions. 
If not, then by the pigeonhole principle you have a propagation
conflict.

If you want to be eager, you can also try to colour your graph, such
that each component gets a unique colour, which is assigned to the
initial P a priori (use BFS to propagate colours).  If a package would
receive two colours from that operation, you again have a propagation
conflict, but you also know which packages cause it.

I think we should be able to implement one of those algorithms in `guix
graph`.  Then people could check whether their change to a propagated
input breaks anything, whether in Python or elsewhere.

Regards,
Leo

Re: python-distlib vs python-distlib/next

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> This broke the *installation* (not the build) of other packages such
> as python-pygenometracks.  The reason is that the closure of
> python-pygenometracks would include both python-distlib and 
> python-distlib/next.  Guix would then prevent the installation of
> python-pygenometracks into user profiles:
>
>> guix package: error: profile contains conflicting entries for
>> python-distlib
>> guix package: error:   first entry: python-distlib@0.3.1
>> /gnu/store/mypg42bass5n61liwyq7llrwla4w8bny-python-distlib-0.3.1
>> guix package: error:    ... propagated from python-virtualenv@20.2.1
>> guix package: error:    ... propagated from python-tox@3.20.0
>> guix package: error:    ... propagated from
>> python-argh@0.26.2-1.dcd3253
>> guix package: error:    ... propagated from
>> python-gffutils@0.9-1.4034c54
>> guix package: error:    ... propagated from
>> python-pygenometracks@3.3
>> guix package: error:   second entry: python-distlib@0.3.0
>> /gnu/store/fwya18dazamh2cjdkgh5wpxi7lqn4fcm-python-distlib-0.3.0
>> guix package: error:    ... propagated from
>> python-sortedcontainers@2.1.0
>> guix package: error:    ... propagated from
>> python-intervaltree@3.0.2
>> guix package: error:    ... propagated from
>> python-pygenometracks@3.3
>
> We can’t easily detect these problems because they only strike at
> installation time.  This means that we may not do partial upgrades 
> in the Python world due to the quirks of propagation.  There may be
> other packages that are broken in a similar fashion right now.

‘guix lint’ catches that:

--8<---------------cut here---------------start------------->8---
$ guix lint -c profile-collisions python-pygenometracks 
gnu/packages/bioinformatics.scm:11896:2: python-pygenometracks@3.3: propagated inputs python-distlib@0.3.1 and python-distlib@0.3.0 collide
--8<---------------cut here---------------end--------------->8---

The Data Service should notice these new warnings, though I was unable
to find it, and
<http://data.guix.gnu.org/revision/b5ed51bdf93de46e0852fda3738690f820ada439/lint-warnings?locale=en_US.UTF-8&package_query=python-virtualenv&linter=profile-collisions&message_query=&field=linter&field=message&field=location>
doesn’t seem to have this warning.

We definitely need automated feedback before a patch can be applied.

Another thing we could do is have ‘guix lint’ check the dependents of a
package in addition to the package itself in some cases (maybe just for
‘profile-collisions’?).

Thoughts?

Ludo’.