Poetry upgrade and related packages

Poetry upgrade and related packages

[Sébastien Lerique]

Dear all,

This thread is an attempt to keep a handle on the various patches 
involved in upgrading Poetry to 1.1.4, and to ask a couple 
questions that crop up.

- Tanguy's original patch http://issues.guix.info/44077 is merged
- But python-packaging had to be downgraded again because it 
  generated too much rebuilds 
  <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00003.html>
- Tanguy submitted another patch for Poetry reducing 
  python-packaging's required version: 
  https://issues.guix.info/45003

Now, there are a couple more hiccups with other packages:
- poetry actually requires a more recent version of python-keyring 
  (>=21.2.0 instead of the current 21.0.0),
- upstream python-keyring is at 21.5.0, which requires 
  python-setuptools >= 42 (guix now has 41.0.1)
- as a sidenote, when I locally added setuptools >=42 explicitely 
  to python-keyring's native-inputs, it solved the version 
  declaration problem described here 
  <https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-xyz.scm?id=04b83678653fda3c66e600e88f54f5108290ec1c#n13228>

Question: upstream setuptools is at 50.3.2, and they have dropped 
python2 support at v45 (see 
<https://github.com/pypa/setuptools/blob/master/CHANGES.rst>). 
Should we simply upgrade to v44, or rather create alternate 
python{,2}-setuptools-44 packages? `guix refresh -l 
python2-setuptools` lists 48 dependent packages.

Happy hacking!
Sébastien

Re: Poetry upgrade and related packages

[Tanguy LE CARROUR]

Hi Sébastien, hi Guix!


Excerpts from Sébastien Lerique's message of December 2, 2020 12:49 pm:
> This thread is an attempt to keep a handle on the various patches 
> involved in upgrading Poetry to 1.1.4, and to ask a couple 
> questions that crop up.
> 
> - Tanguy's original patch http://issues.guix.info/44077 is merged
> - But python-packaging had to be downgraded again because it 
>   generated too much rebuilds 
>   <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00003.html>
> - Tanguy submitted another patch for Poetry reducing 
>   python-packaging's required version: 
>   https://issues.guix.info/45003
> 
> Now, there are a couple more hiccups with other packages:
> - poetry actually requires a more recent version of python-keyring 
>   (>=21.2.0 instead of the current 21.0.0),
> - upstream python-keyring is at 21.5.0, which requires 
>   python-setuptools >= 42 (guix now has 41.0.1)
> - as a sidenote, when I locally added setuptools >=42 explicitely 
>   to python-keyring's native-inputs, it solved the version 
>   declaration problem described here 
>   <https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/python-xyz.scm?id=04b83678653fda3c66e600e88f54f5108290ec1c#n13228>
> 
> Question: upstream setuptools is at 50.3.2, and they have dropped 
> python2 support at v45 (see 
> <https://github.com/pypa/setuptools/blob/master/CHANGES.rst>). 
> Should we simply upgrade to v44, or rather create alternate 
> python{,2}-setuptools-44 packages? `guix refresh -l 
> python2-setuptools` lists 48 dependent packages.

Thank you for summarising the situation!

It's not yet clear to me how to handle (python) package updates:
- when to update;
- when not to update;
- when to introduce "versionned" (`-x.y` suffix) package definitions;
- when to introduce "next" (`/next` suffix) package definitions;
- when to remove the two above suffixes;
- …

So I'm looking forward to reading the answers to this thread! :-)

-- 
Tanguy

Re: Poetry upgrade and related packages

[Ludovic Courtès]

Hello!

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

> It's not yet clear to me how to handle (python) package updates:
> - when to update;
> - when not to update;
> - when to introduce "versionned" (`-x.y` suffix) package definitions;
> - when to introduce "next" (`/next` suffix) package definitions;
> - when to remove the two above suffixes;
> - …
>
> So I'm looking forward to reading the answers to this thread! :-)

When a change introduces too many rebuilds, the convention is to make
that change on a branch that will be merged “later” rather than on
‘master’; this is bullet 8 here:

  https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html

Yet, sometimes we want to introduce new versions that people can get in
their profile, even if the “default” one remains the older version to
avoid world rebuilds.

One example is GDB: gdb@8 has 1,671 dependents, but we added gdb@10 on
the side such that “guix install gdb” gives you version 10.

I hope this helps!

Ludo’.

Re: Poetry upgrade and related packages

[Tanguy LE CARROUR]

Hi,


Excerpts from Ludovic Courtès's message of December 5, 2020 4:44 pm:
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> 
>> It's not yet clear to me how to handle (python) package updates:
>> - when to update;
>> - when not to update;
>> - when to introduce "versionned" (`-x.y` suffix) package definitions;
>> - when to introduce "next" (`/next` suffix) package definitions;
>> - when to remove the two above suffixes;
>> - …
>>
>> So I'm looking forward to reading the answers to this thread! :-)
> 
> When a change introduces too many rebuilds, the convention is to make
> that change on a branch that will be merged “later” rather than on
> ‘master’; this is bullet 8 here:
> 
>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html

Thanks for pointing at, but this "just" tells me on which branch to put
the changeset, not which of the above options should be used when a
package needs to be updated.


> Yet, sometimes we want to introduce new versions that people can get in
> their profile, even if the “default” one remains the older version to
> avoid world rebuilds.

That's exactly my point! If the default one lags behind, then after some
time, nobody will use it any more and we will have introduced one or more
`-x.y` package definitions!
I would consider it to be a "saner" approach to have the default always
"point" to the latest version, but then we would have to "fix" package
depending on older versions by introducing `-x.y` package definitions
for them.

Or am I missing something?!


> One example is GDB: gdb@8 has 1,671 dependents, but we added gdb@10 on
> the side such that “guix install gdb” gives you version 10.

The difference here is that it's a package added to a profile, not a
dependency, so `gdb` means the latest available version of GDB, right?

As you can see, everything is not yet clear to me! Sorry! ^_^'

-- 
Tanguy

Re: Poetry upgrade and related packages

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3324 bytes --]

Tanguy LE CARROUR <tanguy@bioneland.org> skriver:

> Hi,
>
>
> Excerpts from Ludovic Courtès's message of December 5, 2020 4:44 pm:
>> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
>> 
>>> It's not yet clear to me how to handle (python) package updates:
>>> - when to update;
>>> - when not to update;
>>> - when to introduce "versionned" (`-x.y` suffix) package definitions;
>>> - when to introduce "next" (`/next` suffix) package definitions;
>>> - when to remove the two above suffixes;
>>> - …
>>>
>>> So I'm looking forward to reading the answers to this thread! :-)
>> 
>> When a change introduces too many rebuilds, the convention is to make
>> that change on a branch that will be merged “later” rather than on
>> ‘master’; this is bullet 8 here:
>> 
>>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>
> Thanks for pointing at, but this "just" tells me on which branch to put
> the changeset, not which of the above options should be used when a
> package needs to be updated.

There is no "one size fits all" rule.  With rare exceptions, Guix
"wants" to have only have a single version of each package (mainly to
ease maintenance).  As you found, that's not always feasible.

If a package depends on a newer version of something "deep in the graph"
such as Pytest, it's always OK to add a "/next" or "-x.y" variant
(though a convention about which to use would probably be a good idea).

If something depends on a *specific* (older) version of Pytest, it's
better to try and make it work with the newer version; but failing that,
adding a "-x.y" is fine too.

Then there are ecosystems such as Go and Rust that have fairly strict
semver rules, and require carrying many different versions of the same
package.

>> Yet, sometimes we want to introduce new versions that people can get in
>> their profile, even if the “default” one remains the older version to
>> avoid world rebuilds.
>
> That's exactly my point! If the default one lags behind, then after some
> time, nobody will use it any more and we will have introduced one or more
> `-x.y` package definitions!
> I would consider it to be a "saner" approach to have the default always
> "point" to the latest version, but then we would have to "fix" package
> depending on older versions by introducing `-x.y` package definitions
> for them.
>
> Or am I missing something?!

You got it right.  It might be saner to make the unversioned variable
refer to the newest version, but it would often require "pinning"
hundreds of packages to the old version to avoid rebuilds.  Thus, it's
typically more practical to use the "/next" variant until the next
rebuild cycle.

Again there is no hard rule here, I did such a change for 'libcap' in
9e1f5a263e4f6df4d075901c9b58a56f80c8b452 because only two packages
needed to pin the old version.

>> One example is GDB: gdb@8 has 1,671 dependents, but we added gdb@10 on
>> the side such that “guix install gdb” gives you version 10.
>
> The difference here is that it's a package added to a profile, not a
> dependency, so `gdb` means the latest available version of GDB, right?
>
> As you can see, everything is not yet clear to me! Sorry! ^_^'

Hope this clears things up a little more.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: Poetry upgrade and related packages

[Tanguy LE CARROUR]

Hi Marius,


Excerpts from Marius Bakke's message of December 7, 2020 11:59 pm:
> Tanguy LE CARROUR <tanguy@bioneland.org> skriver:
>> Excerpts from Ludovic Courtès's message of December 5, 2020 4:44 pm:
>>> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
>>> 
>>>> It's not yet clear to me how to handle (python) package updates:
>>>> - when to update;
>>>> - when not to update;
>>>> - when to introduce "versionned" (`-x.y` suffix) package definitions;
>>>> - when to introduce "next" (`/next` suffix) package definitions;
>>>> - when to remove the two above suffixes;
>>>> - …
>>>>
>>>> So I'm looking forward to reading the answers to this thread! :-)
>>> 
>>> When a change introduces too many rebuilds, the convention is to make
>>> that change on a branch that will be merged “later” rather than on
>>> ‘master’; this is bullet 8 here:
>>> 
>>>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>>
>> Thanks for pointing at, but this "just" tells me on which branch to put
>> the changeset, not which of the above options should be used when a
>> package needs to be updated.
> 
> There is no "one size fits all" rule.  With rare exceptions, Guix
> "wants" to have only have a single version of each package (mainly to
> ease maintenance).  As you found, that's not always feasible.
> 
> If a package depends on a newer version of something "deep in the graph"
> such as Pytest, it's always OK to add a "/next" or "-x.y" variant
> (though a convention about which to use would probably be a good idea).
>
> If something depends on a *specific* (older) version of Pytest, it's
> better to try and make it work with the newer version; but failing that,
> adding a "-x.y" is fine too.

`-x.y` packages are here to stay.
`/next` are temporary packages.

Is it safe to remove all `-x.y` packages that are not used as inputs?
Is there a (programmatic) way to find those packages?

When `/next` become "current|default|latest", who is in charge of patching
all the package definitions that were using it?!


>>> Yet, sometimes we want to introduce new versions that people can get in
>>> their profile, even if the “default” one remains the older version to
>>> avoid world rebuilds.
>>
>> That's exactly my point! If the default one lags behind, then after some
>> time, nobody will use it any more and we will have introduced one or more
>> `-x.y` package definitions!
>> I would consider it to be a "saner" approach to have the default always
>> "point" to the latest version, but then we would have to "fix" package
>> depending on older versions by introducing `-x.y` package definitions
>> for them.
>>
>> Or am I missing something?!
> 
> You got it right.  It might be saner to make the unversioned variable
> refer to the newest version, but it would often require "pinning"
> hundreds of packages to the old version to avoid rebuilds.  Thus, it's
> typically more practical to use the "/next" variant until the next
> rebuild cycle.

Then, during the rebuild cycle those hundreds of packages get rebuilt
and… some of them fail because they depend on the older version, right?!
But I'm pretty sure it's a problem all distributions have to face.

Wasn't it discussed somewhere else that one should get notified (by email)
when their favourite packages were broken?


> Again there is no hard rule here, I did such a change for 'libcap' in
> 9e1f5a263e4f6df4d075901c9b58a56f80c8b452 because only two packages
> needed to pin the old version.

> Hope this clears things up a little more.  :-)

Yes, thanks! But I guess I need to go through more of those situations
to make sure I understand everything.

-- 
Tanguy

Re: Poetry upgrade and related packages

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 4938 bytes --]

Tanguy LE CARROUR <tanguy@bioneland.org> skriver:

> Hi Marius,
>
>
> Excerpts from Marius Bakke's message of December 7, 2020 11:59 pm:
>> Tanguy LE CARROUR <tanguy@bioneland.org> skriver:
>>> Excerpts from Ludovic Courtès's message of December 5, 2020 4:44 pm:
>>>> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
>>>> 
>>>>> It's not yet clear to me how to handle (python) package updates:
>>>>> - when to update;
>>>>> - when not to update;
>>>>> - when to introduce "versionned" (`-x.y` suffix) package definitions;
>>>>> - when to introduce "next" (`/next` suffix) package definitions;
>>>>> - when to remove the two above suffixes;
>>>>> - …
>>>>>
>>>>> So I'm looking forward to reading the answers to this thread! :-)
>>>> 
>>>> When a change introduces too many rebuilds, the convention is to make
>>>> that change on a branch that will be merged “later” rather than on
>>>> ‘master’; this is bullet 8 here:
>>>> 
>>>>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>>>
>>> Thanks for pointing at, but this "just" tells me on which branch to put
>>> the changeset, not which of the above options should be used when a
>>> package needs to be updated.
>> 
>> There is no "one size fits all" rule.  With rare exceptions, Guix
>> "wants" to have only have a single version of each package (mainly to
>> ease maintenance).  As you found, that's not always feasible.
>> 
>> If a package depends on a newer version of something "deep in the graph"
>> such as Pytest, it's always OK to add a "/next" or "-x.y" variant
>> (though a convention about which to use would probably be a good idea).
>>
>> If something depends on a *specific* (older) version of Pytest, it's
>> better to try and make it work with the newer version; but failing that,
>> adding a "-x.y" is fine too.
>
> `-x.y` packages are here to stay.
> `/next` are temporary packages.

In most cases, yes.  :-)

> Is it safe to remove all `-x.y` packages that are not used as inputs?
> Is there a (programmatic) way to find those packages?

Good questions.  It should be safe to remove unused and versioned "leaf"
packages.  I don't have good answers on how to discover them though.

> When `/next` become "current|default|latest", who is in charge of patching
> all the package definitions that were using it?!

Typically those occur in the same or (closely related) commits.  But it
happens that a "/latest" has been removed on 'core-updates', yet keeps
gaining uses on 'master'.  In those cases deprecating (instead of
removing) the variable on the 'core-updates' branch is the best option,
so that merging master->core-updates won't break the branch.

There is no one "in charge" though, just "someone" has to do it.  :-)

>>>> Yet, sometimes we want to introduce new versions that people can get in
>>>> their profile, even if the “default” one remains the older version to
>>>> avoid world rebuilds.
>>>
>>> That's exactly my point! If the default one lags behind, then after some
>>> time, nobody will use it any more and we will have introduced one or more
>>> `-x.y` package definitions!
>>> I would consider it to be a "saner" approach to have the default always
>>> "point" to the latest version, but then we would have to "fix" package
>>> depending on older versions by introducing `-x.y` package definitions
>>> for them.
>>>
>>> Or am I missing something?!
>> 
>> You got it right.  It might be saner to make the unversioned variable
>> refer to the newest version, but it would often require "pinning"
>> hundreds of packages to the old version to avoid rebuilds.  Thus, it's
>> typically more practical to use the "/next" variant until the next
>> rebuild cycle.
>
> Then, during the rebuild cycle those hundreds of packages get rebuilt
> and… some of them fail because they depend on the older version, right?!
> But I'm pretty sure it's a problem all distributions have to face.

Yes, often you'll get an "upgrading cascade", which may bring new
problems such as circular dependencies, etc (looking at you, Python
ecosystem).

The only remedy I know for that is copious amounts of coffee.

> Wasn't it discussed somewhere else that one should get notified (by email)
> when their favourite packages were broken?

I think cbaines is working on something along those lines.

>> Again there is no hard rule here, I did such a change for 'libcap' in
>> 9e1f5a263e4f6df4d075901c9b58a56f80c8b452 because only two packages
>> needed to pin the old version.
>
>> Hope this clears things up a little more.  :-)
>
> Yes, thanks! But I guess I need to go through more of those situations
> to make sure I understand everything.

For some practical experience, you can try updating to Python 3.9.1 and
the latest Pytest (+ dependencies).  This needs to be done soon for the
'core-updates' branch anyway.  ;-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: Poetry upgrade and related packages

[Tanguy LE CARROUR]

Hi,

Excerpts from Marius Bakke's message of December 9, 2020 12:25 am:
> For some practical experience, you can try updating to Python 3.9.1 and
> the latest Pytest (+ dependencies).  This needs to be done soon for the
> 'core-updates' branch anyway.  ;-)

Let's give it a try! 8-)

Regards,

-- 
Tanguy