[bug#71408] Request for merging "python-team" branch

[bug#71408] Request for merging "python-team" branch

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 114 bytes --]

I haven't been involved in this branch, but I'm opening the request for
merge issue just to keep track of things.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71408] Request for merging "python-team" branch

[Ludovic Courtès]

Hello,

(Cc’ing the Python team.)

Christopher Baines <mail@cbaines.net> skribis:

> I haven't been involved in this branch, but I'm opening the request for
> merge issue just to keep track of things.

I haven’t been involved either.  My understanding is that the main
blocker before merging is assessing the amount of breakage introduced in
this branch¹.

Right now it’s built at 89% on x86_64-linux only, which is a bit less
than on ‘master’ but there are still pending builds.  Interested readers
can browse failed builds at:

  https://ci.guix.gnu.org/eval/latest/dashboard?spec=python-team

(If you find builds that failed due to transient errors like the
infamous “missing derivation”, you can click on “Action -> Restart”—this
is now available in most cases without privileges.)

An important issue seems to be test failures of ‘python-responses’:

  https://ci.guix.gnu.org/build/4738398/details

Excerpt:

--8<---------------cut here---------------start------------->8---
=========================== short test summary info ============================
FAILED responses/tests/test_responses.py::test_auto_calculate_content_length_doesnt_override_existing_value
FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries - Ty...
FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed[True]
FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed[False]
FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed_msg
FAILED responses/tests/test_responses.py::TestMaxRetry::test_adapter_retry_untouched
============ 6 failed, 187 passed, 2 skipped, 20 warnings in 21.86s ============
--8<---------------cut here---------------end--------------->8---

Ideas, Pythonistas?  :-)

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00227.html

[bug#71408] Request for merging "python-team" branch

[Sharlatan Hellseher]

Hi,

All failed tests have the same reason:

--8<---------------cut here---------------start------------->8---
        adapter = requests.adapters.HTTPAdapter(
>           max_retries=Retry(
                total=total,
                backoff_factor=0.1,
                status_forcelist=[500],
                method_whitelist=["GET", "POST", "PATCH"],
                raise_on_status=raise_on_status,
            )
        )
E       TypeError: Retry.__init__() got an unexpected keyword argument
'method_whitelist'
--8<---------------cut here---------------end--------------->8---

It looks like the bump to the latest version would resolve it 0.22.0 -> 0.25.2

--8<---------------cut here---------------start------------->8---
guix refresh --list-dependent python-responses
Building the following 198 packages would ensure 456 dependent
packages are rebuilt: <...>
--8<---------------cut here---------------end--------------->8---

There are some mentioned in GitHub issues with the same simptoms:
https://github.com/GeneralMills/pytrends/issues/591
https://github.com/jamalex/notion-py/issues/386

On Sat, 8 Jun 2024 at 15:27, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hello,
>
> (Cc’ing the Python team.)
>
> Christopher Baines <mail@cbaines.net> skribis:
>
> > I haven't been involved in this branch, but I'm opening the request for
> > merge issue just to keep track of things.
>
> I haven’t been involved either.  My understanding is that the main
> blocker before merging is assessing the amount of breakage introduced in
> this branch¹.
>
> Right now it’s built at 89% on x86_64-linux only, which is a bit less
> than on ‘master’ but there are still pending builds.  Interested readers
> can browse failed builds at:
>
>   https://ci.guix.gnu.org/eval/latest/dashboard?spec=python-team
>
> (If you find builds that failed due to transient errors like the
> infamous “missing derivation”, you can click on “Action -> Restart”—this
> is now available in most cases without privileges.)
>
> An important issue seems to be test failures of ‘python-responses’:
>
>   https://ci.guix.gnu.org/build/4738398/details
>
> Excerpt:
>
> --8<---------------cut here---------------start------------->8---
> =========================== short test summary info ============================
> FAILED responses/tests/test_responses.py::test_auto_calculate_content_length_doesnt_override_existing_value
> FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries - Ty...
> FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed[True]
> FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed[False]
> FAILED responses/tests/test_responses.py::TestMaxRetry::test_max_retries_exceed_msg
> FAILED responses/tests/test_responses.py::TestMaxRetry::test_adapter_retry_untouched
> ============ 6 failed, 187 passed, 2 skipped, 20 warnings in 21.86s ============
> --8<---------------cut here---------------end--------------->8---
>
> Ideas, Pythonistas?  :-)
>
> Ludo’.
>
> ¹ https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00227.html



-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

[bug#71408] Request for merging "python-team" branch

[jgart via Guix-patches via]

Hi Python Team, Guix Team at large, and Parenphilic Pythonistas,

This long lived python-team branch as of today has a lot of git conflicts if you try to rebase and/or merge it on to master.

What do you think if we discuss an alternative team branch policy for the future for feature branches that target master?

Here's a tentative proposal, with an example: 

Instead of having a single python-team branch, with a wide variety of new Python features, what if we had, a python-team feature branch that we work on relatively quickly?

In other words, we avoid long lived branches but try to merge for example, a new python-team-sphinx branch as soon as the "Sphinx feature" is ready. This python-team-sphinx branch will only contain the work required to bump sphinx to the latest version that we'd like to support. The reason I use python-sphinx as an example is because the python-sphinx package requires a lot of rebuilds across many language ecosystems that use Sphinx for documentation purposes. 

I think that keeping the team branches focused on a particular team sub-feature within that team's scope and not using long-lived and largely scoped branches will avoid a ton of frustration trying to fix merge conflicts when/before we announce a request to merge.

We can then focus our focused efforts on iterating over preparing major Python features and packages that require large amounts rebuilds.

WDYT?

jgart

[bug#71408] Request for merging "python-team" branch

[Tanguy LE CARROUR]

Hi jgart,

Quoting jgart (2024-06-18 22:46:47)
> This long lived python-team branch as of today has a lot of git conflicts if
> you try to rebase and/or merge it on to master.
> […]
> Here's a tentative proposal, with an example: 
> […]
> I think that keeping the team branches focused on a particular team
> sub-feature within that team's scope and not using long-lived and largely
> scoped branches will avoid a ton of frustration trying to fix merge conflicts
> when/before we announce a request to merge.
> 
> We can then focus our focused efforts on iterating over preparing major Python
> features and packages that require large amounts rebuilds.

IM(V)HO… the smaller and the more-frequently-rebased the branches, the better!
I still haven’t wrapped my head around the consequences for the CI of rebasing
often —can a simple rebase trigger a world rebuild?—, but this is definitively
something that I do —quite successfully!— on a daily basis in my professional life.

I guess it’s just a matter of giving it a try and see? It’s always possible to
go back to the one-branch-to-rule-them-all `python-team` branch if managing
multiple branches ends up being a pain, right?

I also have no clue how the other teams are managing their branch and their
rebase/merge policy?

-- 
Tanguy

[bug#71408] Request for merging "python-team" branch

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

"jgart" <jgart@dismail.de> writes:

> Hi Python Team, Guix Team at large, and Parenphilic Pythonistas,
>
> This long lived python-team branch as of today has a lot of git
> conflicts if you try to rebase and/or merge it on to master.
>
> What do you think if we discuss an alternative team branch policy for
> the future for feature branches that target master?
>
> Here's a tentative proposal, with an example: 
>
> Instead of having a single python-team branch, with a wide variety of
> new Python features, what if we had, a python-team feature branch that
> we work on relatively quickly?
>
> In other words, we avoid long lived branches but try to merge for
> example, a new python-team-sphinx branch as soon as the "Sphinx
> feature" is ready. This python-team-sphinx branch will only contain
> the work required to bump sphinx to the latest version that we'd like
> to support. The reason I use python-sphinx as an example is because
> the python-sphinx package requires a lot of rebuilds across many
> language ecosystems that use Sphinx for documentation purposes.
>
> I think that keeping the team branches focused on a particular team
> sub-feature within that team's scope and not using long-lived and
> largely scoped branches will avoid a ton of frustration trying to fix
> merge conflicts when/before we announce a request to merge.

This aligns with the current (and previous) guidance on managing
branches [1]. Providing there's a consistent topic for the branch, any
name is fine (e.g. python-team-sphinx is fine).

1: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html

I'm also hoping the requirement to create a guix-patches issue
requesting to merge the branch when it's created will help avoid the
long lived branches that we've had over the last year.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71408] Request for merging "python-team" branch

[jgart via Guix-patches via]

> This aligns with the current (and previous) guidance on managing
> 
> branches [1]. Providing there's a consistent topic for the branch, any
> 
> name is fine (e.g. python-team-sphinx is fine).

Hi Christopher, 

That's great that my proposal already aligns with the current guidance.

I am going to try to start cherry picking uncontroversial and non failing patches on the python-team branch on to master and removing them from the python-team branch in order to reduce the size of the branch. I can then remove those commits from the python-team branch and force push to update it. What do you think of this approach before I attempt it? I'll wait on your guidance before doing this.

Another approach I can take is to break the branch apart into 2 or 3 smaller topic branches and make requests for those branches to merge one at a time.

Note that I have an underpowered old Thinkpad X230 laptop and will only be able to test branches that are not too large. I tried using QA/CI but it seemed to be in an unknown state the last time that I tried. Any suggestions here for how to more effectively use that QA/CI infrastructure would be much appreciated.

I tried spending a bit of time on trying to resolve the merge conflicts but it quickly lead down a long rabbit hole and I only have a limited time to fix so many merge conflicts. The process is also currently very tedious to do with high amounts of accuracy due to their being

1. Lots of commits that I am not familiar
2. Lots of commits
3. The branch has diverged alot since latest master so things don't cleanly apply anymore

What do you think of this current approach of cherry picking good commits off of the python-team branch in order to reduce the size?

Another issue that I ran into on the branch is that there are commits that are failing. For example, at commit fdef5662cce6a57aa6bfa682c7260b65ce3669b8 (the borgmatic changes), borgmatic fails. Should I remove this commit since it is broken?

all the best,

jgart

[bug#71408] Request for merging "python-team" branch

[Sharlatan Hellseher via Guix-patches]

Hi,

@jgart Please let me know if you need a quick test of changes, I've got some
computer recourses at home.

Thanks
Oleg

On Tue, 25 Jun 2024 at 18:04, jgart <jgart@dismail.de> wrote:
>
> > This aligns with the current (and previous) guidance on managing
> >
> > branches [1]. Providing there's a consistent topic for the branch, any
> >
> > name is fine (e.g. python-team-sphinx is fine).
>
> Hi Christopher,
>
> That's great that my proposal already aligns with the current guidance.
>
> I am going to try to start cherry picking uncontroversial and non failing patches on the python-team branch on to master and removing them from the python-team branch in order to reduce the size of the branch. I can then remove those commits from the python-team branch and force push to update it. What do you think of this approach before I attempt it? I'll wait on your guidance before doing this.
>
> Another approach I can take is to break the branch apart into 2 or 3 smaller topic branches and make requests for those branches to merge one at a time.
>
> Note that I have an underpowered old Thinkpad X230 laptop and will only be able to test branches that are not too large. I tried using QA/CI but it seemed to be in an unknown state the last time that I tried. Any suggestions here for how to more effectively use that QA/CI infrastructure would be much appreciated.
>
> I tried spending a bit of time on trying to resolve the merge conflicts but it quickly lead down a long rabbit hole and I only have a limited time to fix so many merge conflicts. The process is also currently very tedious to do with high amounts of accuracy due to their being
>
> 1. Lots of commits that I am not familiar
> 2. Lots of commits
> 3. The branch has diverged alot since latest master so things don't cleanly apply anymore
>
> What do you think of this current approach of cherry picking good commits off of the python-team branch in order to reduce the size?
>
> Another issue that I ran into on the branch is that there are commits that are failing. For example, at commit fdef5662cce6a57aa6bfa682c7260b65ce3669b8 (the borgmatic changes), borgmatic fails. Should I remove this commit since it is broken?
>
> all the best,
>
> jgart



-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

[bug#71408] Request for merging "python-team" branch

[Lars-Dominik Braun]

Hi,

> I am going to try to start cherry picking uncontroversial and non failing patches on the python-team branch on to master and removing them from the python-team branch in order to reduce the size of the branch. I can then remove those commits from the python-team branch and force push to update it. What do you think of this approach before I attempt it? I'll wait on your guidance before doing this.

please go ahead. Divide and conquer is imo a good approach to get this branch merged. 

> Another approach I can take is to break the branch apart into 2 or 3 smaller topic branches and make requests for those branches to merge one at a time.

I’m splitting of the build-system-related changes into the
pyproject-build-system branch (again), because these are relatively
self-contained and thus easy to merge into master.

Lars

[bug#71408] Request for merging "python-team" branch

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3011 bytes --]

"jgart" <jgart@dismail.de> writes:

>> This aligns with the current (and previous) guidance on managing
>> branches [1]. Providing there's a consistent topic for the branch, any
>> name is fine (e.g. python-team-sphinx is fine).
>
> That's great that my proposal already aligns with the current guidance.
>
> I am going to try to start cherry picking uncontroversial and non
> failing patches on the python-team branch on to master and removing
> them from the python-team branch in order to reduce the size of the
> branch. I can then remove those commits from the python-team branch
> and force push to update it. What do you think of this approach before
> I attempt it? I'll wait on your guidance before doing this.
>
> Another approach I can take is to break the branch apart into 2 or 3
> smaller topic branches and make requests for those branches to merge
> one at a time.

If there are commits on the branch that have been submitted as patches,
you might just be able to look at pushing them to master. If there are
commits that haven't been submitted as patches yet, but could be pushed
to master, it would be good to send them as patches so that QA can take
a look.

And splitting off some commits in to a more specific branch is fine as
well.

> Note that I have an underpowered old Thinkpad X230 laptop and will
> only be able to test branches that are not too large. I tried using
> QA/CI but it seemed to be in an unknown state the last time that I
> tried. Any suggestions here for how to more effectively use that QA/CI
> infrastructure would be much appreciated.

QA has been performing recently as it's taking too long to process
revisions. Let me know if things aren't working or if you have questions
though and I can try and help.

> I tried spending a bit of time on trying to resolve the merge
> conflicts but it quickly lead down a long rabbit hole and I only have
> a limited time to fix so many merge conflicts. The process is also
> currently very tedious to do with high amounts of accuracy due to
> their being
>
> 1. Lots of commits that I am not familiar
> 2. Lots of commits
> 3. The branch has diverged alot since latest master so things don't cleanly apply anymore
>
> What do you think of this current approach of cherry picking good
> commits off of the python-team branch in order to reduce the size?

I think making sure commits that can go to master, do go to master, is
really helpful to avoid merge conflicts. It's so much more likely to
have conflicts if there are changes that can be made on master, but are
made on a branch, or changes that should be made on a branch but are
pushed to master.

> Another issue that I ran into on the branch is that there are commits
> that are failing. For example, at commit
> fdef5662cce6a57aa6bfa682c7260b65ce3669b8 (the borgmatic changes),
> borgmatic fails. Should I remove this commit since it is broken?

While ideally we'd care about the state at each commit, this doesn't
really happen so it's up to you what you do there.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]