[bug#75246] Request for merging "python-team" branch

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

The python-team branch is currently tracking the rust-team branch,
because it depends on changes there.  It is shaping up nicely and we
hope to be able to merge it some time soon.

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Lars-Dominik Braun]

Hi Ricardo,

> The python-team branch is currently tracking the rust-team branch,
> because it depends on changes there.  It is shaping up nicely and we
> hope to be able to merge it some time soon.

I recall you wanted to keep python-team small, but do you mind if I
push https://issues.guix.gnu.org/74993 there (it’s a small, but
world-rebuilding change)?

Thanks,
Lars

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Hi Lars,

>> The python-team branch is currently tracking the rust-team branch,
>> because it depends on changes there.  It is shaping up nicely and we
>> hope to be able to merge it some time soon.
>
> I recall you wanted to keep python-team small, but do you mind if I
> push https://issues.guix.gnu.org/74993 there (it’s a small, but
> world-rebuilding change)?

Looks fine to me.  Would this fix the build failure we see in
python-fastapi-pagination-minimal, which seems to be due to a parse
error?

I know I said I wanted to merge this branch quickly and keep the changes
restricted to important fixes, but it turned out that many of the
required fixes are wide-ranging anyway.  I had to upgrade a lot of
packages that led to thousands of rebuilds.  At this point I'd still
like to keep *experimental* changes out, but almost all other
improvements are welcome.

Thank you!

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Lars-Dominik Braun]

Hi,

> Looks fine to me.  Would this fix the build failure we see in
> python-fastapi-pagination-minimal, which seems to be due to a parse
> error?

yes, that looks like the error this patch tries to fix.

Thanks,
Lars

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> The python-team branch is currently tracking the rust-team branch,
> because it depends on changes there.  It is shaping up nicely and we
> hope to be able to merge it some time soon.

Looking at the dashboard for any recent evaluation of the python-team
branch, we still see a lot of failures, e.g. here:

    https://ci.guix.gnu.org/eval/1967649/dashboard

Is anyone here interested in helping to get most of the packages
prefixed with "python-" to build successfully?  

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[jgart via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

I probably won't have time to work on that in the coming weeks.

Could you cherry-pick out the packages that are introducing failures into a WIP branch and make the scope of this request smaller so that it can be merged?

Someone can then address the failures in the next python-team merge request.

WDYT

all the best,

jgart

[-- Attachment #2: Type: text/html, Size: 496 bytes --]

[bug#75246] Last minute python-team fixes: help needed!

[Ricardo Wurmus]

Hi,

I've spent the last weeks whipping the python-team branch into shape,
but I'm running out of steam.  There are still a number of failures, and
I'm about to lose motivation to continue fixing the remaining build
failures.

Here are some of the failures that may be of interest:

  ansible
  python-libcst (needs rust)
  python-pandas 1.5
  python-pytorch 2.0.1
  python-django-q
  python-pyodbc-c (3.1.5 fails with: undefined symbol: PyUnicode_GetMax)
  python-agate (whole wireservice module needs upgrades)

I won't be working on these packages.  Instead I'll focus on preparing
the guix-science channel for the merge.  Please view the dashboard for
the latest evaluation of python-team at
https://ci.guix.gnu.org/jobset/python-team to see if you could
contribute fixes for the remaining failures.  Thanks!

The "python-team" branch has just been rebased on top of "master".  It
will soon be merged --- even with the above failures --- to avoid
falling into our old habits of delaying merges and encouraging the
proliferation of increasingly divergent fixes across several branches.
It is next in line after "libgcrypt-gpg-update"
(https://issues.guix.gnu.org/75180).

(If you find yourself in Cc, you are listed as a member of the python
team.  If you consider this to be outdated, please submit a change to
etc/teams.scm.)

-- 
Ricardo

Re: Last minute python-team fixes: help needed!

[宋文武 via Development of GNU Guix and the GNU System distribution.]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi,
>
> I've spent the last weeks whipping the python-team branch into shape,
> but I'm running out of steam.  There are still a number of failures, and
> I'm about to lose motivation to continue fixing the remaining build
> failures.

Thank you for the work!

> Here are some of the failures that may be of interest:
>
>   ansible
>   python-libcst (needs rust)
>   python-pandas 1.5
>   python-pytorch 2.0.1
>   python-django-q
>   python-pyodbc-c (3.1.5 fails with: undefined symbol: PyUnicode_GetMax)
>   python-agate (whole wireservice module needs upgrades)
>
> I won't be working on these packages.  Instead I'll focus on preparing
> the guix-science channel for the merge.  Please view the dashboard for
> the latest evaluation of python-team at
> https://ci.guix.gnu.org/jobset/python-team to see if you could
> contribute fixes for the remaining failures.  Thanks!

I can contribute some fixes today, it seems many failures are due to
missing python-setuptools and python-wheel in its native-inputs when
using the pyproject-build-system.

[bug#75246] Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]


Hi Ricardo,

I found 2 duplicated packages with different names in bioinformatics:

this
--8<---------------cut here---------------start------------->8---
(define-public fanc
  (let ((commit "354401e52ba2320e6b1ba0d3b5aab3541d31c9f3")
	(revision "1"))
    (package
      (name "fanc")
      (version (git-version "0" revision commit))
      (source (origin
		(method git-fetch)
		(uri (git-reference
		      (url "https://github.com/vaquerizaslab/fanc.git")
		      (commit commit)))
		(file-name (git-file-name name version))
		(sha256
		 (base32
		  "0vp2cak5snla4j2q23d3ixx016bwzvxfiv4z6yp7anl0xaksd5bl"))))
...
      (home-page "https://github.com/vaquerizaslab/fanc")
      (synopsis "Framework for the analysis of C-like data")
      (description "FAN-C provides a pipeline for analysing Hi-C data starting
at mapped paired-end sequencing reads.")
      (license license:gpl3+))))
--8<---------------cut here---------------end--------------->8---

and this

--8<---------------cut here---------------start------------->8---
(define-public python-fanc
  (package
    (name "python-fanc")
    (version "0.9.25")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/vaquerizaslab/fanc")
                    ;; There are no tags.  This commit corresponds to
                    ;; version 0.9.25.
                    (commit "e2205346c13ea5349681dff21adeb271d4ea5261")))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "0rxq24p852iiayi0083fyigvc30as695rha71q6xd4s2ij1k9mqi"))))
...
    (home-page "https://github.com/vaquerizaslab/fanc")
    (synopsis "Framework for the analysis of C-data")
    (description
     "FAN-C provides a pipeline for analysing Hi-C data starting at
mapped paired-end sequencing reads.")
    (license license:gpl3+)))
--8<---------------cut here---------------end--------------->8---

Failing and took too much time on tests I've gave up on 20min :-).

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> I found 2 duplicated packages with different names in bioinformatics:
> [...]

Thanks for letting me know.  I fixed this yesterday and upgraded
python-fanc.

Unfortunately, ci.guix.gnu.org seems to have stopped following our
progress.

https://qa.guix.gnu.org/branch/python-team doesn't know the merge base,
so I will rebase python-team once more on top of master some time today.

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Unfortunately, ci.guix.gnu.org seems to have stopped following our
> progress.

I've fixed this by deleting a stray lock file.

> https://qa.guix.gnu.org/branch/python-team doesn't know the merge base,
> so I will rebase python-team once more on top of master some time today.

I just pushed the rebased branch.

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]


Hi Ricardo,

Do you have a list of packages which you would like to fix/update before
merge?

I try to fix the build of python-stcal which was affected by numpy
upgrade, have no solution yet.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

> Do you have a list of packages which you would like to fix/update before
> merge?

I don't have a list, but I have some ongoing work that I'd like to
complete first.  I'm afraid one of the fixes will cause yet another
world rebuild, but it's best to get it over with now than to introduce
more workarounds.

> I try to fix the build of python-stcal which was affected by numpy
> upgrade, have no solution yet.

I have these things still going on before I would be comfortable with a
merge:

- python-pysnptools (almost done, with lots of Rust)
- python-modin (tests don't work yet)
- python-pynetdicom (lots of tests fail)
- python-asyncua (freezes)

In guix-science I'd like to fix python-tensorflow, but that seems to be
a small problem only -- we could relax requirements there, because it
complains about a patch-level mismatch in the numpy version.

(I noticed this time around that quite a few builds have a failing test suite
only when run on my laptop with more than 4 cores.  botocore is one of
them.  A bit strange.)

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> I have these things still going on before I would be comfortable with a
> merge:
>
> - python-pysnptools (almost done, with lots of Rust)
> - python-modin (tests don't work yet)
> - python-pynetdicom (lots of tests fail)
> - python-asyncua (freezes)

Another important one: python-psycopg-3.1.10 appears to be failing its
test suite now.  I haven't looked into it yet.

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Leo Famulari]

On Wed, Jan 01, 2025 at 06:58:58PM +0100, Ricardo Wurmus wrote:
> The python-team branch is currently tracking the rust-team branch,
> because it depends on changes there.  It is shaping up nicely and we
> hope to be able to merge it some time soon.

I think that we should be bold in merging the branch if there are only
scattered problems.

Already we have had large parts of the Python package graph failing to
build on master (including most of the scientific and numerical
packages), and it was at least partly fixed by cherry-picking from
python-team.

I think it's okay to trade "new with some failures" for "old with some
failures" on the master branch. At least there will be more help to fix
them.

[bug#75246] Request for merging "python-team" branch

[jgart via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

> 
> I think it's okay to trade "new with some failures" for "old with some
> failures" on the master branch. At least there will be more help to fix
> them.
> 
If this the state that we are already in on master then sounds good to me.

In the future, it would be great to have some hooks in place to prevent people from pushing updates to master that break packages or cause too many rebuilds.

In other words, I don't think we should be allowed to push commits to master that break the package graph accidentally.

How can we make such a thing happen? I don't have an answer at the moment :)

all best,

jgart

[-- Attachment #2: Type: text/html, Size: 957 bytes --]

[bug#75246] Request for merging "python-team" branch

[Leo Famulari]

On Fri, Jan 17, 2025 at 08:42:41PM +0000, jgart wrote:
> In the future, it would be great to have some hooks in place to prevent people from pushing updates to master that break packages or cause too many rebuilds.
> 
> In other words, I don't think we should be allowed to push commits to master that break the package graph accidentally.
> 
> How can we make such a thing happen? I don't have an answer at the moment :)

That's basically the premise of qa.guix.gnu.org, and continuous
integration in general.

We "just have to" use those resources in a more disciplined way.

[bug#75246] Request for merging "python-team" branch

[jgart via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

One issue with QA that confuses me though is when it says unknown. For example, here:

https://issues.guix.gnu.org/75635

What does "unknown" mean? Abort looking at QA? 

If the correct response, then, now there's no QA for 75635? ;()

[-- Attachment #2: Type: text/html, Size: 517 bytes --]

[bug#75246] Request for merging "python-team" branch

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

jgart via Guix-patches via <guix-patches@gnu.org> writes:

> One issue with QA that confuses me though is when it says unknown. For example, here:
>
> https://issues.guix.gnu.org/75635
>
> What does "unknown" mean? Abort looking at QA?
>
> If the correct response, then, now there's no QA for 75635? ;()

The overall status is a summary, there should be more information on the
relevant QA page for that issue. Currently on that page [1] the thing
highlighted in red reads:

  Builds for this patch series not yet submitted as master branch
  substitute availability is low for: aarch64-linux armhf-linux
  i686-linux x86_64-linux

1: https://qa.guix.gnu.org/issue/75635

If you look at the page, QA does actually have some information,
e.g. how many rebuilds there are and some information on lint warnings
to, but it can't give an overall status because it is missing build
information for both the master branch and the state after applying the
patches (the "Unknown" column in the "Package changes" table).

Currently my approach so far has been that QA shouldn't be a blocker,
e.g. if it doesn't give the OK then that's not something you have to
wait for, but given how the bordeaux build farm has been struggling to
keep up recently I'm starting to think that if people don't start
waiting for QA, substitute availability might never catch up so the
bordeaux build farm is always going to be behind and therefore QA will
be unable to actually provide any information.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Wed, Jan 01, 2025 at 06:58:58PM +0100, Ricardo Wurmus wrote:
>> The python-team branch is currently tracking the rust-team branch,
>> because it depends on changes there.  It is shaping up nicely and we
>> hope to be able to merge it some time soon.
>
> I think that we should be bold in merging the branch if there are only
> scattered problems.

We had not been able to figure out if there really are only scattered
problems.  Just a few days ago we had to do another world rebuild due to
an upgrade of a small package close to the root of the Python test
graph.  But from what I can tell in test builds of real-world
environments we're in good shape.

> Already we have had large parts of the Python package graph failing to
> build on master (including most of the scientific and numerical
> packages), and it was at least partly fixed by cherry-picking from
> python-team.

I'm pretty much done with what I set out to fix.  Now it's up to the
build farms to successfully complete the evaluations.  I'd like to merge
this all on Monday if the build farms cooperate.

...and then I'd like to start a new shortlived python-team branch with
an upgrade from Python 3.10 to 3.11.

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

> build farms to successfully complete the evaluations.  I'd like to merge
> this all on Monday if the build farms cooperate.

https://qa.guix.gnu.org/branch/python-team sadly doesn't show *anything*
other than "Information unavailable", but ci.guix.gnu.org is happy.  The
dashboard looks much greener than what we currently have for "master",
so I'm feeling good about merging this.

Commit 69d657f9c4ffa59d708356baacbefafb58f09308 is now the tip of the
"master" branch.  The "python-team" branch has been merged.

> ...and then I'd like to start a new shortlived python-team branch with
> an upgrade from Python 3.10 to 3.11.

I see that we have Python 3.12, but let's please target 3.11 first.
It's a small step up from 3.10, and we already know of packages that
won't work with 3.12.  Let's keep the next iteration of this branch
simple, please.

Thanks to everyone who contributed to this branch!

-- 
Ricardo

[bug#75246] Request for merging "python-team" branch

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]


Hi,

Thank you for merging and leading this refresh quest!

> I see that we have Python 3.12, but let's please target 3.11 first.
> It's a small step up from 3.10, and we already know of packages that
> won't work with 3.12.  Let's keep the next iteration of this branch
> simple, please.

While refreshing (now) packages from astronomy module I've faced wit few
failing to build due to hard requirement of Python3.11+ and NumPy 1.27+
or 2.0+ which don't have yet.

I've got a chance to to refresh NumPy to 1.24 but any farther version
failed to build (see notes on the package).

May we give it one more go with NumPy and friends?

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#75246: Request for merging "python-team" branch

[Leo Famulari]

On Mon, Jan 20, 2025 at 09:56:39PM +0100, Ricardo Wurmus wrote:
> Commit 69d657f9c4ffa59d708356baacbefafb58f09308 is now the tip of the
> "master" branch.  The "python-team" branch has been merged.

Hooray! Thanks to the whole team for their work.

[bug#75246] Request for merging "python-team" branch

[Ricardo Wurmus]

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

>> I see that we have Python 3.12, but let's please target 3.11 first.
>> It's a small step up from 3.10, and we already know of packages that
>> won't work with 3.12.  Let's keep the next iteration of this branch
>> simple, please.
>
> While refreshing (now) packages from astronomy module I've faced wit few
> failing to build due to hard requirement of Python3.11+ and NumPy 1.27+
> or 2.0+ which don't have yet.
>
> I've got a chance to to refresh NumPy to 1.24 but any farther version
> failed to build (see notes on the package).
>
> May we give it one more go with NumPy and friends?

I'd like to avoid the kind of sprawling upgrade chain reactions that
we've encountered in the most recent iteration.  My hope is that we've
already upgraded the most important and deeply rooted packages so that
an upgrade to numpy won't lead to an upgrade storm.

Please keep it at Numpy 1.x, though.  As with Pandas I think it makes
sense to first add an extra package for 2.x so that we can use package
transformations to test the impact before switching the default.

Let's open a new issue for the next iteration.

-- 
Ricardo