* [core-updates-frozen] Different variants of Python packages in the same profile?
@ 2021-11-20 10:42 Ricardo Wurmus
2021-11-21 8:51 ` Lars-Dominik Braun
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-20 10:42 UTC (permalink / raw)
To: guix-devel
Hi Guix,
on the core-updates-frozen branch we have a minor problem:
python-numpy has been upgraded to 1.21, but python-numba can only
be built with 1.20[1]. I added python-numpy-1.20 and made
python-numba use it.
Now the problem I have is that numba is used with other packages
(such as python-scikit-learn) that build with the latest numpy.
An example is python-umap-learn, which propagates both
python-numba (with python-numpy-1.20) and python-scikit-learn
(with python-numpy). This doesn’t work because Python only knows
about the name “numpy” and will pick the first matching module on
the load path.
So… since numpy 1.20 is the exception here, could we perhaps …
rename it? And then have python-numba import that renamed module
“totally-not-numpy” instead of “numpy”? Could we thus avoid this
conflict? If renaming is an option — how would it be done? Is it
enough to rename the “numpy” directory with “numpy-1.20”, the
“numpy.py” file with “numpy-1.20.py”, and then update all “import”
statements both by numpy itself and by python-numba?
I’d like to fix this before merging core-updates-frozen, as this
affects at least 16 bioinformatics packages and working around
this will be difficult for users.
--
Ricardo
[1]: https://github.com/numba/numba/issues/7176
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
@ 2021-11-21 8:51 ` Lars-Dominik Braun
2021-11-21 19:02 ` Ricardo Wurmus
2021-11-21 21:27 ` zimoun
2021-11-22 13:25 ` Ludovic Courtès
2 siblings, 1 reply; 7+ messages in thread
From: Lars-Dominik Braun @ 2021-11-21 8:51 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel
Hello Ricardo,
> So… since numpy 1.20 is the exception here, could we perhaps …
> rename it? And then have python-numba import that renamed module
> “totally-not-numpy” instead of “numpy”? Could we thus avoid this
> conflict? If renaming is an option — how would it be done? Is it
> enough to rename the “numpy” directory with “numpy-1.20”, the
> “numpy.py” file with “numpy-1.20.py”, and then update all “import”
> statements both by numpy itself and by python-numba?
I feel this is a dangerous idea. Python is dynamically typed and if we
– somehow – end up with objects from both, numpy 1.20 and numpy 1.21
in the same program, which can still happen when renaming, things may
go wrong very badly. [1] says versions 1.* are ABI compatible, but the
API changes between releases, which is probably why numba cannot be used
with a newer numpy.
Can we rewrite the entire graph to use numpy 1.20 whenever a package
imports numba?
Cheers,
Lars
[1] https://numpy.org/devdocs/user/depending_on_numpy.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-21 8:51 ` Lars-Dominik Braun
@ 2021-11-21 19:02 ` Ricardo Wurmus
0 siblings, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-21 19:02 UTC (permalink / raw)
To: Lars-Dominik Braun; +Cc: guix-devel
Lars-Dominik Braun <lars@6xq.net> writes:
> Hello Ricardo,
>
>> So… since numpy 1.20 is the exception here, could we perhaps …
>> rename it? And then have python-numba import that renamed
>> module
>> “totally-not-numpy” instead of “numpy”? Could we thus avoid
>> this
>> conflict? If renaming is an option — how would it be done? Is
>> it
>> enough to rename the “numpy” directory with “numpy-1.20”, the
>> “numpy.py” file with “numpy-1.20.py”, and then update all
>> “import”
>> statements both by numpy itself and by python-numba?
> I feel this is a dangerous idea. Python is dynamically typed and
> if we
> – somehow – end up with objects from both, numpy 1.20 and numpy
> 1.21
> in the same program, which can still happen when renaming,
> things may
> go wrong very badly. [1] says versions 1.* are ABI compatible,
> but the
> API changes between releases, which is probably why numba cannot
> be used
> with a newer numpy.
>
> Can we rewrite the entire graph to use numpy 1.20 whenever a
> package
> imports numba?
After a quick discussion on IRC we decided to make 1.20 the
default and keep 1.21 as python-numpy-next. Let’s hope we can
switch to 1.21 for good soon.
--
Ricardo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
2021-11-21 8:51 ` Lars-Dominik Braun
@ 2021-11-21 21:27 ` zimoun
2021-11-22 13:25 ` Ludovic Courtès
2 siblings, 0 replies; 7+ messages in thread
From: zimoun @ 2021-11-21 21:27 UTC (permalink / raw)
To: Ricardo Wurmus, guix-devel
Hi Ricardo,
On Sat, 20 Nov 2021 at 10:42, Ricardo Wurmus <rekado@elephly.net> wrote:
> on the core-updates-frozen branch we have a minor problem:
> python-numpy has been upgraded to 1.21, but python-numba can only
> be built with 1.20[1]. I added python-numpy-1.20 and made
> python-numba use it.
I think the upgrade of Numpy had been a mistake. All the Python
scientific stack has to be upgraded in one go. From my understanding,
it is the same issue as Haskell and LTS, for instance. As Lars said,
try to have several versions would probably lead to a mess with weird
behaviours difficult to debug; from my understanding.
The best is to downgrades python-numpy to 1.20 (potentially 1.20.3
released on May 2021, that’s not that old :-)) and wait python-numba
works with Numpy 1.21. IMHO.
Cheers,
simon
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
2021-11-21 8:51 ` Lars-Dominik Braun
2021-11-21 21:27 ` zimoun
@ 2021-11-22 13:25 ` Ludovic Courtès
2021-11-22 19:37 ` Ricardo Wurmus
2021-11-22 19:39 ` Ricardo Wurmus
2 siblings, 2 replies; 7+ messages in thread
From: Ludovic Courtès @ 2021-11-22 13:25 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel
Hi,
Ricardo Wurmus <rekado@elephly.net> skribis:
> on the core-updates-frozen branch we have a minor problem:
> python-numpy has been upgraded to 1.21, but python-numba can only
> be built with 1.20[1]. I added python-numpy-1.20 and made
> python-numba use it.
How big of an effort would it be to modify numba so that it works with
1.21? With luck, that effort has already been made upstream?
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-22 13:25 ` Ludovic Courtès
@ 2021-11-22 19:37 ` Ricardo Wurmus
2021-11-22 19:39 ` Ricardo Wurmus
1 sibling, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-22 19:37 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Hi,
Ludovic Courtès <ludo@gnu.org> writes:
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> on the core-updates-frozen branch we have a minor problem:
>> python-numpy has been upgraded to 1.21, but python-numba can
>> only
>> be built with 1.20[1]. I added python-numpy-1.20 and made
>> python-numba use it.
>
> How big of an effort would it be to modify numba so that it
> works with
> 1.21? With luck, that effort has already been made upstream?
It hasn’t. Upstream doesn’t seem to know exactly how to
accomplish this. There’s a breaking change in numpy (something
relating to typing of ufuncs) and it leads to incorrect type
errors in numba AFAIU. This is not something I’d feel comfortable
just hacking around.
--
Ricardo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
2021-11-22 13:25 ` Ludovic Courtès
2021-11-22 19:37 ` Ricardo Wurmus
@ 2021-11-22 19:39 ` Ricardo Wurmus
1 sibling, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-22 19:39 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Ludovic Courtès <ludo@gnu.org> writes:
> Hi,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> on the core-updates-frozen branch we have a minor problem:
>> python-numpy has been upgraded to 1.21, but python-numba can
>> only
>> be built with 1.20[1]. I added python-numpy-1.20 and made
>> python-numba use it.
>
> How big of an effort would it be to modify numba so that it
> works with
> 1.21? With luck, that effort has already been made upstream?
I spoke too soon:
https://github.com/numba/numba/issues/7175#issuecomment-975470113
“Numba 0.55 release candidates will be out within the next week
or so (hopefully!), these will have NumPy 1.21.x series
support.”
So soon after the merge we will be able to switch back to numpy
1.21 as the default.
--
Ricardo
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-11-22 19:41 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
2021-11-21 8:51 ` Lars-Dominik Braun
2021-11-21 19:02 ` Ricardo Wurmus
2021-11-21 21:27 ` zimoun
2021-11-22 13:25 ` Ludovic Courtès
2021-11-22 19:37 ` Ricardo Wurmus
2021-11-22 19:39 ` Ricardo Wurmus
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).