unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* [core-updates-frozen] Different variants of Python packages in the same profile?
@ 2021-11-20 10:42 Ricardo Wurmus
  2021-11-21  8:51 ` Lars-Dominik Braun
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-20 10:42 UTC (permalink / raw)
  To: guix-devel

Hi Guix,

on the core-updates-frozen branch we have a minor problem: 
python-numpy has been upgraded to 1.21, but python-numba can only 
be built with 1.20[1].  I added python-numpy-1.20 and made 
python-numba use it.

Now the problem I have is that numba is used with other packages 
(such as python-scikit-learn) that build with the latest numpy. 
An example is python-umap-learn, which propagates both 
python-numba (with python-numpy-1.20) and python-scikit-learn 
(with python-numpy).  This doesn’t work because Python only knows 
about the name “numpy” and will pick the first matching module on 
the load path.

So… since numpy 1.20 is the exception here, could we perhaps … 
rename it?  And then have python-numba import that renamed module 
“totally-not-numpy” instead of “numpy”?  Could we thus avoid this 
conflict?  If renaming is an option — how would it be done?  Is it 
enough to rename the “numpy” directory with “numpy-1.20”, the 
“numpy.py” file with “numpy-1.20.py”, and then update all “import” 
statements both by numpy itself and by python-numba?

I’d like to fix this before merging core-updates-frozen, as this 
affects at least 16 bioinformatics packages and working around 
this will be difficult for users.

-- 
Ricardo

[1]: https://github.com/numba/numba/issues/7176


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
@ 2021-11-21  8:51 ` Lars-Dominik Braun
  2021-11-21 19:02   ` Ricardo Wurmus
  2021-11-21 21:27 ` zimoun
  2021-11-22 13:25 ` Ludovic Courtès
  2 siblings, 1 reply; 7+ messages in thread
From: Lars-Dominik Braun @ 2021-11-21  8:51 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel

Hello Ricardo,

> So… since numpy 1.20 is the exception here, could we perhaps … 
> rename it?  And then have python-numba import that renamed module 
> “totally-not-numpy” instead of “numpy”?  Could we thus avoid this 
> conflict?  If renaming is an option — how would it be done?  Is it 
> enough to rename the “numpy” directory with “numpy-1.20”, the 
> “numpy.py” file with “numpy-1.20.py”, and then update all “import” 
> statements both by numpy itself and by python-numba?
I feel this is a dangerous idea. Python is dynamically typed and if we
– somehow – end up with objects from both, numpy 1.20 and numpy 1.21
in the same program, which can still happen when renaming, things may
go wrong very badly. [1] says versions 1.* are ABI compatible, but the
API changes between releases, which is probably why numba cannot be used
with a newer numpy.

Can we rewrite the entire graph to use numpy 1.20 whenever a package
imports numba?

Cheers,
Lars

[1] https://numpy.org/devdocs/user/depending_on_numpy.html



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-21  8:51 ` Lars-Dominik Braun
@ 2021-11-21 19:02   ` Ricardo Wurmus
  0 siblings, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-21 19:02 UTC (permalink / raw)
  To: Lars-Dominik Braun; +Cc: guix-devel


Lars-Dominik Braun <lars@6xq.net> writes:

> Hello Ricardo,
>
>> So… since numpy 1.20 is the exception here, could we perhaps … 
>> rename it?  And then have python-numba import that renamed 
>> module 
>> “totally-not-numpy” instead of “numpy”?  Could we thus avoid 
>> this 
>> conflict?  If renaming is an option — how would it be done?  Is 
>> it 
>> enough to rename the “numpy” directory with “numpy-1.20”, the 
>> “numpy.py” file with “numpy-1.20.py”, and then update all 
>> “import” 
>> statements both by numpy itself and by python-numba?
> I feel this is a dangerous idea. Python is dynamically typed and 
> if we
> – somehow – end up with objects from both, numpy 1.20 and numpy 
> 1.21
> in the same program, which can still happen when renaming, 
> things may
> go wrong very badly. [1] says versions 1.* are ABI compatible, 
> but the
> API changes between releases, which is probably why numba cannot 
> be used
> with a newer numpy.
>
> Can we rewrite the entire graph to use numpy 1.20 whenever a 
> package
> imports numba?

After a quick discussion on IRC we decided to make 1.20 the 
default and keep 1.21 as python-numpy-next.  Let’s hope we can 
switch to 1.21 for good soon.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
  2021-11-21  8:51 ` Lars-Dominik Braun
@ 2021-11-21 21:27 ` zimoun
  2021-11-22 13:25 ` Ludovic Courtès
  2 siblings, 0 replies; 7+ messages in thread
From: zimoun @ 2021-11-21 21:27 UTC (permalink / raw)
  To: Ricardo Wurmus, guix-devel

Hi Ricardo,

On Sat, 20 Nov 2021 at 10:42, Ricardo Wurmus <rekado@elephly.net> wrote:

> on the core-updates-frozen branch we have a minor problem: 
> python-numpy has been upgraded to 1.21, but python-numba can only 
> be built with 1.20[1].  I added python-numpy-1.20 and made 
> python-numba use it.

I think the upgrade of Numpy had been a mistake.  All the Python
scientific stack has to be upgraded in one go.  From my understanding,
it is the same issue as Haskell and LTS, for instance.  As Lars said,
try to have several versions would probably lead to a mess with weird
behaviours difficult to debug; from my understanding.

The best is to downgrades python-numpy to 1.20 (potentially 1.20.3
released on May 2021, that’s not that old :-)) and wait python-numba
works with Numpy 1.21.  IMHO.


Cheers,
simon





^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
  2021-11-21  8:51 ` Lars-Dominik Braun
  2021-11-21 21:27 ` zimoun
@ 2021-11-22 13:25 ` Ludovic Courtès
  2021-11-22 19:37   ` Ricardo Wurmus
  2021-11-22 19:39   ` Ricardo Wurmus
  2 siblings, 2 replies; 7+ messages in thread
From: Ludovic Courtès @ 2021-11-22 13:25 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> on the core-updates-frozen branch we have a minor problem:
> python-numpy has been upgraded to 1.21, but python-numba can only 
> be built with 1.20[1].  I added python-numpy-1.20 and made
> python-numba use it.

How big of an effort would it be to modify numba so that it works with
1.21?  With luck, that effort has already been made upstream?

Ludo’.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-22 13:25 ` Ludovic Courtès
@ 2021-11-22 19:37   ` Ricardo Wurmus
  2021-11-22 19:39   ` Ricardo Wurmus
  1 sibling, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-22 19:37 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> on the core-updates-frozen branch we have a minor problem:
>> python-numpy has been upgraded to 1.21, but python-numba can 
>> only 
>> be built with 1.20[1].  I added python-numpy-1.20 and made
>> python-numba use it.
>
> How big of an effort would it be to modify numba so that it 
> works with
> 1.21?  With luck, that effort has already been made upstream?

It hasn’t.  Upstream doesn’t seem to know exactly how to 
accomplish this.  There’s a breaking change in numpy (something 
relating to typing of ufuncs) and it leads to incorrect type 
errors in numba AFAIU.  This is not something I’d feel comfortable 
just hacking around.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [core-updates-frozen] Different variants of Python packages in the same profile?
  2021-11-22 13:25 ` Ludovic Courtès
  2021-11-22 19:37   ` Ricardo Wurmus
@ 2021-11-22 19:39   ` Ricardo Wurmus
  1 sibling, 0 replies; 7+ messages in thread
From: Ricardo Wurmus @ 2021-11-22 19:39 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> on the core-updates-frozen branch we have a minor problem:
>> python-numpy has been upgraded to 1.21, but python-numba can 
>> only 
>> be built with 1.20[1].  I added python-numpy-1.20 and made
>> python-numba use it.
>
> How big of an effort would it be to modify numba so that it 
> works with
> 1.21?  With luck, that effort has already been made upstream?

I spoke too soon: 
https://github.com/numba/numba/issues/7175#issuecomment-975470113

  “Numba 0.55 release candidates will be out within the next week 
  or so (hopefully!), these will have NumPy 1.21.x series 
  support.”

So soon after the merge we will be able to switch back to numpy 
1.21 as the default.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-11-22 19:41 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-20 10:42 [core-updates-frozen] Different variants of Python packages in the same profile? Ricardo Wurmus
2021-11-21  8:51 ` Lars-Dominik Braun
2021-11-21 19:02   ` Ricardo Wurmus
2021-11-21 21:27 ` zimoun
2021-11-22 13:25 ` Ludovic Courtès
2021-11-22 19:37   ` Ricardo Wurmus
2021-11-22 19:39   ` Ricardo Wurmus

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).