Python, poetry, and pandas

Python, poetry, and pandas

[Sébastien Lerique]

Dear all,

In the course of trying out poetry for data science projects on 
guix, I am running into the problem of some poetry-installed 
packages not finding their desired libraries. (See example below 
with pandas.) I suppose this is because poetry will download 
pre-built binary wheels which have pre-coded library paths.

If this is the problem, would there be any clean way to wrap those 
binary wheels, or patch poetry-on-guix to mandatorily rebuild 
binaries? (Would that make sense?)

(I am aware that python-pandas is packaged for guix, but some 
packages I will need down the road might not be; this project is 
also in collaboration with non guix users, so I am looking for a 
workflow which will work for them too.)

(I am also aware of guix-jupyter, but that is another topic I'd 
like to return to later as it needs some more bugsquashing for 
usability.)

Happy hacking!
Sébastien

#+BEGIN_SRC
$ poetry new poetry-test

Created package poetry_test in poetry-test
$ cd poetry-test/
$ poetry add pandas
Creating virtualenv poetry-test-vOWX4_Vr-py3.8 in 
/home/sl/.cache/pypoetry/virtualenvs
Using version ^1.2.3 for pandas

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 4 installs, 9 updates, 0 removals

  • Updating pyparsing (2.4.6 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 2.4.7)
  • Updating six (1.14.0 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 1.15.0)
  • Updating attrs (19.3.0 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 20.3.0)
  • Updating more-itertools (8.2.0 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 8.7.0)
  • Updating numpy (1.17.3 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 1.20.1)
  • Updating packaging (20.0 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 20.9)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Updating python-dateutil (2.8.1 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 2.8.1)
  • Updating pytz (2020.4 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 2021.1)
  • Updating wcwidth (0.1.8 
  /home/sl/.guix-profile/lib/python3.8/site-packages -> 0.2.5)
  • Installing pandas (1.2.3)
  • Installing pytest (5.4.3)
$ poetry run which python
/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/bin/python
$ poetry run python
Python 3.8.2 (default, Jan  1 1970, 00:00:01)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/__init__.py", 
  line 51, in <module>
    from pandas.core.api import (
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/api.py", 
  line 31, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/groupby/__init__.py", 
  line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, 
    NamedAgg, SeriesGroupBy
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/groupby/generic.py", 
  line 65, in <module>
    from pandas.core.frame import DataFrame
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/frame.py", 
  line 119, in <module>
    from pandas.core import algorithms, common as com, generic, 
    nanops, ops
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/generic.py", 
  line 111, in <module>
    from pandas.core.window import Expanding, 
    ExponentialMovingWindow, Rolling, Window
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/__init__.py", 
  line 1, in <module>
    from pandas.core.window.ewm import (  # noqa:F401
  File 
  "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py", 
  line 9, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: libstdc++.so.6: cannot open shared object file: No 
such file or directory
>>>
#+END_SRC

Re: Python, poetry, and pandas

[Phil]

Hi Sébastien,

Sébastien Lerique writes:

> Dear all,
>
> (I am aware that python-pandas is packaged for guix, but some packages
> I will need down the road might not be; this project is also in
> collaboration with non guix users, so I am looking for a workflow
> which will work for them too.)
>

Under the covers Poetry is just using regular Python virtualenvs I think.
These don't work well with Guix for a number of reasons - but in my
previous attempts to use them I found that Guix's use of PYTHONPATH
meant that virtualenvs where never isolated from packages installed by
Guix.

Guix offers pretty much everything that virtualenvs offers, but it does
things a little differently - you'll need to look at environments,
profiles, manifests, and use of a guix.scm to recreate (IMHO) a better
experience in the end.

When I started using Guix I went to some lengths to harmonize
virtualenvs and Guix - and others have made more substantial efforts
than I did - but you end up creating as many problems as you fix:
https://lists.gnu.org/archive/html/help-guix/2020-11/msg00237.html

If you search the mail archive a couple of other people have asked
similar questions to do with poetry, pyenv, pip, etc - they're all
interesting reading and easy to grep with those keywords:
https://lists.gnu.org/archive/html/help-guix/

The advice I was given at the time was don't mix package managers, and
in hindsight I'm glad repackaged what I needed under Guix.

I've packaged quite a few python libraries now, and most are pretty
trivial to do - if they are in PyPi you can just use this:

guix import pypi <package_name>

If you need a later version than Guix provides (eg pandas), my advice is
to use the ability to "inherit" the current definition in Guix and tweak
the version.  For Pandas I did this for a more recent version I needed
and all I had to do was disable 2 unit tests (which failed for a
non-Guix reason).

Apologies - not exactly the answer you want (I have a stab at that
below), but it's worth considering moving to Guix packaging exclusively
to keep your life simple.


>  File
> "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py",
> line 9, in <module>
>    import pandas._libs.window.aggregations as window_aggregations
> ImportError: libstdc++.so.6: cannot open shared object file: No such
> file or directory

If it's only missing a reference to the underlying C++ library you could
install this library using Guix, or if you're using a foreign distro point to
the non-Guix version under /usr/lib which, at a guess, Poetry may have
picked up - but this is going to cause you more trouble down the line
than repackaging in Guix in my experience.

Re: Python, poetry, and pandas

[zimoun]

Hi,

On Wed, 10 Mar 2021 at 16:38, Sébastien Lerique <sl@eauchat.org> wrote:

> In the course of trying out poetry for data science projects on 
> guix, I am running into the problem of some poetry-installed 
> packages not finding their desired libraries. (See example below 
> with pandas.) I suppose this is because poetry will download 
> pre-built binary wheels which have pre-coded library paths.

How did you installed poetry?  Via Guix or something else?
Which python packages do you have in your profile?
Is Guix System or Guix on foreign distro?

> If this is the problem, would there be any clean way to wrap those 
> binary wheels, or patch poetry-on-guix to mandatorily rebuild 
> binaries? (Would that make sense?)
>
> (I am aware that python-pandas is packaged for guix, but some 
> packages I will need down the road might not be; this project is 
> also in collaboration with non guix users, so I am looking for a 
> workflow which will work for them too.)

What are these packages?


Cheers,
simon

Re: Python, poetry, and pandas

[Sébastien Lerique]

Hi Phil and Simon,

Thanks both for the input and advice (not citing it all here).

First the details. Simon wrote:
> How did you installed poetry?  Via Guix or something else?
> Which python packages do you have in your profile?
> Is Guix System or Guix on foreign distro?

I'm using Guix System, and installed poetry via Guix. I have the 
following python packages installed: python, poetry, 
python-jupyterlab (from guix-science), and python-black. And 
guix-jupyter.

>> (I am aware that python-pandas is packaged for guix, but some
>> packages I will need down the road might not be; this project 
>> is
>> also in collaboration with non guix users, so I am looking for 
>> a
>> workflow which will work for them too.)
>
> What are these packages?

I don't know yet, but I suspect the situation might crop up. Phil 
pointed to the PyPi importer, and I haven't used it yet, but it 
does seem to make things simpler if I need to add a package to 
Guix.

Phil wrote:
>> ImportError: libstdc++.so.6: cannot open shared object file: No 
>> such
>> file or directory
>
> If it's only missing a reference to the underlying C++ library 
> you could
> install this library using Guix, or if you're using a foreign 
> distro point to
> the non-Guix version under /usr/lib which, at a guess, Poetry 
> may have
> picked up - but this is going to cause you more trouble down the 
> line
> than repackaging in Guix in my experience.

Yes, my experience with trying this for some Julia packages 
(GLMakie) convinced me not to try LD_LIBRARY_PATH hacks any more.

Now the broader view. Phil:
> Guix offers pretty much everything that virtualenvs offers, but 
> it does
> things a little differently - you'll need to look at 
> environments,
> profiles, manifests, and use of a guix.scm to recreate (IMHO) a 
> better
> experience in the end.

I just set up a simple guix.scm for my current project, which 
indeed does the job, and of course provides all the great 
guarantees of Guix! Given the number of python packages currently, 
and the existence of the PyPi importer, I'm pretty convinced it 
covers my personal needs.

Now my wish is to find ways to collaborate with others on this 
kind of project, who unfortunately often use macOS. I was 
exploring the possibility of Guix on Darwin, but the best way to 
do that looks like running Guix System inside Docker:

http://logs.guix.gnu.org/guix/2021-03-11.log#044004
https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00111.html

So it seems that to collaborate with non-guix and non-GNU/Linux 
users, the options are 1) maintain a second environment management 
configuration (e.g. using poetry) to provide a stop-gap, or 2) 
convince colleagues to run Guix in Docker. The situation for 
Windows might be a bit better using the Linux Subsystem.

Thanks again all,
Sébastien

On 10 Mar 2021 at 19:02, Phil <phil@beadling.co.uk> wrote:

> Hi Sébastien,
>
> Sébastien Lerique writes:
>
>> Dear all,
>>
>> (I am aware that python-pandas is packaged for guix, but some 
>> packages
>> I will need down the road might not be; this project is also in
>> collaboration with non guix users, so I am looking for a 
>> workflow
>> which will work for them too.)
>>
>
> Under the covers Poetry is just using regular Python virtualenvs 
> I think.
> These don't work well with Guix for a number of reasons - but in 
> my
> previous attempts to use them I found that Guix's use of 
> PYTHONPATH
> meant that virtualenvs where never isolated from packages 
> installed by
> Guix.
>
> Guix offers pretty much everything that virtualenvs offers, but 
> it does
> things a little differently - you'll need to look at 
> environments,
> profiles, manifests, and use of a guix.scm to recreate (IMHO) a 
> better
> experience in the end.
>
> When I started using Guix I went to some lengths to harmonize
> virtualenvs and Guix - and others have made more substantial 
> efforts
> than I did - but you end up creating as many problems as you 
> fix:
> https://lists.gnu.org/archive/html/help-guix/2020-11/msg00237.html
>
> If you search the mail archive a couple of other people have 
> asked
> similar questions to do with poetry, pyenv, pip, etc - they're 
> all
> interesting reading and easy to grep with those keywords:
> https://lists.gnu.org/archive/html/help-guix/
>
> The advice I was given at the time was don't mix package 
> managers, and
> in hindsight I'm glad repackaged what I needed under Guix.
>
> I've packaged quite a few python libraries now, and most are 
> pretty
> trivial to do - if they are in PyPi you can just use this:
>
> guix import pypi <package_name>
>
> If you need a later version than Guix provides (eg pandas), my 
> advice is
> to use the ability to "inherit" the current definition in Guix 
> and tweak
> the version.  For Pandas I did this for a more recent version I 
> needed
> and all I had to do was disable 2 unit tests (which failed for a
> non-Guix reason).
>
> Apologies - not exactly the answer you want (I have a stab at 
> that
> below), but it's worth considering moving to Guix packaging 
> exclusively
> to keep your life simple.
>
>
>>  File
>> "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py",
>> line 9, in <module>
>>    import pandas._libs.window.aggregations as 
>>    window_aggregations
>> ImportError: libstdc++.so.6: cannot open shared object file: No 
>> such
>> file or directory
>
> If it's only missing a reference to the underlying C++ library 
> you could
> install this library using Guix, or if you're using a foreign 
> distro point to
> the non-Guix version under /usr/lib which, at a guess, Poetry 
> may have
> picked up - but this is going to cause you more trouble down the 
> line
> than repackaging in Guix in my experience.

Re: Python, poetry, and pandas

[Phil]

Sébastien Lerique writes:


> Now my wish is to find ways to collaborate with others on this kind of
> project, who unfortunately often use macOS. I was exploring the
> possibility of Guix on Darwin, but the best way to do that looks like
> running Guix System inside Docker:
>

I haven't tried with MacOS yet but a few ideas:

- Try using the QEMU image available (or create your own VirtualBox
image - I've tried this and it worked OK).

- If running a full graphical env is too clunky for you could experiment with multipass.  I
haven't tried this yet on multipass but I've had some success running
Guix inside WSL2 (very similar to multipass) on Windows using busybox
and these instructions:
https://github.com/giuliano108/guix-packages/blob/master/notes/Guix-on-WSL2.md

It's a bit 'beta', but I use this when I want fast local access to a Guix
environment from a Windows desktop without having
yet-another-foreign-distro (eg Ubuntu-on-WSL) in-between. 

- I'm not expert on this but I suspect you could produce a slimmed down
version of Guix more akin to the WSL experience using the 'guix system
vm-image' command.  Check-out the manual.

Having something that worked with multipass and WSL2 directly on the Guix
website would be extremely useful in drawing a crowd from non-Linux
users!  So this would be a noble endeavour IMHO.  If you get anything
working I'd be very interested.

Of course this isn't substitute for running on Darwin as a foreign
distro, but I'm unaware of any current project to do that.  I suspect
it would be challenging to get package definitions working for substantially
different compiler flags etc, it would almost become a separate project.

It has been discussed tho - follow the links in this post  to see discussions on guix-devel too:
https://www.reddit.com/r/GUIX/comments/c2xei5/any_way_to_have_guix_running_as_a_package_manager/