unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
* bug#53258: Python unable to find modules within a Singularity container created with guix pack
@ 2022-01-14 14:20 Marek Felšöci
  2022-01-14 17:45 ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Marek Felšöci @ 2022-01-14 14:20 UTC (permalink / raw)
  To: 53258

Hello folks,

Recently, I have come around this issue. I created a Singularity container using 
the following `guix pack` command:

`guix pack -f squashfs bash coreutils python python-numpy`

Then, I run a bash shell within the container:

`singularity exec test.gz.squashfs bash`

Now, if I want to use numpy,

`python3 -m numpy`

it gives me the following error and the numpy module IS NOT found:

Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: 
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages' 
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named 
numpy

However, if I do,

`export PYTHONPATH=$GUIX_PYTHONPATH`

the numpy module IS found:

Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: 
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages' 
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named 
numpy.__main__; 'numpy' is a package and cannot be directly executed

Still, the sitecustomize error persists.

See the output of `guix describe` below (only the guix channel):

Pokolenie 3    14. január 2022 14:33:06    (súčasné)
   guix 5c5d9e5
     zdroj repozitára: https://git.savannah.gnu.org/git/guix.git
     vetva: master
     úprava: 5c5d9e5a3208fa51351c6ad8267eba3d96fa2232

Thanks,

Marek





^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#53258: Python unable to find modules within a Singularity container created with guix pack
  2022-01-14 14:20 bug#53258: Python unable to find modules within a Singularity container created with guix pack Marek Felšöci
@ 2022-01-14 17:45 ` Ludovic Courtès
  2022-02-04 20:50   ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2022-01-14 17:45 UTC (permalink / raw)
  To: Marek Felšöci; +Cc: 53258, Maxim Cournoyer

Hi Marek,

Marek Felšöci <marek@felsoci.sk> skribis:

> Recently, I have come around this issue. I created a Singularity
> container using the following `guix pack` command:
>
> `guix pack -f squashfs bash coreutils python python-numpy`
>
> Then, I run a bash shell within the container:
>
> `singularity exec test.gz.squashfs bash`
>
> Now, if I want to use numpy,
>
> `python3 -m numpy`
>
> it gives me the following error and the numpy module IS NOT found:
>
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError:
> '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages' 
> is not in list
> /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No
> module named numpy

I can kinda reproduce via a relocatable pack (relocatable packs and
Singularity packs have in common that they use relative symlinks in
their profile, to allow for relocation):

  guix pack -RR python python-numpy -S /bin=bin -S /etc=etc

Then I unpack the thing:

  mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/…-pack.tar.gz

And from there I use the same trick as in
‘tests/guix-pack-relocatable.sh’ to “hide” /gnu/store:

--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan  1 1970, 00:00:01) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>> import os
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages")
['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hack', 'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt', 'pip', 'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache__']
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/numpy")
['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.py', 'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py', '__config__.py', 'core', '_distributor_init.py', 'distutils', 'tests', 'emath.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd', 'conftest.py', '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing', 'LICENSE.txt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py', 'setup.py', '__pycache__']
--8<---------------cut here---------------end--------------->8---

So in this case ‘GUIX_PYTHONPATH’ is correctly set, but it looks as
though ‘sitecustomize.py’ wasn’t loaded.  (I don’t get the “Error in
sitecustomize” message that Marek sees though, weird!)

From Marek’s message, it would seem that somewhere file names are
constructed by appending “..”.  Given Unix’s weird dot-dot semantics,
relative symlinks could be interfering:

--8<---------------cut here---------------start------------->8---
$ ls -l /tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/
totalo 48
lrwxrwxrwx 1 ludo users 102 Jan  1  1970 _distutils_hack -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_distutils_hack
lrwxrwxrwx 1 ludo users 111 Jan  1  1970 distutils-precedence.pth -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/distutils-precedence.pth
lrwxrwxrwx 1 ludo users  99 Jan  1  1970 numpy -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy
lrwxrwxrwx 1 ludo users 121 Jan  1  1970 numpy-1.20.3-py3.9.egg-info -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy-1.20.3-py3.9.egg-info
lrwxrwxrwx 1 ludo users  90 Jan  1  1970 pip -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip
lrwxrwxrwx 1 ludo users 107 Jan  1  1970 pip-21.1.3.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip-21.1.3.dist-info
lrwxrwxrwx 1 ludo users 100 Jan  1  1970 pkg_resources -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_resources
lrwxrwxrwx 1 ludo users  98 Jan  1  1970 __pycache__ -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycache__
lrwxrwxrwx 1 ludo users  97 Jan  1  1970 README.txt -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.txt
lrwxrwxrwx 1 ludo users  97 Jan  1  1970 setuptools -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools
lrwxrwxrwx 1 ludo users 114 Jan  1  1970 setuptools-56.0.0.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools-56.0.0.dist-info
lrwxrwxrwx 1 ludo users 103 Jan  1  1970 sitecustomize.py -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/sitecustomize.py
--8<---------------cut here---------------end--------------->8---

Now look below how dot-dot resolution whereas “lexical” dot-dot
resolution dose the right thing:

--8<---------------cut here---------------start------------->8---
$ ls -l /tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
ls: ne eblas atingi '/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages': Dosiero aŭ dosierujo ne ekzistas
$ ls -l /tmp/pack/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
totalo 40
dr-xr-xr-x 3 ludo users 4096 Jan  1  1970 _distutils_hack/
-r--r--r-- 1 ludo users  152 Jan  1  1970 distutils-precedence.pth
dr-xr-xr-x 5 ludo users 4096 Jan  1  1970 pip/
dr-xr-xr-x 2 ludo users 4096 Jan  1  1970 pip-21.1.3.dist-info/
dr-xr-xr-x 6 ludo users 4096 Jan  1  1970 pkg_resources/
dr-xr-xr-x 2 ludo users 4096 Jan  1  1970 __pycache__/
-r--r--r-- 1 ludo users  119 Jan  1  1970 README.txt
dr-xr-xr-x 7 ludo users 4096 Jan  1  1970 setuptools/
dr-xr-xr-x 2 ludo users 4096 Jan  1  1970 setuptools-56.0.0.dist-info/
-r--r--r-- 1 ludo users 2172 Jan  1  1970 sitecustomize.py
--8<---------------cut here---------------end--------------->8---

In Python’s ‘site.py’, the ‘addsitedir’ method indirectly calls
‘os.path.abspath’, not sure if that is at fault.

I also see that ‘sys.path’ is in fact completely bogus, with “pack”
missing from every directory name:

--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store;  . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan  1 1970, 00:00:01) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python39.zip', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/lib-dynload', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages']
--8<---------------cut here---------------end--------------->8---

So I think something somewhere is manipulating file names and gets
confused.

Does that ring a bell, Maxim?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#53258: Python unable to find modules within a Singularity container created with guix pack
  2022-01-14 17:45 ` Ludovic Courtès
@ 2022-02-04 20:50   ` Ludovic Courtès
  2022-02-05 22:28     ` Josselin Poiret via Bug reports for GNU Guix
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2022-02-04 20:50 UTC (permalink / raw)
  To: Marek Felšöci; +Cc: 53258, Maxim Cournoyer

Hello!

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

>>From Marek’s message, it would seem that somewhere file names are
> constructed by appending “..”.  Given Unix’s weird dot-dot semantics,
> relative symlinks could be interfering:

Some news!  Here’s a recap of the IRC discussion earlier today:
‘sys.prefix’ looks like
/tmp/pack2/bin/../../zmv1alp7dzjd6hkc7l6fjz817n2mfzzw-python-3.9.9R.

‘sys.normpath’ does lexical dot-dot resolution (which is incorrect on
POSIX):

  https://docs.python.org/3.11/library/os.path.html#os.path.normpath

Thus, passing the prefix above to ‘sys.normpath’ leads “pack2” to be
erased.

Josselin mentioned “path_hooks” as a possible solution, though I’m not
sure what that means; could you explain?  :-)

Thanks!

Ludo’.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#53258: Python unable to find modules within a Singularity container created with guix pack
  2022-02-04 20:50   ` Ludovic Courtès
@ 2022-02-05 22:28     ` Josselin Poiret via Bug reports for GNU Guix
  2022-02-05 22:30       ` Josselin Poiret via Bug reports for GNU Guix
  2022-03-07  8:25       ` Ludovic Courtès
  0 siblings, 2 replies; 6+ messages in thread
From: Josselin Poiret via Bug reports for GNU Guix @ 2022-02-05 22:28 UTC (permalink / raw)
  To: Ludovic Courtès, Marek Felšöci; +Cc: 53258, Maxim Cournoyer

Hello everyone,

After the IRC investigation, it turned out the use of abspath in
Lib/site.py was in fact the culprit. Replacing all its uses with
realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S
/bin=bin -S /etc=etc` did in fact resolve the issue!

I've raised the issue upstream at [1].  The fix itself is pretty simple,
but it's theoretically a breaking change, so we'll see how Python
maintainers want to move forward with it.  We could backport this fix
for the next core-updates cycle, whether or not it gets accepted
upstream, and we will also need to modify sitecustomize.py as well to
call os.path.realpath on python_site.

Best,
-- 
Josselin Poiret




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#53258: Python unable to find modules within a Singularity container created with guix pack
  2022-02-05 22:28     ` Josselin Poiret via Bug reports for GNU Guix
@ 2022-02-05 22:30       ` Josselin Poiret via Bug reports for GNU Guix
  2022-03-07  8:25       ` Ludovic Courtès
  1 sibling, 0 replies; 6+ messages in thread
From: Josselin Poiret via Bug reports for GNU Guix @ 2022-02-05 22:30 UTC (permalink / raw)
  To: Ludovic Courtès, Marek Felšöci; +Cc: 53258, Maxim Cournoyer

Silly me forgetting the link, here it is:

[1] https://bugs.python.org/issue46653

Apologies for the noise,
-- 
Josselin Poiret




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#53258: Python unable to find modules within a Singularity container created with guix pack
  2022-02-05 22:28     ` Josselin Poiret via Bug reports for GNU Guix
  2022-02-05 22:30       ` Josselin Poiret via Bug reports for GNU Guix
@ 2022-03-07  8:25       ` Ludovic Courtès
  1 sibling, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2022-03-07  8:25 UTC (permalink / raw)
  To: Josselin Poiret; +Cc: 53258, Marek Felšöci, Maxim Cournoyer

Hi Josselin,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> After the IRC investigation, it turned out the use of abspath in
> Lib/site.py was in fact the culprit. Replacing all its uses with
> realpath and doing `./pre-inst-env guix pack -RR python python-numpy -S
> /bin=bin -S /etc=etc` did in fact resolve the issue!
>
> I've raised the issue upstream at [1].  The fix itself is pretty simple,
> but it's theoretically a breaking change, so we'll see how Python
> maintainers want to move forward with it.  We could backport this fix
> for the next core-updates cycle, whether or not it gets accepted
> upstream, and we will also need to modify sitecustomize.py as well to
> call os.path.realpath on python_site.

> [1] https://bugs.python.org/issue46653

Thanks for the investigation and for reporting it upstream!

Is there a workaround we could apply locally, ideally one that does not
lead to a world rebuild?

Ludo’.




^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-03-07  8:26 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-14 14:20 bug#53258: Python unable to find modules within a Singularity container created with guix pack Marek Felšöci
2022-01-14 17:45 ` Ludovic Courtès
2022-02-04 20:50   ` Ludovic Courtès
2022-02-05 22:28     ` Josselin Poiret via Bug reports for GNU Guix
2022-02-05 22:30       ` Josselin Poiret via Bug reports for GNU Guix
2022-03-07  8:25       ` Ludovic Courtès

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).