From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OHeoAuy24WGSbAEAgWs5BA (envelope-from ) for ; Fri, 14 Jan 2022 18:46:20 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id j//kOuu24WGGrAAAauVa8A (envelope-from ) for ; Fri, 14 Jan 2022 18:46:19 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3840B3E4A4 for ; Fri, 14 Jan 2022 18:46:19 +0100 (CET) Received: from localhost ([::1]:50526 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n8QeY-0002WY-6l for larch@yhetil.org; Fri, 14 Jan 2022 12:46:18 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43932) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n8QeI-0002WE-AU for bug-guix@gnu.org; Fri, 14 Jan 2022 12:46:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:45184) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n8QeI-00056L-0l for bug-guix@gnu.org; Fri, 14 Jan 2022 12:46:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n8QeH-00086j-TS for bug-guix@gnu.org; Fri, 14 Jan 2022 12:46:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#53258: Python unable to find modules within a Singularity container created with guix pack Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 14 Jan 2022 17:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53258 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Marek =?UTF-8?Q?Fel=C5=A1=C3=B6ci?= Received: via spool by 53258-submit@debbugs.gnu.org id=B53258.164218233831123 (code B ref 53258); Fri, 14 Jan 2022 17:46:01 +0000 Received: (at 53258) by debbugs.gnu.org; 14 Jan 2022 17:45:38 +0000 Received: from localhost ([127.0.0.1]:38087 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n8Qdt-00085v-Qa for submit@debbugs.gnu.org; Fri, 14 Jan 2022 12:45:38 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:49480) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n8Qdp-00085a-D2 for 53258@debbugs.gnu.org; Fri, 14 Jan 2022 12:45:37 -0500 X-IronPort-AV: E=Sophos;i="5.88,289,1635199200"; d="scan'208";a="15900436" Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jan 2022 18:45:26 +0100 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <67e7d225-f824-b691-7125-6b37abef8bb0@felsoci.sk> Date: Fri, 14 Jan 2022 18:45:26 +0100 In-Reply-To: <67e7d225-f824-b691-7125-6b37abef8bb0@felsoci.sk> ("Marek =?UTF-8?Q?Fel=C5=A1=C3=B6ci?="'s message of "Fri, 14 Jan 2022 15:20:46 +0100") Message-ID: <87wnj22ouh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 53258@debbugs.gnu.org, Maxim Cournoyer Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1642182379; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=Er/rdUcV1q2DYrKRSkBIWGgaaUIDN3VIxvjLVx6p4mo=; b=Ur7LPZjh+Xd7Pq4lVkV8O0ZrW9XTm3eW3+FRRIJ7IE6fUs/Z8v/vbek/zFdnm8xKdzaoOx azxiJGSR0qnjCtVx0tsrTw67HeZmYMNQS/oYv8pq/eT/NjgpubccrSFdcHXb5gQbDHSrKj 2tD7w38eEkwRYCIScYWPnWuJgqYUI8ATt1HT1Uyt2xkm7uy8pvXtY9CZa2TAwR5nRTu5dU h2unjvahbwLSjBUKZFDtU5wY2xt8UoUU6EnOtQTuK3zYRjQG4Bz7L0aNm6f3juViT5nvMr qoB6lA+dhVCJ4Zl9poC8XZUiD1HeQSmoiohwEMQO7j4hN9QqlPiW4DEnqacNVA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1642182379; a=rsa-sha256; cv=none; b=aLTUCOlHkdkDHKuQ+z3yXq4XoMPD9p/LkdAYhLjlgPobNTO4T/h5rHAUcKd7PHdx5/RkPp n/zlmdlDCrXv7ghz3pBcU+POLS5n0WtTaCPup9tZdh0tvQfqM4pmHwBv+cObB63xrZyX3J Y8Nt5WGRZigM5VVIZtGltBLiDR71rJ6yvlCn0iQZ8qBLHZy3nD53OW1T3IMqj0IWSk13yd TUJqrJ2EgetquofKGF7lw2LUq39gjUHKK+Aqmf7bMO2l2K2Lu41nOFuXFIULeY0s3qgxJ+ QldBah0hJqRYxkNzy3fPb/pmqt/r68+Qq/AQCboWp0uNhe7N2INx917zlQ8bhQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.32 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3840B3E4A4 X-Spam-Score: -3.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: VYduKoqy4Aoj Hi Marek, Marek Fel=C5=A1=C3=B6ci skribis: > Recently, I have come around this issue. I created a Singularity > container using the following `guix pack` command: > > `guix pack -f squashfs bash coreutils python python-numpy` > > Then, I run a bash shell within the container: > > `singularity exec test.gz.squashfs bash` > > Now, if I want to use numpy, > > `python3 -m numpy` > > it gives me the following error and the numpy module IS NOT found: > > Error in sitecustomize; set PYTHONVERBOSE for traceback: > ValueError: > '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnn= p8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'=20 > is not in list > /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No > module named numpy I can kinda reproduce via a relocatable pack (relocatable packs and Singularity packs have in common that they use relative symlinks in their profile, to allow for relocation): guix pack -RR python python-numpy -S /bin=3Dbin -S /etc=3Detc Then I unpack the thing: mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/=E2=80=A6-pack.tar.gz And from there I use the same trick as in =E2=80=98tests/guix-pack-relocatable.sh=E2=80=99 to =E2=80=9Chide=E2=80=9D = /gnu/store: --8<---------------cut here---------------start------------->8--- $ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc= /profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 ' /gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-pack= ages Python 3.9.6 (default, Jan 1 1970, 00:00:01)=20 [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'numpy' >>> import os >>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/pyt= hon3.9/site-packages") ['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hac= k', 'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt',= 'pip', 'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache= __'] >>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/pyt= hon3.9/site-packages/numpy") ['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.= py', 'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py', '= __config__.py', 'core', '_distributor_init.py', 'distutils', 'tests', 'emat= h.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd', 'conftest.py',= '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing', 'LICENSE.t= xt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py', 'setup.p= y', '__pycache__'] --8<---------------cut here---------------end--------------->8--- So in this case =E2=80=98GUIX_PYTHONPATH=E2=80=99 is correctly set, but it = looks as though =E2=80=98sitecustomize.py=E2=80=99 wasn=E2=80=99t loaded. (I don=E2= =80=99t get the =E2=80=9CError in sitecustomize=E2=80=9D message that Marek sees though, weird!) >From Marek=E2=80=99s message, it would seem that somewhere file names are constructed by appending =E2=80=9C..=E2=80=9D. Given Unix=E2=80=99s weird = dot-dot semantics, relative symlinks could be interfering: --8<---------------cut here---------------start------------->8--- $ ls -l /tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/py= thon3.9/site-packages/ totalo 48 lrwxrwxrwx 1 ludo users 102 Jan 1 1970 _distutils_hack -> ../../../../nid= dlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_di= stutils_hack lrwxrwxrwx 1 ludo users 111 Jan 1 1970 distutils-precedence.pth -> ../../= ../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-pac= kages/distutils-precedence.pth lrwxrwxrwx 1 ludo users 99 Jan 1 1970 numpy -> ../../../../mlccgh05bf8cd= inq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy lrwxrwxrwx 1 ludo users 121 Jan 1 1970 numpy-1.20.3-py3.9.egg-info -> ../= ../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.= 9/site-packages/numpy-1.20.3-py3.9.egg-info lrwxrwxrwx 1 ludo users 90 Jan 1 1970 pip -> ../../../../niddlj0qbk5vzk1= mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip lrwxrwxrwx 1 ludo users 107 Jan 1 1970 pip-21.1.3.dist-info -> ../../../.= ./niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-package= s/pip-21.1.3.dist-info lrwxrwxrwx 1 ludo users 100 Jan 1 1970 pkg_resources -> ../../../../niddl= j0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_r= esources lrwxrwxrwx 1 ludo users 98 Jan 1 1970 __pycache__ -> ../../../../niddlj0= qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycac= he__ lrwxrwxrwx 1 ludo users 97 Jan 1 1970 README.txt -> ../../../../niddlj0q= bk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.t= xt lrwxrwxrwx 1 ludo users 97 Jan 1 1970 setuptools -> ../../../../niddlj0q= bk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptoo= ls lrwxrwxrwx 1 ludo users 114 Jan 1 1970 setuptools-56.0.0.dist-info -> ../= ../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-= packages/setuptools-56.0.0.dist-info lrwxrwxrwx 1 ludo users 103 Jan 1 1970 sitecustomize.py -> ../../../../ni= ddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/si= tecustomize.py --8<---------------cut here---------------end--------------->8--- Now look below how dot-dot resolution whereas =E2=80=9Clexical=E2=80=9D dot= -dot resolution dose the right thing: --8<---------------cut here---------------start------------->8--- $ ls -l /tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/..= /../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packag= es ls: ne eblas atingi '/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-p= rofile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.= 9/site-packages': Dosiero a=C5=AD dosierujo ne ekzistas $ ls -l /tmp/pack/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/l= ib/python3.9/site-packages totalo 40 dr-xr-xr-x 3 ludo users 4096 Jan 1 1970 _distutils_hack/ -r--r--r-- 1 ludo users 152 Jan 1 1970 distutils-precedence.pth dr-xr-xr-x 5 ludo users 4096 Jan 1 1970 pip/ dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 pip-21.1.3.dist-info/ dr-xr-xr-x 6 ludo users 4096 Jan 1 1970 pkg_resources/ dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 __pycache__/ -r--r--r-- 1 ludo users 119 Jan 1 1970 README.txt dr-xr-xr-x 7 ludo users 4096 Jan 1 1970 setuptools/ dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 setuptools-56.0.0.dist-info/ -r--r--r-- 1 ludo users 2172 Jan 1 1970 sitecustomize.py --8<---------------cut here---------------end--------------->8--- In Python=E2=80=99s =E2=80=98site.py=E2=80=99, the =E2=80=98addsitedir=E2= =80=99 method indirectly calls =E2=80=98os.path.abspath=E2=80=99, not sure if that is at fault. I also see that =E2=80=98sys.path=E2=80=99 is in fact completely bogus, wit= h =E2=80=9Cpack=E2=80=9D missing from every directory name: --8<---------------cut here---------------start------------->8--- $ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/et= c/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 ' /gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-pack= ages Python 3.9.6 (default, Jan 1 1970, 00:00:01)=20 [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python39.zip'= , '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9', '/tm= p/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/lib-dynload'= , '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-p= ackages'] --8<---------------cut here---------------end--------------->8--- So I think something somewhere is manipulating file names and gets confused. Does that ring a bell, Maxim? Thanks, Ludo=E2=80=99.