From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1) Date: Sat, 14 Jan 2017 22:02:20 +0100 Message-ID: <6bfab194-5796-733f-75dd-f0214e3410f8@crazy-compilers.com> References: <8737gw2fqw.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:34679) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cSVT1-0000eV-IE for guix-devel@gnu.org; Sat, 14 Jan 2017 16:02:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cSVSy-0007Rf-AR for guix-devel@gnu.org; Sat, 14 Jan 2017 16:02:27 -0500 In-Reply-To: <8737gw2fqw.fsf@gmail.com> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Guix-devel Cc: Maxim Cournoyer Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer: > One thing which I discovered while testing pip on Guix was that > depending on how you use Python on Guix the PYTHONPATH differs: I analysed this problem and found set we have another serious problem here. But good news: There is an easy solution for both problems. Result of the analysis (as explained below): We must not extend PYTHONPATH for adding the profile's site-packages directory. Proposed solution: The standard library module "site" already contains a place where we can hook in: # Prefixes for site-packages; add additional prefixes like /usr/local here PREFIXES =3D [sys.prefix, sys.exec_prefix] So we simply prepend $GUIX_ENVIRONMENT to this list: PREFIXES =3D [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix, sys.exec_prefix] Prepending to speed up searches, since within an environment almost all site-packages will end in GUIX_ENVIRONMENT. There is no need to check if $GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged and the non-existing path will be skipped out automatically This would solve both the problem Maxim described and the problem described below, since the Guix site-packages behave exactly like any other global site-packages directory. When following this proposal, we *may* even remove some "wrapping" stuff in the python-build-system. But this has to be investigated first. Analysis (proof see below): The python interpreter was two command line options, which are rarely used, though: -E ignores environment variables, esp. PYTHONPATH -S Disable the import of the module site and the site-dependent manipulations of sys.path. This means: a) When passing -E, the Guix environment's site-packages are ignored (together with PYTHONPATH), while they should not. b) When passing -S, the Guix environment's site-packages should be ignored, but is not (since it is specified in $PYTHONPATH, which is not ignored) Conclusion: We should must not use PYTHONPATH to specify the Guix's environment's site-package directory. Analysis details: (guix)$ which python3 /gnu/store/zcnb=E2=80=A6-profile/bin/python3 Without any options: -> /home/USER/.local/lib/python3.5/site-packages and /gnu/store/zcnb=E2=80=A6-profile/lib/python3.5/site-packages should be in sys.path (base)$ python3 -c 'import sys ; print("\n".join(sys.path))' /usr/lib64/python34.zip /usr/lib64/python3.4 /usr/lib64/python3.4/plat-linux /usr/lib64/python3.4/lib-dynload /home/USER/.local/lib/python3.4/site-packages /usr/lib64/python3.4/site-packages /usr/lib/python3.4/site-packages (guix)$ python3 -c 'import sys ; print("\n".join(sys.path))' /gnu/store/zcnb=E2=80=A6-profile/lib/python3.5/site-packages /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python35.zip /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5 /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/plat-linux /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/lib-dynload /home/USER/.local/lib/python3.5/site-packages /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/site-packages -E Ignore environment variables like PYTHONPATH and PYTHONHOME that modify the behavior of the interpreter. -> /home/USER/.local/lib/python3.5/site-packages and /gnu/store/zcnb=E2=80=A6-profile/lib/python3.5/site-packages should be in sys.path (base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))' /usr/lib64/python34.zip /usr/lib64/python3.4 /usr/lib64/python3.4/plat-linux /usr/lib64/python3.4/lib-dynload /home/USER/.local/lib/python3.4/site-packages /usr/lib64/python3.4/site-packages /usr/lib/python3.4/site-packages (guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))' /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python35.zip /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5 /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/plat-linux /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/lib-dynload /home/USER/.local/lib/python3.5/site-packages /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/site-packages -S =3D Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. -> /home/USER/.local/lib/python3.5/site-packages and /gnu/store/zcnb=E2=80=A6-profile/lib/python3.5/site-packages sould *not* be in sys.path (base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))' /usr/lib64/python34.zip /usr/lib64/python3.4/ /usr/lib64/python3.4/plat-linux /usr/lib64/python3.4/lib-dynload (guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))' /gnu/store/zcnb=E2=80=A6-profile/lib/python3.5/site-packages /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python35.zip /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/ /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/plat-linux /gnu/store/alk9=E2=80=A6-python-3.5.2/lib/python3.5/lib-dynload --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |