From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Re: PYTHONPATH woes Date: Wed, 21 Feb 2018 22:58:57 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="------------82B3ECBE8C7DDAE99146B457" Return-path: Received: from eggs.gnu.org ([208.118.235.92]:42645) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eocQY-0006eJ-Tw for guix-devel@gnu.org; Wed, 21 Feb 2018 17:00:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eocPn-0004LW-Ju for guix-devel@gnu.org; Wed, 21 Feb 2018 16:59:50 -0500 Received: from mail-out.m-online.net ([212.18.0.9]:60470) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eocPn-0004Hr-3F for guix-devel@gnu.org; Wed, 21 Feb 2018 16:59:03 -0500 In-Reply-To: Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org, Ricardo Wurmus , Pjotr Prins This is a multi-part message in MIME format. --------------82B3ECBE8C7DDAE99146B457 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, thanks for raising the PYTHONPATH issue. Looks like we have a major problem here. Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus: > I don=E2=80=99t know why this happens. I find it puzzling that in this > particular case the user=E2=80=99s profile contains an *older* version = of > statsmodels (0.6.1). The wrapper includes the correct version of > statsmodels (0.8.0) in the PYTHONPATH. Here=E2=80=99s the backtrace: You are addressing three issues here: 1) Why is the older package imported, where the newer one is expected to be first in path? 2) We are mixing Python2 and Python3 paths 3) Is the way we use PYTHONPATH in the wrapper the correct way? re 1): This is extremely curious: Why is the older package imported, where the newer one is expected to be first in path? To analyze this in detail, please make a copy of the resp. wrapper script and change the last command into: exec -a "$0" python3 -m site and run the changed wrapper. This should show all paths defined in the wrapper first. If this does not give any insight, change it into exec -a "$0" python3 -v -c "import statsmodels" and then exec -a "$0" python3 -v -c "import ribodiff.estimatedisp" and try to find some insight there. re 2): As soon as Python2 and Python3 are installed in the same profile, we put both site-packages for both versions into PYTHONPATH, which obviously is wrong. $ PYTHONPATH=3D guix environment --ad-hoc python@2 python [=E2=80=A6] [guix] $ echo $PYTHONPATH /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-pa= ckages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/= site-packages: A simple work-around would be to make the profile a (pseudo) virtual environment, which is a easy as creating a file "/gnu/store/=E2=80=A6-profile/pyvenv.cfg". This will trigger a mechanism = in site.py to insert /gnu/store/=E2=80=A6-profile/lib/pythonX.Y/site-package= s" into sys.path - for the current python version only! Try it: sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg $ PYTHONPATH=3D guix environment --ad-hoc python@2 python [=E2=80=A6] [guix] $ PYTHONPATH=3D/tmp/foo:/tmp/bar python3 -m site [=E2=80=A6] =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-profile/lib/python3.5/site-packa= ges', re 3) When running PYTHONPATH=3D/tmp/foo:/tmp/bar python -m site on e.g. Debian, one can see that the order in sys.path is as follows: - $PWD - $PYTHONPATH elements - built-in paths (e.g. /usr/lib64/python3.5) - site-packages The idea seems to be that PYTHONPATH can overwrite all packages, but site-package can not. This can be seen as if in the wrapper-scripts we are not using PYTHONPATH as indented: The user can not overwrite site-packages. We ought to think if this is what we want, as this is how Python works. Depending on the result of the analysis for (1) and if we implement (2), we can investigate how to solve (3). One idea I already looked at this evening is to replace the wrappers by a minimal virtualenv. --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------82B3ECBE8C7DDAE99146B457 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi,

thanks for raising the PYTHONPATH issue. Looks like we have a major problem here.

Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus:
I don=E2=80=99t know why this happens.  I find it pu=
zzling that in this
particular case the user=E2=80=99s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here=E2=80=99s the backtrace:

You are addressing three issues here:
1) Why is the older package imported, where the newer one is expected to be first in path?
2) We are mixing Python2 and Python3 paths
3) Is the way we use PYTHONPATH in the wrapper the correct way?

re 1): This is extremely curious: Why is the older package imported, where the newer one is expected to be first in path?

To analyze this in detail, please make a copy of the resp. wrapper script and change the last command into:

exec -a "$0" python3 -m site

and run the changed wrapper. This should show all paths defined in the wrapper first.

If this does not give any insight, change it into

exec -a "$0" python3 -v -c "import statsmodels"

and then

exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"

and try to find some insight there.


re 2): As soon as Python2 and Python3 are installed in the same profile, we put both site-packages for both versions into PYTHONPATH, which obviously is wrong.

$ PYTHONPATH=3D guix environment --ad-hoc python@2 python
[=E2=80=A6]
[guix] $ echo $PYTHONPATH
/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-pa= ckages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/= site-packages:

A simple work-around would be to make the profile a (pseudo) virtual environment, which is a easy as creating a file "/gnu/store/=E2=80=A6-profile/pyvenv.cfg". This will trigger a mech= anism in site.py to insert /gnu/store/=E2=80=A6-profile/lib/pythonX.Y/site-packages" into sys.= path - for the current python version only!

Try it:

sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg
$ PYTHONPATH=3D guix environment --ad-hoc python@2 python
[=E2=80=A6]
[guix] $ PYTHONPATH=3D/tmp/foo:/tmp/bar python3 -m site
[=E2=80=A6]
=C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-profile/lib/python3.5/site= -packages',


re 3) When running

PYTHONPATH=3D/tmp/foo:/tmp/bar python -m site

on e.g. Debian, one can see that the order in sys.path is as follows:

- $PWD
- $PYTHONPATH elements
- built-in paths (e.g. /usr/lib64/python3.5)
- site-packages

The idea seems to be that PYTHONPATH can overwrite all packages, but site-package can not. This can be seen as if in the wrapper-scripts we are not using PYTHONPATH as indented: The user can not overwrite site-packages. We ought to think if this is what we want, as this is how Python works.

Depending on the result of the analysis for (1) and if we implement (2), we can investigate how to solve (3). One idea I already looked at this evening is to replace the wrappers by a minimal virtualenv.

--=20
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com   =
            |
| www.crazy-compilers.com | compilers which you thought are impo=
ssible |
--------------82B3ECBE8C7DDAE99146B457--