From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: PYTHONPATH issue analysis - part 2 (was: PYTHONPATH woes) Date: Tue, 27 Feb 2018 12:49:46 +0100 Message-ID: <50179a9a-4e29-e729-ee49-ccbb90d9eda9@crazy-compilers.com> References: <87371tqbyb.fsf@elephly.net> <20180223165953.GA6088@thebird.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42420) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eqdla-00062c-G0 for guix-devel@gnu.org; Tue, 27 Feb 2018 06:49:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eqdlX-000579-B4 for guix-devel@gnu.org; Tue, 27 Feb 2018 06:49:54 -0500 Received: from mail-out.m-online.net ([2001:a60:0:28:0:1:25:1]:45887) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eqdlX-00056v-0f for guix-devel@gnu.org; Tue, 27 Feb 2018 06:49:51 -0500 In-Reply-To: Content-Language: de-DE List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: guix-devel@gnu.org Hi, nex part of the analysis: Result =3D=3D=3D=3D=3D=3D=3D The venv-hack I posted a few days ago works as expected only for GUIX_PROFILE, but not for virtual environments. Preliminary Proposal =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D As it stands now, the venv-hack is not a valid solution. It may be the ba= sis for another solution, tough. Rational =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D I tried answering four questions: A: Can virtual environments be stacked in stock Python? =C2=A0=C2=A0 No, they can not, see point 1. and 2. below. C: Given PYTHONPATH is not set, do virtual environments in Guix use =C2=A0=C2=A0 the correct "site" packages (which is the ones in GUIX_PROFI= LE). =C2=A0=C2=A0 No, in guix venvs use the site-packages from =C2=A0=C2=A0 /gnu/store/=E2=80=A6-python-3.6.3/. See points 3. and 4 belo= w. D: Would the venv-hack I posted a view days ago solve the issue? =C2=A0=C2=A0 No, it would not. It would work as expected for python in th= e =C2=A0=C2=A0 profile, but not for virtual environments based on this. See= point =C2=A0=C2=A0 5. below. 1. Set up a virtual environemnt using the system installed python. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D $ pyvenv-3.5 /tmp/venv-1 $ ls -l /tmp/venv-1/bin/python =E2=80=A6 /tmp/venv-1/bin/python -> python3.5 $ ls /tmp/venv-1/bin/ activate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 activate.fish=C2=A0 easy_install-3= .5*=C2=A0 pip3*=C2=A0=C2=A0=C2=A0 python@=C2=A0=C2=A0 python3.5@ activate.csh=C2=A0 easy_install*=C2=A0 pip*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pip3.5*=C2=A0 pyth= on3@ $ ls -l /tmp/venv-1/bin/python* =E2=80=A6 /tmp/venv-1/bin/python -> python3.5 =E2=80=A6 /tmp/venv-1/bin/python3 -> python3.5 =E2=80=A6 /tmp/venv-1/bin/python3.5 -> /usr/bin/python3.5 $ /tmp/venv-1/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/home/hartmut', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python35.zip', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5/plat-linux', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5/lib-dynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-1/lib64/python3.5/site-packages', =C2=A0=C2=A0=C2=A0 '/tmp/venv-1/lib/python3.5/site-packages', ] As expected there are only the venvs' site-packges in sys.path. 2. Now stack venv on top of venv-1. Use --system-site-packages to (hopefully) make venv-1's site-packages available to venv-2. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D $ which pyvenv-3.5 /bin/pyvenv-3.5 $ /tmp/venv-1/bin/python /bin/pyvenv-3.5 /tmp/venv-2 --system-site-packag= es $ ls -l /tmp/venv-2/bin/python* =E2=80=A6 /tmp/venv-2/bin/python -> /tmp/venv-1/bin/python =E2=80=A6 /tmp/venv-2/bin/python3 -> python $ ls /tmp/venv-2/bin/ activate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 activate.fish=C2=A0 easy_install-3= .5*=C2=A0 pip3*=C2=A0=C2=A0=C2=A0 python@ activate.csh=C2=A0 easy_install*=C2=A0 pip*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pip3.5*=C2=A0 pyth= on3@ $ /tmp/venv-2/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python35.zip', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5/plat-linux', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5/lib-dynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-2/lib64/python3.5/site-packages', =C2=A0=C2=A0=C2=A0 '/tmp/venv-2/lib/python3.5/site-packages', =C2=A0=C2=A0=C2=A0 '/usr/lib64/python3.5/site-packages', =C2=A0=C2=A0=C2=A0 '/usr/lib/python3.5/site-packages', ] As you can see (last two entries), the system site-packages are taken from the real system installation, not from the stacked venv-1. This means, venvs can not be stacked. 3. Now let's see how guix-profile installed python works. I used a somewhat current HEAD (7e4e3df4e8) to ensure using the most current wrappers etc. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D $ ./pre-inst-env guix package -i python =E2=80=A6 3a. Do not set PYTHONPATH when setting up the venv. ---------------------------------------------------- $ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3a =E2=80=A6 $ /tmp/venv-3a/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-3a/lib/python3.6/site-packages', ] As expected there are only the venvs' site-packges in sys.path. 3b. Set PYTHONPATH when setting up the venv. ---------------------------------------------------- $ PYTHONPATH=3D"$HOME/.guix-profile/lib/python3.6/site-packages" ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3b =E2=80=A6 $ /tmp/venv-3b/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-3b/lib/python3.6/site-packages', ] Again there are only the venvs' site-packges in sys.path. This is excpected, since PYTHONPATH only effects the run of pyenv-3.6 4. Same as 3, but use --system-site-packages =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 4a Don't set PYTHONPATH when setting up the venv. ---------------------------------------------------- $ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4a --system-site-package =E2=80=A6 $ /tmp/venv-4a/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-4a/lib/python3.6/site-packages', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/site-= packages', ] This is *not* what what a Guix user would expect. For the Guix user's perspective his/her "Python site-packages" are those in $GUIX_PROFILE. Esp. since guix never installs into /gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/site-packages and thus th= is path never contains additional "site" packages. 4b Set PYTHONPATH when setting up the venv. ---------------------------------------------------- $ PYTHONPATH=3D"$HOME/.guix-profile/lib/python3.6/site-packages" ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4b --system $ /tmp/venv-4b/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-4b/lib/python3.6/site-packages', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/site-= packages', ] Result is the same as for 4a), reason as in 3b). 5. Would the venv-hack I posted a view days ago solve the issue? =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 5a. Verify the venv-hack works ---------------------------------------------------- $ cp -r ~/.guix-profile /tmp/guix-profile $ mkdir !$ mkdir /tmp/guix-profile $ cp -r ~/.guix-profile/* /tmp/guix-profile $ echo 'include-system-site-packages =3D false' > /tmp/guix-profile/pyven= v.cfg $ /tmp/guix-profile/bin/python3 -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/guix-profile/lib/python3.6/site-packages', ] As expected, the profile's site-packages are included in sys.path. 5b Build a venv based on this (hacked) profile ---------------------------------------------------- $ /tmp/guix-profile/bin/pyvenv-3.6 /tmp/venv-5b --system-site-packages =E2=80=A6 $ /tmp/venv-5b/bin/python -m site sys.path =3D [ =C2=A0=C2=A0=C2=A0 '/tmp', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python36.zip', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/lib-d= ynload', =C2=A0=C2=A0=C2=A0 '/tmp/venv-5b/lib/python3.6/site-packages', =C2=A0=C2=A0=C2=A0 '/gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/site-= packages', ] As in 4a, this is *not* what what a Guix user would expect. The profile's site-packages should be in sys.path, not /gnu/store/=E2=80=A6-python-3.6.3/lib/python3.6/site-packages. --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |