PYTHONPATH issue analysis - part 2 (was: PYTHONPATH woes)

[Hartmut Goebel <h.goebel@crazy-compilers.com>]

Hi,

nex part of the analysis:

Result
=======

The venv-hack I posted a few days ago works as expected only for
GUIX_PROFILE,
but not for virtual environments.


Preliminary Proposal
=======================

As it stands now, the venv-hack is not a valid solution. It may be the basis
for another solution, tough.


Rational
===========

I tried answering four questions:

A: Can virtual environments be stacked in stock Python?

   No, they can not, see point 1. and 2. below.

C: Given PYTHONPATH is not set, do virtual environments in Guix use
   the correct "site" packages (which is the ones in GUIX_PROFILE).

   No, in guix venvs use the site-packages from
   /gnu/store/…-python-3.6.3/. See points 3. and 4 below.

D: Would the venv-hack I posted a view days ago solve the issue?

   No, it would not. It would work as expected for python in the
   profile, but not for virtual environments based on this. See point
   5. below.


1. Set up a virtual environemnt using the system installed python.
====================================================================

$ pyvenv-3.5 /tmp/venv-1
$ ls -l /tmp/venv-1/bin/python
… /tmp/venv-1/bin/python -> python3.5
$ ls /tmp/venv-1/bin/
activate      activate.fish  easy_install-3.5*  pip3*    python@  
python3.5@
activate.csh  easy_install*  pip*               pip3.5*  python3@
$ ls -l /tmp/venv-1/bin/python*
… /tmp/venv-1/bin/python -> python3.5
… /tmp/venv-1/bin/python3 -> python3.5
… /tmp/venv-1/bin/python3.5 -> /usr/bin/python3.5
$ /tmp/venv-1/bin/python -m site
sys.path = [
    '/home/hartmut',
    '/usr/lib64/python35.zip',
    '/usr/lib64/python3.5',
    '/usr/lib64/python3.5/plat-linux',
    '/usr/lib64/python3.5/lib-dynload',
    '/tmp/venv-1/lib64/python3.5/site-packages',
    '/tmp/venv-1/lib/python3.5/site-packages',
]

As expected there are only the venvs' site-packges in sys.path.


2. Now stack venv on top of venv-1. Use --system-site-packages to
(hopefully) make venv-1's site-packages available to venv-2.
====================================================================


$ which pyvenv-3.5
/bin/pyvenv-3.5
$ /tmp/venv-1/bin/python /bin/pyvenv-3.5 /tmp/venv-2 --system-site-packages
$ ls -l /tmp/venv-2/bin/python*
… /tmp/venv-2/bin/python -> /tmp/venv-1/bin/python
… /tmp/venv-2/bin/python3 -> python
$ ls /tmp/venv-2/bin/
activate      activate.fish  easy_install-3.5*  pip3*    python@
activate.csh  easy_install*  pip*               pip3.5*  python3@
$ /tmp/venv-2/bin/python -m site
sys.path = [
    '/tmp',
    '/usr/lib64/python35.zip',
    '/usr/lib64/python3.5',
    '/usr/lib64/python3.5/plat-linux',
    '/usr/lib64/python3.5/lib-dynload',
    '/tmp/venv-2/lib64/python3.5/site-packages',
    '/tmp/venv-2/lib/python3.5/site-packages',
    '/usr/lib64/python3.5/site-packages',
    '/usr/lib/python3.5/site-packages',
]

As you can see (last two entries), the system site-packages are taken
from the real system installation, not from the stacked venv-1. This
means, venvs can not be stacked.


3. Now let's see how guix-profile installed python works. I used a
somewhat current HEAD (7e4e3df4e8) to ensure using the most current
wrappers etc.
====================================================================


$ ./pre-inst-env guix package -i python
…


3a. Do not set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3a
…
$ /tmp/venv-3a/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-3a/lib/python3.6/site-packages',
]

As expected there are only the venvs' site-packges in sys.path.


3b. Set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ PYTHONPATH="$HOME/.guix-profile/lib/python3.6/site-packages"
~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3b
…
$ /tmp/venv-3b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-3b/lib/python3.6/site-packages',
]

Again there are only the venvs' site-packges in sys.path. This is
excpected, since PYTHONPATH only effects the run of pyenv-3.6


4. Same as 3, but use --system-site-packages
====================================================================

4a Don't set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4a --system-site-package
…
$ /tmp/venv-4a/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-4a/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

This is *not* what what a Guix user would expect. For the Guix user's
perspective his/her "Python site-packages" are those in $GUIX_PROFILE.
Esp. since guix never installs into
/gnu/store/…-python-3.6.3/lib/python3.6/site-packages and thus this
path never contains additional "site" packages.


4b Set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ PYTHONPATH="$HOME/.guix-profile/lib/python3.6/site-packages"
~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4b --system
$ /tmp/venv-4b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-4b/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

Result is the same as for 4a), reason as in 3b).


5. Would the venv-hack I posted a view days ago solve the issue?
====================================================================

5a. Verify the venv-hack works
----------------------------------------------------


$ cp -r ~/.guix-profile /tmp/guix-profile
$ mkdir !$
mkdir /tmp/guix-profile
$ cp -r ~/.guix-profile/* /tmp/guix-profile
$ echo 'include-system-site-packages = false' > /tmp/guix-profile/pyvenv.cfg
$ /tmp/guix-profile/bin/python3 -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/guix-profile/lib/python3.6/site-packages',
]

As expected, the profile's site-packages are included in sys.path.


5b Build a venv based on this (hacked) profile
----------------------------------------------------

$ /tmp/guix-profile/bin/pyvenv-3.6 /tmp/venv-5b --system-site-packages
…
$ /tmp/venv-5b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-5b/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

As in 4a, this is *not* what what a Guix user would expect. The
profile's site-packages should be in sys.path, not
/gnu/store/…-python-3.6.3/lib/python3.6/site-packages.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |