Re: PYTHONPATH woes

[Hartmut Goebel <h.goebel@crazy-compilers.com>]

[-- Attachment #1: Type: text/plain, Size: 3150 bytes --]

Hi,

thanks for raising the PYTHONPATH issue. Looks like we have a major
problem here.

Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus:
> I don’t know why this happens.  I find it puzzling that in this
> particular case the user’s profile contains an *older* version of
> statsmodels (0.6.1).  The wrapper includes the correct version of
> statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

You are addressing three issues here:
1) Why is the older package imported, where the newer one is expected to
be first in path?
2) We are mixing Python2 and Python3 paths
3) Is the way we use PYTHONPATH in the wrapper the correct way?

re 1): This is extremely curious: Why is the older package imported,
where the newer one is expected to be first in path?

To analyze this in detail, please make a copy of the resp. wrapper
script and change the last command into:

exec -a "$0" python3 -m site

and run the changed wrapper. This should show all paths defined in the
wrapper first.

If this does not give any insight, change it into

exec -a "$0" python3 -v -c "import statsmodels"

and then

exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"

and try to find some insight there.


re 2): As soon as Python2 and Python3 are installed in the same profile,
we put both site-packages for both versions into PYTHONPATH, which
obviously is wrong.

$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ echo $PYTHONPATH
/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:

A simple work-around would be to make the profile a (pseudo) virtual
environment, which is a easy as creating a file
"/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
sys.path - for the current python version only!

Try it:

sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg
$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ PYTHONPATH=/tmp/foo:/tmp/bar python3 -m site
[…]
    '/gnu/store/…-profile/lib/python3.5/site-packages',


re 3) When running

PYTHONPATH=/tmp/foo:/tmp/bar python -m site

on e.g. Debian, one can see that the order in sys.path is as follows:

- $PWD
- $PYTHONPATH elements
- built-in paths (e.g. /usr/lib64/python3.5)
- site-packages

The idea seems to be that PYTHONPATH can overwrite all packages, but
site-package can not. This can be seen as if in the wrapper-scripts we
are not using PYTHONPATH as indented: The user can not overwrite
site-packages. We ought to think if this is what we want, as this is how
Python works.

Depending on the result of the analysis for (1) and if we implement (2),
we can investigate how to solve (3). One idea I already looked at this
evening is to replace the wrappers by a minimal virtualenv.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 4393 bytes --]