Re: PYTHONPATH issue explanation

[Hartmut Goebel <h.goebel@crazy-compilers.com>]

Hi,

given the ongoing discussion around Python show that my explanation was
not good enough. I'll try to summarize and give more background.

With regard to Python, guix currently has a major issue, which my
proposals are addressing. There are other issues (like naming the
executables, the "wrapper" script", etc.) which are not addressed here.

When installing Python and some Python packages (e.g. python-simplejson)
in guix, the python interpreter will be linked to
GUIX_PROFILE/bin/pythonX.Y and the packages' files are linked into
GUIX_PROFILE/lib/python-X.Y/site-packages/…, which is perfectly okay.

This python interpreter does not find the site-packages in GUIX_PROFILE
since site-packages are search relative to "sys.base_prefix" (which is
the same as "sys.prefix" except in virtual environments).
"sys.base_prefix" is determined based on the executable's path (argv[0])
by resolving all symlinks.

The python interpreter assumes "site-packages" to be relative to "where
python is installed" - called "sys.base_prefix" (which is the same as
"sys.prefix" except in virtual environments). "sys.base_prefix" is
determined based on the executable's path (argv[0]) by resolving all
symlinks. For Guix this means: "sys.base_prefix" will always point to
/gnu/store/…-python-X.Y, not to GUIX_PROFILE. Thus the site-packages
installed into the guix profile will not be found.

This is why we currently (mis-) use PYTHONPATH: To make the
site-packages installed into the guix profile available.

Using PYTHONPATH for this woes since there is only one PYTHONPATH
variable for all versions of python. This is designed by upstream.

Additionally: When using PYTHONPATH the site-packages are added to the
search path ("sys.path") *in front* of the python standard library,
while they are expected to be added *behind*.

Part 3 of my analysis lists three solutions for this, where only number
2 and 3 are "good choices".

no. 2
suggests using a mechanism already implemented in python: Setting
"PYTHONHOME" will make the interpreter to use this as "sys.base_prefix"
unconditionally. Again there is only one PYTHONHOME variable for all
versions of python (designed by upstream). We could work around this
easily (while keeping upstream compatibility) by using
GUIX-PYTHONHOME-X.Y, to be evaluated just after PYTHONHOME.

This would be easy to implement using Guix's "search-path" capabilities
and a small patch to the python interpreter.

The drawback is: This is implemented using an environment variable,
which might not give the expected results in all cases. E.g. running
/gnu/store/…-profile/bin/python will not load the site-packages of that
profile. Also there might be issues implementing virtual environments.
(Thinking about this, I'm quite sure there will. Ouch!)

no.3
suggests changing the way the python interpreter is resolving symlinks
when searching for "sys.base_prefix". The idea is to stop "at the profile".

The hard part of this is to determine "at the profile". Also this needs
a larger patch. But if we manage to implement this, it would be perfect.
I could contribute a draft for this implemented in Python. The
C-implementation needs to be done by some C programmer.

Which way should we go?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |