PYTHONPATH woes

PYTHONPATH woes

[Ricardo Wurmus]

Hi Guix,

we have a couple of packages that provide scripts that depend on Python
modules.  We wrap them in PYTHONPATH to ensure that the correct Python
modules are found at runtime.

This is not enough.

We don’t wrap them tightly enough; instead we allow for a user-provided
PYTHONPATH value to be added to the PYTHONPATH we specified.  The result
is that a user-set PYTHONPATH can act like LD_LIBRARY_PATH — it causes
chaos.  This is despite the fact that we make sure that the wrapper’s
PYTHONPATH comes first!

Suppose a user installs python@2 and python2-statsmodels; at a later
point the user upgrades Guix, and then installs the ribodiff package.
The user does not know that ribodiff is written in Python, nor should
the user be aware of that.

Because Python is installed in the profile, etc/profile will contain a
definition for PYTHONPATH.  The user may source that etc/profile file to
set up all required environment variables.  But now running the ribodiff
scripts fails!

Here’s what happens: the PYTHONPATH that Guix sets for the profile now
contains an incompatible variant of the python2-statsmodels package.
Guix has been upgraded between installing python2-statsmodels and
ribodiff, so a different version of Python was used to build these
modules.  Since the ribodiff wrapper script gladly accepts any set
PYTHONPATH, it causes the ribodiff scripts to load the old and
incompatible python2-statsmodels package instead of the compatible one
from the wrapper.

I don’t know why this happens.  I find it puzzling that in this
particular case the user’s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 81, in <module>
    main()
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 26, in main
    import ribodiff.estimatedisp as ed
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/estimatedisp.py", line 7, in <module>
    import rawdisp as rd
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/rawdisp.py", line 8, in <module>
    import statsmodels.api as sm
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py", line 8, in <module>
    from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/__init__.py", line 1, in <module>
    from .tools import add_constant, categorical
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/tools.py", line 11, in <module>
    from statsmodels.datasets import webuse
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/__init__.py", line 5, in <module>
    from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/__init__.py", line 1, in <module>
    from .data import *
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/data.py", line 90, in <module>
    from statsmodels.datasets import utils as du
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/utils.py", line 13, in <module>
    from pandas import read_csv, DataFrame, Index
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 31, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/hashtable.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
--8<---------------cut here---------------end--------------->8---

Now you could say that this is the user’s fault for not using manifests.
But consider this: what happens if the user had a manifest and installed
“python-statsmodels” instead of the Python 2 variant?  Guix would still
set PYTHONPATH and the ribodiff wrapper would still prefer the profile’s
PYTHONPATH over the wrapped value, so it would cause Python 2 (from
ribodiff) to load a Python 3 module of statsmodels — these are not
compatible and again we have a runtime crash.

Manifests wouldn’t avoid this problem.

Avoiding this problem now requires that users know what language a tool
is implemented in (e.g. Python 2 for Ribodiff) and make a conscious
effort to install these tools in a separate profile containing no Python
3 modules.  This is not a reasonable burden to put on users.

What can we do to fix this?

Would it be good to make the wrappers for Python scripts stricter and
not accept any user-set PYTHONPATH?

How do we approach the problem of having both Python 2 modules and
Python 3 modules in the same profile?  PYTHONPATH will be set to refer
to the site-packages directories of both versions, which is never good.
Does Python offer us a way to do better?  Can we make use of pth files
to get around this problem somehow?

--
Ricardo

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
> Would it be good to make the wrappers for Python scripts stricter and
> not accept any user-set PYTHONPATH?

I think that is a bad idea. You need to be able to opt out. Also
people need to experiment with modules without understanding Guix per
se. In my upcomping blog I would emphasize packaging at the point you
become a serious user.

That should come with a health warning ;). Similarly we should allow
for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
dangerous.

> How do we approach the problem of having both Python 2 modules and
> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
> to the site-packages directories of both versions, which is never good.
> Does Python offer us a way to do better?  Can we make use of pth files
> to get around this problem somehow?

Python should have created PYTHONPATH2 to split them out. We could
patch python2 to do just that.

Even so, the real solution is separate profiles. I get that with
versions of Ruby too.

Pj.

Re: PYTHONPATH woes

[Andy Wingo]

On Tue 20 Feb 2018 16:01, Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out.

Why?  I am not sure this is the case for programs that just happen to be
written in Python.

> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
not quite the same I don't think.

Andy

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 04:18:40PM +0100, Andy Wingo wrote:
> In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
> not quite the same I don't think.

Not exactly the same, but close enough ;). But it is clearer now that
mixing is the problem. Ricardos .pth may be an option if that works,
or we introduce a separate GUIX_PYTHONPATH2 and GUIX_PYTHONPATH for 3.
Python2 is at end of life, so we may set an example there for others.

Even so, I don't think it will solve the particular conflict that
Ricardo was describing. There will always be hairy mix-ins.

Pj.
-- 

Re: PYTHONPATH woes

[Ricardo Wurmus]

Hi Pjotr,

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out. Also
> people need to experiment with modules without understanding Guix per
> se. In my upcomping blog I would emphasize packaging at the point you
> become a serious user.
>
> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

While I agree that it must be *possible* to opt out, I think the
defaults are wrong here.  We don’t ever set LD_LIBRARY_PATH when
building a profile, but we do set PYTHONPATH.  Having PYTHONPATH set
(even without the knowledge of the user) leads to problems here when it
shouldn’t.

I’d argue that the number of users who need to be able to override
individual Python packages for a tool like Ribodiff is way lower than
the number of those who accidentally get into a situation where
PYTHONPATH is set purely because of the contents of their profile,
leading to breakage of unrelated packages that just happen to use Python
modules.

>> How do we approach the problem of having both Python 2 modules and
>> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
>> to the site-packages directories of both versions, which is never good.
>> Does Python offer us a way to do better?  Can we make use of pth files
>> to get around this problem somehow?
>
> Python should have created PYTHONPATH2 to split them out. We could
> patch python2 to do just that.

I think there’s an alternative, but I don’t know it well.  The official
way is to use “.pth” files instead of setting PYTHONPATH.  Maybe there’s
a way that doesn’t involve setting PYTHONPATH.  Instead we could nudge
Python towards reading the profile’s “.pth” file and read the package
locations from there.

This would require a new profile hook and possibly a patch to Python to
add a way to tell it to read the “.pth” file from a location provided by
a Guix environment variable.

> Even so, the real solution is separate profiles. I get that with
> versions of Ruby too.

Right, but it is not obvious what packages must be treated with extra
care.  When I install the Ribodiff package I don’t know or care that
it’s written in Python.  That shouldn’t matter at all.  But now I
actually have to pay attention to this and install Ribodiff in a profile
that doesn’t contain Python 3 things.

That’s rather complicated and unfriendly for users.  I don’t want the
users here to be anxious about installing software, just because a new
tool may be using Python and thus might break when installed to a
profile containing Python things.  We made these wrappers precisely to
isolate the tools from the current environment.  Allowing them to be
disturbed so easily is making wrappers much less useful.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3150 bytes --]

Hi,

thanks for raising the PYTHONPATH issue. Looks like we have a major
problem here.

Am 20.02.2018 um 11:53 schrieb Ricardo Wurmus:
> I don’t know why this happens.  I find it puzzling that in this
> particular case the user’s profile contains an *older* version of
> statsmodels (0.6.1).  The wrapper includes the correct version of
> statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

You are addressing three issues here:
1) Why is the older package imported, where the newer one is expected to
be first in path?
2) We are mixing Python2 and Python3 paths
3) Is the way we use PYTHONPATH in the wrapper the correct way?

re 1): This is extremely curious: Why is the older package imported,
where the newer one is expected to be first in path?

To analyze this in detail, please make a copy of the resp. wrapper
script and change the last command into:

exec -a "$0" python3 -m site

and run the changed wrapper. This should show all paths defined in the
wrapper first.

If this does not give any insight, change it into

exec -a "$0" python3 -v -c "import statsmodels"

and then

exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"

and try to find some insight there.


re 2): As soon as Python2 and Python3 are installed in the same profile,
we put both site-packages for both versions into PYTHONPATH, which
obviously is wrong.

$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ echo $PYTHONPATH
/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:

A simple work-around would be to make the profile a (pseudo) virtual
environment, which is a easy as creating a file
"/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
sys.path - for the current python version only!

Try it:

sudo touch $GUIX_ENVIRONMENT/pyvenv.cfg
$ PYTHONPATH= guix environment --ad-hoc python@2 python
[…]
[guix] $ PYTHONPATH=/tmp/foo:/tmp/bar python3 -m site
[…]
    '/gnu/store/…-profile/lib/python3.5/site-packages',


re 3) When running

PYTHONPATH=/tmp/foo:/tmp/bar python -m site

on e.g. Debian, one can see that the order in sys.path is as follows:

- $PWD
- $PYTHONPATH elements
- built-in paths (e.g. /usr/lib64/python3.5)
- site-packages

The idea seems to be that PYTHONPATH can overwrite all packages, but
site-package can not. This can be seen as if in the wrapper-scripts we
are not using PYTHONPATH as indented: The user can not overwrite
site-packages. We ought to think if this is what we want, as this is how
Python works.

Depending on the result of the analysis for (1) and if we implement (2),
we can investigate how to solve (3). One idea I already looked at this
evening is to replace the wrappers by a minimal virtualenv.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 4393 bytes --]

Re: PYTHONPATH woes

[Ricardo Wurmus]

Hi Hartmut,

I was hoping for you to show up and give your input.  Thank you!  I
appreciate you taking the time.

> re 1): This is extremely curious: Why is the older package imported,
> where the newer one is expected to be first in path?
>
> To analyze this in detail, please make a copy of the resp. wrapper
> script and change the last command into:
>
> exec -a "$0" python3 -m site
>
> and run the changed wrapper. This should show all paths defined in the
> wrapper first.

Since this is Python 2 I used this line:

    exec -a "$0" /gnu/store/0n8ni2l…-python-2.7.13/bin/python -m site

The full wrapper looks like this:

--8<---------------cut here---------------start------------->8---
#!/gnu/store/f8k940vy9gck66m9r4id5m098w3hxgka-bash-minimal-4.4.12/bin/bash
export PYTHONPATH="/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages:/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages:/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages:/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages:/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages:/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages:/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages:/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages:/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages:/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages:/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages:/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages:/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages:/gnu/store/8jwxgbcym5i1accf88bq7i8fgzg2z68q-python2-pyparsing-2.2.0/lib/python2.7/site-packages:/gnu/store/ikx5his9njw7r7df06gkwlayhgl2780a-python2-cycler-0.10.0/lib/python2.7/site-packages:/gnu/store/jknf4w4s9w5w71ampcymgn4d14hfwx3h-python-2.7.13-tk/lib/python2.7/site-packages:/gnu/store/gdgy38ylfm4jaz4cmq6c6650i8iga21l-python2-subprocess32-3.2.7/lib/python2.7/site-packages:/gnu/store/pdlc7mwmm1vfrkgcpscps5kj0p1gwa14-python2-pygobject-2.28.6/lib/python2.7/site-packages:/gnu/store/s8sn8r41jyn35aginp63z2232sw9g23a-python2-functools32-3.2.3-2/lib/python2.7/site-packages:/gnu/store/bj4jlrnai5qlic1arjijyp47isvipca7-python2-pycairo-1.10.0/lib/python2.7/site-packages:/gnu/store/rhc5rpihy35ss7i9fvc0knqa798br9yj-python2-patsy-0.4.1/lib/python2.7/site-packages:/gnu/store/mrm8hn8rv04hm80sl63c1w33xs5gg147-python2-pandas-0.19.2/lib/python2.7/site-packages:/gnu/store/q977dgwlvmak15qn0w2kjk3q322mbrwi-python2-xcffib-0.5.1/lib/python2.7/site-packages:/gnu/store/gmdlgpm3jyfi608fyjmj6g7svpp5y8lc-python2-cffi-1.11.2/lib/python2.7/site-packages:/gnu/store/7fziyn0m8wjc98j0g056bms4yv94mi5q-python2-pycparser-2.17/lib/python2.7/site-packages${PYTHONPATH:+:}$PYTHONPATH"
#exec -a "$0" "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real" "$@"
exec -a "$0" /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/bin/python -m site
--8<---------------cut here---------------end--------------->8---

While PYTHONPATH is set this prints *nothing* at all.  Only after “unset
PYTHONPATH” I get this:

--8<---------------cut here---------------start------------->8---
sys.path = [
    '/home/uzinnal',
    '/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site-packages',
    '/gnu/store/95xlp80dp36m6nllaifndvc1vspnxwb2-python2-mock-1.0.1/lib/python2.7/site-packages',
    '/gnu/store/8dgqb88wabnqscri4brwhasar7i1an1h-python2-nose-1.3.7/lib/python2.7/site-packages',
    '/gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages',
    '/gnu/store/l1gdwsx4gr23gzy8zw6n588icdhhw8z4-python2-matplotlib-2.0.2/lib/python2.7/site-packages',
    '/gnu/store/prvq91x42x7wpxbjgx83y78n23jx3jd5-python2-scipy-0.19.1/lib/python2.7/site-packages',
    '/gnu/store/ng6srvhfmkz9i6g2ilbg6zksbh0v9yik-python2-statsmodels-0.8.0/lib/python2.7/site-packages',
    '/gnu/store/vls5bci2wk15sz080g6wgycj6fmm44w9-python2-cairocffi-0.8.0/lib/python2.7/site-packages',
    '/gnu/store/lfnzcj977hldqhkcpivag1hwqqqzk4gr-python2-six-1.10.0/lib/python2.7/site-packages',
    '/gnu/store/vcs6yjy3851zn350gc90ipw2if3rg2vj-python2-pytz-2017.3/lib/python2.7/site-packages',
    '/gnu/store/5w9r2c3dcz353n9rp56pcwvbks8s2hva-python2-pillow-3.3.3/lib/python2.7/site-packages',
    '/gnu/store/hl1ly62q6gsiwd287gqy354dbgrq5sxk-python2-dateutil-2.6.0/lib/python2.7/site-packages',
    '/gnu/store/8jwxgbcym5i1accf88bq7i8fgzg2z68q-python2-pyparsing-2.2.0/lib/python2.7/site-packages',
    '/gnu/store/ikx5his9njw7r7df06gkwlayhgl2780a-python2-cycler-0.10.0/lib/python2.7/site-packages',
    '/gnu/store/jknf4w4s9w5w71ampcymgn4d14hfwx3h-python-2.7.13-tk/lib/python2.7/site-packages',
    '/gnu/store/gdgy38ylfm4jaz4cmq6c6650i8iga21l-python2-subprocess32-3.2.7/lib/python2.7/site-packages',
    '/gnu/store/pdlc7mwmm1vfrkgcpscps5kj0p1gwa14-python2-pygobject-2.28.6/lib/python2.7/site-packages',
    '/gnu/store/s8sn8r41jyn35aginp63z2232sw9g23a-python2-functools32-3.2.3-2/lib/python2.7/site-packages',
    '/gnu/store/bj4jlrnai5qlic1arjijyp47isvipca7-python2-pycairo-1.10.0/lib/python2.7/site-packages',
    '/gnu/store/rhc5rpihy35ss7i9fvc0knqa798br9yj-python2-patsy-0.4.1/lib/python2.7/site-packages',
    '/gnu/store/mrm8hn8rv04hm80sl63c1w33xs5gg147-python2-pandas-0.19.2/lib/python2.7/site-packages',
    '/gnu/store/q977dgwlvmak15qn0w2kjk3q322mbrwi-python2-xcffib-0.5.1/lib/python2.7/site-packages',
    '/gnu/store/gmdlgpm3jyfi608fyjmj6g7svpp5y8lc-python2-cffi-1.11.2/lib/python2.7/site-packages',
    '/gnu/store/7fziyn0m8wjc98j0g056bms4yv94mi5q-python2-pycparser-2.17/lib/python2.7/site-packages',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python27.zip',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/plat-linux2',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/lib-tk',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/lib-old',
    '/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/lib-dynload',
    '/gnu/store/pdlc7mwmm1vfrkgcpscps5kj0p1gwa14-python2-pygobject-2.28.6/lib/python2.7/site-packages/gtk-2.0',
]
USER_BASE: '/home/uzinnal/.local' (exists)
USER_SITE: '/home/uzinnal/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
--8<---------------cut here---------------end--------------->8---


> If this does not give any insight, change it into
>
> exec -a "$0" python3 -v -c "import statsmodels"

Again with “python3” replaced as “/gnu/store/…-python-2.7.13/bin/python”
I ran this and wasn’t prepared for the deluge of output.  The first time
statsmodels is mentioned is when the prompt appears and “import
statsmodels” is executed.  Interestingly it goes straight to the old
statsmodels-0.6.1 instead of the newer one (0.8.0) which the wrapper
added to PYTHONPATH.

--8<---------------cut here---------------start------------->8---
…
Python 2.7.13 (default, Jan  1 1970, 00:00:01)
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import statsmodels # directory /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels
# /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.pyc matches /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py
import statsmodels # precompiled from /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/__future__.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/__future__.py
import __future__ # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/__future__.pyc
import numpy # directory /gnu/store/xd796qc9kjaslw85jk7f81fxkql021x0-python2-numpy-1.12.0/lib/python2.7/site-packages/numpy
…
--8<---------------cut here---------------end--------------->8---

Here’s what happens before the prompt appears:

--8<---------------cut here---------------start------------->8---
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
import site # from /home/uzinnal/.guix-profile/lib/python3.4/site-packages/site.py
# can't create /home/uzinnal/.guix-profile/lib/python3.4/site-packages/site.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/os.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/os.py
import os # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/posixpath.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/posixpath.py
import posixpath # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/posixpath.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/stat.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/stat.py
import stat # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/stat.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/genericpath.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/genericpath.py
import genericpath # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/genericpath.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/warnings.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/warnings.py
import warnings # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/warnings.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/linecache.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/linecache.py
import linecache # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/linecache.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/types.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/types.py
import types # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/types.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/UserDict.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/UserDict.py
import UserDict # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/UserDict.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_abcoll.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_abcoll.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/abc.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/abc.py
import abc # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/abc.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_weakrefset.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/copy_reg.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/copy_reg.pyc
import imp # builtin
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site.py
import site # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/site.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/traceback.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/traceback.py
import traceback # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/traceback.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sysconfig.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sysconfig.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/re.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/re.py
import re # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/re.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_compile.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_parse.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_parse.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_constants.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/sre_constants.pyc
dlopen("/gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/lib-dynload/_locale.so", 2);
import _locale # dynamically loaded from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/lib-dynload/_locale.so
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_sysconfigdata.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/_sysconfigdata.pyc
# zipimport: found 32 names in /home/uzinnal/.guix-profile/lib/python3.4/site-packages/cairocffi-0.6-py3.4.egg
# zipimport: found 609 names in /gnu/store/jjx8l3mxszklsq80hdpjv08yd9j14hc8-python-pytz-2016.3/lib/python3.4/site-packages/pytz-2016.3-py3.4.egg
# zipimport: found 31 names in /gnu/store/sdddqkk41g9x895kmsd28v1mniiccwpi-python-dateutil-2.5.2/lib/python3.4/site-packages/python_dateutil-2.5.2-py3.4.egg
# zipimport: found 32 names in /home/uzinnal/.guix-profile/lib/python2.7/site-packages/cairocffi-0.6-py2.7.egg
# zipimport: found 31 names in /gnu/store/xdmyyznc45vcfgrygrq50lbk0z89321k-python2-dateutil-2.5.2/lib/python2.7/site-packages/python_dateutil-2.5.2-py2.7.egg
import encodings # directory /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/__init__.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/__init__.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/codecs.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/codecs.py
import codecs # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/codecs.pyc
import _codecs # builtin
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/aliases.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/aliases.pyc
# /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/utf_8.pyc matches /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /gnu/store/0n8ni2ldvyz5yd488cidzi3via7jk7pw-python-2.7.13/lib/python2.7/encodings/utf_8.pyc
--8<---------------cut here---------------end--------------->8---

Note that “import site” is satisfied by Python 3, but the behaviour is
the same when I only keep Python 2 things on the PYTHONPATH.

With “-vvvv” I see that the directory containing statsmodels-0.8.0 is in
fact accessed multiple times for
“{site,os,posixpath,stat,genericpath,warnings}.{so,py,pyc}” and many
more; but when the time comes to “import statsmodels” it loads it
from the 0.6.1 directory; it does not look it up in the 0.8.0 directory.

--8<---------------cut here---------------start------------->8---
Python 2.7.13 (default, Jan  1 1970, 00:00:01)
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# trying statsmodels.so
# trying statsmodelsmodule.so
# trying statsmodels.py
# trying statsmodels.pyc
# trying /gnu/store/b0qkrrkiw0gszw6hdxcbyly6nr9ln7n3-python2-cffi-1.4.2/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.so
# trying /gnu/store/b0qkrrkiw0gszw6hdxcbyly6nr9ln7n3-python2-cffi-1.4.2/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodelsmodule.so
# trying /gnu/store/b0qkrrkiw0gszw6hdxcbyly6nr9ln7n3-python2-cffi-1.4.2/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.py
# trying /gnu/store/b0qkrrkiw0gszw6hdxcbyly6nr9ln7n3-python2-cffi-1.4.2/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.pyc
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodelsmodule.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.py
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/cffi-1.4.2-py2.7-linux-x86_64.egg/statsmodels.pyc
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/matplotlib-1.4.3-py2.7-linux-x86_64.egg/statsmodels.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/matplotlib-1.4.3-py2.7-linux-x86_64.egg/statsmodelsmodule.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/matplotlib-1.4.3-py2.7-linux-x86_64.egg/statsmodels.py
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/matplotlib-1.4.3-py2.7-linux-x86_64.egg/statsmodels.pyc
# trying /gnu/store/8707rsnrfi3f747pj1jvxr7bwb0kk982-python2-nose-1.3.7/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/statsmodels.so
# trying /gnu/store/8707rsnrfi3f747pj1jvxr7bwb0kk982-python2-nose-1.3.7/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/statsmodelsmodule.so
# trying /gnu/store/8707rsnrfi3f747pj1jvxr7bwb0kk982-python2-nose-1.3.7/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/statsmodels.py
# trying /gnu/store/8707rsnrfi3f747pj1jvxr7bwb0kk982-python2-nose-1.3.7/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/statsmodels.pyc
# trying /gnu/store/g6pi1k1py80ah53ig7g34pgp0gr2kr74-python2-six-1.10.0/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.so
# trying /gnu/store/g6pi1k1py80ah53ig7g34pgp0gr2kr74-python2-six-1.10.0/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodelsmodule.so
# trying /gnu/store/g6pi1k1py80ah53ig7g34pgp0gr2kr74-python2-six-1.10.0/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.py
# trying /gnu/store/g6pi1k1py80ah53ig7g34pgp0gr2kr74-python2-six-1.10.0/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.pyc
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/statsmodels.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/statsmodelsmodule.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/statsmodels.py
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/statsmodels.pyc
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodelsmodule.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.py
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/six-1.10.0-py2.7.egg/statsmodels.pyc
import statsmodels # directory /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__module.so
# trying /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py
# /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.pyc matches /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py
import statsmodels # precompiled from /home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.pyc
--8<---------------cut here---------------end--------------->8---

Note that lib/python2.7/site-packages (which is on the user’s
PYTHONPATH) contains a file “python2-statsmodels-0.6.1.pth”.  Does this
play a role here?

> exec -a "$0" python3 -v -c "import ribodiff.estimatedisp"
>
> and try to find some insight there.

Unfortunately, this didn’t tell me much more than the previous command.
After numpy is loaded, the old statsmodels is loaded.


> re 2): As soon as Python2 and Python3 are installed in the same profile,
> we put both site-packages for both versions into PYTHONPATH, which
> obviously is wrong.
>
> $ PYTHONPATH= guix environment --ad-hoc python@2 python
> […]
> [guix] $ echo $PYTHONPATH
> /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:
>
> A simple work-around would be to make the profile a (pseudo) virtual
> environment, which is a easy as creating a file
> "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
> site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
> sys.path - for the current python version only!

This is very good to know.  I haven’t tested this (as I’m not editing
the store by hand), but I suppose we could create an empty pyvenv.cfg in
a profile hook when Python packages are installed.

(This wouldn’t help us much for wrapper scripts, though.)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Hartmut Goebel]

Hi Ricardo,

you are welcome, I've taken the challenge :-) I always have fun digging
in the internals of Python - if I find time.

Thanks a lot for the data. I started investigating it and this raised
some more questions. Some behavior is curious. E.g. why does "python -c
'import statmodules'" start

 Thus it would be easiest if I would work interactively with that
environment. Can you please send me the information I need to rebuild
this environment (see below) - assistance would be great!

Alternatively I could download a export/copy of that profile - if guix
has some means for this.

Could I get interactive access to that profile? (Details to be discussed
of-list) This would allow me to investigate some of the curious
behavior. Nevertheless I would also like to rebuild the profile, so I
can hack the store-objects to work towards a solution.

For rebuilding  the I assume I need the following installation
- relevant parts of "guix package --list-installed"
- relevant installation dates from "guix package --list-generations"
- guix version used would be helpful

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH woes

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

Hi,

Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus:
>> re 2): As soon as Python2 and Python3 are installed in the same profile,
>> we put both site-packages for both versions into PYTHONPATH, which
>> obviously is wrong.
>>
>> $ PYTHONPATH= guix environment --ad-hoc python@2 python
>> […]
>> [guix] $ echo $PYTHONPATH
>> /gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python2.7/site-packages:/gnu/store/jkwp041kjy6li85n66ymxkfrr0hr2psj-profile/lib/python3.5/site-packages:
>>
>> A simple work-around would be to make the profile a (pseudo) virtual
>> environment, which is a easy as creating a file
>> "/gnu/store/…-profile/pyvenv.cfg". This will trigger a mechanism in
>> site.py to insert /gnu/store/…-profile/lib/pythonX.Y/site-packages" into
>> sys.path - for the current python version only!
> This is very good to know.  I haven’t tested this (as I’m not editing
> the store by hand), but I suppose we could create an empty pyvenv.cfg in
> a profile hook when Python packages are installed.

Below please find a simple package which adds this hack. After
installing it, unset PYTHONPATH:
export PYTHONPATH=

In the long run both python@2 and python@3 could propagate this
hack-package, thus is would be installed whenever python is installed.
(This need further investigation or a guix guru to help.)

> (This wouldn’t help us much for wrapper scripts, though.)

I have another hack in petto :) But one step at a time :-)


Now here is the

cat > guix-python-venv-hack.scm <<"EOF"
;;; Copyright © 2018 Hartmut Goebel <h.goebel@crazy-compilers.com>

(use-modules (guix)
             (guix build-system trivial)
             (guix licenses))

(package
  (name "guix-python-venv-hack")
  (version "0.1")
  (source #f)
  (build-system trivial-build-system)
  (arguments
   `(#:modules ((guix build utils))
     #:builder
     (begin
       (use-modules (guix build utils))
       (let* ((out       (assoc-ref %outputs "out"))
              (pyvenv.cfg (string-append out "/pyvenv.cfg")))
         (mkdir-p out)
         (call-with-output-file pyvenv.cfg
           (lambda (p)
             (format p "#")))
         ;;(chmod index.php #o555)
         ))))
  (synopsis "Python venc-hack for Guix")
  (description "See ...")
  (home-page #f)
  (license gpl3+))
EOF
guix package --install-from-file guix-python-venv-hack.scm
unset PYTHONPATH

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 3758 bytes --]

Re: PYTHONPATH woes

[Vincent Legoll]

Hello,

On Thu, Feb 22, 2018 at 9:42 PM, Hartmut Goebel
<h.goebel@crazy-compilers.com> wrote:
>   (synopsis "Python venc-hack for Guix")

Looks like a typo...

s/venc/venv/

-- 
Vincent Legoll

Re: PYTHONPATH woes

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

Am 22.02.2018 um 16:30 schrieb Ricardo Wurmus:
> (This wouldn’t help us much for wrapper scripts, though.)

Attached please fins an approach for solving this issue.

The basic idea is make the application/script use a python within a
virtual environment. I nee to rethink some details and check whether
this will work out.

See the comments in the code for how it is intended to work.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test-new-build2.scm --]
[-- Type: text/x-scheme; name="test-new-build2.scm", Size: 3213 bytes --]

;;; Copyright © 2018 Hartmut Goebel <h.goebel@crazy-compilers.com>

(use-modules (guix)
             (gnu packages python)
             (guix licenses))

;; TODO decide which path to use
(define VENV-DIR "/share/guix-venv/") ; trailing slash!

;; python-uniseg is a reasonable small package without any dependencies and
;; including a script
(package
  (inherit python-uniseg)
  (name "my-python-uniseg")
  (version (package-version python-uniseg))
  (arguments
   `(#:tests? #f  ; copied from base package
     #:phases
     (modify-phases %standard-phases
       (replace 'wrap
         (lambda* (#:key inputs outputs #:allow-other-keys)
           ;; This sets up a small virtual environment for the package
           ;; and wraps the scripts to 
           ;; ../share/guix-venv/my-site-display-0.1
           ;; +- pyvenv.cfg
           ;; +- bin/python
           ;; +- lib/pythonX.Y/site-packages/NAME.pth
  (let* ((out  (assoc-ref outputs "out"))
         (venv (string-append out ,VENV-DIR ,name "-" ,version))
         (python (assoc-ref inputs "python"))
         (site-dir (string-append "/lib/python"
                                  "3.5" ;; FIXME (get-python-version python)
                                  "/site-packages"))
         (PYTHONPATH ;; FIXME: Why is ther only one entry?
          (cons (string-append out site-dir)
                 (search-path-as-string->list
                  (or (getenv "PYTHONPATH") "")))))
    (mkdir-p (string-append venv "/bin"))
    (mkdir-p (string-append venv site-dir))
    ;; The existance of a pyvenv.cfg file marks this as being a virtual
    ;; environment
    (call-with-output-file (string-append venv "/pyvenv.cfg")
      (lambda (p)
        (format p "#include-system-site-packages = false")))
    ;; Link to all required packages using a .pth file
    (call-with-output-file (string-append venv site-dir "/" ,name ".pth")
      (lambda (p)
        (for-each (lambda (pkg)
                    (format p "~a~%" pkg))
                  PYTHONPATH)))
    ;; Create the python "executable" within the virtual environment
    (for-each
     (lambda (name)
       (symlink (string-append python "/bin/" name)
                (string-append venv  "/bin/" name)))
     `("python" ,"python3"))
    )))
       ;; Re-wrap the scripts to use the python within the virtual env
       (add-after 'patch-shebangs 'patch-python-venv-shebang
         (lambda* (#:key outputs inputs #:allow-other-keys)
  
           (let* ((out (assoc-ref outputs "out"))
                  (venv (string-append out ,VENV-DIR ,name "-" ,version))
                  (python (assoc-ref inputs "python")))
         (define (venv-program file)
             (substitute* file
               (((string-append "#!" python "/bin/python([0-9](\\.[0-9])?)?"))
                (string-append "#!" venv "/bin/python"))))
         ;; FIXME: Use logic from python-build-system to find scripts
         (venv-program (string-append out "/bin/uniseg-dbpath"))
         ))))))
  ;; Some requirement to check if all required pathes are included into the
  ;; .pth-file.
  (inputs
   `(("req" ,python-simplejson)))
  )

Re: PYTHONPATH woes

[Pjotr Prins]

On Fri, Feb 23, 2018 at 01:36:17PM +0100, Hartmut Goebel wrote:
> The basic idea is make the application/script use a python within a
> virtual environment. I nee to rethink some details and check whether
> this will work out.

I may misunderstand how you are doing this, but I think it is going to
cause problems. Python makes assumptions about the environment and
virtualenv kinda redirects those. On top of that we have guix profiles
and people who need to redirect PYTHONPATH (temporarily). Adding
virtualenv into the mix will complicate things. Then there are people
using virtualenv on top of Guix ...

I think the problem of mixing module versions has to be resolved
through profiles. That should just work(tm).

The problem of mixing interpreter versions can be resolved through
profiles. Though there is the danger that people mix them into one
profile.

For this I suggest we tell Python2 to only use PYTHONPATH2. That way
there is no interference. Python2 is being phased out (it is obsolete)
and upstream should consider such a solution too.

With Ruby we have a similar interpreter issue - even more fine grained
between versions. It is a pain. But there is no real solution other
than using profiles properly.

Pj.

Re: PYTHONPATH woes

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Then there are people
> using virtualenv on top of Guix ...

Pjotr brings up a good point: can virtualenvs be composed?  Do users
still have a way to override modules via PYTHONPATH or some other
mechanism if we were to create virtualenvs by default?

> I think the problem of mixing module versions has to be resolved
> through profiles. That should just work(tm).

@Pjotr: But we have seen that this doesn’t work as is and currently
requires user intervention.  We cannot expect users to separate all
Python 2 things from all Python 3 things because it is not even obvious
in all cases that a tool results in Python modules to be installed to
the profile.

> For this I suggest we tell Python2 to only use PYTHONPATH2. That way
> there is no interference. Python2 is being phased out (it is obsolete)
> and upstream should consider such a solution too.

I’d like to avoid radical patching when Python seems to have some
support for ignoring directories on the PYTHONPATH that don’t include
the correct version number.

Patching Python 2 is still an option, but I’d like to explore (and
understand) upstream mechanisms first.

> With Ruby we have a similar interpreter issue - even more fine grained
> between versions. It is a pain. But there is no real solution other
> than using profiles properly.

Do Ruby *executables* also suffer from accidental dependency injection
as Ribodiff does in this case?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Pjotr Prins]

On Fri, Feb 23, 2018 at 08:36:46PM +0100, Ricardo Wurmus wrote:
> Patching Python 2 is still an option, but I’d like to explore (and
> understand) upstream mechanisms first.

I don't think it will be radical and something upstream can adopt. We
can at least suggest it ;)

> > With Ruby we have a similar interpreter issue - even more fine grained
> > between versions. It is a pain. But there is no real solution other
> > than using profiles properly.
> 
> Do Ruby *executables* also suffer from accidental dependency injection
> as Ribodiff does in this case?

Main problem is that gems are stored in major versions, e.g. 2.4. So
Ruby 2.4.1 stores modules in the same path as 2.4.2. I think that is a
mistake. Theoretically gems are compatible... But you can see the
potential mess.

Re: PYTHONPATH woes

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 4730 bytes --]

Am 23.02.2018 um 17:59 schrieb Pjotr Prins:
> I may misunderstand how you are doing this, but I think it is going to
> cause problems. Python makes assumptions about the environment and
> virtualenv kinda redirects those. On top of that we have guix profiles
> and people who need to redirect PYTHONPATH (temporarily). Adding
> virtualenv into the mix will complicate things. Then there are people
> using virtualenv on top of Guix ...

I have to admit that my description was a bit terse. I understand your
concerns, but don't worry. This answer is a bit lengthy and you may read
all of the mail as the techniques are a bit complex.

0) I'm proposing to replace the wrapper scripts by a private virtual
environment.

1) This only effects the ways scripts are run, the package is still
available as normal.

2) The venv-hack I posted a few days ago is proposing something
different and is aiming to solve a different problem.

3) AFAIK virtualenvs can NOT be layered, they can only share the system
site-packages. This needs to be investigated  further, but only effects
the venv-hack I posted a few days ago.

Re 1):

This only effects the way *scripts* are run. Instead of using a wrapper
script, this uses a private virtual env - for the script only! Any
package is still available as normal.

Instead of using the python-executable of the profile, the script uses
the one in its private virtual env. This will make the script being run
in its own environment, which we also could name "profile".

The generated layout is as follows:

/gnu/store/…-my-app-0.1
+ lib/python3.5/site-packages/my_package/__init__.py  # unchanged
+ bin/my-app   # no wrapper! uses private venv's python
+ share/guix-venv/my-app-0.1
  +- pyvenv.cfg  # tells bin/python this is a virtual environment
  +- bin/python -> /gnu/store/…-python-3.5.3/bin/python
  +- lib/python3.5/site-packages/my-app.pth  # simulates PYTHONPATH

This comes down to something like

pyvenv-3.5 /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1
/gnu/store/…-my-app-0.1/bin/pip install my-app-0.1.tar.gz
ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/bin/my-app \
      /gnu/store/…-my-app-0.1/bin/my-app
ln -s /gnu/store/…-my-app-0.1/share/guix-venv/my-app-0.1/lib/python3.5/site-packages/my-app \
      /gnu/store/…-my-app-0.1/lib/python3.5/site-packages/my-app

This solves several issues:

  * - python-applications in one profile can use conflicting
    python-packages, since each application's dependencies are enclosed
    in this application's private environment
  * no more problems since the scripts file-name is .my-app-real (see
    e.g. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26752)
  * packages include references to their dependencies the gc can now
    pick up (only valid if the package includes a script, though)


Re 2) and 3):

> I think the problem of mixing module versions has to be resolved
> through profiles. That should just work(tm).

Mixing different package-versions for the same python version can never
ever being solved by profile-means Python supports only a single package
version in sys.path. Python has this
pkg_resources.get_distribution()-stuff, but this never really took of
and was superseded by virtual environments.

> The problem of mixing interpreter versions can be resolved through
> profiles. Though there is the danger that people mix them into one
> profile.
Mixing versions of different Python version can by solved by the
venv-hack I posted a few days ago. This basically makes the profile into
a virtual environment. Thus there is no need for setting PYTHONPATH in
the profile anymore (we can even remove the search-path) and packages of
different Python versions will no longer be mixed.

> For this I suggest we tell Python2 to only use PYTHONPATH2. That way
> there is no interference. Python2 is being phased out (it is obsolete)
> and upstream should consider such a solution too.

No other distribution is using something like PYTHONPATH2, neither does
upstream. Upstream will never introduce such a heavy incompatible change.

Our problems are caused by misusing PYTHONPATH for collection the
system/profile site-packages. We could of course implement search-paths
GUIX-PYTHON-SITE-PACKAGE-2 and GUIX-PYTHON-SITE-PACKAGE-3 (instead of
PYTHONPATH) and change "site.py" to honor them. I'll look into whether
this or my venv-hack are a better solution, esp. regarding "stacked"
virtual environments.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 6261 bytes --]

Re: PYTHONPATH woes

[Hartmut Goebel]

Am 24.02.2018 um 11:44 schrieb Hartmut Goebel:
>  you may read all of the mail as the techniques are a bit complex.

No offense meant, please ignore the sentence. When I started writing the
mail I though my explanation would be much more complex.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

PYTHONPATH issue analysis - part 1 (was: PYTHONPATH woes)

[Hartmut Goebel]

Hi,


I'm going to analyse the PYTHONPATH issues systematically. This is part 1 of
the analysis.


Result
=======

PYTHONPATH intented (by upstream) for adding site-packages.  PYTHONPATH
elements are added in front of essential packages, while site-packages are
added behind.

This means the way we are using PYTHONPATH should be changed.


Preliminary Proposal
=======================

This proposal is under the limitation of the current state of the
analysis. Further parts of the analysis may yield more
insights, and propose different solutions.

1. The search-path-specification PYTHONPATH should be replaced by
   GUIX-PYTHON2-SITE-PACKAGES resp. GUIX-PYTHON3-SITE-PACKAGES (names
   tbd).

2. 'site.py' will be patched to add theses entries to sys.path.

Limitations:

This only solves the issue related to version-specific site-packages. python
still thinks its "home" is in /gnu/store while it actually should be in the
guix-profile.


Rational
=================

According to the documentation [1,2,PEP370], sys.path is composed as
follows:

  $PWD resp. dir containing the script
  $PYTHONPATH elements
  default search path PREFIX/lib/pythonX.Y
  user site-packages  ~/.local/lib/pythonX.Y/site-packages
  system-site packages PREFIX/lib/python2.6/site-packages

The .pth-files are processed in user and system site-packages only,
but not in $PYTHONPATH.

[PEP370] explicitly says: "The […] site directory is added […] after
Python's search paths and PYTHONPATH. This setup […] prevents […]
overwriting a stdlib module. Stdlib modules can still be overwritten
with PYTHONPATH."

This means: PYTHONPATH is not intented to set system site-packages.

Example:

$ cd ~
$ PYTHONPATH=/tmp/aaa:/tmp/bbb /usr/bin/python -m site
sys.path = [
    '/home/htgoebel,
    '/tmp/aaa',
    '/tmp/bbb',
    '/usr/lib/python27.zip',
    '/usr/lib64/python2.7',
    '/usr/lib64/python2.7/plat-linux2',
    '/usr/lib64/python2.7/lib-tk',
    '/usr/lib64/python2.7/lib-old',
    '/usr/lib64/python2.7/lib-dynload',
    '/home/htgoebel/.local/lib/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages/gtk-2.0',
    '/usr/lib64/python2.7/site-packages/wx-3.0-gtk2',
    '/usr/lib/python2.7/site-packages',
]


[1] https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
[2] https://docs.python.org/3/library/site.html#module-site
[PEP370] https://www.python.org/dev/peps/pep-0370/#implementation

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue analysis - part 1

[Hartmut Goebel]

Hi, a small correction:

> Result
> =======
>
> PYTHONPATH intented (by upstream) for adding site-packages.  
This should be "PYTHONPATH is NOT intended …"

> PYTHONPATH
> elements are added in front of essential packages, while site-packages are
> added behind.
>
> This means the way we are using PYTHONPATH should be changed.

The remaining text is correct.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

PYTHONPATH issue analysis - part 2 (was: PYTHONPATH woes)

[Hartmut Goebel]

Hi,

nex part of the analysis:

Result
=======

The venv-hack I posted a few days ago works as expected only for
GUIX_PROFILE,
but not for virtual environments.


Preliminary Proposal
=======================

As it stands now, the venv-hack is not a valid solution. It may be the basis
for another solution, tough.


Rational
===========

I tried answering four questions:

A: Can virtual environments be stacked in stock Python?

   No, they can not, see point 1. and 2. below.

C: Given PYTHONPATH is not set, do virtual environments in Guix use
   the correct "site" packages (which is the ones in GUIX_PROFILE).

   No, in guix venvs use the site-packages from
   /gnu/store/…-python-3.6.3/. See points 3. and 4 below.

D: Would the venv-hack I posted a view days ago solve the issue?

   No, it would not. It would work as expected for python in the
   profile, but not for virtual environments based on this. See point
   5. below.


1. Set up a virtual environemnt using the system installed python.
====================================================================

$ pyvenv-3.5 /tmp/venv-1
$ ls -l /tmp/venv-1/bin/python
… /tmp/venv-1/bin/python -> python3.5
$ ls /tmp/venv-1/bin/
activate      activate.fish  easy_install-3.5*  pip3*    python@  
python3.5@
activate.csh  easy_install*  pip*               pip3.5*  python3@
$ ls -l /tmp/venv-1/bin/python*
… /tmp/venv-1/bin/python -> python3.5
… /tmp/venv-1/bin/python3 -> python3.5
… /tmp/venv-1/bin/python3.5 -> /usr/bin/python3.5
$ /tmp/venv-1/bin/python -m site
sys.path = [
    '/home/hartmut',
    '/usr/lib64/python35.zip',
    '/usr/lib64/python3.5',
    '/usr/lib64/python3.5/plat-linux',
    '/usr/lib64/python3.5/lib-dynload',
    '/tmp/venv-1/lib64/python3.5/site-packages',
    '/tmp/venv-1/lib/python3.5/site-packages',
]

As expected there are only the venvs' site-packges in sys.path.


2. Now stack venv on top of venv-1. Use --system-site-packages to
(hopefully) make venv-1's site-packages available to venv-2.
====================================================================


$ which pyvenv-3.5
/bin/pyvenv-3.5
$ /tmp/venv-1/bin/python /bin/pyvenv-3.5 /tmp/venv-2 --system-site-packages
$ ls -l /tmp/venv-2/bin/python*
… /tmp/venv-2/bin/python -> /tmp/venv-1/bin/python
… /tmp/venv-2/bin/python3 -> python
$ ls /tmp/venv-2/bin/
activate      activate.fish  easy_install-3.5*  pip3*    python@
activate.csh  easy_install*  pip*               pip3.5*  python3@
$ /tmp/venv-2/bin/python -m site
sys.path = [
    '/tmp',
    '/usr/lib64/python35.zip',
    '/usr/lib64/python3.5',
    '/usr/lib64/python3.5/plat-linux',
    '/usr/lib64/python3.5/lib-dynload',
    '/tmp/venv-2/lib64/python3.5/site-packages',
    '/tmp/venv-2/lib/python3.5/site-packages',
    '/usr/lib64/python3.5/site-packages',
    '/usr/lib/python3.5/site-packages',
]

As you can see (last two entries), the system site-packages are taken
from the real system installation, not from the stacked venv-1. This
means, venvs can not be stacked.


3. Now let's see how guix-profile installed python works. I used a
somewhat current HEAD (7e4e3df4e8) to ensure using the most current
wrappers etc.
====================================================================


$ ./pre-inst-env guix package -i python
…


3a. Do not set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3a
…
$ /tmp/venv-3a/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-3a/lib/python3.6/site-packages',
]

As expected there are only the venvs' site-packges in sys.path.


3b. Set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ PYTHONPATH="$HOME/.guix-profile/lib/python3.6/site-packages"
~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-3b
…
$ /tmp/venv-3b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-3b/lib/python3.6/site-packages',
]

Again there are only the venvs' site-packges in sys.path. This is
excpected, since PYTHONPATH only effects the run of pyenv-3.6


4. Same as 3, but use --system-site-packages
====================================================================

4a Don't set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ ~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4a --system-site-package
…
$ /tmp/venv-4a/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-4a/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

This is *not* what what a Guix user would expect. For the Guix user's
perspective his/her "Python site-packages" are those in $GUIX_PROFILE.
Esp. since guix never installs into
/gnu/store/…-python-3.6.3/lib/python3.6/site-packages and thus this
path never contains additional "site" packages.


4b Set PYTHONPATH when setting up the venv.
----------------------------------------------------

$ PYTHONPATH="$HOME/.guix-profile/lib/python3.6/site-packages"
~/.guix-profile/bin/pyvenv-3.6 /tmp/venv-4b --system
$ /tmp/venv-4b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-4b/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

Result is the same as for 4a), reason as in 3b).


5. Would the venv-hack I posted a view days ago solve the issue?
====================================================================

5a. Verify the venv-hack works
----------------------------------------------------


$ cp -r ~/.guix-profile /tmp/guix-profile
$ mkdir !$
mkdir /tmp/guix-profile
$ cp -r ~/.guix-profile/* /tmp/guix-profile
$ echo 'include-system-site-packages = false' > /tmp/guix-profile/pyvenv.cfg
$ /tmp/guix-profile/bin/python3 -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/guix-profile/lib/python3.6/site-packages',
]

As expected, the profile's site-packages are included in sys.path.


5b Build a venv based on this (hacked) profile
----------------------------------------------------

$ /tmp/guix-profile/bin/pyvenv-3.6 /tmp/venv-5b --system-site-packages
…
$ /tmp/venv-5b/bin/python -m site
sys.path = [
    '/tmp',
    '/gnu/store/…-python-3.6.3/lib/python36.zip',
    '/gnu/store/…-python-3.6.3/lib/python3.6',
    '/gnu/store/…-python-3.6.3/lib/python3.6/lib-dynload',
    '/tmp/venv-5b/lib/python3.6/site-packages',
    '/gnu/store/…-python-3.6.3/lib/python3.6/site-packages',
]

As in 4a, this is *not* what what a Guix user would expect. The
profile's site-packages should be in sys.path, not
/gnu/store/…-python-3.6.3/lib/python3.6/site-packages.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

PYTHONPATH issue analysis - part 3 (was: PYTHONPATH woes)

[Hartmut Goebel]

Hi,

here is my third part of the analysis:

Result
==========

We can avoid all of the problems related to how Guix is using PYTHONPATH
quite simple. This will work for virtual environments, too.


Preliminary Proposal
=======================

To be able to install different minor versions of Python in the same
profile, any environment variable should contain the minor version, too.
E.g. …-3.5.

Option 2 (GUIX-PYTHONHOME-X.Y) should be implemented since it is simple.

If we can get option 3 (stop resolving sysmlinks at the correct
iteration) to work, this might be a better solution.



Rational
===========

1. Is setting GUIX-PYTHON-X.Y-SITE-PACKAGES enough?

   This should work for most cases, but might break Python appications
   expecting site-packages to be below sys.prefix. Thus setting
   (GUIX-)PYTHONHOME-X.Y is a better solution.

2. Would GUIX-PYTHONHOME-2.7, …-3.4, …-3.5 work?

   Yes, this would work, but still an environment variable would be
   required.

3. Can we get without any environment variable?

   Yes, if we manage to to resolving symlinks at the correct iteration.
   This might be complicated to achieve.


4. How does Path-handling in Python's start-up sequence work?

   See detailed analysis below.



1. How could GUIX-PYTHON-X.Y-SITE-PACKAGES be implemented?
=============================================================

Given the analysis below, it would be possible to patch site.py to make
it use these environment variables. I did not look at the details yet,
but since the site-package paths are only set by site.py, this should be
not much of an issue.

To be able to install different minor versions of Python in the same
profile, the variables should contain the minor version, too. E.g. …-3.5.

Drawbacks:

- sys.prefix and sys.exec_prefix would still point to the store, not to
  the profile.  This might break Python appications expecting
  site-packages to be below sys.prefix.



2. How could GUIX-PYTHONHOME-X.Y be implemented?
=================================================


Given the analysis below, it should be okay to implement GUIX-PYTHONHOME-X.Y
like this:

- In Py_GetPythonHome() (Python/pythonrun.c), after checking for
  PYTHONHOME, check for GUIX-PYTHONHOME-X.Y. This will effect below
  step 2 for non-venvs and step 3 for venvs.

  This should be save for virtual environments, too, since pyvenv.cfg is
  searched based on argv0_path.

  This should be save for stacked virtual environments, too, since
  pyvenv.cfg will still point to the prior venv and sys.prefix and
  sys.exec_prefix will be set correctly in site.py.

- Implement a "search-path" GUIX-PYTHONHOME-X.Y

- To be able to install different minor versions of Python in the same
  profile, the variables should contain the minor version, too. E.g.
  …-3.5.

Drawbacks:

- We need to ensure GUIX-PYTHONHOME-X.Y is a single path, not a list of
  paths. Or we split the variable in Py_GetPythonHome().

- Requires GUIX-PYTHONHOME-X.Y to be set in the respective environment.


3. How to avoid GUIX-PYTHONHOME[23]?
=========================================

We could avoid GUIX-PYTHONHOME[23] if we stop resolving the symlinks at
the correct point in iteration. Something like this:

    # resolve the symlink
    last = progpath
    while 1:
        if (last.startswith('/gnu/store/') and \
            last[43:52] == '-profile/'):
            # links to a profile
            break
        try:
            next = os.readlink(last)
        except OSError:
            # not a link anymore
            break
        if not next.startswith(SEP):
            # Interpret relative to last
            next = os.path.join(os.path.dirname(last), next)
        if next == GUIX_PYTHON_OUTPUT: # out "/bin/python" compile-time
            # "next" points to the python binary of the guix-output
            break
        last = next
    argv0_path = last

Drawbacks:

- More complicated patch.

- More comparison within a look, this will slow down start-up a bit.

Open questions:

- Which are the correct paths to check to stop iteration?
- How to handle the "pythonX" -> "pythonX.Y" link?
- How to handle "python-wrapper", which links python -> python3



4. Path-handling in Python's start-up sequence
===============================================

No venv
------------

In getpath.c:

0. "progpath" will be set based on argv[0]. This is expected to be the
   fully quallified path of the argv[0].

1. argv0_path is search based on progpath and all symlinks resolved.

2. prefix and exec_prefix are searched based on argv0_path *) **)

3. sys.path is set based on prefix and exec_prefix.

In sysmodule.c:

4. sys.executable is set to "progpath" (see step 0 above).

5. sys.prefix, sys.base_prefix, sys.exec_prefix and sys.base_exec_prefix
   are set to prefix resp. exec_prefix evaluated in step 3 above.

In site.py;

6. When site.py is loaded, system site-packages are added from
   sys.prefix and sys.exec_prefix.


*) There are two special cases, guix does not need to handle: If an
   (embedding) application did call (Py_SetPythonHome) or "PYTHONHOME"
   was set, this overrules argv0_path. But this is where
   GUIX-PYTHONHOME-X.Y could step in.

**) If some "landmark" file is not found, the build-time PREFIX
    resp. EXEC_PREFIX is used. For guix this should not happen.


For a venv:
--------------------

In getpath.c:

0. "progpath" will be set based on argv[0]. This is expected to be the
   fully qualified path of the argv[0].  Try this in a venv::

    $ lm -s /tmp/venv/bin/python /tmp/qqq
    $ /tmp/qqq -S -c 'import sys; print(sys.executable)'
    /tmp/qqq

1. argv0_path is search based on progpath and all symlinks resolved,
   like above.

2. If pyvenf.cfg exists in argv0_path's directory or one level above,
   argv0_path is taken from the "home" entry in this file.

3. prefix and exec_prefix are searched based on argv0_path (resp.
   PYTHONHAOME, GUIX-PYTHONHOME-X.Y). Notably both are now pointing to
   the "home" - not to the virtual env.

4. sys.path is set based on prefix and exec_prefix. Notably this is
   adding the standard library based on "home". Try this in a venv (here
   on a foreign distribution)::

    $ /tmp/venv/bin/python -S -c 'import sys; print(sys.path)'
    ['', '/usr/lib64/python35.zip', '/usr/lib64/python3.5',
     '/usr/lib64/python3.5/plat-linux',
     '/usr/lib64/python3.5/lib-dynload']

In sysmodule.c:

5. sys.executable is set to "progpath" (see step 0 above).

6. sys.prefix, sys.base_prefix, sys.exec_prefix and sys.base_exec_prefix
   are set to prefix resp. exec_prefix evaluated in step 3 above. All of
   these are pointing to "home"! sys.prefix and sys.exec_prefix will be
   adjusted in site.py.

   Try this in a venv (here on a foreign distribution)::

     $ /tmp/venv/bin/python -S -c 'import sys; print(sys.prefix)'
     /usr

In site.py:

7. If pyvenf.cfg exists in the executable's directory or one level
   above, site.py assumes a virtuel environment and will execute the
   following steps.

8. When site.py is loaded, sys.prefix and sys.exec_prefix will be set
   based on sys.executable - which is in the virtual env.

9. sys.base_prefix and sys.base_exec_prefix will not be changes and thus
   always be the "real" prefixes of the Python installation - see step 3
   above.

10. sys._home (not documented) will be set to the "home" entry from
    pyvenv.cfg, if the entry exisits.


-> sys.base_prefix and sys.base_exec_prefix should should point to
   GUIX_PROFILE


pyvenv.cfg - as of Python 3.5 venv
---------------------------------------

The "home" entry in pyvenv.cfg is based on sys.executable. This means if
you are stacking venvs, the "home" entry is pointing to the prior venv,
not to the original prefix.

Nevertheless, sys.path is based on the *orginal* prefix, not the "home":
At the beginning of site.py, the list of prefixes to be searched is set
to sys.prefix, which - see step 6 above - is the same as sys.base_prefix
and thus the orginal prefix, not the "home". site.py will change
sys.prefix later, but not the presets in the list of prefixes to be
searched.



From Modules/getpath.c
============================

 * Before any searches are done, the location of the executable is
 * determined.  If argv[0] has one or more slashes in it, it is used
 * unchanged.  Otherwise, it must have been invoked from the shell's path,
 * so we search $PATH for the named executable and use that.  If the
 * executable was not found on $PATH (or there was no $PATH environment
 * variable), the original argv[0] string is used.
 *
 * Next, the executable location is examined to see if it is a symbolic
 * link.  If so, the link is chased (correctly interpreting a relative
 * pathname if one is found) and the directory of the link target is used.
 *
 * Finally, argv0_path is set to the directory containing the executable
 * (i.e. the last component is stripped).

Re: PYTHONPATH issue analysis - part 3

[Ludovic Courtès]

Hello,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Result
> ==========
>
> We can avoid all of the problems related to how Guix is using PYTHONPATH
> quite simple. This will work for virtual environments, too.

I may well have missed something (sorry about that!), but what are “the
problems related to how Guix is using PYTHONPATH”?

My first reaction is that Guix is not doing anything special with
PYTHONPATH, and only defining it as documented by upstream.

For example:

> Preliminary Proposal
> =======================
>
> To be able to install different minor versions of Python in the same
> profile, any environment variable should contain the minor version, too.
> E.g. …-3.5.

If you’re suggesting to have a ‘PYTHONPATH3-5’ environment variable
instead of ‘PYTHONPATH’, I agree it could be helpful if we are to
install different versions of Python in one profile.  However, it’s a
choice for upstream to make, and upstream did not make that choice.

There’s one case where we went our way instead of following upstream,
and that is ‘GUIX_LOCPATH’.  There are strong justifications, though.

Thanks,
Ludo’.

Re: PYTHONPATH issue analysis - part 3

[Pjotr Prins]

On Tue, Mar 13, 2018 at 10:23:08PM +0100, Ludovic Courtès wrote:
> > Preliminary Proposal
> > =======================
> >
> > To be able to install different minor versions of Python in the same
> > profile, any environment variable should contain the minor version, too.
> > E.g. …-3.5.
> 
> If you’re suggesting to have a ‘PYTHONPATH3-5’ environment variable
> instead of ‘PYTHONPATH’, I agree it could be helpful if we are to
> install different versions of Python in one profile.  

Another problem is that it does not cover special cases where, for
example you compile Python with SSL and without. You don't want them
to share user installed libs. That is why I stick in the Hash value.

Ruby example

  https://gitlab.com/pjotrp/guix-notes/blob/master/scripts/ruby-guix-env

admittedly I no longer use that much. But I would if I just quickly
wanted to try a gem/module and make sure it is isolated.

> However, it’s a choice for upstream to make, and upstream did not
> make that choice.

Sometimes upstream can do with good ideas ;). The shared use of
PYTHONPATH between versions of Python2 and 3 is as brain dead as it
can be. I see people trip over it regularly, including myself. Great
time waster, if nothing else.

> There’s one case where we went our way instead of following upstream,
> and that is ‘GUIX_LOCPATH’.  There are strong justifications, though.

Would not work with multiple versions in one profile though. Anyway,
Hartmut covered all that.

Pj.
-- 

Re: PYTHONPATH issue analysis - part 3

[Hartmut Goebel]

Am 13.03.2018 um 22:44 schrieb Pjotr Prins:
> Another problem is that it does not cover special cases where, for
> example you compile Python with SSL and without. You don't want them
> to share user installed libs.

Upstream does not handle this case, so I do not see a need to handle
this in guix. If we find a way, this would be find. But prior to solving
the optional we should solve the compulsory :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue analysis - part 3

[Pjotr Prins]

On Tue, Mar 13, 2018 at 11:02:03PM +0100, Hartmut Goebel wrote:
> Am 13.03.2018 um 22:44 schrieb Pjotr Prins:
> > Another problem is that it does not cover special cases where, for
> > example you compile Python with SSL and without. You don't want them
> > to share user installed libs.
> 
> Upstream does not handle this case, so I do not see a need to handle
> this in guix. If we find a way, this would be find. But prior to solving
> the optional we should solve the compulsory :-)

  https://github.com/python/cpython/blob/master/configure#L1543

and there are many other options which define the behaviour of the
interpreter. We don't use them in GNU Guix, but it does not mean we
should not think about it/allow it.

I am nit picking a bit, but the problem is that we don't have a full
solution unless we can isolate the instances and the packages they can
install (themselves).

Using one PYTHONPATH is ultimately Python's failure - I agree with
Ludo that people should do what they want with it. Bringing in an
extra GUIX_PYTHONPATH_$VER will confuse things (and probably break it
for some people).

I propose we patch the interpreter to tell about the Guix paths. It
probably only needs to be done in one place. They should go after the
PYTHONPATH as it is done by Python itself, like you suggested earlier.

That is an acceptable approach. Just a little annoyance with every
Python upgrade.

Pj.

Re: PYTHONPATH issue analysis - part 3

[Hartmut Goebel]

Hi Pjotr,

no offense meant, but I have the impression you did not read my analysis
and proposals, did you?

> I propose we patch the interpreter to tell about the Guix paths. It
> probably only needs to be done in one place. They should go after the
> PYTHONPATH as it is done by Python itself, like you suggested earlier.

See option 3 of my analysis part 3.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue analysis - part 3

[Pjotr Prins]

On Wed, Mar 14, 2018 at 10:04:21AM +0100, Hartmut Goebel wrote:
> Hi Pjotr,
> 
> no offense meant, but I have the impression you did not read my analysis
> and proposals, did you?

No offense taken. I did and do read them, but the analysis is not
exactly accessible for my small brain, especially when you go into
symlinks in part 3.

No need to over-analyse. Sometimes you just have to try and see what
breaks. We probably are talking at cross-purposes.

Pj.

Re: PYTHONPATH issue analysis - part 3

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]

Am 14.03.2018 um 08:49 schrieb Pjotr Prins:
> On Tue, Mar 13, 2018 at 11:02:03PM +0100, Hartmut Goebel wrote:
>> Am 13.03.2018 um 22:44 schrieb Pjotr Prins:
>>> Another problem is that it does not cover special cases where, for
>>> example you compile Python with SSL and without. You don't want them
>>> to share user installed libs.
>> Upstream does not handle this case, so I do not see a need to handle
>> this in guix. If we find a way, this would be find. But prior to solving
>> the optional we should solve the compulsory :-)
>   https://github.com/python/cpython/blob/master/configure#L1543
>
> and there are many other options which define the behaviour of the
> interpreter. We don't use them in GNU Guix, but it does not mean we
> should not think about it/allow it.

Now I understand you point: If there are two variants of e.g Python 3.5
available, but there is only one GUIX-PYTHONHOME-3.5 variable, this will
intermix the environments again.

You are right!

Adding the hash would indeed be a good solution :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 2159 bytes --]

Re: PYTHONPATH issue analysis - part 3

[Hartmut Goebel]

Am 13.03.2018 um 22:23 schrieb Ludovic Courtès:
> I may well have missed something (sorry about that!), but what are “the
> problems related to how Guix is using PYTHONPATH”?

In short:

We are using PYTHONPATH for something it is not meant for: PYTHONPATH is
NOT intended (by upstream) for adding site-packages.  PYTHONPATH
elements are added in front of essential packages, while site-packages
are added behind.

Dor details please see part 1 of my analysis.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue analysis - part 3

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 13.03.2018 um 22:23 schrieb Ludovic Courtès:
>> I may well have missed something (sorry about that!), but what are “the
>> problems related to how Guix is using PYTHONPATH”?
>
> In short:
>
> We are using PYTHONPATH for something it is not meant for: PYTHONPATH is
> NOT intended (by upstream) for adding site-packages.  PYTHONPATH
> elements are added in front of essential packages, while site-packages
> are added behind.

Oh right, thanks for the reminder.

Ludo’.

Re: PYTHONPATH issue analysis - part 3

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

Am 13.03.2018 um 22:23 schrieb Ludovic Courtès:
>> Preliminary Proposal
>> =======================
>>
>> To be able to install different minor versions of Python in the same
>> profile, any environment variable should contain the minor version, too.
>> E.g. …-3.5.
> If you’re suggesting to have a ‘PYTHONPATH3-5’ environment variable
> instead of ‘PYTHONPATH’, I agree it could be helpful if we are to

This suggestion is related to Guix-specific variables only, like
GUIX-PYTHONHOME-2.7. (I hae to admit that this is not clear enough).

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1399 bytes --]

Re: PYTHONPATH issue analysis - part 3

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> I may well have missed something (sorry about that!), but what are “the
> problems related to how Guix is using PYTHONPATH”?
>
> My first reaction is that Guix is not doing anything special with
> PYTHONPATH, and only defining it as documented by upstream.

The problem might be that we are using PYTHONPATH at all.  On other
distributions this is usually not required and thus doesn’t cause any
problems.

As outlined in my first email in this thread, our use of PYTHONPATH in
wrappers seems to not have the desired effect in the presence of
incompatible packages that are *later* in the PYTHONPATH.

We also have problems when PYTHONPATH includes modules for both Python 2
and 3, which happens automatically when these modules are installed into
the same Guix profile.  Since merely installing packages for different
Python versions is not a problem on traditional distros (on my Fedora
workstation I have site-packages directories for 4 different versions of
Python) I think we should do better here.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH issue analysis - part 3

[Ludovic Courtès]

Hello,

Ricardo Wurmus <rekado@elephly.net> skribis:

> The problem might be that we are using PYTHONPATH at all.  On other
> distributions this is usually not required and thus doesn’t cause any
> problems.

It’s not required because Python modules live at a fixed location, no?

How does pip deal with that?  I suppose it needs to modify the search
path somehow.

> As outlined in my first email in this thread, our use of PYTHONPATH in
> wrappers seems to not have the desired effect in the presence of
> incompatible packages that are *later* in the PYTHONPATH.
>
> We also have problems when PYTHONPATH includes modules for both Python 2
> and 3, which happens automatically when these modules are installed into
> the same Guix profile.  Since merely installing packages for different
> Python versions is not a problem on traditional distros (on my Fedora
> workstation I have site-packages directories for 4 different versions of
> Python) I think we should do better here.

I agree, though I must say that if PYTHONPATH is not up to the task, I’m
not sure what can be done on our side.

Thanks,
Ludo’.

Re: PYTHONPATH issue explanation

[Hartmut Goebel]

Hi,

given the ongoing discussion around Python show that my explanation was
not good enough. I'll try to summarize and give more background.

With regard to Python, guix currently has a major issue, which my
proposals are addressing. There are other issues (like naming the
executables, the "wrapper" script", etc.) which are not addressed here.

When installing Python and some Python packages (e.g. python-simplejson)
in guix, the python interpreter will be linked to
GUIX_PROFILE/bin/pythonX.Y and the packages' files are linked into
GUIX_PROFILE/lib/python-X.Y/site-packages/…, which is perfectly okay.

This python interpreter does not find the site-packages in GUIX_PROFILE
since site-packages are search relative to "sys.base_prefix" (which is
the same as "sys.prefix" except in virtual environments).
"sys.base_prefix" is determined based on the executable's path (argv[0])
by resolving all symlinks.

The python interpreter assumes "site-packages" to be relative to "where
python is installed" - called "sys.base_prefix" (which is the same as
"sys.prefix" except in virtual environments). "sys.base_prefix" is
determined based on the executable's path (argv[0]) by resolving all
symlinks. For Guix this means: "sys.base_prefix" will always point to
/gnu/store/…-python-X.Y, not to GUIX_PROFILE. Thus the site-packages
installed into the guix profile will not be found.

This is why we currently (mis-) use PYTHONPATH: To make the
site-packages installed into the guix profile available.

Using PYTHONPATH for this woes since there is only one PYTHONPATH
variable for all versions of python. This is designed by upstream.

Additionally: When using PYTHONPATH the site-packages are added to the
search path ("sys.path") *in front* of the python standard library,
while they are expected to be added *behind*.

Part 3 of my analysis lists three solutions for this, where only number
2 and 3 are "good choices".

no. 2
suggests using a mechanism already implemented in python: Setting
"PYTHONHOME" will make the interpreter to use this as "sys.base_prefix"
unconditionally. Again there is only one PYTHONHOME variable for all
versions of python (designed by upstream). We could work around this
easily (while keeping upstream compatibility) by using
GUIX-PYTHONHOME-X.Y, to be evaluated just after PYTHONHOME.

This would be easy to implement using Guix's "search-path" capabilities
and a small patch to the python interpreter.

The drawback is: This is implemented using an environment variable,
which might not give the expected results in all cases. E.g. running
/gnu/store/…-profile/bin/python will not load the site-packages of that
profile. Also there might be issues implementing virtual environments.
(Thinking about this, I'm quite sure there will. Ouch!)

no.3
suggests changing the way the python interpreter is resolving symlinks
when searching for "sys.base_prefix". The idea is to stop "at the profile".

The hard part of this is to determine "at the profile". Also this needs
a larger patch. But if we manage to implement this, it would be perfect.
I could contribute a draft for this implemented in Python. The
C-implementation needs to be done by some C programmer.

Which way should we go?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue explanation

[宋文武]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

Hello,

> Hi,
>
> given the ongoing discussion around Python show that my explanation was
> not good enough. I'll try to summarize and give more background.

Thanks for the explanations and this one!

So I have more understanding of it and ideas...

>
> With regard to Python, guix currently has a major issue, which my
> proposals are addressing. There are other issues (like naming the
> executables, the "wrapper" script", etc.) which are not addressed
> here.
> [...]

Okay, the "major issue" is that we're using "PYTHONPATH", which will add
entries into "sys.path" before builtin ones.  It's semantically wrong
and may (or had?) cause issues.


> Part 3 of my analysis lists three solutions for this, where only number
> 2 and 3 are "good choices".

Option 2, "GUIX_PYTHONHOME_X_Y" can not be used in the build-system
unless we make a union of python inputs, so I think we should go for 1
and optional (later) add 3 too:

- "GUIX_PYTHON_X_Y_SITE_PACKAGES" (X.Y is not a valid env identifier
  in bash) is necessary for the "build" environment.

  We don't make a union of all the inputs in the "build" environment, so
  a PATH (contains multiples directories) like env have to be used to
  let python find all its "site-packages" from inputs.

  > Drawbacks: This might break Python appications expecting
  > site-packages to be below sys.prefix.

  We have a patch named "python-2.7-site-prefixes.patch" seems to handle
  this, maybe we should do it for python3 too?


- Avoid any environment variable for the "profile" environment.

  We have a union "profile" for all the python packages, so environment
  variables can be totally avoided with the help of "venv".

  > We could avoid GUIX-PYTHONHOME[23] if we stop resolving the symlinks
  > at the correct point in iteration.

  This is exactly what "venv" does!  We only need to make the "profile"
  a "venv" for python.  For python3, a simple "pyvenv.cfg" file is
  enough, for python2 I guess we have to make a union or copy files like
  what "virtualenv" does.


I plan to implement option 1 by adding a "sitecustomize.py" (better
than modify "site.py") into the python packages, and modify
"search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".

How's that sound?

Re: PYTHONPATH issue explanation

[Ricardo Wurmus]

宋文武 <iyzsong@member.fsf.org> writes:

> Option 2, "GUIX_PYTHONHOME_X_Y" can not be used in the build-system
> unless we make a union of python inputs

For texlive we create a temporary union in the build system, so this
shouldn’t be an unsurmountable obstacle.

What I don’t like about this solution is that PYTHONHOME can only hold a
single directory, so composing profiles (that use the same Python
variant) would no longer work.  I prefer the
GUIX_PYTHON_X_Y_SITE_PACKAGES solution, because it is an actual search
path.

> - "GUIX_PYTHON_X_Y_SITE_PACKAGES" (X.Y is not a valid env identifier
>   in bash) is necessary for the "build" environment.
>
>   We don't make a union of all the inputs in the "build" environment, so
>   a PATH (contains multiples directories) like env have to be used to
>   let python find all its "site-packages" from inputs.

I think this might be a good solution as it is a drop-in replacement for
our current use PYTHONPATH.

Hartmut wrote this:

> - sys.prefix and sys.exec_prefix would still point to the store, not to
>   the profile.  This might break Python appications expecting
>   site-packages to be below sys.prefix.

Is this an actual problem?  Do you know of applications that make this
assumption?  If so, is this unfixable?

>   We have a union "profile" for all the python packages, so environment
>   variables can be totally avoided with the help of "venv".
>
>   > We could avoid GUIX-PYTHONHOME[23] if we stop resolving the symlinks
>   > at the correct point in iteration.
>
>   This is exactly what "venv" does!  We only need to make the "profile"
>   a "venv" for python.  For python3, a simple "pyvenv.cfg" file is
>   enough, for python2 I guess we have to make a union or copy files like
>   what "virtualenv" does.

I’m not too hopeful about this variant, but I’m rather ignorant about
venvs.  My main concern is about whether it will still be possible for
users to create venvs from a subset of their installed packages when we
generate a pyvenv.cfg by default.

> I plan to implement option 1 by adding a "sitecustomize.py" (better
> than modify "site.py") into the python packages, and modify
> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".
>
> How's that sound?

This sounds good to me.  Thank you and thanks again, Hartmut, for laying
out our options and analysing their advantages and drawbacks!

Before working on this, though, I would like to have the above question
answered to avoid wasting your time.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH issue explanation

[Hartmut Goebel]

Am 17.03.2018 um 11:07 schrieb Ricardo Wurmus:
> What I don’t like about this solution is that PYTHONHOME can only hold a
> single directory, so composing profiles (that use the same Python
> variant) would no longer work.  I prefer the

What exactly do you mean with "composing profiles"? This fails:

guix environment --ad-hoc python
echo $GUIX_ENVIRONMENT
# /gnu/store/0d8vp2h…-profile
echo $PYTHONPATH
# /gnu/store/0d8vp2h…-profile/lib/python3.5/site-packages
guix environment --ad-hoc python-simplejson
echo $GUIX_ENVIRONMENT
# /gnu/store/5xgfisg…-profile
echo $PYTHONPATH
# /gnu/store/0d8vp2h…-profile/lib/python3.5/site-packages
python3 -s -c 'import simplejson'
# import error

('-s' avoids leaking packages from §HOME/.local/…)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue explanation

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]

Am 17.03.2018 um 11:07 schrieb Ricardo Wurmus:
>
>> - sys.prefix and sys.exec_prefix would still point to the store, not to
>>   the profile.  This might break Python appications expecting
>>   site-packages to be below sys.prefix.
> Is this an actual problem?  Do you know of applications that make this
> assumption?  If so, is this unfixable?

I'm not aware of any actual problem.

> I’m not too hopeful about this variant, but I’m rather ignorant about
> venvs.  My main concern is about whether it will still be possible for
> users to create venvs from a subset of their installed packages when we
> generate a pyvenv.cfg by default.

venvs never contain a "subset of installed packages". They either
include all system site-packages or none of them.

But as I've already written, generating a pyvenv.cfg for this case will
not work as we need it.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1874 bytes --]

[PATCH] gnu: python: Honor 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]


> I plan to implement option 1 by adding a "sitecustomize.py" (better
> than modify "site.py") into the python packages, and modify
> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".

Patch coming:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-2.7-python-3.6-Honor-GUIX_PYTHON_X_Y_SITE.patch --]
[-- Type: text/x-patch, Size: 4962 bytes --]

From d9c273c0ee8c5e87b12b37a325c649f8df808af3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 17 Mar 2018 18:46:55 +0800
Subject: [PATCH] gnu: python-2.7, python-3.6: Honor
 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

This replace the use of 'PYTHONPATH' as search path specification, as
suggested by Hartmut Goebel <h.goebel@crazy-compilers.com>.  See
<https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00178.html> for
details.

* gnu/packages/python.scm (python-guix-search-path-specification)
(python-guix-sitecustomize.py): New procedures.
(python-2.7, python-3.6):
[native-search-paths]: Use 'python-guix-search-path-specification'.
[arguments]: Add 'install-sitecustomize.py' phase.
---
 gnu/packages/python.scm | 67 ++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 58 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 6639e6c9e..2ce8db710 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -136,6 +136,41 @@
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1))
 
+(define (python-guix-search-path-specification version)
+  "Return the search path specification for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES"))
+         (files       (list (string-append
+                             "lib/python" major.minor "/site-packages"))))
+    (search-path-specification
+     (variable variable)
+     (files files))))
+
+(define (python-guix-sitecustomize.py version)
+  "Return the content of @file{sitecustomize.py} for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES")))
+    (format #f "# Append module search paths for guix packages to sys.path.
+import os
+import site
+
+SITE_PACKAGES = os.environ.get('~a')
+
+if SITE_PACKAGES is None:
+    SITE_PACKAGES = []
+else:
+    SITE_PACKAGES = SITE_PACKAGES.split(os.pathsep)
+
+for i in SITE_PACKAGES:
+    site.addsitedir(i)
+" variable)))
+
 (define-public python-2.7
   (package
     (name "python2")
@@ -304,6 +339,16 @@
                                      "/site-packages")))
                        (install-file tkinter.so target)
                        (delete-file tkinter.so)))))
+                #t)))
+          (add-after 'install 'install-sitecustomize.py
+            (lambda* (#:keys outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (sitedir (car (find-files out "^site-packages$"
+                                               #:directories #t))))
+                (with-output-to-file
+                    (string-append sitedir "/sitecustomize.py")
+                  (lambda ()
+                    (display ,(python-guix-sitecustomize.py version))))
                 #t))))))
     (inputs
      `(("bzip2" ,bzip2)
@@ -318,9 +363,7 @@
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+     (list (python-guix-search-path-specification version)))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -427,13 +470,19 @@ data types.")
                                          "-x" "(lib2to3|test/bad.*)"
                                          ,file)))
                               (find-files out "\\.py$")))
-                  (list '() '("-O") '("-OO"))))))))))
+                  (list '() '("-O") '("-OO"))))))
+           (replace 'install-sitecustomize.py
+             (lambda* (#:keys outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (sitedir (car (find-files out "^site-packages$"
+                                                #:directories #t))))
+                 (with-output-to-file
+                     (string-append sitedir "/sitecustomize.py")
+                   (lambda ()
+                     (display ,(python-guix-sitecustomize.py version))))
+                 #t)))))))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))))))
+     (list (python-guix-search-path-specification version)))))
 
 ;; Current 3.x version.
 (define-public python-3 python-3.6)
-- 
2.13.3


[-- Attachment #3: Type: text/plain, Size: 86 bytes --]


This targets 'core-updates' and will rebuild the world, I can't afford
to test it...

Re: [PATCH] gnu: python: Honor 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

Am 17.03.2018 um 12:18 schrieb 宋文武:
>> I plan to implement option 1 by adding a "sitecustomize.py" (better
>> than modify "site.py") into the python packages, and modify
>> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".

Sorry, do say, but does not work in a virtual environment, since
GUIX_PYTHON_X_Y_SITE_PACKAGES will be added unconditionally. Also I
assume this will execute site.main() twice.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1258 bytes --]

Re: [PATCH] gnu: python: Honor 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

[宋文武]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 17.03.2018 um 12:18 schrieb 宋文武:
>
>  I plan to implement option 1 by adding a "sitecustomize.py" (better
> than modify "site.py") into the python packages, and modify
> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".
>
> Sorry, do say, but does not work in a virtual environment, since GUIX_PYTHON_X_Y_SITE_PACKAGES will be added unconditionally. Also I assume this will execute site.main
> () twice.

Okay, and maybe it actually works?  :-)

It turns out that 'sitecustomize.py' will be imported (executed) at most
once, and won't be added unconditionally:

- with "include-system-site-packages = false", a python3 created venv
  will have a "sys.path" like:

  ['',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python36.zip',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6/lib-dynload',
  '/tmp/venv36/lib/python3.6/site-packages']

  Since "sitecustomize.py" is in
  "/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6/site-packages",
  it's not executed at all.


- with "include-system-site-packages = true", the python3 created venv
  will have a "sys.path" like:

  ['',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python36.zip',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6/lib-dynload',
  '/tmp/lib/python3.6/site-packages',
  '/gnu/store/pppycfhs5gc7dsx7g099l9p6ncw3m6d9-python-3.6.4/lib/python3.6/site-packages',
  ...... (entries added by GUIX_PYTHON_3_6_SITE_PACKAGES)]

  I think this is the wanted result.

  I haven't try "virtualenv" and python2 (need time to build...), but
  I guess the results should be the same?
  

Re: [PATCH] gnu: python: Honor 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

iyzsong@member.fsf.org (宋文武) writes:

>> I plan to implement option 1 by adding a "sitecustomize.py" (better
>> than modify "site.py") into the python packages, and modify
>> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".
>
> Patch coming:
>
> [patch with typo...]
>
>
> This targets 'core-updates' and will rebuild the world, I can't afford
> to test it...

Updated with typo fixed:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-2.7-python-3.6-Honor-GUIX_PYTHON_X_Y_SITE.patch --]
[-- Type: text/x-patch, Size: 4962 bytes --]

From d9c273c0ee8c5e87b12b37a325c649f8df808af3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 17 Mar 2018 18:46:55 +0800
Subject: [PATCH] gnu: python-2.7, python-3.6: Honor
 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

This replace the use of 'PYTHONPATH' as search path specification, as
suggested by Hartmut Goebel <h.goebel@crazy-compilers.com>.  See
<https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00178.html> for
details.

* gnu/packages/python.scm (python-guix-search-path-specification)
(python-guix-sitecustomize.py): New procedures.
(python-2.7, python-3.6):
[native-search-paths]: Use 'python-guix-search-path-specification'.
[arguments]: Add 'install-sitecustomize.py' phase.
---
 gnu/packages/python.scm | 67 ++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 58 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 6639e6c9e..2ce8db710 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -136,6 +136,41 @@
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1))
 
+(define (python-guix-search-path-specification version)
+  "Return the search path specification for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES"))
+         (files       (list (string-append
+                             "lib/python" major.minor "/site-packages"))))
+    (search-path-specification
+     (variable variable)
+     (files files))))
+
+(define (python-guix-sitecustomize.py version)
+  "Return the content of @file{sitecustomize.py} for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES")))
+    (format #f "# Append module search paths for guix packages to sys.path.
+import os
+import site
+
+SITE_PACKAGES = os.environ.get('~a')
+
+if SITE_PACKAGES is None:
+    SITE_PACKAGES = []
+else:
+    SITE_PACKAGES = SITE_PACKAGES.split(os.pathsep)
+
+for i in SITE_PACKAGES:
+    site.addsitedir(i)
+" variable)))
+
 (define-public python-2.7
   (package
     (name "python2")
@@ -304,6 +339,16 @@
                                      "/site-packages")))
                        (install-file tkinter.so target)
                        (delete-file tkinter.so)))))
+                #t)))
+          (add-after 'install 'install-sitecustomize.py
+            (lambda* (#:keys outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (sitedir (car (find-files out "^site-packages$"
+                                               #:directories #t))))
+                (with-output-to-file
+                    (string-append sitedir "/sitecustomize.py")
+                  (lambda ()
+                    (display ,(python-guix-sitecustomize.py version))))
                 #t))))))
     (inputs
      `(("bzip2" ,bzip2)
@@ -318,9 +363,7 @@
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+     (list (python-guix-search-path-specification version)))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -427,13 +470,19 @@ data types.")
                                          "-x" "(lib2to3|test/bad.*)"
                                          ,file)))
                               (find-files out "\\.py$")))
-                  (list '() '("-O") '("-OO"))))))))))
+                  (list '() '("-O") '("-OO"))))))
+           (replace 'install-sitecustomize.py
+             (lambda* (#:keys outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (sitedir (car (find-files out "^site-packages$"
+                                                #:directories #t))))
+                 (with-output-to-file
+                     (string-append sitedir "/sitecustomize.py")
+                   (lambda ()
+                     (display ,(python-guix-sitecustomize.py version))))
+                 #t)))))))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))))))
+     (list (python-guix-search-path-specification version)))))
 
 ;; Current 3.x version.
 (define-public python-3 python-3.6)
-- 
2.13.3

Re: PYTHONPATH issue explanation

[Hartmut Goebel]

Hi,

I agree with Ricardo: We first should agree on what we want to implement.

I created a pad at [1] for collecting all test-cases and the expected
results. Please add you test-cases there. Thanks!

[1] https://semestriel.framapad.org/p/guix-python-site-packages-test-cases

Am 17.03.2018 um 02:41 schrieb 宋文武:

> - "GUIX_PYTHON_X_Y_SITE_PACKAGES" […] is necessary for the "build" environment.
For the build environment we could easily work around using PYTHONPATH.
Since the build-system is clearly defined and does not interfere with
any user-definitions, this is save to do.

> - Avoid any environment variable for the "profile" environment.
>
>   We have a union "profile" for all the python packages, so environment
>   variables can be totally avoided with the help of "venv".
[…]
>  We only need to make the "profile"
>   a "venv" for python.  For python3, a simple "pyvenv.cfg" file is
>   enough, for python2 I guess we have to make a union or copy files like
>   what "virtualenv" does.

This would be a very elegant solution. Unfortunately this does not work
as shown in part 2 of my analysis, esp. point 4a.

>   > We could avoid GUIX-PYTHONHOME[23] if we stop resolving the symlinks
>   > at the correct point in iteration.
>
>   This is exactly what "venv" does! 

Unfortunately venv works quite different: system site-packages are
always taken from sys.base_exec. See part 3 of my analysis, esp. the
"pyvenv.cfg" section.

> I plan to implement option 1 by adding a "sitecustomize.py" (better
> than modify "site.py") into the python packages, and modify
> "search-path-specification" to use "GUIX_PYTHON_X_Y_SITE_PACKAGES".

When implementing this in sitecustomize.py, you will end up
re-implementing the complete venv mechanism.

When going the GUIX_PYTHON_X_Y_SITE_PACKAGES route, we should look where
the best place will be: Maybe site.PREFIXES, maybe
site.getsitepackages(), maybe site.venv().

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH issue explanation

[宋文武]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> I agree with Ricardo: We first should agree on what we want to
> implement.

Okay.

>
> I created a pad at [1] for collecting all test-cases and the expected
> results. Please add you test-cases there. Thanks!
>
> [1] https://semestriel.framapad.org/p/guix-python-site-packages-test-cases

I have append some text, it's available to all in realtime?
not sure how it works...

>
> Am 17.03.2018 um 02:41 schrieb 宋文武:
>
>> - "GUIX_PYTHON_X_Y_SITE_PACKAGES" […] is necessary for the "build" environment.
> For the build environment we could easily work around using PYTHONPATH.
> Since the build-system is clearly defined and does not interfere with
> any user-definitions, this is save to do.

Yes, but if "GUIX_PYTHON_X_Y_SITE_PACKAGES" does works (i hope so) in
the "profile" side, it's better to replace PYTHONPATH for consistent.

>
>> - Avoid any environment variable for the "profile" environment.
>>
>>   We have a union "profile" for all the python packages, so environment
>>   variables can be totally avoided with the help of "venv".
> […]
>>  We only need to make the "profile"
>>   a "venv" for python.  For python3, a simple "pyvenv.cfg" file is
>>   enough, for python2 I guess we have to make a union or copy files like
>>   what "virtualenv" does.
>
> This would be a very elegant solution. Unfortunately this does not work
> as shown in part 2 of my analysis, esp. point 4a.

A workaround for the broke case maybe tell the user to create a
"sitecustomize.py" in the created venv, and add the search paths of
profile himself.


I'd like do more tests with the GUIX_PYTHON_X_Y_SITE_PACKAGES option
(patch sent), hope it works :-)

Re: PYTHONPATH issue explanation

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

iyzsong@member.fsf.org (宋文武) writes:

> [...]
>
> I'd like do more tests with the GUIX_PYTHON_X_Y_SITE_PACKAGES option
> (patch sent), hope it works :-)

Hello, I have write a shell script to do some tests, it looks good to me!


Updated 'GUIX_PYTHON_X_Y_SITE_PACKAGES' patch, target 'core-updates' at
commit 171a117c (you also have to comment out the "manual-database"
profile hook in the "guix/profiles.scm", as it's broken in that commit):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-2.7-python-3.6-Honor-GUIX_PYTHON_X_Y_SITE.patch --]
[-- Type: text/x-patch, Size: 4858 bytes --]

From d807306d02aab0a84de4fa3ff457a5b97ac15520 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 17 Mar 2018 18:46:55 +0800
Subject: [PATCH] gnu: python-2.7, python-3.6: Honor
 'GUIX_PYTHON_X_Y_SITE_PACKAGES'.

This replace the use of 'PYTHONPATH' as search path specification, as
suggested by Hartmut Goebel <h.goebel@crazy-compilers.com>.  See
<https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00178.html> for
details.

* gnu/packages/python.scm (python-guix-search-path-specification)
(python-guix-sitecustomize.py): New procedures.
(python-2.7, python-3.6):
[native-search-paths]: Use 'python-guix-search-path-specification'.
[arguments]: Add 'install-sitecustomize.py' phase.
---
 gnu/packages/python.scm | 65 +++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 57 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index f3a75c30e..45de8c527 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -136,6 +136,41 @@
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1))
 
+(define (python-guix-search-path-specification version)
+  "Return the search path specification for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES"))
+         (files       (list (string-append
+                             "lib/python" major.minor "/site-packages"))))
+    (search-path-specification
+     (variable variable)
+     (files files))))
+
+(define (python-guix-sitecustomize.py version)
+  "Return the content of @file{sitecustomize.py} for python VERSION."
+  (let* ((major.minor (version-major+minor version))
+         (variable    (string-append
+                       "GUIX_PYTHON_"
+                       (string-replace-substring major.minor "." "_")
+                       "_SITE_PACKAGES")))
+    (format #f "# Append module search paths for guix packages to sys.path.
+import os
+import site
+
+SITE_PACKAGES = os.environ.get('~a')
+
+if SITE_PACKAGES is None:
+    SITE_PACKAGES = []
+else:
+    SITE_PACKAGES = SITE_PACKAGES.split(os.pathsep)
+
+for i in SITE_PACKAGES:
+    site.addsitedir(i)
+" variable)))
+
 (define-public python-2.7
   (package
     (name "python2")
@@ -304,6 +339,16 @@
                                      "/site-packages")))
                        (install-file tkinter.so target)
                        (delete-file tkinter.so)))))
+                #t)))
+          (add-after 'install 'install-sitecustomize.py
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (sitedir (car (find-files out "^site-packages$"
+                                               #:directories? #t))))
+                (with-output-to-file
+                    (string-append sitedir "/sitecustomize.py")
+                  (lambda ()
+                    (display ,(python-guix-sitecustomize.py version))))
                 #t))))))
     (inputs
      `(("bzip2" ,bzip2)
@@ -318,9 +363,7 @@
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+     (list (python-guix-search-path-specification version)))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -428,13 +471,19 @@ data types.")
                                          ,file)))
                               (find-files out "\\.py$")))
                   (list '() '("-O") '("-OO")))
+                 #t)))
+           (replace 'install-sitecustomize.py
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (sitedir (car (find-files out "^site-packages$"
+                                                #:directories? #t))))
+                 (with-output-to-file
+                     (string-append sitedir "/sitecustomize.py")
+                   (lambda ()
+                     (display ,(python-guix-sitecustomize.py version))))
                  #t)))))))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))))))
+     (list (python-guix-search-path-specification version)))))
 
 ;; Current 3.x version.
 (define-public python-3 python-3.6)
-- 
2.13.3


[-- Attachment #3: Type: text/plain, Size: 31 bytes --]



And here is the test script:

[-- Attachment #4: test-python-site-packages.sh --]
[-- Type: application/x-sh, Size: 3483 bytes --]

[-- Attachment #5: Type: text/plain, Size: 32 bytes --]



Do I miss something?  Thanks!

Re: PYTHONPATH issue explanation

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 8558 bytes --]

Hi Hartmut,

Awesome analysis!  Thank you for taking point on this.  I will offer
some feedback.  I hope it is useful.

The short version is: I think Python should let us explicitly tell it
where its system site directory is.  If Python provided such a feature,
then I think we could use it and avoid all these problems.  I think this
would be better than modifying the heuristics that Python uses for
finding its system site during start-up (although I think that is a good
back-up plan), since those heuristics are complicated and difficult to
control.  It would just be simpler if we could explicitly tell Python
where its site directory is, instead of indirectly arranging for Python
to find its site directory via its module-lookup Rube-Goldberg machine.

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> This python interpreter does not find the site-packages in GUIX_PROFILE
> since site-packages are search relative to "sys.base_prefix" (which is
> the same as "sys.prefix" except in virtual environments).
> "sys.base_prefix" is determined based on the executable's path (argv[0])
> by resolving all symlinks.

I am familiar with this problem.  Any time you want to deploy Python and
its libraries by building up a symlink tree, and you put Python in a
part of the file system that lives far away from the libraries
themselves, Python will punish you cruelly with this behavior.  It is no
fun at all.  :-( You always have to come up with silly hacks to work
around it, and those hacks don't work generally in every case.

Question: Why does Python insist on canonicalizing its executable path?
It always seemed to me like if Python just used the original path, these
problems would not occur.  People who use symlink trees to deploy Python
would be happy.  Perhaps I am missing some information.  What is the
intent behind Python's decision to canonicalize the executable path?
What problems occur if Python doesn't do that?

> The python interpreter assumes "site-packages" to be relative to "where
> python is installed" - called "sys.base_prefix" (which is the same as
> "sys.prefix" except in virtual environments). "sys.base_prefix" is
> determined based on the executable's path (argv[0]) by resolving all
> symlinks. For Guix this means: "sys.base_prefix" will always point to
> /gnu/store/…-python-X.Y, not to GUIX_PROFILE. Thus the site-packages
> installed into the guix profile will not be found.

Yes.  This is a problem.  As you know, this heuristic fails
spectacularly when you try to deploy Python in a symlink tree.

Question: Why does Python not supply a way to "inject" the system site
directory?  In Guix-deployed systems, we are the masters of reality.  We
control ALL the paths.  We can tell Python exactly where its "system
site" is - we can build a symlink tree of its system site in the store
and then tell Python to use that site specifically.  For example, if
Python would let us specify this path via a PYTHON_SYSTEM_SITE
environment variable, then I think it would solve many (all?) of our
problems.  Perhaps this is similar to what you are suggesting regarding
GUIX_PYTHON_X.Y_SITE_PACKAGES and GUIX_PYTHONHOME_X.Y.

> This is why we currently (mis-) use PYTHONPATH: To make the
> site-packages installed into the guix profile available.

I agree that this is a mis-use.  People do it because Python doesn't
provide any better way.  And then people find out about all its terrible
down-sides, like for example the fact that .pth files will not be
processed if they appear on the PYTHONPATH.  And then they do stuff like
hack site.py to walk the PYTHONPATH and evaluate all the .pth files,
which is gross but sort of works.  Just thinking about the pain I have
experienced with this stuff makes my blood boil.

> no. 2
> suggests using a mechanism already implemented in python: Setting
> "PYTHONHOME" will make the interpreter to use this as "sys.base_prefix"
> unconditionally. Again there is only one PYTHONHOME variable for all
> versions of python (designed by upstream). We could work around this
> easily (while keeping upstream compatibility) by using
> GUIX-PYTHONHOME-X.Y, to be evaluated just after PYTHONHOME.

Are there legitimate use cases where a user wants to set their own
PYTHONHOME?  If so, would our use of PYTHONHOME prevent them from doing
that?  If so, that seems bad.

In the past, I have used PYTHONUSERBASE (or maybe it was PYTHONUSERSITE,
I can't remember exactly which) to make Python find libraries in a
symlink tree.  However, because that is intended for users to use, I
don't think it's a good solution for us here.  If we co-opt these
environment variables, then users would not be able to use them.

> The drawback is: This is implemented using an environment variable,
> which might not give the expected results in all cases. E.g. running
> /gnu/store/…-profile/bin/python will not load the site-packages of that
> profile. Also there might be issues implementing virtual environments.
> (Thinking about this, I'm quite sure there will. Ouch!)

I wouldn't be surprised if that's true, but right now, I can't think of
any specific virtualenv-related problems that would occur by using
PYTHONHOME.

> no.3
> suggests changing the way the python interpreter is resolving symlinks
> when searching for "sys.base_prefix". The idea is to stop "at the profile".
>
> The hard part of this is to determine "at the profile". Also this needs
> a larger patch. But if we manage to implement this, it would be perfect.
> I could contribute a draft for this implemented in Python. The
> C-implementation needs to be done by some C programmer.

This seems a little tricky, mainly because it's going to rely again on
heuristics that may not always be accurate.  As I mentioned above, in
Guix we are the masters of reality, so why can't we just tell Python
exactly where its system site path is?  If Python needs to be taught how
to be informed of such things, perhaps that is the patch we should
write: a patch that enables us to tell Python exactly where its system
site directory will be found.

> Which way should we go?

I think we should figure out a way to tell Python EXACTLY where its
system site directory is.  If that isn't viable, then I think the next
best thing will be to adjust the site-finding heuristics (your proposal
No. 3).

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> As it stands now, the venv-hack is not a valid solution. It may be the basis
> for another solution, tough.

I agree.  We need a solution that allows users to use virtualenv the way
they would normally on any other foreign distro, if they want to.

> 1. How could GUIX-PYTHON-X.Y-SITE-PACKAGES be implemented?
> =============================================================
>
> [...]
>
> 2. How could GUIX-PYTHONHOME-X.Y be implemented?
> =================================================

How do these two methods (GUIX-PYTHON-X.Y-SITE-PACKAGES
vs. GUIX-PYTHONHOME-X.Y) differ?  They seem to serve basically the same
purpose.


> 3. How to avoid GUIX-PYTHONHOME[23]?
> =========================================
>
> We could avoid GUIX-PYTHONHOME[23] if we stop resolving the symlinks at
> the correct point in iteration.
>
> [...]
> 
> Drawbacks:
>
> - More complicated patch.
>
> - More comparison within a look, this will slow down start-up a bit.
>
> Open questions:
>
> - Which are the correct paths to check to stop iteration?
> - How to handle the "pythonX" -> "pythonX.Y" link?
> - How to handle "python-wrapper", which links python -> python3

Instead of modifying Python's heuristics for finding its site, it'd be
better if Python just exposed a way for us to explicitly tell it where
its site directory is.

However, if we really want to modify the heuristics, I can think of some
possible ideas for how to do it:

* Don't canonicalize the path in the first place.
* Stop just before the first path that is in the store.
* Stop at the first path that is in the store.
* Stop at a path that matches a special pattern that we control,
  like "guix-python-site" or something.  We could create 

> 4. Path-handling in Python's start-up sequence

As you've shown, the way Python handles paths when it starts up is quite
complicated.  This is another reason why I would prefer not to change
the heuristics, but instead to expose a way for us to explicitly tell
Python where its site is.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

PYTHONPATH - let's systematically tame the baest

[Hartmut Goebel]

Hi,

let's pick up on this issue and systematically design the test-cases to
benchmark the proposed solutions. I already prepared a test-script to
simplify this and will provide a full description as later.

**Please comment if any relevant case is missing or if any case can be
skipped**

1) Test-cases

For all environments (see below) these cases must give the expected
output - which is defined by what a "foreign distribution's" python
would do:
- "installed" python
- venv with and without --system-site-packages
- stacked venv with and without --system-site-packages

2) Environments to be tested.

The proposed solution must pass the test-suite in all of these environments:

2.1 guix environment:

     guix environment --ad-hoc python -- python3 testit
      --> Expected outcome: site-packages from GUIX_ENVIRONEMENT

2.2 guix environment with container:

     guix environment -C --ad-hoc python -- python3 testit
      --> Expected outcome: site-packages from GUIX_ENVIRONEMENT

2.3 Installed package *without setting the environment variables!*

     guix package -i python && ~/.guix-profile/bin/python3 testit
    --> Expected outcome: site-packages from ~/.guix-profile/
    --> Shall this work, too? Is it nice-to-have or useless?

2.4 running from /gnu/store (directly)

    $(readlink -f ~/.guix-profile/bin/python3) testit
    --> Expected outcome: site-packages from /gnu/store
    --> What is the expected outcome? What is the expected

2.5 running from /gnu/store (via link)

    ln -s $(readlink -f ~/.guix-profile/bin/python3)
/tmp/test-guix-pythonA.exe ;
    /tmp/test-guix-pythonA.exe testit
    --> Expected outcome: site-packages from /gnu/store

2.6 Installed in GuixSD

    --> Do we need to test this? Or is this already covered by one of
the other cases?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH - let's systematically tame the baest

[宋文武]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,

Hello!

>
> let's pick up on this issue and systematically design the test-cases to
> benchmark the proposed solutions. I already prepared a test-script to
> simplify this and will provide a full description as later.
>
> **Please comment if any relevant case is missing or if any case can be
> skipped**
>
> 1) Test-cases
>
> For all environments (see below) these cases must give the expected
> output - which is defined by what a "foreign distribution's" python
> would do:
> - "installed" python
> - venv with and without --system-site-packages
> - stacked venv with and without --system-site-packages

We should consider both python2 and python3, and virtual environments
created by the 'virtualenv' package.

>
> 2) Environments to be tested.
>
> The proposed solution must pass the test-suite in all of these environments:
>
> 2.1 guix environment:
>
>      guix environment --ad-hoc python -- python3 testit
>       --> Expected outcome: site-packages from GUIX_ENVIRONEMENT
>
> 2.2 guix environment with container:
>
>      guix environment -C --ad-hoc python -- python3 testit
>       --> Expected outcome: site-packages from GUIX_ENVIRONEMENT
>
> 2.3 Installed package *without setting the environment variables!*
>
>      guix package -i python && ~/.guix-profile/bin/python3 testit
>     --> Expected outcome: site-packages from ~/.guix-profile/
>     --> Shall this work, too? Is it nice-to-have or useless?

Yeah, it's nice to have (to avoid introducing an environment variable),
but not necessary.

>
> 2.4 running from /gnu/store (directly)
>
>     $(readlink -f ~/.guix-profile/bin/python3) testit
>     --> Expected outcome: site-packages from /gnu/store
>     --> What is the expected outcome? What is the expected

I think if we use environment variable to specify all the site-packages,
it should be the same as running from profile.  It maybe different if we
resolve site-packages by the executable location...

>
> 2.5 running from /gnu/store (via link)
>
>     ln -s $(readlink -f ~/.guix-profile/bin/python3)
> /tmp/test-guix-pythonA.exe ;
>     /tmp/test-guix-pythonA.exe testit
>     --> Expected outcome: site-packages from /gnu/store

True when we're not use the environment variable.

>
> 2.6 Installed in GuixSD
>
>     --> Do we need to test this? Or is this already covered by one of
> the other cases?

For this, there is nothing special about GuixSD.


Had you review my 'GUIX_PYTHON_X_Y_SITE_PACKAGES' patch?  I think it's
enough to support both python2 and python3 in the same profile:

http://lists.gnu.org/archive/html/guix-devel/2018-03/msg00238.html


Thanks!

Re: PYTHONPATH - let's systematically tame the baest

[Hartmut Goebel]

Am 17.04.2018 um 03:47 schrieb 宋文武:
> Had you review my 'GUIX_PYTHON_X_Y_SITE_PACKAGES' patch?  I think it's
> enough to support both python2 and python3 in the same profile:

Yes I had. But we should first decide on he expected results, then
decide which solution is adequate :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: PYTHONPATH - let's systematically tame the baest

[Ricardo Wurmus]

Hi Hartmut,

> let's pick up on this issue and systematically design the test-cases to
> benchmark the proposed solutions. I already prepared a test-script to
> simplify this and will provide a full description as later.

Thank you for picking up the work on this!

In all of the tests do we only care about the reported value of
site-packages?  Should the tests include loading non-trivial packages
that have other Python packages as dependencies?

> 2.3 Installed package *without setting the environment variables!*
>
>  guix package -i python && ~/.guix-profile/bin/python3 testit
>  --> Expected outcome: site-packages from ~/.guix-profile/
>  --> Shall this work, too? Is it nice-to-have or useless?

2.3b is to install the package into a separate profile with

    guix package -p /path/to/somewhere -i python

> 2.4 running from /gnu/store (directly)
>
>   $(readlink -f ~/.guix-profile/bin/python3) testit
>  --> Expected outcome: site-packages from /gnu/store
>   --> What is the expected outcome? What is the expected
>
> 2.5 running from /gnu/store (via link)
>
>   ln -s $(readlink -f ~/.guix-profile/bin/python3)
> /tmp/test-guix-pythonA.exe ;
>  /tmp/test-guix-pythonA.exe testit
>  --> Expected outcome: site-packages from /gnu/store

I think these two cases should yield the same result.

> 2.6 Installed in GuixSD
>
>  --> Do we need to test this? Or is this already covered by one of
> the other cases?

I don’t think we need to test this as GuixSD does not have any special
behaviour for Python and the system profile is just another profile.
This would be the same as 2.3b.

--
Ricardo