Re: Anomalies in Python search-path

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7576 bytes --]


Hi Hartmut! I was hoping you'd join this thread!

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
>> One thing which I discovered while testing pip on Guix was that
>> depending on how you use Python on Guix the PYTHONPATH differs:
>
> I analysed this problem and found set we have another serious problem
> here. But good news: There is an easy solution for both problems.
>
> Result of the analysis (as explained below): We must not extend
> PYTHONPATH for adding the profile's site-packages directory.

Right! site.py will put prepend any previously set PYTHONPATH to the one
it forms. If a site-packages is present there it will appear before the
user site location, which is not good. I had missed this particular
problem before by failing to restart my user session after installing
python (the custom PYTHONPATH we set was not yet in effect).

>
> Proposed solution: The standard library module "site" already contains a
> place where we can hook in:
>
>    # Prefixes for site-packages; add additional prefixes like /usr/local
> here
>     PREFIXES = [sys.prefix, sys.exec_prefix]
>
> So we simply prepend $GUIX_ENVIRONMENT to this list:
>
>     PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
> sys.exec_prefix]
>

I think your approach is correct! Any directory we add to PREFIXES will
get picked by the `addsitepackages` function in site.py, and this
function is called *following* the `addusersitepackages` function in
site.main(), which means those entries will appear after the user site
location in sys.path, which is what we want.

What is currently responsible for exporting PYTHONPATH to
".guix-profile/lib/python2.7/site-packages" ? It looks like it might be
the `native-search-paths' fields of our python-2.7 package?

    (native-search-paths
     (list (search-path-specification
            (variable "PYTHONPATH")
            (files '("lib/python2.7/site-packages")))))

This should rather be added with a patch adding it to the PREFIXES
variable of the site.py module, as you suggested.

This is the value of PYTHONPATH in my profile, after installing python@2
and re-login:

$ echo $PYTHONPATH
/home/maxim/.guix-profile/lib/python2.7/site-packages

Also, if we no longer rely on manipulating the PYTHONPATH directly, we
should also remove the "python-2.7-site-prefixes.patch" patch, which
adds /gnu/store entries found in PYTHONPATH to the PREFIXES variable.

> Prepending to speed up searches, since within an environment almost all
> site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
> $GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
> and the non-existing path will be skipped out automatically
>
> This would solve both the problem Maxim described and the problem
> described below, since the Guix site-packages behave exactly like any
> other global site-packages directory.
>
> When following this proposal, we *may* even remove some "wrapping" stuff
> in the python-build-system. But this has to be investigated first.
>

Yes, it will be interesting to see if the changes proposed here would
remove the need for the wrap phase. I hope it does :).

>
> Analysis (proof see below):
>
> The python interpreter was two command line options, which are rarely
> used, though:
>
> -E ignores environment variables, esp. PYTHONPATH
> -S Disable  the  import  of the module site and the site-dependent
> manipulations of sys.path.
>
> This means:
> a) When passing -E, the Guix environment's site-packages are ignored
> (together with PYTHONPATH), while they should not.
> b) When passing -S, the Guix environment's site-packages should be
> ignored, but is not (since it is specified in $PYTHONPATH, which is not
> ignored)
>
> Conclusion: We should must not use PYTHONPATH to specify the Guix's
> environment's site-package directory.
>

Right. In general, we can generalize the PYTHONPATH rule to "we must not use
PYTHONPATH" ;)

>
> Analysis details:
>
> (guix)$ which python3
> /gnu/store/zcnb…-profile/bin/python3
>
>
> Without any options:
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    should be in sys.path
>
> (base)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
> -E     Ignore environment variables  like  PYTHONPATH  and  PYTHONHOME
>        that modify the behavior of the interpreter.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    should be in sys.path
>
> (base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
>
> -S = Disable  the  import  of the module site and the site-dependent
>      manipulations of sys.path that it entails.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    sould *not* be in sys.path
>
> (base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4/
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
>
>
> (guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload

Thanks for the feedback and detailed analysis!

Here's what I will attempt from now:

In the python-2.7 variable:

1. Remove the current "python-2.7-site-prefixes" patch.

2. Apply a new patch to site.py to add "$HOME/.guix-profile"
    as well as "$GUIX_ENVIRONMENT".

3. Remove the 'native-search-paths' field.

As a bonus, now that we won't be using PYTHONPATH anymore, there
shouldn't be any crosstalk between Python 2 & Python 3, so we ought to
be able to install both in the same profile/environment at the same time
(currently installing Python 3 "updates" Python 2).

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]