[PATCH] Update python-pip to 9.0.1

[PATCH] Update python-pip to 9.0.1

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Hello Guix!

This is my 2nd and hopefully correct attempt at updating python-pip to
9.0.1. Test inputs were removed as the tarball available from pypi is
stripped from any test.

It also incorporates suggestions that were previously made [1]; for
example we no longer need to use setuptools as an input since our Python
package definitions were recently updated to always be built with
pip/setuptools (the ensurepip flag).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-pip-Update-to-9.0.1.patch --]
[-- Type: text/x-patch, Size: 1711 bytes --]

From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Mon, 28 Nov 2016 16:30:07 -0800
Subject: [PATCH] gnu: python-pip: Update to 9.0.1

* gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1.
---
 gnu/packages/python.scm | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5faebae3d9..028cda6292 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -6773,26 +6773,21 @@ library.")
 (define-public python-pip
   (package
     (name "python-pip")
-    (version "8.0.2")
+    (version "9.0.1")
     (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pip" version))
        (sha256
         (base32
-         "08cm8d4228fj0qnrysy3qv1a6022zr3dcs25amd14lgxil6vvx26"))))
+         "03clr9c1dih5n9c00c592zzvf6r1ffimywkaq9agcqdllzhl7wh9"))))
     (build-system python-build-system)
-    (native-inputs
-      `(;; Tests
-        ("python-virtualenv" ,python-virtualenv)
-        ("python-mock" ,python-mock)
-        ("python-pytest" ,python-pytest)
-        ("python-scripttest" ,python-scripttest)))
+    (arguments
+     '(#:tests? #f))          ; there are no tests in the pypi archive.
     (home-page "https://pip.pypa.io/")
-    (synopsis
-      "Package manager for Python software")
+    (synopsis "Package manager for Python software")
     (description
-      "Pip is a package manager for Python software, that finds packages on the
+     "Pip is a package manager for Python software, that finds packages on the
 Python Package Index (PyPI).")
     (license license:expat)))
 
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 2189 bytes --]


One thing which I discovered while testing pip on Guix was that
depending on how you use Python on Guix the PYTHONPATH differs:

# PYTHONPATH in an environment
$ guix environment python-wrapper
[env]$ python3 -c 'import sys; print(sys.path)'
['',
'/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
'/home/maxim/.local/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']
[env]$ pip install --user --upgrade setuptools
Collecting setuptools
  Using cached setuptools-32.3.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-20.10.1


# PYTHONPATH when installed in your user profile
$ guix package -i python-wrapper
$ python3 -c 'import sys; print(sys.path)'
['',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
'/home/maxim/.local/lib/python3.5/site-packages',
'/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']
$ pip3 uninstall setuptools
...
$ pip3 --user --upgrade setuptools
Collecting setuptools
  Using cached setuptools-32.3.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-32.3.1


Conclusion: When using python in a "guix environment", the *system*
site-packages directory appears _before_ the *user* site-packages
directory in the PYTHONPATH, which is wrong and causes pip to not work
as intended. [2]

[1] https://lists.gnu.org/archive/html/guix-devel/2016-11/msg01242.html
[2] https://pip.pypa.io/en/stable/user_guide/#user-installs

Thanks,

Maxim

Re: [PATCH] Update python-pip to 9.0.1

[Ludovic Courtès]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> This is my 2nd and hopefully correct attempt at updating python-pip to
> 9.0.1. Test inputs were removed as the tarball available from pypi is
> stripped from any test.
>
> It also incorporates suggestions that were previously made [1]; for
> example we no longer need to use setuptools as an input since our Python
> package definitions were recently updated to always be built with
> pip/setuptools (the ensurepip flag).

Sounds good.

> From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001
> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date: Mon, 28 Nov 2016 16:30:07 -0800
> Subject: [PATCH] gnu: python-pip: Update to 9.0.1
>
> * gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1.

I applied and expounded the commit log a bit.

One thing which I discovered while testing pip on Guix was that
depending on how you use Python on Guix the PYTHONPATH differs:

> # PYTHONPATH in an environment
> $ guix environment python-wrapper
> [env]$ python3 -c 'import sys; print(sys.path)'
> ['',
> '/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages',
> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
> '/home/maxim/.local/lib/python3.5/site-packages',
> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']

The above command creates an environment containing the dependencies of
python-wrapper.

Instead, you probably wanted to run this:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix environment --ad-hoc python-wrapper -- python -c 'import sys; print(sys.path)'
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/bxr0cw01qhgd9r741a6nnm5zwx3y12rr-profile.drv
   /gnu/store/l91wf51q5ravikki293baljfx0bacqm8-info-dir.drv
   /gnu/store/a4mz2l85sc11w6llh56zxlcc1bmfmzsr-ca-certificate-bundle.drv
   /gnu/store/3z9hsnrwvqw70h0ssxqw0vy5y2dvzcj4-fonts-dir.drv
['', '/gnu/store/w63slncr9gyzjrs09n8ls81xac9fs804-profile/lib/python3.5/site-packages', '/home/ludo/.guix-profile/lib/python3.5/site-packages', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python35.zip', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/plat-linux', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/lib-dynload', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages']
--8<---------------cut here---------------end--------------->8---

… although it doesn’t make much of a difference in this particular case
since python-wrapper has python as its dependency.

> Conclusion: When using python in a "guix environment", the *system*
> site-packages directory appears _before_ the *user* site-packages
> directory in the PYTHONPATH, which is wrong and causes pip to not work
> as intended. [2]

‘guix environment’ without --pure adds its own entries to the front of
the search paths, which is why we observe this behavior.

In most cases this is what we want (we want to create an environment
that prevails over the user’s profile), but I can see why this is
problematic here.

I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted
anyway, but then again I’m no Python expert.  :-)

Thoughts?

Thanks!

Ludo’.

Re: [PATCH] Update python-pip to 9.0.1

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 3840 bytes --]

Hi Ludovic!

Apologies for my delayed answer.

ludo@gnu.org (Ludovic Courtès) writes:

>> From bfd91a59acdf3105505d7dd8483bb9cb97137cbf Mon Sep 17 00:00:00 2001
>> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> Date: Mon, 28 Nov 2016 16:30:07 -0800
>> Subject: [PATCH] gnu: python-pip: Update to 9.0.1
>>
>> * gnu/packages/python.scm (python-pip, python2-pip): Update to 9.0.1.
>
> I applied and expounded the commit log a bit.

Thanks!

>
> One thing which I discovered while testing pip on Guix was that
> depending on how you use Python on Guix the PYTHONPATH differs:
>
>> # PYTHONPATH in an environment
>> $ guix environment python-wrapper
>> [env]$ python3 -c 'import sys; print(sys.path)'
>> ['',
>> '/gnu/store/ar7k7ds90ikxv40a6lif6jv2g39l7mls-profile/lib/python3.5/site-packages',
>> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python35.zip',
>> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5',
>> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/plat-linux',
>> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/lib-dynload',
>> '/home/maxim/.local/lib/python3.5/site-packages',
>> '/gnu/store/b7zbbavbk1jv40b9virwmglck9bdj43a-python-3.5.2/lib/python3.5/site-packages']
>
> The above command creates an environment containing the dependencies of
> python-wrapper.
>
> Instead, you probably wanted to run this:
>
> $ ./pre-inst-env guix environment --ad-hoc python-wrapper -- python -c
> 'import sys; print(sys.path)'

Right! Thanks for pointing that out.

> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
> The following derivations will be built:
>    /gnu/store/bxr0cw01qhgd9r741a6nnm5zwx3y12rr-profile.drv
>    /gnu/store/l91wf51q5ravikki293baljfx0bacqm8-info-dir.drv
>    /gnu/store/a4mz2l85sc11w6llh56zxlcc1bmfmzsr-ca-certificate-bundle.drv
>    /gnu/store/3z9hsnrwvqw70h0ssxqw0vy5y2dvzcj4-fonts-dir.drv
> ['', '/gnu/store/w63slncr9gyzjrs09n8ls81xac9fs804-profile/lib/python3.5/site-packages', '/home/ludo/.guix-profile/lib/python3.5/site-packages', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python35.zip', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/plat-linux', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/lib-dynload', '/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages']
>
> … although it doesn’t make much of a difference in this particular case
> since python-wrapper has python as its dependency.
>
>> Conclusion: When using python in a "guix environment", the *system*
>> site-packages directory appears _before_ the *user* site-packages
>> directory in the PYTHONPATH, which is wrong and causes pip to not work
>> as intended. [2]
>
> ‘guix environment’ without --pure adds its own entries to the front of
> the search paths, which is why we observe this behavior.
>

Is this specific to 'guix environment' with --pure vs without --pure?
I'll test it out!

Also, would this behavior be observed for any 3rd
party package management system, say, the one for ruby, or <some>
language? If so maybe we could document this behavior somewhere.

> In most cases this is what we want (we want to create an environment
> that prevails over the user’s profile), but I can see why this is
> problematic here.
>
> I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted
> anyway, but then again I’m no Python expert.  :-)
>

Yes, it is a bit convoluted. But with environments being so convenient,
I can think that people other than me would attempt this ;)

Thanks for the reply!

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH] Update python-pip to 9.0.1

[Ludovic Courtès]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> ‘guix environment’ without --pure adds its own entries to the front of
>> the search paths, which is why we observe this behavior.
>>
>
> Is this specific to 'guix environment' with --pure vs without --pure?

Without ‘--pure’.  Conversely, ‘--pure’ clears the environment
variables altogether:

  https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-environment.html

> Also, would this behavior be observed for any 3rd
> party package management system, say, the one for ruby, or <some>
> language? If so maybe we could document this behavior somewhere.

What would you add to the document of ‘guix environment’?  It seems
quite clear to me at first sight.

>> In most cases this is what we want (we want to create an environment
>> that prevails over the user’s profile), but I can see why this is
>> problematic here.
>>
>> I would argue that using ‘pip’ in ‘guix environment’ is a bit convoluted
>> anyway, but then again I’m no Python expert.  :-)
>>
>
> Yes, it is a bit convoluted. But with environments being so convenient,
> I can think that people other than me would attempt this ;)

Yes I can imagine.  :-)

Thanks,
Ludo’.

Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1)

[Hartmut Goebel]

Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
> One thing which I discovered while testing pip on Guix was that
> depending on how you use Python on Guix the PYTHONPATH differs:

I analysed this problem and found set we have another serious problem
here. But good news: There is an easy solution for both problems.

Result of the analysis (as explained below): We must not extend
PYTHONPATH for adding the profile's site-packages directory.

Proposed solution: The standard library module "site" already contains a
place where we can hook in:

   # Prefixes for site-packages; add additional prefixes like /usr/local
here
    PREFIXES = [sys.prefix, sys.exec_prefix]

So we simply prepend $GUIX_ENVIRONMENT to this list:

    PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
sys.exec_prefix]

Prepending to speed up searches, since within an environment almost all
site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
$GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
and the non-existing path will be skipped out automatically

This would solve both the problem Maxim described and the problem
described below, since the Guix site-packages behave exactly like any
other global site-packages directory.

When following this proposal, we *may* even remove some "wrapping" stuff
in the python-build-system. But this has to be investigated first.


Analysis (proof see below):

The python interpreter was two command line options, which are rarely
used, though:

-E ignores environment variables, esp. PYTHONPATH
-S Disable  the  import  of the module site and the site-dependent
manipulations of sys.path.

This means:
a) When passing -E, the Guix environment's site-packages are ignored
(together with PYTHONPATH), while they should not.
b) When passing -S, the Guix environment's site-packages should be
ignored, but is not (since it is specified in $PYTHONPATH, which is not
ignored)

Conclusion: We should must not use PYTHONPATH to specify the Guix's
environment's site-package directory.


Analysis details:

(guix)$ which python3
/gnu/store/zcnb…-profile/bin/python3


Without any options:

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   should be in sys.path

(base)$ python3 -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages

(guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages


-E     Ignore environment variables  like  PYTHONPATH  and  PYTHONHOME
       that modify the behavior of the interpreter.

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   should be in sys.path

(base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/USER/.local/lib/python3.4/site-packages
/usr/lib64/python3.4/site-packages
/usr/lib/python3.4/site-packages

(guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
/home/USER/.local/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages



-S = Disable  the  import  of the module site and the site-dependent
     manipulations of sys.path that it entails.

-> /home/USER/.local/lib/python3.5/site-packages and
   /gnu/store/zcnb…-profile/lib/python3.5/site-packages
   sould *not* be in sys.path

(base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'

/usr/lib64/python34.zip
/usr/lib64/python3.4/
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload


(guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'

/gnu/store/zcnb…-profile/lib/python3.5/site-packages
/gnu/store/alk9…-python-3.5.2/lib/python35.zip
/gnu/store/alk9…-python-3.5.2/lib/python3.5/
/gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
/gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Anomalies in Python search-path (was: [PATCH] Update python-pip to 9.0.1)

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 2344 bytes --]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> ‘guix environment’ without --pure adds its own entries to the front of
>>> the search paths, which is why we observe this behavior.
>>>

After more testing, the Python search-path we form is always problematic
for pip since the system site-packages will always appear in front of the
user site location (~/.local/...).

My last reporting that pip was working correctly in non-environment was
wrong because I had failed to logout/login after installing the python
package; after doing so, the behavior is the same as in an environment:
the user site location (~/.local/...) comes after site-packages rather
than before, as would normally be the case. Apparently this is due to
setting a site-packages location with the environment variable
PYTHONPATH, which has precedence over the user site location.

All of these precedence rules ared defined by the module site.py which
comes with Python. It can print information about the Python search
paths with:

$ python -m site

Normally the user site-packages location should come before any other
site-packages location so that user installed Python packages can take
precedence over packages visible at the system level (or in our profile
in the the case Guix).

>>
>> Is this specific to 'guix environment' with --pure vs without --pure?
>
> Without ‘--pure’.  Conversely, ‘--pure’ clears the environment
> variables altogether:
>
>   https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-environment.html
>
>> Also, would this behavior be observed for any 3rd
>> party package management system, say, the one for ruby, or <some>
>> language? If so maybe we could document this behavior somewhere.
>
> What would you add to the document of ‘guix environment’?  It seems
> quite clear to me at first sight.
>

I've re-read the documentation about it and yes, it is quite clear about
what it does. I'll revisit any problems with environments later, after
the root problem (which lies in the way we manage the Python search
paths in our Python package definition/python build system) is fixed.

Thanks for you time, and for Guix!

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Anomalies in Python search-path

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 7576 bytes --]


Hi Hartmut! I was hoping you'd join this thread!

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
>> One thing which I discovered while testing pip on Guix was that
>> depending on how you use Python on Guix the PYTHONPATH differs:
>
> I analysed this problem and found set we have another serious problem
> here. But good news: There is an easy solution for both problems.
>
> Result of the analysis (as explained below): We must not extend
> PYTHONPATH for adding the profile's site-packages directory.

Right! site.py will put prepend any previously set PYTHONPATH to the one
it forms. If a site-packages is present there it will appear before the
user site location, which is not good. I had missed this particular
problem before by failing to restart my user session after installing
python (the custom PYTHONPATH we set was not yet in effect).

>
> Proposed solution: The standard library module "site" already contains a
> place where we can hook in:
>
>    # Prefixes for site-packages; add additional prefixes like /usr/local
> here
>     PREFIXES = [sys.prefix, sys.exec_prefix]
>
> So we simply prepend $GUIX_ENVIRONMENT to this list:
>
>     PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
> sys.exec_prefix]
>

I think your approach is correct! Any directory we add to PREFIXES will
get picked by the `addsitepackages` function in site.py, and this
function is called *following* the `addusersitepackages` function in
site.main(), which means those entries will appear after the user site
location in sys.path, which is what we want.

What is currently responsible for exporting PYTHONPATH to
".guix-profile/lib/python2.7/site-packages" ? It looks like it might be
the `native-search-paths' fields of our python-2.7 package?

    (native-search-paths
     (list (search-path-specification
            (variable "PYTHONPATH")
            (files '("lib/python2.7/site-packages")))))

This should rather be added with a patch adding it to the PREFIXES
variable of the site.py module, as you suggested.

This is the value of PYTHONPATH in my profile, after installing python@2
and re-login:

$ echo $PYTHONPATH
/home/maxim/.guix-profile/lib/python2.7/site-packages

Also, if we no longer rely on manipulating the PYTHONPATH directly, we
should also remove the "python-2.7-site-prefixes.patch" patch, which
adds /gnu/store entries found in PYTHONPATH to the PREFIXES variable.

> Prepending to speed up searches, since within an environment almost all
> site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
> $GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
> and the non-existing path will be skipped out automatically
>
> This would solve both the problem Maxim described and the problem
> described below, since the Guix site-packages behave exactly like any
> other global site-packages directory.
>
> When following this proposal, we *may* even remove some "wrapping" stuff
> in the python-build-system. But this has to be investigated first.
>

Yes, it will be interesting to see if the changes proposed here would
remove the need for the wrap phase. I hope it does :).

>
> Analysis (proof see below):
>
> The python interpreter was two command line options, which are rarely
> used, though:
>
> -E ignores environment variables, esp. PYTHONPATH
> -S Disable  the  import  of the module site and the site-dependent
> manipulations of sys.path.
>
> This means:
> a) When passing -E, the Guix environment's site-packages are ignored
> (together with PYTHONPATH), while they should not.
> b) When passing -S, the Guix environment's site-packages should be
> ignored, but is not (since it is specified in $PYTHONPATH, which is not
> ignored)
>
> Conclusion: We should must not use PYTHONPATH to specify the Guix's
> environment's site-package directory.
>

Right. In general, we can generalize the PYTHONPATH rule to "we must not use
PYTHONPATH" ;)

>
> Analysis details:
>
> (guix)$ which python3
> /gnu/store/zcnb…-profile/bin/python3
>
>
> Without any options:
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    should be in sys.path
>
> (base)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
> -E     Ignore environment variables  like  PYTHONPATH  and  PYTHONHOME
>        that modify the behavior of the interpreter.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    should be in sys.path
>
> (base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
>
> -S = Disable  the  import  of the module site and the site-dependent
>      manipulations of sys.path that it entails.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
>    /gnu/store/zcnb…-profile/lib/python3.5/site-packages
>    sould *not* be in sys.path
>
> (base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4/
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
>
>
> (guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload

Thanks for the feedback and detailed analysis!

Here's what I will attempt from now:

In the python-2.7 variable:

1. Remove the current "python-2.7-site-prefixes" patch.

2. Apply a new patch to site.py to add "$HOME/.guix-profile"
    as well as "$GUIX_ENVIRONMENT".

3. Remove the 'native-search-paths' field.

As a bonus, now that we won't be using PYTHONPATH anymore, there
shouldn't be any crosstalk between Python 2 & Python 3, so we ought to
be able to install both in the same profile/environment at the same time
(currently installing Python 3 "updates" Python 2).

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Anomalies in Python search-path

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> After more testing, the Python search-path we form is always problematic
> for pip since the system site-packages will always appear in front of the
> user site location (~/.local/...).

You’re talking about the search path formed by ‘guix environment’,
right?

> My last reporting that pip was working correctly in non-environment was
> wrong because I had failed to logout/login after installing the python
> package; after doing so, the behavior is the same as in an environment:
> the user site location (~/.local/...) comes after site-packages rather
> than before, as would normally be the case. Apparently this is due to
> setting a site-packages location with the environment variable
> PYTHONPATH, which has precedence over the user site location.

~/.guix-profile/etc/profile prepends things to the search paths, but
it’s up to the user whether to source it before or after other
definitions have been provided (in .bash_profile or similar).

On GuixSD, for instance, my GUILE_LOAD_PATH has:

  ~/.local and similar entries
  then ~/.guix-profile/share/guile/site/2.0
  then /run/current-system/profile/share/guile/site/2.0

Ludo’.

Re: Anomalies in Python search-path

[Hartmut Goebel]

Am 15.01.2017 um 20:23 schrieb Maxim Cournoyer:
> 1. Remove the current "python-2.7-site-prefixes" patch.

I'm afraid, we'll still need this patch. Since "native-inputs" will not
be available at build time otherwise. But you way give it a try.

> 2. Apply a new patch to site.py to add "$HOME/.guix-profile"
>     as well as "$GUIX_ENVIRONMENT".

I already tried this and unfortunately the test-case "test_site" failed:
The user-site-packages directory was not in sys.path. I had not time for
investigating this.

BTW: I'd not use a patch, but simply "substitute" it in a phase. But
that's a matter of taste.

> 3. Remove the 'native-search-paths' field.

I assume we can remove this. But I do not really understand what
"native-search-paths" *really* does. Ludo?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Anomalies in Python search-path

[Hartmut Goebel]

Am 15.01.2017 um 23:10 schrieb Ludovic Courtès:
>> > My last reporting that pip was working correctly in non-environment was
>> > wrong because I had failed to logout/login after installing the python
>> > package; after doing so, the behavior is the same as in an environment:
>> > the user site location (~/.local/...) comes after site-packages rather
>> > than before, as would normally be the case. Apparently this is due to
>> > setting a site-packages location with the environment variable
>> > PYTHONPATH, which has precedence over the user site location.
> ~/.guix-profile/etc/profile prepends things to the search paths, but
> it’s up to the user whether to source it before or after other
> definitions have been provided (in .bash_profile or similar).

Maxim is talking about how python is processing the paths.

The "normal" order is:
- $PYTHONPATH  (which normaly does not include site-packages)
- Python Standard library (not including site-packages)
- user's site-packages (~/.local/lib/python3.4/site-packages)
- global site-packages (/usr/lib/python3.4/site-packages)

The user's site-packages take precedence over the global ones.

In Guix we currently use PYTHONPATH to add global site-packages. Since
PYTHONPATH parts are put in front of the search list, they take
precedence over the user's site-packages. And this behaviour is wrong.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Anomalies in Python search-path

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 15.01.2017 um 23:10 schrieb Ludovic Courtès:
>>> > My last reporting that pip was working correctly in non-environment was
>>> > wrong because I had failed to logout/login after installing the python
>>> > package; after doing so, the behavior is the same as in an environment:
>>> > the user site location (~/.local/...) comes after site-packages rather
>>> > than before, as would normally be the case. Apparently this is due to
>>> > setting a site-packages location with the environment variable
>>> > PYTHONPATH, which has precedence over the user site location.
>> ~/.guix-profile/etc/profile prepends things to the search paths, but
>> it’s up to the user whether to source it before or after other
>> definitions have been provided (in .bash_profile or similar).
>
> Maxim is talking about how python is processing the paths.
>
> The "normal" order is:
> - $PYTHONPATH  (which normaly does not include site-packages)
> - Python Standard library (not including site-packages)
> - user's site-packages (~/.local/lib/python3.4/site-packages)
> - global site-packages (/usr/lib/python3.4/site-packages)
>
> The user's site-packages take precedence over the global ones.
>
> In Guix we currently use PYTHONPATH to add global site-packages. Since
> PYTHONPATH parts are put in front of the search list, they take
> precedence over the user's site-packages. And this behaviour is wrong.

Oh, I see.

I’m not sure what can be done on our side.  PYTHONPATH is the only thing
we can influence.

It looks like the notion of “global site-packages” only applies to FHS
systems and makes little sense for us.  If there were, say, a
PYTHONSITEPATH in lieu of this “global site-packages” thing, we could
use that.

Ludo’.