[bug#46028] [PATCH core-updates 0/2] Farewell to PYTHONPATH.

[bug#46028] [PATCH core-updates 0/2] Farewell to PYTHONPATH.

[Maxim Cournoyer]

Farewell to PYTHONPATH.

Maxim Cournoyer (2):
  utils: Add a version-minor procedure.
  gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH_X_Y.

 gnu/packages/python.scm | 37 +++++++++++++++++++++++++++++++++----
 guix/utils.scm          | 15 ++++++++++++---
 2 files changed, 45 insertions(+), 7 deletions(-)

-- 
2.29.2

[bug#46028] [PATCH core-updates 1/2] utils: Add a version-minor procedure.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 2268 bytes --]

* guix/utils.scm (version-components): New procedure.
(version-prefix): Use it.
(version-major): Likewise.
(version-minor): New procedure.
---
 guix/utils.scm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/guix/utils.scm b/guix/utils.scm
index 678954dbfa..cd4e595f7e 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -8,7 +8,7 @@
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2018, 2020 Marius Bakke <marius@gnu.org>
 ;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
-;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2020, 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -93,6 +93,7 @@
             version-major+minor+point
             version-major+minor
             version-major
+            version-minor
             guile-version>?
             version-prefix?
             string-replace-substring
@@ -573,10 +574,14 @@ or '= when they denote equal versions."
               ((negative? result) '<)
               (else '=))))))
 
+(define (version-components version-string)
+  "Return the dot-separated version components as a list."
+  (string-split version-string #\.))
+
 (define (version-prefix version-string num-parts)
   "Truncate version-string to the first num-parts components of the version.
 For example, (version-prefix \"2.1.47.4.23\" 3) returns \"2.1.47\""
-  (string-join (take (string-split version-string #\.) num-parts) "."))
+  (string-join (take (version-components version-string) num-parts) "."))
 
 (define (version-major+minor+point version-string)
   "Return \"major>.<minor>.<point>\", where major, minor and point are the
@@ -595,7 +600,11 @@ minor version numbers from version-string."
 
 (define (version-major version-string)
   "Return the major version number as string from the version-string."
-  (version-prefix version-string 1))
+  (first (version-components version-string)))
+
+(define (version-minor version-string)
+  "Return the minor version number as string from the version-string."
+  (second (version-components version-string)))
 
 (define (version>? a b)
   "Return #t when A denotes a version strictly newer than B."
-- 
2.29.2

[bug#46028] [PATCH core-updates 2/2] gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH_X_Y.

[Maxim Cournoyer]

Using PYTHONPATH as a mean to discover the Python packages had the following
issues:

        1. It is not versioned, so different versions of Python would clash if
        installed in a shared profile.

        2. It would interfere with the host Python site on foreign
        distributions, sometimes preventing a a user to login their GDM
        session (!).

        3. It would take precedence over user installed Python packages
        installed through pip.

        4. It would leak into Python virtualenvs, which are supposed to create
        isolated Python environments.

This changes fixes the above issues by making use of a sitecustomize.py
module.  The newly introduced GUIX_PYTHONPATH_X_Y environment variable, where
X is the major and Y the minor version numbers, is read from the environment
and spliced in 'sys.path' just before Python's own site location, which
provides the expected behavior.

* gnu/packages/python.scm (python-2.7)[phases]{customize-site}: Add phase.
[native-search-paths]: Modify so that GUIX_PYTHONPATH_X_Y is used.
* gnu/packages/python.scm (python-3.9)[native-search-paths]: Likewise.
---
 gnu/packages/python.scm | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d3df1a4dab..79cea3d43a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -313,7 +313,30 @@
                                     "/site-packages")))
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
-               #t))))))
+               #t)))
+         (add-after 'install 'customize-site
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (site-packages (string-append
+                                    out "/lib/python"
+                                    ,(version-major+minor version)
+                                    "/site-packages"))
+                    (pythonpath (format #f "GUIX_PYTHONPATH_~a_~a"
+                                        ,(version-major version)
+                                        ,(version-minor version)))
+                    (sitecustomize.py (string-append site-packages
+                                                     "/sitecustomize.py")))
+               (with-output-to-file sitecustomize.py
+                 (lambda _
+                   (format #t "\
+import os
+import sys
+
+# Insert the GUIX_PYTHONPATH_X_Y values just before Python's own site.
+sys_path_absolute = [os.path.realpath(p) for p in sys.path]
+index = sys_path_absolute.index(~s)
+sys.path = sys.path[:index] + os.environ[~s].split(':') + sys.path[index:]
+" site-packages pythonpath)))))))))
     (inputs
      `(("bzip2" ,bzip2)
        ("expat" ,expat)
@@ -334,8 +357,12 @@
              '())))
     (native-search-paths
      (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+            (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                              (version-major version)
+                              (version-minor version)))
+            (files (list (string-append "lib/python"
+                                        (version-major+minor version)
+                                        "/site-packages"))))))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -472,7 +499,9 @@ data types.")
        ,@(package-native-inputs python-2)))
     (native-search-paths
      (list (search-path-specification
-            (variable "PYTHONPATH")
+            (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                              (version-major version)
+                              (version-minor version)))
             (files (list (string-append "lib/python"
                                         (version-major+minor version)
                                         "/site-packages"))))
-- 
2.29.2

[bug#46028] [PATCH core-updates v2] gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH_X_Y.

[Maxim Cournoyer]

Using PYTHONPATH as a mean to discover the Python packages had the following
issues:

        1. It is not versioned, so different versions of Python would clash if
        installed in a shared profile.

        2. It would interfere with the host Python site on foreign
        distributions, sometimes preventing a a user to login their GDM
        session (!).

        3. It would take precedence over user installed Python packages
        installed through pip.

        4. It would leak into Python virtualenvs, which are supposed to create
        isolated Python environments.

This changes fixes the above issues by making use of a sitecustomize.py
module.  The newly introduced GUIX_PYTHONPATH_X_Y environment variable, where
X is the major and Y the minor version numbers, is read from the environment
and spliced in 'sys.path' just before Python's own site location, which
provides the expected behavior.

* gnu/packages/python.scm (customize-site)
(guix-pythonpath-search-path): New procedures.
* gnu/packages/python.scm (python-2.7)[phases]{customize-site}: New phase.
[native-search-paths]: Adjust.
* gnu/packages/python.scm (python-3.9)[native-search-paths]: Likewise.
[phases]{customize-site}: Override the inherited phase, so that it uses the
correct version.
* gnu/packages/commencement.scm (python-boot0)
[phases]{customize-site}: Likewise.
[native-search-paths]: Override the inherited one.
---
 gnu/packages/commencement.scm |  4 ++-
 gnu/packages/python.scm       | 57 +++++++++++++++++++++++++++--------
 2 files changed, 48 insertions(+), 13 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 0713bcb843..6d7593e223 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -3176,6 +3176,7 @@ memoized as a function of '%current-system'."
                      ;; built, since it requires Linux headers.
                      (("'linux', ") ""))))
                (delete 'set-TZDIR)
+               (replace 'customize-site ,(customize-site version))
                ,@(if (hurd-system?)
                      `((add-before 'build 'fix-regen
                          (lambda* (#:key inputs #:allow-other-keys)
@@ -3184,7 +3185,8 @@ memoized as a function of '%current-system'."
                                (("/usr/include/")
                                 (string-append libc "/include/")))))))
                      '())))
-           ((#:tests? _ #f) #f))))))
+           ((#:tests? _ #f) #f))))
+    (native-search-paths (list (guix-pythonpath-search-path version)))))
 
 (define/system-dependent ld-wrapper-boot0
   ;; The first 'ld' wrapper, defined with 'define/system-dependent' because
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d3df1a4dab..c3453d63a9 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -102,7 +102,44 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-26))
+  #:use-module (srfi srfi-26)
+  #:export (customize-site
+            guix-pythonpath-search-path))
+
+(define* (customize-site version)
+  "Generate a sitecustomize.py phase, using VERSION."
+  `(lambda* (#:key outputs #:allow-other-keys)
+     (let* ((out (assoc-ref outputs "out"))
+            (site-packages (string-append
+                            out "/lib/python"
+                            ,(version-major+minor version)
+                            "/site-packages"))
+            (pythonpath (format #f "GUIX_PYTHONPATH_~a_~a"
+                                ,(version-major version)
+                                ,(version-minor version)))
+            (sitecustomize.py (string-append site-packages
+                                             "/sitecustomize.py")))
+       (with-output-to-file sitecustomize.py
+         (lambda _
+           (format #t "\
+import os
+import sys
+
+# Insert the GUIX_PYTHONPATH_X_Y values just before Python's own site.
+sys_path_absolute = [os.path.realpath(p) for p in sys.path]
+index = sys_path_absolute.index(~s)
+sys.path = sys.path[:index] + os.environ[~s].split(':') + sys.path[index:]
+" site-packages pythonpath))))))
+
+(define (guix-pythonpath-search-path version)
+  "Generate a GUIX_PYTHONPATH_X_Y search path specification, using VERSION."
+  (search-path-specification
+   (variable (format #f "GUIX_PYTHONPATH_~a_~a"
+                     (version-major version)
+                     (version-minor version)))
+   (files (list (string-append "lib/python"
+                               (version-major+minor version)
+                               "/site-packages")))))
 
 (define-public python-2.7
   (package
@@ -313,7 +350,9 @@
                                     "/site-packages")))
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
-               #t))))))
+               #t)))
+         (add-after 'install 'customize-site
+           ,(customize-site version)))))
     (inputs
      `(("bzip2" ,bzip2)
        ("expat" ,expat)
@@ -332,10 +371,7 @@
              `(("python2" ,this-package)
                ("which" ,which))
              '())))
-    (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+    (native-search-paths (list (guix-pythonpath-search-path version)))
     (home-page "https://www.python.org")
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -463,7 +499,8 @@ data types.")
                                "-x" "lib2to3/.*"
                                ,out))))
                   (list "none" "-O" "-OO"))
-                 #t)))))))
+                 #t)))
+           (replace 'customize-site ,(customize-site version))))))
     (native-inputs
      `(("tzdata" ,tzdata-for-tests)
        ,@(if (%current-target-system)
@@ -471,11 +508,7 @@ data types.")
              '())
        ,@(package-native-inputs python-2)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))
+     (list (guix-pythonpath-search-path version)
            ;; Used to locate tzdata by the zoneinfo module introduced in
            ;; Python 3.9.
            (search-path-specification
-- 
2.29.2

[bug#46028] [PATCH core-updates v2] gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH_X_Y.

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Using PYTHONPATH as a mean to discover the Python packages had the following
> issues:
>
>         1. It is not versioned, so different versions of Python would clash if
>         installed in a shared profile.
>
>         2. It would interfere with the host Python site on foreign
>         distributions, sometimes preventing a a user to login their GDM
>         session (!).
>
>         3. It would take precedence over user installed Python packages
>         installed through pip.
>
>         4. It would leak into Python virtualenvs, which are supposed to create
>         isolated Python environments.

I've adapted the most problematic uses of PYTHONPATH in the code base
(mostly (getenv "PYTHONPATH"), which would typically now return #f and
pushed the branch to cu/farewell-to-pythonpath (it contains ~160
commits, so I thought it'd be easier to review as a branch).

Thanks!

Maxim

[bug#46028] [PATCH core-updates 0/2] Farewell to PYTHONPATH.

[Maxim Cournoyer]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Using PYTHONPATH as a mean to discover the Python packages had the following
>> issues:
>>
>>         1. It is not versioned, so different versions of Python would clash if
>>         installed in a shared profile.
>>
>>         2. It would interfere with the host Python site on foreign
>>         distributions, sometimes preventing a a user to login their GDM
>>         session (!).
>>
>>         3. It would take precedence over user installed Python packages
>>         installed through pip.
>>
>>         4. It would leak into Python virtualenvs, which are supposed to create
>>         isolated Python environments.
>
> I've adapted the most problematic uses of PYTHONPATH in the code base
> (mostly (getenv "PYTHONPATH"), which would typically now return #f and
> pushed the branch to cu/farewell-to-pythonpath (it contains ~160
> commits, so I thought it'd be easier to review as a branch).

There's now a third version of this that I pushed to the
cu/farewell-to-pythonpath branch.  It now uses GUIX_PYTHONPATH instead
of GUIX_PYTHONPATH_X_Y, which makes it easier to use in any package
definition (as there's no longer a need to import the (guix build
python-build-system) module for packages not using the Python build
system).

I'll merge it soon if nobody has an objection.

Thanks,

Maxim

bug#46028: [PATCH core-updates 0/2] Farewell to PYTHONPATH.

[Maxim Cournoyer]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>>> Using PYTHONPATH as a mean to discover the Python packages had the following
>>> issues:
>>>
>>>         1. It is not versioned, so different versions of Python would clash if
>>>         installed in a shared profile.
>>>
>>>         2. It would interfere with the host Python site on foreign
>>>         distributions, sometimes preventing a a user to login their GDM
>>>         session (!).
>>>
>>>         3. It would take precedence over user installed Python packages
>>>         installed through pip.
>>>
>>>         4. It would leak into Python virtualenvs, which are supposed to create
>>>         isolated Python environments.
>>
>> I've adapted the most problematic uses of PYTHONPATH in the code base
>> (mostly (getenv "PYTHONPATH"), which would typically now return #f and
>> pushed the branch to cu/farewell-to-pythonpath (it contains ~160
>> commits, so I thought it'd be easier to review as a branch).
>
> There's now a third version of this that I pushed to the
> cu/farewell-to-pythonpath branch.  It now uses GUIX_PYTHONPATH instead
> of GUIX_PYTHONPATH_X_Y, which makes it easier to use in any package
> definition (as there's no longer a need to import the (guix build
> python-build-system) module for packages not using the Python build
> system).
>
> I'll merge it soon if nobody has an objection.

I've now pushed this to core-updates.

Closing!

Maxim