[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
  to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image
created by Guix. See https://issues.guix.gnu.org/53258.

Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
---
 gnu/packages/aux-files/python/sitecustomize.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index e2348e0356..82ea91104d 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -37,7 +37,7 @@ import sys
 major_minor = '{}.{}'.format(*sys.version_info)
 site_packages_prefix = os.path.join(
     'lib', 'python' + major_minor, 'site-packages')
-python_site = os.path.join(sys.prefix, site_packages_prefix)
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
 
 try:
     all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
-- 
2.41.0

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
>   to deal with situations where it contains "../"

This patch leads to massive rebuilds for non-trivial environments. I
will submit an alternative that patches Python via a graft.

Cheers,
  Konrad.

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
  to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image
created by Guix. See https://issues.guix.gnu.org/53258.

This is a variant of https://issues.guix.gnu.org/68241 that implements the
fix as a graft rather than as a patch to the Python package. Since many
packages in Guix depend on Python, the basic patch leads to enormous
build times. A graft should be sufficient since the fix has no impact
on Python being used within Guix.

Change-Id: I3b07f4edf474a12fcabfbf90f37c4c0223278330
---
 .../aux-files/python/sitecustomize-fixed.py   | 65 +++++++++++++++++++
 gnu/packages/python.scm                       | 22 ++++++-
 2 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/aux-files/python/sitecustomize-fixed.py

diff --git a/gnu/packages/aux-files/python/sitecustomize-fixed.py b/gnu/packages/aux-files/python/sitecustomize-fixed.py
new file mode 100644
index 0000000000..82ea91104d
--- /dev/null
+++ b/gnu/packages/aux-files/python/sitecustomize-fixed.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import site
+import sys
+
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages.  File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path.  To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+    'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
+
+try:
+    all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
+except KeyError:
+    all_sites_raw = []
+# Normalize paths, otherwise a trailing slash would cause it to not match.
+all_sites_norm = [os.path.normpath(p) for p in all_sites_raw]
+matching_sites = [p for p in all_sites_norm
+                  if p.endswith(site_packages_prefix)]
+
+if matching_sites:
+    # Deduplicate the entries, append them to sys.path, and handle any
+    # .pth files they contain.
+    for s in matching_sites:
+        site.addsitedir(s)
+
+    # Move the entries that were appended to sys.path in front of
+    # Python's own site-packages directory.  This enables Guix
+    # packages to override Python's bundled packages, such as 'pip'.
+    python_site_index = sys.path.index(python_site)
+    new_site_start_index = sys.path.index(matching_sites[0])
+    if python_site_index < new_site_start_index:
+        sys.path = (sys.path[:python_site_index]
+                    + sys.path[new_site_start_index:]
+                    + sys.path[python_site_index:new_site_start_index])
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..3cc3c08360 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -414,7 +414,16 @@ (define-public python-2.7
 packages; exception-based error handling; and very high level dynamic
 data types.")
     (properties '((cpe-name . "python")))
-    (license license:psfl)))
+    (license license:psfl)
+    (replacement python-2.7-fixed)))
+
+(define-public python-2.7-fixed
+  (package
+    (inherit python-2.7)
+    (native-inputs
+     (modify-inputs (package-native-inputs python-2.7)
+       (replace "sitecustomize.py"
+         (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
 
 ;; Current 2.x version.
 (define-public python-2 python-2.7)
@@ -594,7 +603,16 @@ (define-public python-3.10
            ;; Python 3.9.
            (search-path-specification
             (variable "PYTHONTZPATH")
-            (files (list "share/zoneinfo")))))))
+            (files (list "share/zoneinfo")))))
+    (replacement python-3.10-fixed)))
+
+(define-public python-3.10-fixed
+  (package
+    (inherit python-3.10)
+    (native-inputs
+     (modify-inputs (package-native-inputs python-3.10)
+       (replace "sitecustomize.py"
+         (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
 
 ;; Current 3.x version.
 (define-public python-3 python-3.10)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
-- 
2.41.0

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Ludovic Courtès]

Hi Konrad!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
>   to deal with situations where it contains "../"
>
> This happens in particular when running Python from a Singularity image
> created by Guix. See https://issues.guix.gnu.org/53258.
>
> Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
> ---
>  gnu/packages/aux-files/python/sitecustomize.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
> index e2348e0356..82ea91104d 100644
> --- a/gnu/packages/aux-files/python/sitecustomize.py
> +++ b/gnu/packages/aux-files/python/sitecustomize.py
> @@ -37,7 +37,7 @@ import sys
>  major_minor = '{}.{}'.format(*sys.version_info)
>  site_packages_prefix = os.path.join(
>      'lib', 'python' + major_minor, 'site-packages')
> -python_site = os.path.join(sys.prefix, site_packages_prefix)
> +python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))

IIRC Josselin had a patch for <https://issues.guix.gnu.org/53258>, but I
can’t find it.  Josselin?

Also, do I get it right that the patch you sent fixes more than #53258?

Thanks!

Ludo’.

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

Hi Ludo,

> Also, do I get it right that the patch you sent fixes more than #53258?

#53258 is two things: a weird error message, and a failure to run valid
Python code, the two being possibly related.

My patch fixes the error message. The Python example that Marek gave
already works fine in today's Guix, so it was probably fixed by
something else as a side effect.

Could the bug that causes the error message also mess up valid Python
code? In theory, yes, but I don't have an example at hand.

Cheers,
  Konrad.

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Maxim Cournoyer]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Hi Ludo,
>
>> Also, do I get it right that the patch you sent fixes more than #53258?
>
> #53258 is two things: a weird error message, and a failure to run valid
> Python code, the two being possibly related.
>
> My patch fixes the error message. The Python example that Marek gave
> already works fine in today's Guix, so it was probably fixed by
> something else as a side effect.

Oh?  And you tested using the exact same reproducer, using a relocatable
pack?

-- 
Thanks,
Maxim

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> My patch fixes the error message. The Python example that Marek gave
>> already works fine in today's Guix, so it was probably fixed by
>> something else as a side effect.
>
> Oh?  And you tested using the exact same reproducer, using a relocatable
> pack?

Yes, I tested it in a Singularity container.

Cheers,
  konrad.

bug#68241: [PATCH] gnu: python: Fix sitecustomize.py

[Maxim Cournoyer]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> My patch fixes the error message. The Python example that Marek gave
>>> already works fine in today's Guix, so it was probably fixed by
>>> something else as a side effect.
>>
>> Oh?  And you tested using the exact same reproducer, using a relocatable
>> pack?
>
> Yes, I tested it in a Singularity container.

Great, it's now on core-updates.

-- 
Thanks,
Maxim

[bug#68241] [PATCH] gnu: python: Fix sitecustomize.py

[Konrad Hinsen]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Great, it's now on core-updates.

Thanks!

Konrad.