bug#50105: [core-updates] Python sitecustomize issue.

bug#50105: [core-updates] Python sitecustomize issue.

[Mathieu Othacehe]

Hello,

I tried to upgrade glade to the 3.38.2 release on core-update and
encountered the following test issue:

--8<---------------cut here---------------start------------->8---
2/5 modules               FAIL            0.29s   killed by signal 5 SIGTRAP
>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stdout:
# random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
# GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
stderr:
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list

(/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
--8<---------------cut here---------------end--------------->8---

I think the issue here lies in the sitecustomize.py file introduced with
cb72f9a773e0931ee3758c851d96007ded034e4c.

--8<---------------cut here---------------start------------->8---
python_root = os.path.realpath(sys.executable).split('/bin/')[0]
--8<---------------cut here---------------end--------------->8---

When using the Python C library, sys.executable is not the expected
python binary but the test binary.

So when we try to find the associated libraries in the Python search
path, this way:

--8<---------------cut here---------------start------------->8---
index = sys_path_absolute.index(python_site)
sys.path = sys.path[:index] + matching_sites + sys.path[index:]
--8<---------------cut here---------------end--------------->8---

we get:

--8<---------------cut here---------------start------------->8---
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
--8<---------------cut here---------------end--------------->8---

Running the test with the PYTHONPATH set to:

--8<---------------cut here---------------start------------->8---
PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH 
--8<---------------cut here---------------end--------------->8---

is a way to work around this issue.

The fix here would be to replace sys.executable in the sitecustomize.py
with something more adequate, but I have no idea what could it be.

Thanks,

Mathieu

bug#50105: [core-updates] Python sitecustomize issue.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

Hi Mathieu,

[...]

> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.
>
> python_root = os.path.realpath(sys.executable).split('/bin/')[0]
>
>
> When using the Python C library, sys.executable is not the expected
> python binary but the test binary.
>
> So when we try to find the associated libraries in the Python search
> path, this way:
>
> index = sys_path_absolute.index(python_site)
> sys.path = sys.path[:index] + matching_sites + sys.path[index:]
>
>
> we get:
>
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
>
> Running the test with the PYTHONPATH set to:
>
> PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH 
>
> is a way to work around this issue.
>
> The fix here would be to replace sys.executable in the sitecustomize.py
> with something more adequate, but I have no idea what could it be.

From the doc [0], sys.prefix seems to be more correct.  This had been
suggested by Hartmut some time ago and I had this patch ready recently.
Perhaps it already addresses the issue at hand?

You'll find it attached.  It hasn't gotten much testing yet.

[0]  https://docs.python.org/3/library/sys.html#sys.prefix


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-aux-files-sitecustomize-Cleanup-and-add-explanatory-.patch --]
[-- Type: text/x-patch, Size: 2831 bytes --]

From e693b38a6d5c370e13f795c6303f50871ec78ae4 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Tue, 3 Aug 2021 20:41:20 -0400
Subject: [PATCH] aux-files: sitecustomize: Cleanup and add explanatory
 comments.

* gnu/packages/aux-files/python/sitecustomize.py: Add a comment explaining the
general idea.

(major_minor): Use the unpacking operator (*) to provide the arguments.
(site_packages_prefix): Use os.path.join to form the path.
(python_site): Likewise.
(all_sites_raw): Split on os.path.pathsep.
(sys.path): Directly splice the result in the list.

Suggested-by: Hartmut Goebel <h.goebel@crazy-compilers.com>
---
 .../aux-files/python/sitecustomize.py         | 28 ++++++++++++++-----
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index 65d3c7d554..71e328b9ac 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -20,13 +20,26 @@
 import os
 import sys
 
-python_root = os.path.realpath(sys.executable).split('/bin/')[0]
-major_minor = '{}.{}'.format(sys.version_info[0], sys.version_info[1])
-site_packages_prefix = 'lib/python' + major_minor + '/site-packages'
-python_site = python_root + '/' + site_packages_prefix
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages.  File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path.  To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+    'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.join(sys.prefix, site_packages_prefix)
 
 try:
-    all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(':')
+    all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
 except KeyError:
     all_sites_raw = []
 # Normalize paths, otherwise a trailing slash would cause it to not match.
@@ -35,7 +48,8 @@ matching_sites = [p for p in all_sites_norm
                   if p.endswith(site_packages_prefix)]
 
 # Insert sites matching the current version into sys.path, right before
-# Python's own site.
+# Python's own site.  This way, the user can override the libraries provided
+# by Python itself.
 sys_path_absolute = [os.path.realpath(p) for p in sys.path]
 index = sys_path_absolute.index(python_site)
-sys.path = sys.path[:index] + matching_sites + sys.path[index:]
+sys.path[index:index] = matching_sites
-- 
2.32.0

bug#50105: [core-updates] Python sitecustomize issue.

[Maxim Cournoyer]

Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hello,
>
> I tried to upgrade glade to the 3.38.2 release on core-update and
> encountered the following test issue:
>
> 2/5 modules               FAIL            0.29s   killed by signal 5 SIGTRAP
>>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
> ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> stdout:
> # random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
> # GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
> Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
> stderr:
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
> (/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
>
>
> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.

[...]

Is there a simple reproducer to test my fix?  Would just building
current Glade and its test suite trigger it?

Thanks,

Maxim

bug#50105: [core-updates] Python sitecustomize issue.

[Maxim Cournoyer]

Hi again,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> Is there a simple reproducer to test my fix?  Would just building
> current Glade and its test suite trigger it?

OK, nevermind, I found that the test case was deleting the 'fix-tests'
build phase from glade3, and building the package.  I confirm the fix
works!

I'll send that patch series I've been talking about.

Thanks,

Maxim

bug#50105: [core-updates] Python sitecustomize issue.

[Mathieu Othacehe]

Hello Maxim,

> OK, nevermind, I found that the test case was deleting the 'fix-tests'
> build phase from glade3, and building the package.  I confirm the fix
> works!

Great, that's exactly what I was trying to test but you beat me to it
:). I'll be afk for a month so I won't be able to confirm that I have
the same result locally. However, this fix looks good to me.

Thanks,

Mathieu

bug#50105: [core-updates] Python sitecustomize issue.

[Maxim Cournoyer]

Hello,

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hello Maxim,
>
>> OK, nevermind, I found that the test case was deleting the 'fix-tests'
>> build phase from glade3, and building the package.  I confirm the fix
>> works!
>
> Great, that's exactly what I was trying to test but you beat me to it
> :). I'll be afk for a month so I won't be able to confirm that I have
> the same result locally. However, this fix looks good to me.

I had forgotten to close this.  The fix has been in core-updates-frozen
for a while.

Thank you,

Maxim