all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* PYTHONPATH woes
@ 2018-02-20 10:53 Ricardo Wurmus
  2018-02-20 15:01 ` Pjotr Prins
  2018-02-21 21:58 ` Hartmut Goebel
  0 siblings, 2 replies; 49+ messages in thread
From: Ricardo Wurmus @ 2018-02-20 10:53 UTC (permalink / raw)
  To: guix-devel

Hi Guix,

we have a couple of packages that provide scripts that depend on Python
modules.  We wrap them in PYTHONPATH to ensure that the correct Python
modules are found at runtime.

This is not enough.

We don’t wrap them tightly enough; instead we allow for a user-provided
PYTHONPATH value to be added to the PYTHONPATH we specified.  The result
is that a user-set PYTHONPATH can act like LD_LIBRARY_PATH — it causes
chaos.  This is despite the fact that we make sure that the wrapper’s
PYTHONPATH comes first!

Suppose a user installs python@2 and python2-statsmodels; at a later
point the user upgrades Guix, and then installs the ribodiff package.
The user does not know that ribodiff is written in Python, nor should
the user be aware of that.

Because Python is installed in the profile, etc/profile will contain a
definition for PYTHONPATH.  The user may source that etc/profile file to
set up all required environment variables.  But now running the ribodiff
scripts fails!

Here’s what happens: the PYTHONPATH that Guix sets for the profile now
contains an incompatible variant of the python2-statsmodels package.
Guix has been upgraded between installing python2-statsmodels and
ribodiff, so a different version of Python was used to build these
modules.  Since the ribodiff wrapper script gladly accepts any set
PYTHONPATH, it causes the ribodiff scripts to load the old and
incompatible python2-statsmodels package instead of the compatible one
from the wrapper.

I don’t know why this happens.  I find it puzzling that in this
particular case the user’s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 81, in <module>
    main()
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 26, in main
    import ribodiff.estimatedisp as ed
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/estimatedisp.py", line 7, in <module>
    import rawdisp as rd
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/rawdisp.py", line 8, in <module>
    import statsmodels.api as sm
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py", line 8, in <module>
    from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/__init__.py", line 1, in <module>
    from .tools import add_constant, categorical
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/tools.py", line 11, in <module>
    from statsmodels.datasets import webuse
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/__init__.py", line 5, in <module>
    from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/__init__.py", line 1, in <module>
    from .data import *
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/data.py", line 90, in <module>
    from statsmodels.datasets import utils as du
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/utils.py", line 13, in <module>
    from pandas import read_csv, DataFrame, Index
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 31, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/hashtable.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
--8<---------------cut here---------------end--------------->8---

Now you could say that this is the user’s fault for not using manifests.
But consider this: what happens if the user had a manifest and installed
“python-statsmodels” instead of the Python 2 variant?  Guix would still
set PYTHONPATH and the ribodiff wrapper would still prefer the profile’s
PYTHONPATH over the wrapped value, so it would cause Python 2 (from
ribodiff) to load a Python 3 module of statsmodels — these are not
compatible and again we have a runtime crash.

Manifests wouldn’t avoid this problem.

Avoiding this problem now requires that users know what language a tool
is implemented in (e.g. Python 2 for Ribodiff) and make a conscious
effort to install these tools in a separate profile containing no Python
3 modules.  This is not a reasonable burden to put on users.

What can we do to fix this?

Would it be good to make the wrappers for Python scripts stricter and
not accept any user-set PYTHONPATH?

How do we approach the problem of having both Python 2 modules and
Python 3 modules in the same profile?  PYTHONPATH will be set to refer
to the site-packages directories of both versions, which is never good.
Does Python offer us a way to do better?  Can we make use of pth files
to get around this problem somehow?

--
Ricardo

^ permalink raw reply	[flat|nested] 49+ messages in thread

end of thread, other threads:[~2018-04-19  8:23 UTC | newest]

Thread overview: 49+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-02-20 10:53 PYTHONPATH woes Ricardo Wurmus
2018-02-20 15:01 ` Pjotr Prins
2018-02-20 15:18   ` Andy Wingo
2018-02-20 16:40     ` Pjotr Prins
2018-02-20 15:30   ` Ricardo Wurmus
2018-02-21 21:58 ` Hartmut Goebel
2018-02-22 15:30   ` Ricardo Wurmus
2018-02-22 18:35     ` Hartmut Goebel
2018-02-22 20:42     ` Hartmut Goebel
2018-02-23  8:45       ` Vincent Legoll
2018-02-23 12:36     ` Hartmut Goebel
2018-02-23 16:59       ` Pjotr Prins
2018-02-23 19:36         ` Ricardo Wurmus
2018-02-23 23:54           ` Pjotr Prins
2018-02-24 10:44         ` Hartmut Goebel
2018-02-24 10:49           ` Hartmut Goebel
2018-02-27 11:43           ` PYTHONPATH issue analysis - part 1 (was: PYTHONPATH woes) Hartmut Goebel
2018-03-13 21:54             ` PYTHONPATH issue analysis - part 1 Hartmut Goebel
2018-02-27 11:49           ` PYTHONPATH issue analysis - part 2 (was: PYTHONPATH woes) Hartmut Goebel
2018-03-11 21:47           ` PYTHONPATH issue analysis - part 3 " Hartmut Goebel
2018-03-13 21:23             ` PYTHONPATH issue analysis - part 3 Ludovic Courtès
2018-03-13 21:44               ` Pjotr Prins
2018-03-13 22:02                 ` Hartmut Goebel
2018-03-14  7:49                   ` Pjotr Prins
2018-03-14  9:04                     ` Hartmut Goebel
2018-03-14 18:21                       ` Pjotr Prins
2018-03-15 19:48                     ` Hartmut Goebel
2018-03-13 21:47               ` Hartmut Goebel
2018-03-14  9:41                 ` Ludovic Courtès
2018-03-13 21:51               ` Hartmut Goebel
2018-03-14  0:10               ` Ricardo Wurmus
2018-03-15  9:09                 ` Ludovic Courtès
2018-03-15 19:30             ` PYTHONPATH issue explanation Hartmut Goebel
2018-03-17  1:41               ` 宋文武
2018-03-17 10:07                 ` Ricardo Wurmus
2018-03-17 22:46                   ` Hartmut Goebel
2018-03-17 22:53                   ` Hartmut Goebel
2018-03-17 11:18                 ` [PATCH] gnu: python: Honor 'GUIX_PYTHON_X_Y_SITE_PACKAGES' 宋文武
2018-03-17 21:53                   ` Hartmut Goebel
2018-03-18  0:04                     ` 宋文武
2018-03-18  0:07                   ` 宋文武
2018-03-17 22:04                 ` PYTHONPATH issue explanation Hartmut Goebel
2018-03-18  0:57                   ` 宋文武
2018-03-18 10:05                     ` 宋文武
2018-03-24 20:47               ` Chris Marusich
2018-04-16 14:21             ` PYTHONPATH - let's systematically tame the baest Hartmut Goebel
2018-04-17  1:47               ` 宋文武
2018-04-17  7:03                 ` Hartmut Goebel
2018-04-18  8:34               ` Ricardo Wurmus

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.