PYTHONPATH woes

[Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>]

Hi Guix,

we have a couple of packages that provide scripts that depend on Python
modules.  We wrap them in PYTHONPATH to ensure that the correct Python
modules are found at runtime.

This is not enough.

We don’t wrap them tightly enough; instead we allow for a user-provided
PYTHONPATH value to be added to the PYTHONPATH we specified.  The result
is that a user-set PYTHONPATH can act like LD_LIBRARY_PATH — it causes
chaos.  This is despite the fact that we make sure that the wrapper’s
PYTHONPATH comes first!

Suppose a user installs python@2 and python2-statsmodels; at a later
point the user upgrades Guix, and then installs the ribodiff package.
The user does not know that ribodiff is written in Python, nor should
the user be aware of that.

Because Python is installed in the profile, etc/profile will contain a
definition for PYTHONPATH.  The user may source that etc/profile file to
set up all required environment variables.  But now running the ribodiff
scripts fails!

Here’s what happens: the PYTHONPATH that Guix sets for the profile now
contains an incompatible variant of the python2-statsmodels package.
Guix has been upgraded between installing python2-statsmodels and
ribodiff, so a different version of Python was used to build these
modules.  Since the ribodiff wrapper script gladly accepts any set
PYTHONPATH, it causes the ribodiff scripts to load the old and
incompatible python2-statsmodels package instead of the compatible one
from the wrapper.

I don’t know why this happens.  I find it puzzling that in this
particular case the user’s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 81, in <module>
    main()
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", line 26, in main
    import ribodiff.estimatedisp as ed
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/estimatedisp.py", line 7, in <module>
    import rawdisp as rd
  File "/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/rawdisp.py", line 8, in <module>
    import statsmodels.api as sm
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py", line 8, in <module>
    from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/__init__.py", line 1, in <module>
    from .tools import add_constant, categorical
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/tools.py", line 11, in <module>
    from statsmodels.datasets import webuse
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/__init__.py", line 5, in <module>
    from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/__init__.py", line 1, in <module>
    from .data import *
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/data.py", line 90, in <module>
    from statsmodels.datasets import utils as du
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/utils.py", line 13, in <module>
    from pandas import read_csv, DataFrame, Index
  File "/home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 31, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/hashtable.so: undefined symbol: PyUnicodeUCS2_FromStringAndSize not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
--8<---------------cut here---------------end--------------->8---

Now you could say that this is the user’s fault for not using manifests.
But consider this: what happens if the user had a manifest and installed
“python-statsmodels” instead of the Python 2 variant?  Guix would still
set PYTHONPATH and the ribodiff wrapper would still prefer the profile’s
PYTHONPATH over the wrapped value, so it would cause Python 2 (from
ribodiff) to load a Python 3 module of statsmodels — these are not
compatible and again we have a runtime crash.

Manifests wouldn’t avoid this problem.

Avoiding this problem now requires that users know what language a tool
is implemented in (e.g. Python 2 for Ribodiff) and make a conscious
effort to install these tools in a separate profile containing no Python
3 modules.  This is not a reasonable burden to put on users.

What can we do to fix this?

Would it be good to make the wrappers for Python scripts stricter and
not accept any user-set PYTHONPATH?

How do we approach the problem of having both Python 2 modules and
Python 3 modules in the same profile?  PYTHONPATH will be set to refer
to the site-packages directories of both versions, which is never good.
Does Python offer us a way to do better?  Can we make use of pth files
to get around this problem somehow?

--
Ricardo