Re: Python and propagation

[Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>]

Andreas Enge <andreas@enge.fr> writes:

> On Thu, Feb 18, 2016 at 01:21:48PM +0100, Ricardo Wurmus wrote:
>> 2) avoid PYTHONPATH, patch all Python files invasively!
>>     import sys
>>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>
> That sounds great! It appears to me as if you have found the equivalent of
> a RUNPATH for python scripts.
>
> It could more or less be done like the patch-shebangs phase. To me it
> does not look difficult to implement. Assume it is done in a separate
> phase after patching shebangs:
>
> - Create a list of all the python inputs of the package; these are the
>   inputs the package name of which starts with "python-". Or the directory
>   names "/gnu/store/xxx-python-xxx", depending on what is available in
>   the phase.

I think this might have to operate on the bag.  I’m not sure if a Python
package can (or should be able to expect to) import a package that it
has not declared as a direct input.  So far we have just propagated
inputs, which makes it unclear if an input in the bag is really directly
used by the Python package itself.

If we only used direct inputs (each of which would alter the “sys.path”
for its own needs on load) we would probably have to clean up the inputs
of many Python packages, because we can no longer rely on the
side-effect of propagation (which blurs the line between what a package
is and what a bag is).  I would consider this a good thing, but it
sounds like a lot of work.

> - Construct the corresponding string to add to the python scripts.
> - Look for files starting with a shebang and having "python" as part
>   of the interpreter. Then add the string after the first, shebang line
>   in all these files.

I think this must be done to all Python files, not just scripts.  If I
install just numpy and I want to use it as a library it should be able
to find all its (fixed) dependencies by itself.  So even though the
entry point for numpy does not have a shebang, it should also contain
the magic string on its first line to set the “sys.path”.

> Or is it more complicated than that?

Maybe it is, but I don’t know enough about Python to answer the
following questions:

* What is the cumulative effect of setting or appending to “sys.path”?
  Does this affect performance?  Can we do this at compile time only?
  (Probably not as we do not guarantee that “.py” files are compiled to
  “.pyc”.)

* What happens when we have more than one package modifying the
  “sys.path”?  Is the effect local to the package or global for the
  Python process?  Will a package that only sets “sys.path” to find the
  modules it imports affect the operation of modules and their own
  imports?

* Should we overwrite “sys.path” rather than appending to it?  Would
  that still allow users to override pieces with PYTHONPATH?  Should
  they be able to do so?

* Could the load order of Python files become important (e.g. when one
  file in one package sets a different store item for a Python module
  than another)?

~~ Ricardo