Python and propagation

Python and propagation

[Ricardo Wurmus]

Hi Guix,

I’m dealing with Python in Guix pretty often, because numpy and scipy
are very popular here — and I have been bitten by propagated inputs a
little too often, so I’d like to discuss an alternative.

A user has the following packages installed in the default profile:

    python-2.7.10
    python2-numpy
    python2-scipy

This installation is already a little stale as it was done a couple of
months ago, so the versions of numpy and scipy are not the latest.  Now
the user chooses to install the innocuous-looking “htseq” package.  Guix
downloads the substitute (and more) and then proceeds to build a new
generation of the profile.

But, oh, what is *that*?  There are hundreds of lines in which Guix
warns about numpy conflicts, shrugs and picks one or the other version
of the conflicting file.  “How could this have happened?”, the user
cries out in dispair.

Well, the user did not see that “htseq” is not only an executable, but
also a Python library.  As a Python library it depends on numpy, and
thus propagates numpy as it is customary in Guix.  This is necessary
because Python libraries do not have a RUNPATH feature and must be able
to find named imports in some directory in the PYTHONPATH.  To make
packages available on a simple user-controlled PYTHONPATH, we propagate
all dependent Python packages, i.e. we install them into the very same
profile, so that the user only has to do this

    export PYTHONPATH=$HOME/.guix-profile/lib/python2.7

instead of the unmanageable

    export PYTHONPATH=/gnu/store/...dep1/...:/gnu/store/...dep2...:...

I wonder if we could do better than this.  Here are two proposals for
discussion; one is probably not very contentious, the other is straight
from my dream diary in which I collect all sorts of strange and
unrealistic ideas.

1) print a warning when a collision is expected to happen, not when a
collision has happened.

Guix knows what is currently installed in a profile, so it knows that,
say, “python2-numpy” is installed.  It also knows that installing
“htseq” is also going to install “python2-numpy” into that profile.  It
also knows that the propagated “python2-numpy” is different from the
installed “python2-numpy”.  Knowing all that, it should also know that
there are going to be file conflicts, no?

If that’s all true, could we make Guix print a warning about impending
doom *before* it creates a new profile generation?  It’s a much nicer
user experience (at least to me) when I’m being told about possible
conflicts (at the package level) before Guix creates a new profile and
nonchalantly informs me about arbitrarily resolving conflicts (at the
file level).  The difference is in the amount of warnings I get (package
vs individual files) and about the time I would otherwise have to waste
to roll back the profile and upgrade already installed libraries or
choose to install “htseq” into a new profile.


2) avoid PYTHONPATH, patch all Python files invasively!

Python does not have any feature that is comparable to RUNPATH.  It is
only concerned with finding libraries/modules by *name* in one of the
directories specified by the PYTHONPATH environment variable.

But actually the PYTHONPATH variable is not the only means to affect the
search path for modules.  It is possible to change the search path
programmatically:

    import sys
    sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
    import numpy

The first two lines add an explicit store item path to the search path;
the third line just imports the numpy module by name (as usual).  Even
without setting the PYTHONPATH to include the numpy in the profile the
third line won’t fail.

I wonder if we could design a phase that — very much like the
“wrap-program” phase — modifies *every* Python file and injects lines
like the first two in the example above, appending explicit store item
paths, so that all dependent Python libraries can be found without the
need to have them installed in the same profile and without the need to
set PYTHONPATH.

Maybe this is crazy, and maybe this causes other annoying problems, but
I think it is the closest we can get to a RUNPATH-like feature in
Python.

What do you think?  Do I need a long vacation or is this something we
might realistically do in an automated fashion?

~~ Ricardo

Re: Python and propagation

[Andreas Enge]

On Thu, Feb 18, 2016 at 01:21:48PM +0100, Ricardo Wurmus wrote:
> 2) avoid PYTHONPATH, patch all Python files invasively!
>     import sys
>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")

That sounds great! It appears to me as if you have found the equivalent of
a RUNPATH for python scripts.

It could more or less be done like the patch-shebangs phase. To me it
does not look difficult to implement. Assume it is done in a separate
phase after patching shebangs:

- Create a list of all the python inputs of the package; these are the
  inputs the package name of which starts with "python-". Or the directory
  names "/gnu/store/xxx-python-xxx", depending on what is available in
  the phase.
- Construct the corresponding string to add to the python scripts.
- Look for files starting with a shebang and having "python" as part
  of the interpreter. Then add the string after the first, shebang line
  in all these files.

Or is it more complicated than that?

Andreas

Re: Python and propagation

[Ricardo Wurmus]

Andreas Enge <andreas@enge.fr> writes:

> On Thu, Feb 18, 2016 at 01:21:48PM +0100, Ricardo Wurmus wrote:
>> 2) avoid PYTHONPATH, patch all Python files invasively!
>>     import sys
>>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>
> That sounds great! It appears to me as if you have found the equivalent of
> a RUNPATH for python scripts.
>
> It could more or less be done like the patch-shebangs phase. To me it
> does not look difficult to implement. Assume it is done in a separate
> phase after patching shebangs:
>
> - Create a list of all the python inputs of the package; these are the
>   inputs the package name of which starts with "python-". Or the directory
>   names "/gnu/store/xxx-python-xxx", depending on what is available in
>   the phase.

I think this might have to operate on the bag.  I’m not sure if a Python
package can (or should be able to expect to) import a package that it
has not declared as a direct input.  So far we have just propagated
inputs, which makes it unclear if an input in the bag is really directly
used by the Python package itself.

If we only used direct inputs (each of which would alter the “sys.path”
for its own needs on load) we would probably have to clean up the inputs
of many Python packages, because we can no longer rely on the
side-effect of propagation (which blurs the line between what a package
is and what a bag is).  I would consider this a good thing, but it
sounds like a lot of work.

> - Construct the corresponding string to add to the python scripts.
> - Look for files starting with a shebang and having "python" as part
>   of the interpreter. Then add the string after the first, shebang line
>   in all these files.

I think this must be done to all Python files, not just scripts.  If I
install just numpy and I want to use it as a library it should be able
to find all its (fixed) dependencies by itself.  So even though the
entry point for numpy does not have a shebang, it should also contain
the magic string on its first line to set the “sys.path”.

> Or is it more complicated than that?

Maybe it is, but I don’t know enough about Python to answer the
following questions:

* What is the cumulative effect of setting or appending to “sys.path”?
  Does this affect performance?  Can we do this at compile time only?
  (Probably not as we do not guarantee that “.py” files are compiled to
  “.pyc”.)

* What happens when we have more than one package modifying the
  “sys.path”?  Is the effect local to the package or global for the
  Python process?  Will a package that only sets “sys.path” to find the
  modules it imports affect the operation of modules and their own
  imports?

* Should we overwrite “sys.path” rather than appending to it?  Would
  that still allow users to override pieces with PYTHONPATH?  Should
  they be able to do so?

* Could the load order of Python files become important (e.g. when one
  file in one package sets a different store item for a Python module
  than another)?

~~ Ricardo

Re: Python and propagation

[Jookia]

On Thu, Feb 18, 2016 at 03:28:23PM +0100, Andreas Enge wrote:
> Or is it more complicated than that?

I'd imagine things could get a bit hairy since you can override the 'import'
command as such: https://pypi.python.org/pypi/demandimport

> Andreas

Re: Python and propagation

[宋文武]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> [...]
>
> 2) avoid PYTHONPATH, patch all Python files invasively!
>
> Python does not have any feature that is comparable to RUNPATH.  It is
> only concerned with finding libraries/modules by *name* in one of the
> directories specified by the PYTHONPATH environment variable.
>
> But actually the PYTHONPATH variable is not the only means to affect the
> search path for modules.  It is possible to change the search path
> programmatically:
>
>     import sys
>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>     import numpy
>
> The first two lines add an explicit store item path to the search path;
> the third line just imports the numpy module by name (as usual).  Even
> without setting the PYTHONPATH to include the numpy in the profile the
> third line won’t fail.
>
I packaged python-hy, and it does find all depends without propagations.
In its output, the trick is done by ‘python-hy-0.11.1.pth’ and ‘site.py’.

After read:
  <https://docs.python.org/2/library/site.html>
  <http://stackoverflow.com/questions/15208615/using-pth-files>

I try:
--8<---------------cut here---------------start------------->8---
$ mkdir -p /tmp/o
$ mkdir -p /tmp/a
$ echo /tmp/a > /tmp/o/x.pth
$ cat > /tmp/o/sitecustomize.py << EOF
> import os, site
> dir = os.path.dirname(os.path.abspath(__fil__))
> site.addsitedir(dir)
> EOF
$ PYTHONPATH=/tmp/o python3 -c 'import sys; print(sys.path)
--8<---------------cut here---------------end--------------->8---
And ‘/tmp/a’ is at the end of the list.

So, I think if we get all the pth files correct, no need to use
propagate-inputs :-)

Re: Python and propagation

[Andreas Enge]

Just a quick reply to the first part of your message, as I do not have
time to read more right now ;-)

On Thu, Feb 18, 2016 at 03:45:36PM +0100, Ricardo Wurmus wrote:
> I think this might have to operate on the bag.  I’m not sure if a Python
> package can (or should be able to expect to) import a package that it
> has not declared as a direct input.  So far we have just propagated
> inputs, which makes it unclear if an input in the bag is really directly
> used by the Python package itself.
> 
> If we only used direct inputs (each of which would alter the “sys.path”
> for its own needs on load) we would probably have to clean up the inputs
> of many Python packages, because we can no longer rely on the
> side-effect of propagation (which blurs the line between what a package
> is and what a bag is).  I would consider this a good thing, but it
> sounds like a lot of work.

I think this is a good thing.

If package A has package B as an input, and B has package C as a propagated
input, then C should only be used from inside B.

If A also needs C directly, it should be declared (also) as a direct input
of A. By accident, it is true that things work without since A, B and C
all live in the same world eventually, but logically this is not what
propagated inputs are for.

So if we discover such cases, I would consider them as bugs, and needing
to fix them can be seen as a nice side-effect of the proposed change.

Andreas

Re: Python and propagation

[Christopher Allan Webber]

Ricardo Wurmus writes:

> 1) print a warning when a collision is expected to happen, not when a
> collision has happened.
>
> Guix knows what is currently installed in a profile, so it knows that,
> say, “python2-numpy” is installed.  It also knows that installing
> “htseq” is also going to install “python2-numpy” into that profile.  It
> also knows that the propagated “python2-numpy” is different from the
> installed “python2-numpy”.  Knowing all that, it should also know that
> there are going to be file conflicts, no?
>
> If that’s all true, could we make Guix print a warning about impending
> doom *before* it creates a new profile generation?  It’s a much nicer
> user experience (at least to me) when I’m being told about possible
> conflicts (at the package level) before Guix creates a new profile and
> nonchalantly informs me about arbitrarily resolving conflicts (at the
> file level).  The difference is in the amount of warnings I get (package
> vs individual files) and about the time I would otherwise have to waste
> to roll back the profile and upgrade already installed libraries or
> choose to install “htseq” into a new profile.

This would be nice, or at least, is there some kind of
"guix package --show-collisions" or something?

> 2) avoid PYTHONPATH, patch all Python files invasively!
>
> Python does not have any feature that is comparable to RUNPATH.  It is
> only concerned with finding libraries/modules by *name* in one of the
> directories specified by the PYTHONPATH environment variable.
>
> But actually the PYTHONPATH variable is not the only means to affect the
> search path for modules.  It is possible to change the search path
> programmatically:
>
>     import sys
>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>     import numpy
>
> The first two lines add an explicit store item path to the search path;
> the third line just imports the numpy module by name (as usual).  Even
> without setting the PYTHONPATH to include the numpy in the profile the
> third line won’t fail.
>
> I wonder if we could design a phase that — very much like the
> “wrap-program” phase — modifies *every* Python file and injects lines
> like the first two in the example above, appending explicit store item
> paths, so that all dependent Python libraries can be found without the
> need to have them installed in the same profile and without the need to
> set PYTHONPATH.
>
> Maybe this is crazy, and maybe this causes other annoying problems, but
> I think it is the closest we can get to a RUNPATH-like feature in
> Python.
>
> What do you think?  Do I need a long vacation or is this something we
> might realistically do in an automated fashion?
>
> ~~ Ricardo

So... that may work!  Except... well, I've raised this on another
thread, but I worry about late-binding based plugins or programs that
themselves call other python scripts (same with Guile, etc).  The
previous example I gave was, what if I write a Haunt extension library,
and I need to call "haunt.scm" and pull in that stuff?

  http://lists.gnu.org/archive/html/help-guix/2016-02/msg00032.html

What also about MediaGoblin plugins?  MediaGoblin allows you to define
plugins for different media types, and etc.  If you're adding a
"gmg_frobnicator" plugin, can MediaGoblin find gmg_frobnicator on its
PYTHONPATH when run?  And the reverse question: can gmg_frobnicator
import back into MediaGoblin and its dependencies?

I'm not sure...

Re: Python and propagation

[宋文武]

iyzsong@member.fsf.org (宋文武) writes:

>> [...]
>
> I packaged python-hy, and it does find all depends without propagations.
> In its output, the trick is done by ‘python-hy-0.11.1.pth’ and ‘site.py’.
>
> After read:
>   <https://docs.python.org/2/library/site.html>
>   <http://stackoverflow.com/questions/15208615/using-pth-files>
>
> I try:
> $ mkdir -p /tmp/o
> $ mkdir -p /tmp/a
> $ echo /tmp/a > /tmp/o/x.pth
> $ cat > /tmp/o/sitecustomize.py << EOF
>> import os, site
>> dir = os.path.dirname(os.path.abspath(__fil__))
>> site.addsitedir(dir)
>> EOF
> $ PYTHONPATH=/tmp/o python3 -c 'import sys; print(sys.path)
> And ‘/tmp/a’ is at the end of the list.
>
> So, I think if we get all the pth files correct, no need to use
> propagate-inputs :-)
Oops, I’m naive.  Be able to handle indirect inputs,  pth files should
be loaded recursively.  And this was the way in nixpkgs [1], but
deprecated now [2].

It’s safe to ignore me :-)

[1] https://github.com/NixOS/nixpkgs/pull/258
[2] https://github.com/NixOS/nixpkgs/pull/11086

Re: Python and propagation

[Federico Beffa]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> 1) print a warning when a collision is expected to happen, not when a
> collision has happened.

+1

> 2) avoid PYTHONPATH, patch all Python files invasively!
>
> Python does not have any feature that is comparable to RUNPATH.  It is
> only concerned with finding libraries/modules by *name* in one of the
> directories specified by the PYTHONPATH environment variable.
>
> But actually the PYTHONPATH variable is not the only means to affect the
> search path for modules.  It is possible to change the search path
> programmatically:
>
>     import sys
>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>     import numpy
>
> The first two lines add an explicit store item path to the search path;
> the third line just imports the numpy module by name (as usual).  Even
> without setting the PYTHONPATH to include the numpy in the profile the
> third line won’t fail.
>
> I wonder if we could design a phase that — very much like the
> “wrap-program” phase — modifies *every* Python file and injects lines
> like the first two in the example above, appending explicit store item
> paths, so that all dependent Python libraries can be found without the
> need to have them installed in the same profile and without the need to
> set PYTHONPATH.
>
> Maybe this is crazy, and maybe this causes other annoying problems, but
> I think it is the closest we can get to a RUNPATH-like feature in
> Python.
>
> What do you think?  Do I need a long vacation or is this something we
> might realistically do in an automated fashion?

Isn't the problem left of how the python interpreter can find those
patched libraries?

Maybe something along this lines could do:

* Instead of installing all files related to a module into profiles,
  only install a .pth file in 'site-packages' with a unique name. The
  name could even include the Guix package hash. This file should
  include the full path to the store ('real') package. This could maybe
  be achieved with 'python-build-system' creating two derivations: a
  derivation including only the .pth file and the 'real' package
  derivation (similar to a wrapped program pointing to the real one).

  For example

  $cd ~/tmp/ttmp
  $echo -e "aspell\nmechanics" > test.pth
  $cd
  $python
  >>> import sys
  >>> import site
  >>> site.addsitedir("/home/beffa/tmp/ttmp")
  >>> sys.path[-3:]
  ['/home/beffa/tmp/ttmp', '/home/beffa/tmp/ttmp/aspell',
  '/home/beffa/tmp/ttmp/mechanics']

* Then you can install many different versions of a package and they
  will not mask each other.

  To select a specific version we could then possibly use pkg_resources.

  For example:

  >>> import pkg_resources
  >>> pkg_resources.require("numpy==1.9.1")
  [numpy 1.9.1 (/gnu/store/hbdzccnvnlf54mflgcigqw2jv4ybippv-profile/lib/python3.4/site-packages)]
  >>> import numpy

  If this always works (I'm not sure), we could patch this info into the
  packages at compile time.

Regards,
Fede

Re: Python and propagation

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> As a Python library it depends on numpy, and thus propagates numpy as
> it is customary in Guix.  This is necessary because Python libraries
> do not have a RUNPATH feature

Does the newfangled “wheels” thing support something like a RUNPATH?

> 1) print a warning when a collision is expected to happen, not when a
> collision has happened.

Sounds like a good idea, indeed.

The file-by-file warnings may still be useful in other cases though, for
example when different packages genuinely provide same-named files.

> 2) avoid PYTHONPATH, patch all Python files invasively!

Sounds like an interesting option.  :-)  There are open issues though,
as you and others note (‘import’ can overridden!).

(This reminds me that, since Guile 2.2 uses ELF, it should be made to
honor RUNPATH!)

Ludo’.

Re: Python and propagation

[Ricardo Wurmus]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Ricardo Wurmus writes:
>> 2) avoid PYTHONPATH, patch all Python files invasively!

[...]

> So... that may work!  Except... well, I've raised this on another
> thread, but I worry about late-binding based plugins or programs that
> themselves call other python scripts (same with Guile, etc).  The
> previous example I gave was, what if I write a Haunt extension library,
> and I need to call "haunt.scm" and pull in that stuff?
>
>   http://lists.gnu.org/archive/html/help-guix/2016-02/msg00032.html
>
> What also about MediaGoblin plugins?  MediaGoblin allows you to define
> plugins for different media types, and etc.

I don’t understand the issues, primarily because I don’t understand if
the load path is a global mutable variable for the Python process, or if
it is local to the file, or if maybe it’s something else entirely.

I only proposed manipulating the load path before an “import” statement
because that’s how we can make the “import” statement succeed even if
the PYTHONPATH at programme start does not contain the directory holding
a needed library’s files.  “import” doesn’t allow us to use full paths,
so the idea was to “wrap” the “import” statement in an environment in
which the load path has been sufficiently augmented.  It would not be
feasible to replace every “import foo” statement with something like
this:

    old_load_path = sys.path
    sys.path.append("/gnu/store/cabba9e...-foo.../lib/...")
    import foo
    sys.path = old_load_path

so sys.path is instead just changed once per file.  It would be worth
figuring out how far these changes to sys.path reach — do they have a
global, process-wide effect?  (That would be inconvenient as it’s hard
to reason about a change like this, because ordering becomes important.)

How exactly to manipulate the load path — I don’t know.  We could indeed
recursively load .pth files if they exist, thereby adding things to the
search path.  (I did this in “(@@ (gnu packages music) solfege)” to make
pygtk work, for example.)  Or we could create a list of paths at build
time.

> If you're adding a "gmg_frobnicator" plugin, can MediaGoblin find
> gmg_frobnicator on its PYTHONPATH when run?

It would not affect the loading of plugins, as far as I imagine it,
because plugins would deal with their search paths by themselves (e.g. a
user adds them to the PYTHONPATH).  Since we only augment the load path
instead of replacing it, $PYTHONPATH will always be available.  So the
answer to this question is “yes”, I think.

> And the reverse question: can gmg_frobnicator import back into
> MediaGoblin and its dependencies?

I don’t know what “import back into” means.

I’m not a Python programmer, so I cannot come up with a simple example
project that would allow us to test such an invasive substitution
scheme.  Is anyone else here willing to give this a try?

~~ Ricardo

Re: Python and propagation

[Christopher Allan Webber]

Ricardo Wurmus writes:

>> If you're adding a "gmg_frobnicator" plugin, can MediaGoblin find
>> gmg_frobnicator on its PYTHONPATH when run?
>
> It would not affect the loading of plugins, as far as I imagine it,
> because plugins would deal with their search paths by themselves (e.g. a
> user adds them to the PYTHONPATH).  Since we only augment the load path
> instead of replacing it, $PYTHONPATH will always be available.  So the
> answer to this question is “yes”, I think.

Sounds like my worries are unfounded then.  I'm all for it!

Re: Python and propagation

[Danny Milosavljevic]

Or:

3) Use Python importlib[1] metapath functionality[2] in order to register your own module finder early[3].

You can override the entire module loading functionality like this, both the finding of modules and the loading them (off a stone tablet for example ^^).
You can even override builtin (!!) module loaders that way.

The default sys.metapath is [<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.PathFinder'>].

(All these classes do not need to be instantiated since you use them via classmethods)

The frozen importer is used in order to find dependencies in order to build a self-contained special python interpreter which includes all the dependencies you use (see Tools/freeze/ [4]).

So while I'm not sure what you want to achieve in detail (I've read some of the E-Mails in this thread but not every word), I think you can either use "freeze" or modify sitecustomize.py in order to make it magically modify metapath so it does what you want - without patching any of the other files.

(For comparison, check out what pip freeze does (despite the name, it just determines dependencies - a la ldd - and prints them))

On the other hand, virtualenv also builds a special Python interpreter.

Note that if you want to do something equivalent to 

  sys.path = ["a"]
  import x
  sys.path = ["b"]
  import x

if both a/x.py and b/x.py exist (and are different) and have the latter "import" import another x (after the former "import" succeeded), that's impossible and Python's semantics forbid that that works in any case. The latter import must use the already-loaded module (because of the same name). Also, the name "x" is visible in the remainder of the script that did the import, so no renaming either. Since Python does dynamic binding, you also can't just have the first "x" be visible in the first half and the second "x" in the second half - although (only) Python3 changed some stuff to allow lexical binding in some special cases.

Also not:

  sys.path = ["a"]
  import x
  sys.path = ["c"]
  import d

where
  c/d.py:
    sys.path = ["../b"]
    import x # nope! It's still the old one!

Also, it's customary to do this:

  a/__init__.py:
    x = 3
    import b
    y = 4
  a/b.py:
    import a
    print(a.x)
  main.py:
    import a
    print(a.y)

and it should work fine, printing 3 and 4. So while a module is being loaded, it should be able to import modules. (here, a/__init__.py is running, importing b, and b is using "a" although it's only half-there - b at this point in time also only sees the first half - i.e. it doesn't see y yet).

[1] https://docs.python.org/3/library/importlib.html
[2] https://www.python.org/dev/peps/pep-0302/ (search for "metapath")
[3] in lib/python.../site-packages/sitecustomize.py
[4] https://wiki.python.org/moin/Freeze