Note for Python packages: packages do not have "inputs" (most of the time)

Note for Python packages: packages do not have "inputs" (most of the time)

[Hartmut Goebel]

Hi,

I just found some Python packages recent checked in to master, which
have "inputs" defined.

All Python packages, please keep in mind:

*The default for including other Python packages is "propagated-inputs".*

guix import is wrong here, as it defaults to "inputs". Please refer to
the section "Specifying Dependencies" in info doc/guix "Python Modules".

Thanks!

Note: Currently specifying dependencies as "input" works by mere luck
and all these packages will fail with the new python build system. (And
I'm having a hard time cleaning up all these wrong inputs :-((

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Note for Python packages: packages do not have "inputs" (most of the time)

[ng0]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> I just found some Python packages recent checked in to master, which
> have "inputs" defined.
>
> All Python packages, please keep in mind:
>
> *The default for including other Python packages is "propagated-inputs".*
>
> guix import is wrong here, as it defaults to "inputs". Please refer to
> the section "Specifying Dependencies" in info doc/guix "Python Modules".
>
> Thanks!
>
> Note: Currently specifying dependencies as "input" works by mere luck
> and all these packages will fail with the new python build system. (And
> I'm having a hard time cleaning up all these wrong inputs :-((

I think this is wrong. My assumption is I work with the old system as
long as the new system is not in place. I see no changes which fix this,
so why should I do work in advance when this must be fixed afterwards?

Will the current system report packages as broken, or are they entirely
broken? From my perspective this reads weird. I have not read the new
documentation section, but I assumed this is not yet in place?

> -- 
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel          | h.goebel@crazy-compilers.com               |
> | www.crazy-compilers.com | compilers which you thought are impossible |
>
>
>

Re: Note for Python packages: packages do not have "inputs" (most of the time)

[Hartmut Goebel]

Am 23.10.2016 um 21:52 schrieb ng0:
> I think this is wrong. My assumption is I work with the old system as
> long as the new system is not in place. I see no changes which fix this,
> so why should I do work in advance when this must be fixed afterwards?
>
> Will the current system report packages as broken, or are they entirely
> broken? From my perspective this reads weird. I have not read the new
> documentation section, but I assumed this is not yet in place?

The new documentation section is already in place, since it is totally
independent from the new or the old build system.

Please note that I'm only asking to stop introducing more broken
packages (this is using "inputs" for python packages). I'm taking care
of the existing packages on the wip-python-build-system branch.

The current system does not detect that these packages are broken
(neither does the new one), but using "inputs" is conceptually broken,
as I'll describe the following:

Depending on the way the python packages are installed (pip,
easy_install, setup.py install with distutils, setup.py install with
setuptools, setup.py easy_install) what ends up in …/site-packages/ is
quite different. When using setup.py install with setuptools (which is
the most common way), a .pth-file will be created containing a link to
the required packages. See [Example 1 below]

Now imagine this:

Package P has packet "R" as input. When "P" gets installed, version 0.2
of "R" is in guix. The .pth-file (of packages P) refers to
/gnu/store/abcd…-R-0.2/… Everything is fine. Some time later, you
install into the same profile packet "Q" which required "R", too, but to
function correctly it requires at least version 0.4 of "R". The
.pth-file of packages Q refers to /gnu/store/1234…-R-0.4/…. But "R"
itself will not show up in the site-packages directory.

So you have two .pth-files in your site-packages directory, pointing to
different versions of the same package "R". Now when running python, all
.pth-files in site-packages are processed in alphabetical order (as
documented in [1]). So P.pth is processed prior to Q.pth and R-0.2.egg
will end up in front of R-0.4.egg. Package Q will not function correctly
since the old version of R will be imported.

[Example 2] is a real-world example for this. As you can seen, there are
*two* versions of werkzeug installed and version 0.11.5 imported. (In
this very case additionally "Werkzeug" will be in site-packages since it
is a propagated input for flask, resulting in werkzeug being here
*three* time.)

But it is even worse: Now install some package "A", requiring "R-0.4".
You will get a A.pth file, which will be processed even prior to P.pth,
and not (magically) Q functions correctly again. Huh!

So you have massive conflicts, and guix is not able to warn you about
the conflict. Since from the guix point of view everything is right.

And this is why using "inputs" is wrong and must be replaces by
"propagated-inputs".

When using propagated inputs, in the example above when installing Q,
R-0.4 would have been installed into site-packages. This alone would not
solve the import issue described above, but the problem would not be
hidden inside these .pth files (which barely anybody understands).


Example 1:
$ # Using guix 0.11.0
$ guix package -i python-flask
$ # should install mprfzzs3zy2z9zmlidh93g63nf70ki6z-python-flask-0.10.1
$ ls /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/
Flask-0.10.1-py3.4.egg/  __pycache__/  python-flask-0.10.1.pth  site.py
$ cat
/gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/*.pth  #
Output shortened
./Flask-0.10.1-py3.4.egg
/gnu/store/2h4fsdw…-python-itsdangerous-0.24/lib/python3.4/site-packages/itsdangerous-0.24-py3.4.egg
/gnu/store/pw81lwj…-python-jinja2-2.8/lib/python3.4/site-packages/Jinja2-2.8-py3.4.egg
/gnu/store/0r2wdiciw…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/brz263mir…-python-markupsafe-0.23/lib/python3.4/site-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
p+len(new)


[Example 2]
$ # Using guix 0.11.0
$ guix package -i python python-flask
$ # Using guix master as of today
$ ./pre-inst-env guix package -i python-pytest-localserver
$ grep -h werkzeug ~/.guix-profile/lib/python3.4/site-packages/*.pth  #
output shortend
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/r59535dq0…-python-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg
$ ~/.guix-profile/bin/python3 -c "import sys
for p in sys.path: 'werkzeug' in p and print(p)"
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/r59535dq0…-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg
$ ~/.guix-profile/bin/python3 -c "import werkzeug ;
print(werkzeug.__file__)"
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg/werkzeug/__init__.py

[1] https://docs.python.org/3/library/site.html

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Note for Python packages: packages do not have "inputs" (most of the time)

[Leo Famulari]

On Tue, Oct 25, 2016 at 11:48:42AM +0200, Hartmut Goebel wrote:
> Am 23.10.2016 um 21:52 schrieb ng0:
> > I think this is wrong. My assumption is I work with the old system as
> > long as the new system is not in place. I see no changes which fix this,
> > so why should I do work in advance when this must be fixed afterwards?
> >
> > Will the current system report packages as broken, or are they entirely
> > broken? From my perspective this reads weird. I have not read the new
> > documentation section, but I assumed this is not yet in place?
> 
> The new documentation section is already in place, since it is totally
> independent from the new or the old build system.
> 
> Please note that I'm only asking to stop introducing more broken
> packages (this is using "inputs" for python packages). I'm taking care
> of the existing packages on the wip-python-build-system branch.

Changing the subject: Is that branch stable? Can I start testing core
Python package upgrades on top of it?

Re: Note for Python packages: packages do not have "inputs" (most of the time)

[Hartmut Goebel]

Am 25.10.2016 um 14:49 schrieb Leo Famulari:
> Changing the subject: Is that branch stable? Can I start testing core
> Python package upgrades on top of it?

I worked on this branch, just fixed some last (hopefully) errors and
IMHO the branch is stable.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Note for Python packages: packages do not have "inputs" (most of the time)

[Leo Famulari]

On Tue, Oct 25, 2016 at 07:59:07PM +0200, Hartmut Goebel wrote:
> Am 25.10.2016 um 14:49 schrieb Leo Famulari:
> > Changing the subject: Is that branch stable? Can I start testing core
> > Python package upgrades on top of it?
> 
> I worked on this branch, just fixed some last (hopefully) errors and
> IMHO the branch is stable.

Great, I'm going to test a pytest update on top of it (locally). But
feel free to rebase the branch if necessary.