From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Re: Note for Python packages: packages do not have "inputs" (most of the time) Date: Tue, 25 Oct 2016 11:48:42 +0200 Message-ID: <580F2A7A.9080406@crazy-compilers.com> References: <580CF174.1000201@crazy-compilers.com> <87wpgyonb2.fsf@we.make.ritual.n0.is> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:59748) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1byyLi-0000kR-4r for guix-devel@gnu.org; Tue, 25 Oct 2016 05:48:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1byyLf-0000PZ-0r for guix-devel@gnu.org; Tue, 25 Oct 2016 05:48:50 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:50481) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1byyLe-0000P9-NH for guix-devel@gnu.org; Tue, 25 Oct 2016 05:48:46 -0400 In-Reply-To: <87wpgyonb2.fsf@we.make.ritual.n0.is> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: ng0 , guix-devel Am 23.10.2016 um 21:52 schrieb ng0: > I think this is wrong. My assumption is I work with the old system as > long as the new system is not in place. I see no changes which fix this= , > so why should I do work in advance when this must be fixed afterwards? > > Will the current system report packages as broken, or are they entirely= > broken? From my perspective this reads weird. I have not read the new > documentation section, but I assumed this is not yet in place? The new documentation section is already in place, since it is totally independent from the new or the old build system. Please note that I'm only asking to stop introducing more broken packages (this is using "inputs" for python packages). I'm taking care of the existing packages on the wip-python-build-system branch. The current system does not detect that these packages are broken (neither does the new one), but using "inputs" is conceptually broken, as I'll describe the following: Depending on the way the python packages are installed (pip, easy_install, setup.py install with distutils, setup.py install with setuptools, setup.py easy_install) what ends up in =E2=80=A6/site-package= s/ is quite different. When using setup.py install with setuptools (which is the most common way), a .pth-file will be created containing a link to the required packages. See [Example 1 below] Now imagine this: Package P has packet "R" as input. When "P" gets installed, version 0.2 of "R" is in guix. The .pth-file (of packages P) refers to /gnu/store/abcd=E2=80=A6-R-0.2/=E2=80=A6 Everything is fine. Some time la= ter, you install into the same profile packet "Q" which required "R", too, but to function correctly it requires at least version 0.4 of "R". The =2Epth-file of packages Q refers to /gnu/store/1234=E2=80=A6-R-0.4/=E2=80= =A6. But "R" itself will not show up in the site-packages directory. So you have two .pth-files in your site-packages directory, pointing to different versions of the same package "R". Now when running python, all =2Epth-files in site-packages are processed in alphabetical order (as documented in [1]). So P.pth is processed prior to Q.pth and R-0.2.egg will end up in front of R-0.4.egg. Package Q will not function correctly since the old version of R will be imported. [Example 2] is a real-world example for this. As you can seen, there are *two* versions of werkzeug installed and version 0.11.5 imported. (In this very case additionally "Werkzeug" will be in site-packages since it is a propagated input for flask, resulting in werkzeug being here *three* time.) But it is even worse: Now install some package "A", requiring "R-0.4". You will get a A.pth file, which will be processed even prior to P.pth, and not (magically) Q functions correctly again. Huh! So you have massive conflicts, and guix is not able to warn you about the conflict. Since from the guix point of view everything is right. And this is why using "inputs" is wrong and must be replaces by "propagated-inputs". When using propagated inputs, in the example above when installing Q, R-0.4 would have been installed into site-packages. This alone would not solve the import issue described above, but the problem would not be hidden inside these .pth files (which barely anybody understands). Example 1: $ # Using guix 0.11.0 $ guix package -i python-flask $ # should install mprfzzs3zy2z9zmlidh93g63nf70ki6z-python-flask-0.10.1 $ ls /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/ Flask-0.10.1-py3.4.egg/ __pycache__/ python-flask-0.10.1.pth site.py $ cat /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/*.pth # Output shortened =2E/Flask-0.10.1-py3.4.egg /gnu/store/2h4fsdw=E2=80=A6-python-itsdangerous-0.24/lib/python3.4/site-p= ackages/itsdangerous-0.24-py3.4.egg /gnu/store/pw81lwj=E2=80=A6-python-jinja2-2.8/lib/python3.4/site-packages= /Jinja2-2.8-py3.4.egg /gnu/store/0r2wdiciw=E2=80=A6-python-werkzeug-0.11.5/lib/python3.4/site-p= ackages/Werkzeug-0.11.5-py3.4.egg /gnu/store/brz263mir=E2=80=A6-python-markupsafe-0.23/lib/python3.4/site-p= ackages/MarkupSafe-0.23-py3.4-linux-x86_64.egg import sys; new=3Dsys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=3Dgetattr(sys,'__egginsert',0); sys.path[p:p]=3Dnew; sys.__egginsert =3D= p+len(new) [Example 2] $ # Using guix 0.11.0 $ guix package -i python python-flask $ # Using guix master as of today $ ./pre-inst-env guix package -i python-pytest-localserver $ grep -h werkzeug ~/.guix-profile/lib/python3.4/site-packages/*.pth # output shortend /gnu/store/0r2wdiciw1=E2=80=A6-python-werkzeug-0.11.5/lib/python3.4/site-= packages/Werkzeug-0.11.5-py3.4.egg /gnu/store/r59535dq0=E2=80=A6-python-werkzeug-0.11.11/lib/python3.4/site-= packages/Werkzeug-0.11.11-py3.4.egg $ ~/.guix-profile/bin/python3 -c "import sys for p in sys.path: 'werkzeug' in p and print(p)" /gnu/store/0r2wdiciw1=E2=80=A6-python-werkzeug-0.11.5/lib/python3.4/site-= packages/Werkzeug-0.11.5-py3.4.egg /gnu/store/r59535dq0=E2=80=A6-werkzeug-0.11.11/lib/python3.4/site-package= s/Werkzeug-0.11.11-py3.4.egg $ ~/.guix-profile/bin/python3 -c "import werkzeug ; print(werkzeug.__file__)" /gnu/store/0r2wdiciw1=E2=80=A6-python-werkzeug-0.11.5/lib/python3.4/site-= packages/Werkzeug-0.11.5-py3.4.egg/werkzeug/__init__.py [1] https://docs.python.org/3/library/site.html --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |