* Note for Python packages: packages do not have "inputs" (most of the time) @ 2016-10-23 17:20 Hartmut Goebel 2016-10-23 19:52 ` ng0 0 siblings, 1 reply; 6+ messages in thread From: Hartmut Goebel @ 2016-10-23 17:20 UTC (permalink / raw) To: guix-devel Hi, I just found some Python packages recent checked in to master, which have "inputs" defined. All Python packages, please keep in mind: *The default for including other Python packages is "propagated-inputs".* guix import is wrong here, as it defaults to "inputs". Please refer to the section "Specifying Dependencies" in info doc/guix "Python Modules". Thanks! Note: Currently specifying dependencies as "input" works by mere luck and all these packages will fail with the new python build system. (And I'm having a hard time cleaning up all these wrong inputs :-(( -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Note for Python packages: packages do not have "inputs" (most of the time) 2016-10-23 17:20 Note for Python packages: packages do not have "inputs" (most of the time) Hartmut Goebel @ 2016-10-23 19:52 ` ng0 2016-10-25 9:48 ` Hartmut Goebel 0 siblings, 1 reply; 6+ messages in thread From: ng0 @ 2016-10-23 19:52 UTC (permalink / raw) To: Hartmut Goebel, guix-devel Hartmut Goebel <h.goebel@crazy-compilers.com> writes: > Hi, > > I just found some Python packages recent checked in to master, which > have "inputs" defined. > > All Python packages, please keep in mind: > > *The default for including other Python packages is "propagated-inputs".* > > guix import is wrong here, as it defaults to "inputs". Please refer to > the section "Specifying Dependencies" in info doc/guix "Python Modules". > > Thanks! > > Note: Currently specifying dependencies as "input" works by mere luck > and all these packages will fail with the new python build system. (And > I'm having a hard time cleaning up all these wrong inputs :-(( I think this is wrong. My assumption is I work with the old system as long as the new system is not in place. I see no changes which fix this, so why should I do work in advance when this must be fixed afterwards? Will the current system report packages as broken, or are they entirely broken? From my perspective this reads weird. I have not read the new documentation section, but I assumed this is not yet in place? > -- > Regards > Hartmut Goebel > > | Hartmut Goebel | h.goebel@crazy-compilers.com | > | www.crazy-compilers.com | compilers which you thought are impossible | > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Note for Python packages: packages do not have "inputs" (most of the time) 2016-10-23 19:52 ` ng0 @ 2016-10-25 9:48 ` Hartmut Goebel 2016-10-25 12:49 ` Leo Famulari 0 siblings, 1 reply; 6+ messages in thread From: Hartmut Goebel @ 2016-10-25 9:48 UTC (permalink / raw) To: ng0, guix-devel Am 23.10.2016 um 21:52 schrieb ng0: > I think this is wrong. My assumption is I work with the old system as > long as the new system is not in place. I see no changes which fix this, > so why should I do work in advance when this must be fixed afterwards? > > Will the current system report packages as broken, or are they entirely > broken? From my perspective this reads weird. I have not read the new > documentation section, but I assumed this is not yet in place? The new documentation section is already in place, since it is totally independent from the new or the old build system. Please note that I'm only asking to stop introducing more broken packages (this is using "inputs" for python packages). I'm taking care of the existing packages on the wip-python-build-system branch. The current system does not detect that these packages are broken (neither does the new one), but using "inputs" is conceptually broken, as I'll describe the following: Depending on the way the python packages are installed (pip, easy_install, setup.py install with distutils, setup.py install with setuptools, setup.py easy_install) what ends up in …/site-packages/ is quite different. When using setup.py install with setuptools (which is the most common way), a .pth-file will be created containing a link to the required packages. See [Example 1 below] Now imagine this: Package P has packet "R" as input. When "P" gets installed, version 0.2 of "R" is in guix. The .pth-file (of packages P) refers to /gnu/store/abcd…-R-0.2/… Everything is fine. Some time later, you install into the same profile packet "Q" which required "R", too, but to function correctly it requires at least version 0.4 of "R". The .pth-file of packages Q refers to /gnu/store/1234…-R-0.4/…. But "R" itself will not show up in the site-packages directory. So you have two .pth-files in your site-packages directory, pointing to different versions of the same package "R". Now when running python, all .pth-files in site-packages are processed in alphabetical order (as documented in [1]). So P.pth is processed prior to Q.pth and R-0.2.egg will end up in front of R-0.4.egg. Package Q will not function correctly since the old version of R will be imported. [Example 2] is a real-world example for this. As you can seen, there are *two* versions of werkzeug installed and version 0.11.5 imported. (In this very case additionally "Werkzeug" will be in site-packages since it is a propagated input for flask, resulting in werkzeug being here *three* time.) But it is even worse: Now install some package "A", requiring "R-0.4". You will get a A.pth file, which will be processed even prior to P.pth, and not (magically) Q functions correctly again. Huh! So you have massive conflicts, and guix is not able to warn you about the conflict. Since from the guix point of view everything is right. And this is why using "inputs" is wrong and must be replaces by "propagated-inputs". When using propagated inputs, in the example above when installing Q, R-0.4 would have been installed into site-packages. This alone would not solve the import issue described above, but the problem would not be hidden inside these .pth files (which barely anybody understands). Example 1: $ # Using guix 0.11.0 $ guix package -i python-flask $ # should install mprfzzs3zy2z9zmlidh93g63nf70ki6z-python-flask-0.10.1 $ ls /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/ Flask-0.10.1-py3.4.egg/ __pycache__/ python-flask-0.10.1.pth site.py $ cat /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/*.pth # Output shortened ./Flask-0.10.1-py3.4.egg /gnu/store/2h4fsdw…-python-itsdangerous-0.24/lib/python3.4/site-packages/itsdangerous-0.24-py3.4.egg /gnu/store/pw81lwj…-python-jinja2-2.8/lib/python3.4/site-packages/Jinja2-2.8-py3.4.egg /gnu/store/0r2wdiciw…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg /gnu/store/brz263mir…-python-markupsafe-0.23/lib/python3.4/site-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) [Example 2] $ # Using guix 0.11.0 $ guix package -i python python-flask $ # Using guix master as of today $ ./pre-inst-env guix package -i python-pytest-localserver $ grep -h werkzeug ~/.guix-profile/lib/python3.4/site-packages/*.pth # output shortend /gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg /gnu/store/r59535dq0…-python-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg $ ~/.guix-profile/bin/python3 -c "import sys for p in sys.path: 'werkzeug' in p and print(p)" /gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg /gnu/store/r59535dq0…-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg $ ~/.guix-profile/bin/python3 -c "import werkzeug ; print(werkzeug.__file__)" /gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg/werkzeug/__init__.py [1] https://docs.python.org/3/library/site.html -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Note for Python packages: packages do not have "inputs" (most of the time) 2016-10-25 9:48 ` Hartmut Goebel @ 2016-10-25 12:49 ` Leo Famulari 2016-10-25 17:59 ` Hartmut Goebel 0 siblings, 1 reply; 6+ messages in thread From: Leo Famulari @ 2016-10-25 12:49 UTC (permalink / raw) To: Hartmut Goebel; +Cc: guix-devel On Tue, Oct 25, 2016 at 11:48:42AM +0200, Hartmut Goebel wrote: > Am 23.10.2016 um 21:52 schrieb ng0: > > I think this is wrong. My assumption is I work with the old system as > > long as the new system is not in place. I see no changes which fix this, > > so why should I do work in advance when this must be fixed afterwards? > > > > Will the current system report packages as broken, or are they entirely > > broken? From my perspective this reads weird. I have not read the new > > documentation section, but I assumed this is not yet in place? > > The new documentation section is already in place, since it is totally > independent from the new or the old build system. > > Please note that I'm only asking to stop introducing more broken > packages (this is using "inputs" for python packages). I'm taking care > of the existing packages on the wip-python-build-system branch. Changing the subject: Is that branch stable? Can I start testing core Python package upgrades on top of it? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Note for Python packages: packages do not have "inputs" (most of the time) 2016-10-25 12:49 ` Leo Famulari @ 2016-10-25 17:59 ` Hartmut Goebel 2016-10-25 18:05 ` Leo Famulari 0 siblings, 1 reply; 6+ messages in thread From: Hartmut Goebel @ 2016-10-25 17:59 UTC (permalink / raw) To: Leo Famulari; +Cc: guix-devel Am 25.10.2016 um 14:49 schrieb Leo Famulari: > Changing the subject: Is that branch stable? Can I start testing core > Python package upgrades on top of it? I worked on this branch, just fixed some last (hopefully) errors and IMHO the branch is stable. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Note for Python packages: packages do not have "inputs" (most of the time) 2016-10-25 17:59 ` Hartmut Goebel @ 2016-10-25 18:05 ` Leo Famulari 0 siblings, 0 replies; 6+ messages in thread From: Leo Famulari @ 2016-10-25 18:05 UTC (permalink / raw) To: Hartmut Goebel; +Cc: guix-devel On Tue, Oct 25, 2016 at 07:59:07PM +0200, Hartmut Goebel wrote: > Am 25.10.2016 um 14:49 schrieb Leo Famulari: > > Changing the subject: Is that branch stable? Can I start testing core > > Python package upgrades on top of it? > > I worked on this branch, just fixed some last (hopefully) errors and > IMHO the branch is stable. Great, I'm going to test a pytest update on top of it (locally). But feel free to rebase the branch if necessary. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-10-25 18:05 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-10-23 17:20 Note for Python packages: packages do not have "inputs" (most of the time) Hartmut Goebel 2016-10-23 19:52 ` ng0 2016-10-25 9:48 ` Hartmut Goebel 2016-10-25 12:49 ` Leo Famulari 2016-10-25 17:59 ` Hartmut Goebel 2016-10-25 18:05 ` Leo Famulari
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.