* No Explicit Python Version Dependency In Package Definitions? @ 2021-01-04 23:34 Phil 2021-01-04 23:51 ` Christopher Baines 0 siblings, 1 reply; 5+ messages in thread From: Phil @ 2021-01-04 23:34 UTC (permalink / raw) To: help-guix Hi, It seems standard not to declare python2 or python3 as a dependency on python package definitions - however other dependent python libraries are stated. eg python-scipy will declare dependencies on python-numpy and python-matplotlib - but not on a specific version of python package required to use it. I'm guessing this is to avoid tying packages to specific python releases, but I'm curious about the mechanics. It looks like 'package-with-python2' might be used to allow us to distinguish between python2 and python3, but ignoring the python2 case, I have the following python3 questions: Can we can install python-scipy without installing python3, given python isn't an explicit dependency in the package? Which site-packages directory under what python3 version will be used? A quick check looks that /path/to/profile/lib/python3.8/site-packages is currently used but what makes the decision to put them under python3.8 - especially if python3.8 isn't installed in the profile? What happens if Guix upgrades python3 from 3.8 -> 3.9? How are packages already installed under the 3.8, moved to the new 3.9 python version, is this seamless? If I'm using Guix on top of a foreign distro and don't have python3 installed as part of Guix, will my python-scipy end-up installed for my foreign distro's python install? Any pointers welcome! Thanks, Phil. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: No Explicit Python Version Dependency In Package Definitions? 2021-01-04 23:34 No Explicit Python Version Dependency In Package Definitions? Phil @ 2021-01-04 23:51 ` Christopher Baines 2021-01-05 14:59 ` Phil 0 siblings, 1 reply; 5+ messages in thread From: Christopher Baines @ 2021-01-04 23:51 UTC (permalink / raw) To: Phil; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 3554 bytes --] Phil <phil@beadling.co.uk> writes: > Hi, > > It seems standard not to declare python2 or python3 as a dependency on > python package definitions - however other dependent python libraries are stated. > > eg python-scipy will declare dependencies on python-numpy and > python-matplotlib - but not on a specific version of python package > required to use it. > > I'm guessing this is to avoid tying packages to specific python > releases, but I'm curious about the mechanics. Build systems are a mechanic to deduplicate common steps, but also common inputs between packages, and the python-build-system will include a default Python as an input. https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138 > It looks like 'package-with-python2' might be used to allow us to > distinguish between python2 and python3, but ignoring the python2 case, > I have the following python3 questions: > > Can we can install python-scipy without installing python3, given python > isn't an explicit dependency in the package? Guix packages don't have dependencies like other package managers, there's propagated-inputs which can act like dependencies where Guix will attempt to add propagated inputs to the profile when you install a package. But generally "dependencies" as in other things you should have when you have a given store item, are just references between store items: → guix gc --references /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2 /gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib /gnu/store/741057r2x06zwg6zcmqmdyv51spm6n9i-gfortran-7.5.0-lib /gnu/store/bs9pl1f805ins80xaf4s3n35a0x2lyq3-openblas-0.3.9 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31 /gnu/store/h8jw9qhyfp6fm6nb3cgh4335qhr31wfz-python-wrapper-3.8.2 /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2 /gnu/store/jxx8fr78jrcvpid5aplmkplbm1dk6czs-python-3.8.2 → grep -r jxx8fr78jrcvpid5aplmkplbm1dk6czs /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2 Binary file /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2/lib/python3.8/site-packages/scipy/optimize/_zeros.cpython-38-x86_64-linux-gnu.so matches ... So, you can't have this particular python-scipy output in your store without python as well, as it's referenced by some shared libraries, which I guess makes sense. > Which site-packages directory under what python3 version will be used? > A quick check looks that /path/to/profile/lib/python3.8/site-packages is > currently used but what makes the decision to put them under python3.8 - > especially if python3.8 isn't installed in the profile? The python build system takes care of where files end up in the package outputs at least: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/python-build-system.scm#n38 > What happens if Guix upgrades python3 from 3.8 -> 3.9? How are packages > already installed under the 3.8, moved to the new 3.9 python version, is this seamless? When the default python version is changed, the build system will change accordingly. > If I'm using Guix on top of a foreign distro and don't have python3 > installed as part of Guix, will my python-scipy end-up installed for my > foreign distro's python install? Installing Guix things is just ensuring things are in the /gnu/store, plus maybe some environment variables for the profile. So it won't end up where your (not Guix) distro installs Python things at least. Hope that helps! Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: No Explicit Python Version Dependency In Package Definitions? 2021-01-04 23:51 ` Christopher Baines @ 2021-01-05 14:59 ` Phil 2021-01-05 21:46 ` Christopher Baines 2021-01-05 22:56 ` Ricardo Wurmus 0 siblings, 2 replies; 5+ messages in thread From: Phil @ 2021-01-05 14:59 UTC (permalink / raw) To: Christopher Baines; +Cc: help-guix Thanks for the reply. Christopher Baines writes: > Build systems are a mechanic to deduplicate common steps, but also > common inputs between packages, and the python-build-system will include > a default Python as an input. > > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138 Got it - so the version is set behind the scenes depending on the current python package definition. > So, you can't have this particular python-scipy output in your store > without python as well, as it's referenced by some shared libraries, > which I guess makes sense. Yep - so even tho python 3.8 is not installed by installing eg python-scipy, the package is made available in the store as it's referenced. > When the default python version is changed, the build system will change > accordingly. Yep makes sense - so the python package will now reference the 3.9 package here instead of 3.8: ;; Current 3.x version. (define-public python-3 python-3.8) I'm guessing my local Guix would stay on 3.8 until I did a guix upgrade. At this point the new version of python-3 would force any python packages I had to reinstall against 3.9. I assume my local 3.8 system would be left untouched, such that I could rollback both the python version and my packages if I wanted? What would happen if I installed a new python package after pulling the latest package definitions tho? So I have a system say with python3.8 and python-scipy, and I decide I want to then install python-pandas (for example). Will it not then build this for python 3.9 (due to the new definition), if the version of python has incremented between the installs of python-scipy and python-pandas? Would I then have to manually reinstall python-scipy to have it under 3.9 (as well as 3.8) (or do a guix upgrade)? Last question, if today I wanted to create a profile that installed python-scipy against the python 3.9 package definition (which already exists in Guix it's just not the default). Do I have to change the python-3 definition as per above, or is there another way of saying "use python3.9 just for this profile". I suspect I could use a manifest to install 3.9 rather than the default, but won't any packages still depend on 3.8 unless I switch the python-3 definition? Thanks again! ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: No Explicit Python Version Dependency In Package Definitions? 2021-01-05 14:59 ` Phil @ 2021-01-05 21:46 ` Christopher Baines 2021-01-05 22:56 ` Ricardo Wurmus 1 sibling, 0 replies; 5+ messages in thread From: Christopher Baines @ 2021-01-05 21:46 UTC (permalink / raw) To: Phil; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 4093 bytes --] Phil <phil@beadling.co.uk> writes: > Thanks for the reply. > > Christopher Baines writes: > >> Build systems are a mechanic to deduplicate common steps, but also >> common inputs between packages, and the python-build-system will include >> a default Python as an input. >> >> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138 > > Got it - so the version is set behind the scenes depending on the > current python package definition. > >> So, you can't have this particular python-scipy output in your store >> without python as well, as it's referenced by some shared libraries, >> which I guess makes sense. > > Yep - so even tho python 3.8 is not installed by installing eg > python-scipy, the package is made available in the store as it's referenced. > >> When the default python version is changed, the build system will change >> accordingly. > > Yep makes sense - so the python package will now reference the 3.9 > package here instead of 3.8: > > ;; Current 3.x version. > (define-public python-3 python-3.8) > > I'm guessing my local Guix would stay on 3.8 until I did a guix > upgrade. At this point the new version of python-3 would force any > python packages I had to reinstall against 3.9. I assume my local 3.8 system > would be left untouched, such that I could rollback both the python > version and my packages if I wanted? What you say is roughly right, but it's not very representative of what happens. Upgrading to Guix that has Python 3.9 as the default from a version where 3.8 is the default will affect the build system behaviour as you've seen. Upgrading your profile will generate you a new generation where the Python packages are built with 3.9 (as that's what the updated Guix provides). Changes to profiles are not destructive, so you can roll back. > What would happen if I installed a new python package after pulling the > latest package definitions tho? Guix profiles work with search paths, and the PYTHONPATH is version specifc, so python@3.9 will look in lib/python3.9/site-packages. You can see this in action here: → guix environment --ad-hoc python python-pandas --search-paths export PATH="/gnu/store/08y11hsyflh1fdpkvs8f8snydkr9vq36-profile/bin${PATH:+:}$PATH" export PYTHONPATH="/gnu/store/08y11hsyflh1fdpkvs8f8snydkr9vq36-profile/lib/python3.8/site-packages${PYTHONPATH:+:}$PYTHONPATH" If you have a python@3.8 plus one python library built with python@3.8 and another python library built with python@3.9, the PYTHONPATH Guix generates will just include lib/python3.8/site-packages. > So I have a system say with python3.8 and python-scipy, and I decide I > want to then install python-pandas (for example). Will it not then > build this for python 3.9 (due to the new definition), if the version of > python has incremented between the installs of python-scipy and > python-pandas? Would I then have to manually reinstall python-scipy to > have it under 3.9 (as well as 3.8) (or do a guix upgrade)? Yep. > Last question, if today I wanted to create a profile that installed > python-scipy against the python 3.9 package definition (which already > exists in Guix it's just not the default). Do I have to change the > python-3 definition as per above, or is there another way of saying "use > python3.9 just for this profile". You can pass the python package to use to the python-build-system through the #:python argument. Look for examples of this in Guix, there's plenty for Python 2 only libraries. There are quite a few transformation options that you can use when building/installing packages, but I'm not aware of one that works with build system arguments. Maybe that could be added though. Without that, you'd need to either change the default python, or generate a variant of the library which uses the python you intend. > I suspect I could use a manifest to install 3.9 rather than the default, > but won't any packages still depend on 3.8 unless I switch the python-3 > definition? Yes. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: No Explicit Python Version Dependency In Package Definitions? 2021-01-05 14:59 ` Phil 2021-01-05 21:46 ` Christopher Baines @ 2021-01-05 22:56 ` Ricardo Wurmus 1 sibling, 0 replies; 5+ messages in thread From: Ricardo Wurmus @ 2021-01-05 22:56 UTC (permalink / raw) To: Phil; +Cc: help-guix Hi Phil, > So I have a system say with python3.8 and python-scipy, and I decide I > want to then install python-pandas (for example). Will it not then > build this for python 3.9 (due to the new definition), if the version of > python has incremented between the installs of python-scipy and > python-pandas? Would I then have to manually reinstall python-scipy to > have it under 3.9 (as well as 3.8) (or do a guix upgrade)? Yes. It’s important to upgrade your Python packages together to avoid weirdness. I suggest using a manifest. > Last question, if today I wanted to create a profile that installed > python-scipy against the python 3.9 package definition (which already > exists in Guix it's just not the default). Do I have to change the > python-3 definition as per above, or is there another way of saying "use > python3.9 just for this profile". > > I suspect I could use a manifest to install 3.9 rather than the default, > but won't any packages still depend on 3.8 unless I switch the python-3 > definition? Inside a manifest you can use the Guix package transformation API to replace all instances of one Python variant with another — recursively throughout the whole package graph. You’ll have to compile all those packages locally, though, because these package variants haven’t been built by the build farm. Luckily Guix does this for you. It just takes time (and electricity). -- Ricardo ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-01-05 22:56 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-04 23:34 No Explicit Python Version Dependency In Package Definitions? Phil 2021-01-04 23:51 ` Christopher Baines 2021-01-05 14:59 ` Phil 2021-01-05 21:46 ` Christopher Baines 2021-01-05 22:56 ` Ricardo Wurmus
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).