unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* No Explicit Python Version Dependency In Package Definitions?
@ 2021-01-04 23:34 Phil
  2021-01-04 23:51 ` Christopher Baines
  0 siblings, 1 reply; 5+ messages in thread
From: Phil @ 2021-01-04 23:34 UTC (permalink / raw)
  To: help-guix

Hi,

It seems standard not to declare python2 or python3 as a dependency on
python package definitions - however other dependent python libraries are stated.

eg python-scipy will declare dependencies on python-numpy and
python-matplotlib - but not on a specific version of python package
required to use it.

I'm guessing this is to avoid tying packages to specific python
releases, but I'm curious about the mechanics.

It looks like 'package-with-python2' might be used to allow us to
distinguish between python2 and python3, but ignoring the python2 case,
I have the following python3 questions:

Can we can install python-scipy without installing python3, given python
isn't an explicit dependency in the package?

Which site-packages directory under what python3 version will be used?
A quick check looks that /path/to/profile/lib/python3.8/site-packages is
currently used but what makes the decision to put them under python3.8 -
especially if python3.8 isn't installed in the profile?

What happens if Guix upgrades python3 from 3.8 -> 3.9?  How are packages
already installed under the 3.8, moved to the new 3.9 python version, is this seamless?

If I'm using Guix on top of a foreign distro and don't have python3
installed as part of Guix, will my python-scipy end-up installed for my
foreign distro's python install?

Any pointers welcome!

Thanks,
Phil.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: No Explicit Python Version Dependency In Package Definitions?
  2021-01-04 23:34 No Explicit Python Version Dependency In Package Definitions? Phil
@ 2021-01-04 23:51 ` Christopher Baines
  2021-01-05 14:59   ` Phil
  0 siblings, 1 reply; 5+ messages in thread
From: Christopher Baines @ 2021-01-04 23:51 UTC (permalink / raw)
  To: Phil; +Cc: help-guix

[-- Attachment #1: Type: text/plain, Size: 3554 bytes --]


Phil <phil@beadling.co.uk> writes:

> Hi,
>
> It seems standard not to declare python2 or python3 as a dependency on
> python package definitions - however other dependent python libraries are stated.
>
> eg python-scipy will declare dependencies on python-numpy and
> python-matplotlib - but not on a specific version of python package
> required to use it.
>
> I'm guessing this is to avoid tying packages to specific python
> releases, but I'm curious about the mechanics.

Build systems are a mechanic to deduplicate common steps, but also
common inputs between packages, and the python-build-system will include
a default Python as an input.

https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138

> It looks like 'package-with-python2' might be used to allow us to
> distinguish between python2 and python3, but ignoring the python2 case,
> I have the following python3 questions:
>
> Can we can install python-scipy without installing python3, given python
> isn't an explicit dependency in the package?

Guix packages don't have dependencies like other package managers,
there's propagated-inputs which can act like dependencies where Guix
will attempt to add propagated inputs to the profile when you install a
package. But generally "dependencies" as in other things you should have
when you have a given store item, are just references between store
items:

→ guix gc --references /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2
/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib
/gnu/store/741057r2x06zwg6zcmqmdyv51spm6n9i-gfortran-7.5.0-lib
/gnu/store/bs9pl1f805ins80xaf4s3n35a0x2lyq3-openblas-0.3.9
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
/gnu/store/h8jw9qhyfp6fm6nb3cgh4335qhr31wfz-python-wrapper-3.8.2
/gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2
/gnu/store/jxx8fr78jrcvpid5aplmkplbm1dk6czs-python-3.8.2

→ grep -r jxx8fr78jrcvpid5aplmkplbm1dk6czs /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2
Binary file /gnu/store/jj1915czcy6wlf6riajapl1j6dfgwwii-python-scipy-1.3.2/lib/python3.8/site-packages/scipy/optimize/_zeros.cpython-38-x86_64-linux-gnu.so matches
...

So, you can't have this particular python-scipy output in your store
without python as well, as it's referenced by some shared libraries,
which I guess makes sense.

> Which site-packages directory under what python3 version will be used?
> A quick check looks that /path/to/profile/lib/python3.8/site-packages is
> currently used but what makes the decision to put them under python3.8 -
> especially if python3.8 isn't installed in the profile?

The python build system takes care of where files end up in the package
outputs at least:

https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/python-build-system.scm#n38

> What happens if Guix upgrades python3 from 3.8 -> 3.9?  How are packages
> already installed under the 3.8, moved to the new 3.9 python version, is this seamless?

When the default python version is changed, the build system will change
accordingly.

> If I'm using Guix on top of a foreign distro and don't have python3
> installed as part of Guix, will my python-scipy end-up installed for my
> foreign distro's python install?

Installing Guix things is just ensuring things are in the /gnu/store,
plus maybe some environment variables for the profile. So it won't end
up where your (not Guix) distro installs Python things at least.

Hope that helps!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: No Explicit Python Version Dependency In Package Definitions?
  2021-01-04 23:51 ` Christopher Baines
@ 2021-01-05 14:59   ` Phil
  2021-01-05 21:46     ` Christopher Baines
  2021-01-05 22:56     ` Ricardo Wurmus
  0 siblings, 2 replies; 5+ messages in thread
From: Phil @ 2021-01-05 14:59 UTC (permalink / raw)
  To: Christopher Baines; +Cc: help-guix

Thanks for the reply.

Christopher Baines writes:

> Build systems are a mechanic to deduplicate common steps, but also
> common inputs between packages, and the python-build-system will include
> a default Python as an input.
>
> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138

Got it - so the version is set behind the scenes depending on the
current python package definition.

> So, you can't have this particular python-scipy output in your store
> without python as well, as it's referenced by some shared libraries,
> which I guess makes sense.

Yep - so even tho python 3.8 is not installed by installing eg
python-scipy, the package is made available in the store as it's referenced.

> When the default python version is changed, the build system will change
> accordingly.

Yep makes sense - so the python package will now reference the 3.9
package here instead of 3.8:

;; Current 3.x version.
(define-public python-3 python-3.8)

I'm guessing my local Guix would stay on 3.8 until I did a guix
upgrade.  At this point the new version of python-3 would force any
python packages I had to reinstall against 3.9.  I assume my local 3.8 system
would be left untouched, such that I could rollback both the python
version and my packages if I wanted?

What would happen if I installed a new python package after pulling the
latest package definitions tho?

So I have a system say with python3.8 and python-scipy, and I decide I
want to then install python-pandas (for example).  Will it not then
build this for python 3.9 (due to the new definition), if the version of
python has incremented between the installs of python-scipy and
python-pandas?  Would I then have to manually reinstall python-scipy to
have it under 3.9 (as well as 3.8) (or do a guix upgrade)?

Last question, if today I wanted to create a profile that installed
python-scipy against the python 3.9 package definition (which already
exists in Guix it's just not the default).  Do I have to change the
python-3 definition as per above, or is there another way of saying "use
python3.9 just for this profile".

I suspect I could use a manifest to install 3.9 rather than the default,
but won't any packages still depend on 3.8 unless I switch the python-3
definition?

Thanks again!





^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: No Explicit Python Version Dependency In Package Definitions?
  2021-01-05 14:59   ` Phil
@ 2021-01-05 21:46     ` Christopher Baines
  2021-01-05 22:56     ` Ricardo Wurmus
  1 sibling, 0 replies; 5+ messages in thread
From: Christopher Baines @ 2021-01-05 21:46 UTC (permalink / raw)
  To: Phil; +Cc: help-guix

[-- Attachment #1: Type: text/plain, Size: 4093 bytes --]


Phil <phil@beadling.co.uk> writes:

> Thanks for the reply.
>
> Christopher Baines writes:
>
>> Build systems are a mechanic to deduplicate common steps, but also
>> common inputs between packages, and the python-build-system will include
>> a default Python as an input.
>>
>> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm#n138
>
> Got it - so the version is set behind the scenes depending on the
> current python package definition.
>
>> So, you can't have this particular python-scipy output in your store
>> without python as well, as it's referenced by some shared libraries,
>> which I guess makes sense.
>
> Yep - so even tho python 3.8 is not installed by installing eg
> python-scipy, the package is made available in the store as it's referenced.
>
>> When the default python version is changed, the build system will change
>> accordingly.
>
> Yep makes sense - so the python package will now reference the 3.9
> package here instead of 3.8:
>
> ;; Current 3.x version.
> (define-public python-3 python-3.8)
>
> I'm guessing my local Guix would stay on 3.8 until I did a guix
> upgrade.  At this point the new version of python-3 would force any
> python packages I had to reinstall against 3.9.  I assume my local 3.8 system
> would be left untouched, such that I could rollback both the python
> version and my packages if I wanted?

What you say is roughly right, but it's not very representative of what
happens. Upgrading to Guix that has Python 3.9 as the default from a
version where 3.8 is the default will affect the build system behaviour
as you've seen. Upgrading your profile will generate you a new
generation where the Python packages are built with 3.9 (as that's what
the updated Guix provides).

Changes to profiles are not destructive, so you can roll back.

> What would happen if I installed a new python package after pulling the
> latest package definitions tho?

Guix profiles work with search paths, and the PYTHONPATH is version
specifc, so python@3.9 will look in lib/python3.9/site-packages. You can
see this in action here:

→ guix environment --ad-hoc python python-pandas --search-paths
export PATH="/gnu/store/08y11hsyflh1fdpkvs8f8snydkr9vq36-profile/bin${PATH:+:}$PATH"
export PYTHONPATH="/gnu/store/08y11hsyflh1fdpkvs8f8snydkr9vq36-profile/lib/python3.8/site-packages${PYTHONPATH:+:}$PYTHONPATH"

If you have a python@3.8 plus one python library built with python@3.8
and another python library built with python@3.9, the PYTHONPATH Guix
generates will just include lib/python3.8/site-packages.

> So I have a system say with python3.8 and python-scipy, and I decide I
> want to then install python-pandas (for example).  Will it not then
> build this for python 3.9 (due to the new definition), if the version of
> python has incremented between the installs of python-scipy and
> python-pandas?  Would I then have to manually reinstall python-scipy to
> have it under 3.9 (as well as 3.8) (or do a guix upgrade)?

Yep.

> Last question, if today I wanted to create a profile that installed
> python-scipy against the python 3.9 package definition (which already
> exists in Guix it's just not the default).  Do I have to change the
> python-3 definition as per above, or is there another way of saying "use
> python3.9 just for this profile".

You can pass the python package to use to the python-build-system
through the #:python argument. Look for examples of this in Guix,
there's plenty for Python 2 only libraries.

There are quite a few transformation options that you can use when
building/installing packages, but I'm not aware of one that works with
build system arguments. Maybe that could be added though. Without that,
you'd need to either change the default python, or generate a variant of
the library which uses the python you intend.

> I suspect I could use a manifest to install 3.9 rather than the default,
> but won't any packages still depend on 3.8 unless I switch the python-3
> definition?

Yes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: No Explicit Python Version Dependency In Package Definitions?
  2021-01-05 14:59   ` Phil
  2021-01-05 21:46     ` Christopher Baines
@ 2021-01-05 22:56     ` Ricardo Wurmus
  1 sibling, 0 replies; 5+ messages in thread
From: Ricardo Wurmus @ 2021-01-05 22:56 UTC (permalink / raw)
  To: Phil; +Cc: help-guix


Hi Phil,

> So I have a system say with python3.8 and python-scipy, and I decide I
> want to then install python-pandas (for example).  Will it not then
> build this for python 3.9 (due to the new definition), if the version of
> python has incremented between the installs of python-scipy and
> python-pandas?  Would I then have to manually reinstall python-scipy to
> have it under 3.9 (as well as 3.8) (or do a guix upgrade)?

Yes.  It’s important to upgrade your Python packages together to avoid
weirdness.  I suggest using a manifest.

> Last question, if today I wanted to create a profile that installed
> python-scipy against the python 3.9 package definition (which already
> exists in Guix it's just not the default).  Do I have to change the
> python-3 definition as per above, or is there another way of saying "use
> python3.9 just for this profile".
>
> I suspect I could use a manifest to install 3.9 rather than the default,
> but won't any packages still depend on 3.8 unless I switch the python-3
> definition?

Inside a manifest you can use the Guix package transformation API to
replace all instances of one Python variant with another — recursively
throughout the whole package graph.  You’ll have to compile all those
packages locally, though, because these package variants haven’t been
built by the build farm.  Luckily Guix does this for you.  It just takes
time (and electricity).

-- 
Ricardo


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-01-05 22:56 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-04 23:34 No Explicit Python Version Dependency In Package Definitions? Phil
2021-01-04 23:51 ` Christopher Baines
2021-01-05 14:59   ` Phil
2021-01-05 21:46     ` Christopher Baines
2021-01-05 22:56     ` Ricardo Wurmus

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).