installing python 2 and python 3 in the same profile

installing python 2 and python 3 in the same profile

[Ricardo Wurmus]

Hi Guix,

with the introduction of the collision avoidance feature that prevents
you from installing different variants of the same package into a
profile we made it impossible to install “python” and “python@2” into
the same profile.

It still works with ad-hoc environments, but manifests containing both
Python versions cannot be instantiated any more.  This is too strict,
because we know that these two variants don’t cause conflicts.

What can we do to make this feature a little smarter?

How about a package property that defines a “conflicts?” predicate that
takes two packages of the same name and determines (e.g. by checking the
major version) if these two packages are conflicting?  If no such
predicate is provided we assume that packages with the same name cause a
conflict and prevent installation.

--
Ricardo

Re: installing python 2 and python 3 in the same profile

[Pjotr Prins]

On Wed, Mar 07, 2018 at 08:50:48AM +0100, Ricardo Wurmus wrote:
> Hi Guix,
> 
> with the introduction of the collision avoidance feature that prevents
> you from installing different variants of the same package into a
> profile we made it impossible to install “python” and “python@2” into
> the same profile.

I am happy with this idea. Maybe guix can give an informative message
in the case of Python.

Generally, mixing Python2 and 3 in one profile makes no sense. During a
course we gave this week we saw multiple occurances of students
messing up with python versions inside notebooks(!) That is how bad it
gets. I think the Python community rather screwed up on that one. That
is why people use virtualenv which is a full copy of a Python setup.

> It still works with ad-hoc environments, but manifests containing both
> Python versions cannot be instantiated any more.  This is too strict,
> because we know that these two variants don’t cause conflicts.

That is not my experience. Any mix is a problem.

> What can we do to make this feature a little smarter?
> 
> How about a package property that defines a “conflicts?” predicate that
> takes two packages of the same name and determines (e.g. by checking the
> major version) if these two packages are conflicting?  If no such
> predicate is provided we assume that packages with the same name cause a
> conflict and prevent installation.

I suggest not to go there. Teach people about profiles and
environments and all should be fine. You can support Python 3, Python
2.7 and Python 2.4 (etc), but it is better not to put them in one
profile. A profile represents a sane environment. That is how it
should be. 

I must add that in most of my Guix experience I relied on
$HOME/.guix-profile. But these days I hardly use that anymore. Just
for some basic desktop tooling. Using multiple profiles is the only
way to mix versions and keep a level of sanity.

Pj.
-- 

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

On 10/03/2018 09:34, Pjotr Prins wrote:

>> It still works with ad-hoc environments, but manifests containing both
>> Python versions cannot be instantiated any more.  This is too strict,
>> because we know that these two variants don’t cause conflicts.
> 
> That is not my experience. Any mix is a problem.

I agree. Even if "it works" superficially, it's hard to do serious work 
with Python when you have the two versions in the same profile.

In fact, I find that Guix is the best package manager for dealing with 
this precisely because it allows me to have multiple independent profiles.

> I suggest not to go there. Teach people about profiles and
> environments and all should be fine. You can support Python 3, Python

Yes, and perhaps make working with profiles a bit easier. For example, 
I'd appreciate a "guix profile" resembling "guix environment":

    guix profile ~/profiles/python2

would start a shell with the python2 profile, and

    guix profile ~/profiles/python2 command

would run "command" in the context of the profile. This is actually on 
my to-do list.

> I must add that in most of my Guix experience I relied on
> $HOME/.guix-profile. But these days I hardly use that anymore. Just
> for some basic desktop tooling. Using multiple profiles is the only
> way to mix versions and keep a level of sanity.

Same here.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Wed, Mar 07, 2018 at 08:50:48AM +0100, Ricardo Wurmus wrote:
>> Hi Guix,
>>
>> with the introduction of the collision avoidance feature that prevents
>> you from installing different variants of the same package into a
>> profile we made it impossible to install “python” and “python@2” into
>> the same profile.
>
> I am happy with this idea. Maybe guix can give an informative message
> in the case of Python.
>
> Generally, mixing Python2 and 3 in one profile makes no sense.

Obviously, I strongly disagree :)

On my CentOS and Fedora machines at work both versions are installed
(with the system package manager) and nobody complains.  I can conjure
up a Python 2 environment by calling “python2” and a Python 3
environment by calling “python3” or “python”.  Neither variant will load
up modules for the other variant — unlike the behaviour on Guix.

Even in Guix we allow these packages to be added to the same environment
with “guix environment --ad-hoc” – and there are no collisions, because
upstream made the effort to ensure that these packages don’t provide
files of the same names.

It is an unnecessary restriction to *prevent* users from installing
Python 2 and 3 interpreters into the same profile.  Any errors we see
with mixing 2 and 3 is due to a bug in Guix which is due to our use of
PYTHONPATH.

In large environments Python 2 and 3 modules and interpreters can
peacefully coexist as long as the tools use the appropriate names for
the interpreters (“python2” or “python” for Python 2, “python3” for
the interpreter for Python 3).

In other places in Guix we don’t even prevent users from doing things
that are generally ill-advised (e.g. the use of LD_LIBRARY_PATH); why
would we prevent a perfectly valid use-case when it comes to different
versions of Python?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing python 2 and python 3 in the same profile

[Pjotr Prins]

On Sun, Mar 11, 2018 at 12:10:09PM +0100, Ricardo Wurmus wrote:
> Obviously, I strongly disagree :)
 
You should ;)

> On my CentOS and Fedora machines at work both versions are installed
> (with the system package manager) and nobody complains.  I can conjure
> up a Python 2 environment by calling “python2” and a Python 3
> environment by calling “python3” or “python”.  Neither variant will load
> up modules for the other variant — unlike the behaviour on Guix.
> 
> Even in Guix we allow these packages to be added to the same environment
> with “guix environment --ad-hoc” – and there are no collisions, because
> upstream made the effort to ensure that these packages don’t provide
> files of the same names.
> 
> It is an unnecessary restriction to *prevent* users from installing
> Python 2 and 3 interpreters into the same profile.  Any errors we see
> with mixing 2 and 3 is due to a bug in Guix which is due to our use of
> PYTHONPATH.

Alright, if you choose to use them as two different tools/languages.

But even in minor versions there are problems. Ruby has those too, I
know for a fact, though Python may be worse since it uses compiled
sources.

> In large environments Python 2 and 3 modules and interpreters can
> peacefully coexist as long as the tools use the appropriate names for
> the interpreters (“python2” or “python” for Python 2, “python3” for
> the interpreter for Python 3).
> 
> In other places in Guix we don’t even prevent users from doing things
> that are generally ill-advised (e.g. the use of LD_LIBRARY_PATH); why
> would we prevent a perfectly valid use-case when it comes to different
> versions of Python?

Sure. Note that Python2 is quickly becoming obsolete. It is escalating
because packages are now dropping support. After 10 years of joy
mixing versions...

Pj.

Re: installing python 2 and python 3 in the same profile

[Hartmut Goebel]

Am 11.03.2018 um 13:42 schrieb Pjotr Prins:
> But even in minor versions there are problems.

Yes, but it's not guix's job to solve the language programmer's problems.

Since quite some time, then python itnerpreters get installed as
"pythonX.Y", in addition to "pythonX" and "python". Thus it is perfectly
possible to install several version of python in parallel, give that one
decides which should mak to "pythonX" and which to "python".

The libraries are always installed into version-specific directories.

>  Ruby has those too, I
> know for a fact, though Python may be worse since it uses compiled
> sources.

As I've been told, it's worse in Ruby, since rube is using the same
gem-directories for all versions. In contrast, python uses different
(see above) and if you need to pin specific package versions, virtual
environments are for the rescue and quite light-weight.

Neverthelesse these are problems out of scope for guix :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: installing python 2 and python 3 in the same profile

[Pjotr Prins]

On Sun, Mar 11, 2018 at 02:30:25PM +0100, Hartmut Goebel wrote:
> As I've been told, it's worse in Ruby, since rube is using the same
> gem-directories for all versions. 

That is not accurate.  Ruby uses even 'stable' numbers of minor
versions to split out. So 2.4.x and 2.6.x get installed in different
lib site dirs. The assumption is that everything in between is
compatible. If fact, very similar to what Python does.

I think they are on par when it comes to that part of the solution.
Ruby is arguably a bit better off because sources are compiled every
time from source.  Thim means that potential conflicts may be a bit
likelier to be caught at runtime.

> In contrast, python uses different (see above) and if you need to
> pin specific package versions, virtual environments are for the
> rescue and quite light-weight.

I know. Still, it is not completely independent per *installation*.

> Neverthelesse these are problems out of scope for guix :-)

Actually the Python and Ruby mess is fixable using (1) Guix profiles
and (2) including the hash in the local $HOME install path. That is
what I do.  That enforces perfect isolation between versions. No more
pain after and no need for Ruby bundler or Python virtualenv. I have
no use for those tools.

Pj.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

Ricardo Wurmus <rekado@elephly.net> writes:

> It is an unnecessary restriction to *prevent* users from installing
> Python 2 and 3 interpreters into the same profile.  Any errors we see

I agree. But the current question is not if we should allow people to
shoot themselves into the foot, but how much effort we should invest to
make this possible.


Pjotr Prins <pjotr.public12@thebird.nl> writes:

>> It is an unnecessary restriction to *prevent* users from installing
>> Python 2 and 3 interpreters into the same profile.  Any errors we see
>> with mixing 2 and 3 is due to a bug in Guix which is due to our use of
>> PYTHONPATH.
>
> Alright, if you choose to use them as two different tools/languages.

Which is what they are - the biggest mistake in the Python 2->3
transition was, in my opinion, to pretend they are the same language.
It is possible to write programs that are valid in both languages but
produce different results; this even happens regularly by mistake
due to the change of division semantics. So by my definition, they are
two similar but distinct languages.

> Sure. Note that Python2 is quickly becoming obsolete. It is escalating
> because packages are now dropping support. After 10 years of joy
> mixing versions...

I expect Python 2 to remain widely used for another 10 years after
official support ends. Applications that have no security concerns can
simply continue to build on the last official release of Python 2 and
its library ecosystem. For applications with a high migration cost, it's
still not worth the effort to port to Python 3.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> It is an unnecessary restriction to *prevent* users from installing
>> Python 2 and 3 interpreters into the same profile.  Any errors we see
>
> I agree. But the current question is not if we should allow people to
> shoot themselves into the foot, but how much effort we should invest to
> make this possible.

It hasn’t always been like this.  We added a feature to anticipate
conflicts that would happen at the time a profile is built.  This
feature now prevents you from installing variants of the same package
into the same profile — this is what we want.  Python 2 and 3 are not
variants of the same package, so preventing their installation into the
same profile is a bug introduced by that new feature.

My proposal would require very little effort to implement.  It is just a
question of whether there is a more elegant approach.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing python 2 and python 3 in the same profile

[Hartmut Goebel]

Am 10.03.2018 um 09:34 schrieb Pjotr Prins:
> Generally, mixing Python2 and 3 in one profile makes no sense. During a

I strongly disagree with the same arguments as Ricardo.

The only reason we have problems installing both python2 and python3 in
the same profile is: We are using PYTHONPATH in a way it is not meant to
be used. See my analysis posted a few days ago - which I need to finish.

We need to fix the way guix is installing Python!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: installing python 2 and python 3 in the same profile

[Pjotr Prins]

On Sun, Mar 11, 2018 at 02:00:40PM +0100, Hartmut Goebel wrote:
> Am 10.03.2018 um 09:34 schrieb Pjotr Prins:
> > Generally, mixing Python2 and 3 in one profile makes no sense. During a
> 
> I strongly disagree with the same arguments as Ricardo.
> 
> The only reason we have problems installing both python2 and python3 in
> the same profile is: We are using PYTHONPATH in a way it is not meant to
> be used. See my analysis posted a few days ago - which I need to finish.
> 
> We need to fix the way guix is installing Python!

OK, ok. Do note that it won't fix issues around minor versions where
the PYTHONPATH does not disambiguate.

Therefore it is not as clean as it is made out to be. Nothing Guix
specific, but solvable through profiles.

Pj.

Re: installing python 2 and python 3 in the same profile

[Ludovic Courtès]

Hello!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> with the introduction of the collision avoidance feature that prevents
> you from installing different variants of the same package into a
> profile we made it impossible to install “python” and “python@2” into
> the same profile.
>
> It still works with ad-hoc environments,

That’s because we explicitly disable the check in ‘guix environment’
(commit afd06f605bf88a796acefc7ed598b43879346a6b).

> but manifests containing both Python versions cannot be instantiated
> any more.  This is too strict, because we know that these two variants
> don’t cause conflicts.

More or less: ‘PYTHONPATH’ is version-independent, which I suppose could
be problematic.

> What can we do to make this feature a little smarter?
>
> How about a package property that defines a “conflicts?” predicate that
> takes two packages of the same name and determines (e.g. by checking the
> major version) if these two packages are conflicting?  If no such
> predicate is provided we assume that packages with the same name cause a
> conflict and prevent installation.

Unfortunately it wouldn’t work in the general case where some profile
entries may be coming from ~/.guix-profile/manifest, as is the case when
you do:

  guix package -i python@2
  guix package -i python@3

I would suggest short-term solutions, which are not exclusive:

  1. Add a ‘--allow-collisions’ flag to ‘guix profile’ (we discussed it
     before but didn’t get around to doing it apparently!).

  2. Use different package names when we know things can be
     parallel-installed: “python2” vs. “python” (I’m talking about the
     package name, not its version string.)  That’s what distros usually
     do, and I think it’s good enough.

A longer-term solution would be to add a property like you suggest,
except that it should be serializable (a regexp, an sexp, or something),
such that we can keep it in ~/.guix-profile/manifest and honor it later.
That adds complexity for something we could easily address through
conventions.

Thoughts?

Ludo’.

Re: installing python 2 and python 3 in the same profile

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> How about a package property that defines a “conflicts?” predicate that
>> takes two packages of the same name and determines (e.g. by checking the
>> major version) if these two packages are conflicting?  If no such
>> predicate is provided we assume that packages with the same name cause a
>> conflict and prevent installation.
>
> Unfortunately it wouldn’t work in the general case where some profile
> entries may be coming from ~/.guix-profile/manifest, as is the case when
> you do:
>
>   guix package -i python@2
>   guix package -i python@3
>
> I would suggest short-term solutions, which are not exclusive:
>
>   1. Add a ‘--allow-collisions’ flag to ‘guix profile’ (we discussed it
>      before but didn’t get around to doing it apparently!).
>
>   2. Use different package names when we know things can be
>      parallel-installed: “python2” vs. “python” (I’m talking about the
>      package name, not its version string.)  That’s what distros usually
>      do, and I think it’s good enough.

Both of these are desirable solutions, I think.  “--allow-collisions”
would be an escape hatch, and renaming packages is much easier than the
“serialisable predicate” idea.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing python 2 and python 3 in the same profile

[Hartmut Goebel]

Am 13.03.2018 um 22:52 schrieb Ludovic Courtès:
>   2. Use different package names when we know things can be
>      parallel-installed: “python2” vs. “python” (I’m talking about the
>      package name, not its version string.)  That’s what distros usually
>      do, and I think it’s good enough.

I'd prefer this.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

On 14/03/2018 12:39, Hartmut Goebel wrote:
> Am 13.03.2018 um 22:52 schrieb Ludovic Courtès:
>>    2. Use different package names when we know things can be
>>       parallel-installed: “python2” vs. “python” (I’m talking about the
>>       package name, not its version string.)  That’s what distros usually
>>       do, and I think it’s good enough.
> 
> I'd prefer this.

That sounds like a good basis. But perhaps "python" for Python 2 and 
"python3" for Python 3 would make more sense, since those are the names 
of the executables.

This does of course raise the question of how this will evolve in the 
long run, but since so many bad decisions were already taken, I am not 
trying to guess what will happen. For now, the upstream recommendation 
remains to use "python" and "python3" to distinguish the executables. 
But what will happen in 2020? The Python community might be tempted to 
change the naming to mark the end of Python 2 support, but that would be 
at the price of another round of breaking everybody's scripts.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> On 14/03/2018 12:39, Hartmut Goebel wrote:
>> Am 13.03.2018 um 22:52 schrieb Ludovic Courtès:
>>>    2. Use different package names when we know things can be
>>>       parallel-installed: “python2” vs. “python” (I’m talking about the
>>>       package name, not its version string.)  That’s what distros usually
>>>       do, and I think it’s good enough.
>>
>> I'd prefer this.
>
> That sounds like a good basis. But perhaps "python" for Python 2 and
> "python3" for Python 3 would make more sense, since those are the
> names of the executables.

Yes. OTOH we use the “python2-” prefix for 2.x packages and “python-”
for 3.x packages.

At any rate, that’s a change we should do now in ‘core-updates’.

Ricardo?

> This does of course raise the question of how this will evolve in the
> long run, but since so many bad decisions were already taken, I am not
> trying to guess what will happen. For now, the upstream recommendation
> remains to use "python" and "python3" to distinguish the
> executables. But what will happen in 2020? The Python community might
> be tempted to change the naming to mark the end of Python 2 support,
> but that would be at the price of another round of breaking
> everybody's scripts.

Not our business I’d say.  :-)

Ludo’.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

Ludovic Courtès <ludo@gnu.org> writes:

> Yes. OTOH we use the “python2-” prefix for 2.x packages and “python-”
> for 3.x packages.

Indeed. What a mess!

>> This does of course raise the question of how this will evolve in the
>> long run, but since so many bad decisions were already taken, I am not
...

> Not our business I’d say.  :-)

Right now, no. But we might one day have to decide how to respect
upstream recommendations that change over time, if that change has a
potentially undesirable impact on Guix users.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
>
>> On 14/03/2018 12:39, Hartmut Goebel wrote:
>>> Am 13.03.2018 um 22:52 schrieb Ludovic Courtès:
>>>>    2. Use different package names when we know things can be
>>>>       parallel-installed: “python2” vs. “python” (I’m talking about the
>>>>       package name, not its version string.)  That’s what distros usually
>>>>       do, and I think it’s good enough.
>>>
>>> I'd prefer this.
>>
>> That sounds like a good basis. But perhaps "python" for Python 2 and
>> "python3" for Python 3 would make more sense, since those are the
>> names of the executables.
>
> Yes. OTOH we use the “python2-” prefix for 2.x packages and “python-”
> for 3.x packages.
>
> At any rate, that’s a change we should do now in ‘core-updates’.
>
> Ricardo?

Done in a7714d42de2c3082f3609d1e63c83d703fb39cf9 and
23c0d40e1312663ef553ba7b6415a0ac483b591e on core-updates.

Thanks for the ideas!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net