Python 3 binaries

Python 3 binaries

[Andreas Enge]

Hello,

python 3 does not ship a python binary any more, just a binary called
python3. That could be useful, since it would allow to install python 2 and 3
side by side. However, all packages relying on a shebang substitution with
a python binary now fail.

I see two general possibilities to solve the problem:
- The simplest one, add a symlink python->python3 in the python 3 package.
  But then we lose the possibility of installing python 2 and 3 at the same
  time. Which maybe does not matter? It would only be a problem for a user
  wanting to install both in the user profile, while all other packages
  would internally have rewritten their calls to either of the two python
  versions
- In the patch-shebang phase, we could, if the python binary is not
  available, look for the python 3 binary. But that would mean special code
  only for python.

Personally, I would tend to the first solution. But what do you think?

Andreas

Re: Python 3 binaries

[Cyril Roelandt]

On 08/31/2013 05:30 PM, Andreas Enge wrote:
> Hello,
>
> python 3 does not ship a python binary any more, just a binary called
> python3. That could be useful, since it would allow to install python 2 and 3
> side by side. However, all packages relying on a shebang substitution with
> a python binary now fail.
>
> I see two general possibilities to solve the problem:
> - The simplest one, add a symlink python->python3 in the python 3 package.
>    But then we lose the possibility of installing python 2 and 3 at the same
>    time. Which maybe does not matter? It would only be a problem for a user
>    wanting to install both in the user profile, while all other packages
>    would internally have rewritten their calls to either of the two python
>    versions

I think lots of users would want that. On Debian, I really like having 
multiple versions of Python, since some of my scripts are not Python3 
compatible. It's also useful to run test suites against many versions of 
Python. I think users must be able to install both.

Cyril.

Re: Python 3 binaries

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 08/31/2013 05:30 PM, Andreas Enge wrote:
>> Hello,
>>
>> python 3 does not ship a python binary any more, just a binary called
>> python3. That could be useful, since it would allow to install python 2 and 3
>> side by side. However, all packages relying on a shebang substitution with
>> a python binary now fail.
>>
>> I see two general possibilities to solve the problem:
>> - The simplest one, add a symlink python->python3 in the python 3 package.
>>    But then we lose the possibility of installing python 2 and 3 at the same
>>    time. Which maybe does not matter? It would only be a problem for a user
>>    wanting to install both in the user profile, while all other packages
>>    would internally have rewritten their calls to either of the two python
>>    versions
>
> I think lots of users would want that. On Debian, I really like having
> multiple versions of Python, since some of my scripts are not Python3
> compatible. It's also useful to run test suites against many versions
> of Python. I think users must be able to install both.

Yes, and that’s visibly what upstream wants.  So I would just leave
things as is, without the symlink.

Then again I know nothing about Python.  :-)

Ludo’.

Re: Python 3 binaries

[Brandon Invergo]

[-- Attachment #1: Type: text/plain, Size: 2635 bytes --]

Hi everyone,

> python 3 does not ship a python binary any more, just a binary called
> python3. That could be useful, since it would allow to install python
> 2 and 3 side by side. However, all packages relying on a shebang
> substitution with a python binary now fail.

Here's what it looks like on my Parabola system, which I think is a sane
way of doing it (ok, there's no *truly* sane way of doing it):

$ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python -> python3
lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python-config -> python3-config
lrwxrwxrwx 1 root root  1K May 12 18:01 /usr/bin/python2 -> python2.7
lrwxrwxrwx 1 root root  1K May 12 18:01 /usr/bin/python2-config -> python2.7-config
-rwxr-xr-x 1 root root  7K May 12 18:01 /usr/bin/python2.7
-rwxr-xr-x 1 root root  2K May 12 18:01 /usr/bin/python2.7-config
lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3 -> python3.3
lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3-config -> python3.3-config
-rwxr-xr-x 2 root root 11K May 21 17:50 /usr/bin/python3.3
lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3.3-config -> python3.3m-config
-rwxr-xr-x 2 root root 11K May 21 17:50 /usr/bin/python3.3m
-rwxr-xr-x 1 root root  2K May 21 17:50 /usr/bin/python3.3m-config

$ ls -l /usr/bin/idle*
lrwxrwxrwx 1 root root 1K May 21 17:50 /usr/bin/idle -> idle3
-rwxr-xr-x 1 root root 1K May 12 18:01 /usr/bin/idle2
lrwxrwxrwx 1 root root 1K May 21 17:50 /usr/bin/idle3 -> idle3.3
-rwxr-xr-x 1 root root 1K May 21 17:50 /usr/bin/idle3.3

$ ls -l /usr/bin/pydoc*
lrwxrwxrwx 1 root root 1K May 21 17:50 /usr/bin/pydoc -> pydoc3
-rwxr-xr-x 1 root root 1K May 12 18:01 /usr/bin/pydoc2
lrwxrwxrwx 1 root root 1K May 21 17:50 /usr/bin/pydoc3 -> pydoc3.3
-rwxr-xr-x 1 root root 1K May 21 17:50 /usr/bin/pydoc3.3

$ ls -l /usr/bin/2to3*
lrwxrwxrwx 1 root root 1K May 21 17:50 /usr/bin/2to3 -> 2to3-3.3
-rwxr-xr-x 1 root root 1K May 12 18:01 /usr/bin/2to3-2.7
-rwxr-xr-x 1 root root 1K May 21 17:50 /usr/bin/2to3-3.3

I think Debian is similar, but with Python 2 being the default rather
than Python 3.

Shebangs are routinely changed in Parabola to accommodate this but since
shebangs are also routinely being changed in Guix anyway, this shouldn't
be a problem. 

So, the install scripts will require a bit of post-processing to
generate the symlinks and to move around some binaries to avoid
conflicts, but I think this is the only thing you can do.

Cheers, 
-brandon

ps - great meeting some of you last week in Paris!

-- 
Brandon Invergo
http://brandon.invergo.net

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Python 3 binaries

[Andreas Enge]

Very well. If we do not wish to make the symlink python->python3,
then we will need special code in patch-shebangs, so that things like
"#!/usr/bin/python" may get rewritten to
"#!/nix/store/...python-3.3.2/bin/python3" if no binary named "python"
is found.

I think the following patch does this:

--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -516,7 +516,10 @@ FILE are kept unchanged."
                                  (rest (match:substring m 3))
                                  (has-env (string-suffix? "/env" interp))
                                  (cmd (if has-env arg1 (basename interp)))
-                                 (bin (search-path path cmd)))
+                                 (bin (or (search-path path cmd)
+                                          (if (string=? cmd "python")
+                                              (search-path path "python3")
+                                              #f))))
                             (if bin
                                 (if (string=? bin interp)
                                     #f            ; nothing to do

But it triggers a full rebuild, so it will take me a while to test it.

If you agree to this patch, we would need to revive core-updates, and should
revert to python 2 in master. Could we then build core-updates also on
hydra.gnu.org? This would make it easier to add python modules for the two
different versions.

Andreas

Re: Python 3 binaries

[Andreas Enge]

On Sun, Sep 01, 2013 at 11:28:17AM +0200, Andreas Enge wrote:
> Could we then build core-updates also on hydra.gnu.org?

This part is solved; it was already there, just apparently did not connect
correctly to git.

Andreas

Re: Python 3 binaries

[Andreas Enge]

On Sun, Sep 01, 2013 at 10:20:18AM +0200, Brandon Invergo wrote:
> Here's what it looks like on my Parabola system, which I think is a sane
> way of doing it (ok, there's no *truly* sane way of doing it):
> $ ls -l /usr/bin/python*
> lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python -> python3
> lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python-config -> python3-config
> lrwxrwxrwx 1 root root  1K May 12 18:01 /usr/bin/python2 -> python2.7
> lrwxrwxrwx 1 root root  1K May 12 18:01 /usr/bin/python2-config -> python2.7-config
> -rwxr-xr-x 1 root root  7K May 12 18:01 /usr/bin/python2.7
> -rwxr-xr-x 1 root root  2K May 12 18:01 /usr/bin/python2.7-config
> lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3 -> python3.3
> lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3-config -> python3.3-config
> -rwxr-xr-x 2 root root 11K May 21 17:50 /usr/bin/python3.3
> lrwxrwxrwx 1 root root  1K May 21 17:50 /usr/bin/python3.3-config -> python3.3m-config
> -rwxr-xr-x 2 root root 11K May 21 17:50 /usr/bin/python3.3m
> -rwxr-xr-x 1 root root  2K May 21 17:50 /usr/bin/python3.3m-config

This is more or less what happens now, with the default implicitly being
Python 2, since the Python 2 package ships a binary "python" and the Python 3
package ships a binary "python3". Since we promise to follow upstream as
closely as possible, it would make sense for us to keep this situation.

> Shebangs are routinely changed in Parabola to accommodate this but since
> shebangs are also routinely being changed in Guix anyway, this shouldn't
> be a problem. 

This is what my patch that I sent previously to the list should accomplish.

Thanks for the input!

Andreas

Re: Python 3 binaries

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Very well. If we do not wish to make the symlink python->python3,
> then we will need special code in patch-shebangs, so that things like
> "#!/usr/bin/python" may get rewritten to
> "#!/nix/store/...python-3.3.2/bin/python3" if no binary named "python"
> is found.

Ah, so packages that work with Python 3 expect a ‘python’ (and not
‘python3’) executable?

Then that’s a different story (I thought ‘python3’ was the official name
for the binary.)

I’d rather not have specific things like that in ‘patch-shebangs’.  So,
what we could do is:

  • Leave ‘python-3’ as is, without the symlink.

  • Add a ‘python-3-wrapper’ package that just contains ‘bin/python’
    pointing to ‘…/bin/python3’ (using ‘trivial-build-system’.)

  • When building Python 3 packages, we’d use the wrapper, not the real
    one; however, users would install the real one in their environment.

WDYT?

Ludo’.

Re: Python 3 binaries

[Andreas Enge]

On Sun, Sep 01, 2013 at 04:03:47PM +0200, Ludovic Courtès wrote:
> Ah, so packages that work with Python 3 expect a ‘python’ (and not
> ‘python3’) executable?

Apparently so. And anyway, packages that work with both versions usually
start their scripts with #!/usr/bin/python.

> Then that’s a different story (I thought ‘python3’ was the official name
> for the binary.)
> 
> I’d rather not have specific things like that in ‘patch-shebangs’.  So,
> what we could do is:
>   • Leave ‘python-3’ as is, without the symlink.
>   • Add a ‘python-3-wrapper’ package that just contains ‘bin/python’
>     pointing to ‘…/bin/python3’ (using ‘trivial-build-system’.)
>   • When building Python 3 packages, we’d use the wrapper, not the real
>     one; however, users would install the real one in their environment.

This would be a possibility.

Personally, I find the solution rewriting the shebangs cleaner, but this
is more a matter of taste than anything.

The wrapper would require the user to install both python-3 and the wrapper
(or contain python-3 as a propagated input), so that the user has all files
in the python-3 package. This is slightly ugly.

The solution with the wrapper has the advantage that users who want only
Python 3 and not Python 2 would then get a binary named "python" pointing
to version 3, useful also for their own code. But somewhere it would have
to be documented that they then have to install python-3-wrapper and not just
python or python-3.

But the naming is not very clear; why do I need to install python-wrapper
if I just want the latest version of Python?
How about calling the package python-default and having it contain python-3
as a propagated input? We could even have a version 2 of this package, which
would be empty with only python-2 as a propagated input. Then the user could:

- install python-default = python-default-3 and get only Python 3,
  with binaries "python" and "pydoc" pointing to "python3" and "pydoc3",
  respectively;
- install python-default-2 and get only Python 2.
- install python-2 and python = python-3 to get both of them, with
  "python" pointing to Python 2.

What do you think?

Andreas

Re: Python 3 binaries

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Sun, Sep 01, 2013 at 04:03:47PM +0200, Ludovic Courtès wrote:
>> Ah, so packages that work with Python 3 expect a ‘python’ (and not
>> ‘python3’) executable?
>
> Apparently so. And anyway, packages that work with both versions usually
> start their scripts with #!/usr/bin/python.

OK.

>> Then that’s a different story (I thought ‘python3’ was the official name
>> for the binary.)
>> 
>> I’d rather not have specific things like that in ‘patch-shebangs’.  So,
>> what we could do is:
>>   • Leave ‘python-3’ as is, without the symlink.
>>   • Add a ‘python-3-wrapper’ package that just contains ‘bin/python’
>>     pointing to ‘…/bin/python3’ (using ‘trivial-build-system’.)
>>   • When building Python 3 packages, we’d use the wrapper, not the real
>>     one; however, users would install the real one in their environment.
>
> This would be a possibility.
>
> Personally, I find the solution rewriting the shebangs cleaner, but this
> is more a matter of taste than anything.
>
> The wrapper would require the user to install both python-3 and the wrapper
> (or contain python-3 as a propagated input), so that the user has all files
> in the python-3 package. This is slightly ugly.

Users would typically only need to install the wrapper; that’s enough
since it holds a reference to the real Python.

However, my understanding from what Cyril and Brandon said is that users
may prefer to have it called ‘python3’ by default, so they can install
both Python 2 and Python 3 in parallel.  Furthermore, they can choose to
have (say) an alias python=python3 if that’s what they want.

Based on that, I thought the wrapper would be mostly for internal
consumption.

Did I get it right?

> The solution with the wrapper has the advantage that users who want only
> Python 3 and not Python 2 would then get a binary named "python" pointing
> to version 3, useful also for their own code. But somewhere it would have
> to be documented that they then have to install python-3-wrapper and not just
> python or python-3.
>
> But the naming is not very clear; why do I need to install python-wrapper
> if I just want the latest version of Python?
> How about calling the package python-default and having it contain python-3
> as a propagated input? We could even have a version 2 of this package, which
> would be empty with only python-2 as a propagated input. Then the user could:
>
> - install python-default = python-default-3 and get only Python 3,
>   with binaries "python" and "pydoc" pointing to "python3" and "pydoc3",
>   respectively;
> - install python-default-2 and get only Python 2.
> - install python-2 and python = python-3 to get both of them, with
>   "python" pointing to Python 2.
>
> What do you think?

My understanding was that users (really: Python developers) would expect
to get a ‘python3’ binary when they install the latest, and a ‘python’
binary otherwise.

Then that means we don’t really have to worry, and just document that
the python-3.x package is an unmodified upstream package, with its
binary is called ‘python3’.

WDYT?

Ludo’, who’d really appreciate feedback from the Python-savvy.  :-)

Re: Python 3 binaries

[Cyril Roelandt]

On 09/01/2013 07:34 PM, Ludovic Courtès wrote:
> However, my understanding from what Cyril and Brandon said is that users
> may prefer to have it called ‘python3’ by default, so they can install
> both Python 2 and Python 3 in parallel.  Furthermore, they can choose to
> have (say) an alias python=python3 if that’s what they want.
>
> Based on that, I thought the wrapper would be mostly for internal
> consumption.
>
> Did I get it right?

On Debian:

$ ls -l $(which python)
lrwxrwxrwx 1 root root 9 May  6 02:58 /usr/bin/python -> python2.7
$ ls -l $(which python3)
lrwxrwxrwx 1 root root 9 Aug 18 18:12 /usr/bin/python3 -> python3.3

Packages usually exist in two different versions: python-foo and 
python3-foo.

I think this is quite a good way of packaging both Python 2 and 3. One 
day, maybe nobody will use Python 2.x any more, and we'll just use 
"python" instead of "python3", but until then, I'm really happy to have 
"python" and "python3".


Cyril.

Re: Python 3 binaries

[Andreas Enge]

On Sun, Sep 01, 2013 at 07:34:03PM +0200, Ludovic Courtès wrote:
> However, my understanding from what Cyril and Brandon said is that users
> may prefer to have it called ‘python3’ by default, so they can install
> both Python 2 and Python 3 in parallel.  Furthermore, they can choose to
> have (say) an alias python=python3 if that’s what they want.
> 
> Based on that, I thought the wrapper would be mostly for internal
> consumption.
> 
> Did I get it right?
> 
> My understanding was that users (really: Python developers) would expect
> to get a ‘python3’ binary when they install the latest, and a ‘python’
> binary otherwise.

My impression was that most people would like to install the latest and
greatest (python 3), but with the binary still called "python".
These people could install python-default (the wrapper, which I would then
expect to be the package that the average user would install).

Developers who want both versions can install python-2 and python-3.

The Debian python policy stipulates the following:
"Python scripts depending on the default Python version (...) or not
depending on a specific Python version should use python (without a version)
as the interpreter name.
Python scripts that only work with a specific Python version must explicitly
use the versioned interpreter name (pythonX.Y)."

Following this policy (which we may or may not do), if we declare Python 3
to be the default python version, then our python-3 package should contain
a binary named "python", and we would have to delete the "python" binary
from the version 2 package and keep only those named "python2" and "python2.7".
And we should, for internal use, create a wrapper package python-default-2...

But indeed, it would be nice to get our python specialists' opinion.

Andreas

Re: Python 3 binaries

[Andreas Enge]

On Sun, Sep 01, 2013 at 07:40:18PM +0200, Cyril Roelandt wrote:
> Packages usually exist in two different versions: python-foo and
> python3-foo.
> 
> I think this is quite a good way of packaging both Python 2 and 3.
> One day, maybe nobody will use Python 2.x any more, and we'll just
> use "python" instead of "python3", but until then, I'm really happy
> to have "python" and "python3".

I think it is not compatible with our policy of defaulting to always the
latest version, if possible, while debian usually defaults to something
old and very stable. (I notice that python-3.0 dates from 2008).

So do I understand correctly that you would suggest two packages, python
(containing version 2.x) and python3 (containing version 3.x), and the same
for all modules? Otherwise, having python-2 and python-3, upgrading and
installing without giving a version number would automatically switch
to python-3, and then we would lose the "python" binary.

With a package python (version 2) and a package python3, without a wrapper
package, all our packages containing some #!/usr/bin/python would have to use
Python version 2. Is that what we want?

I think the suggestions with a wrapper package make it easier to switch to
Python 3 wherever possible, while your suggestion, if I understand it
correctly, seems to force us to stay with Python 2 until the last minute.

Worse, if there are packages requiring Python 3 but containing
#!/usr/bin/python (do such programs exist?), we would need to treat them
on a case-by-case basis.

Andreas

Re: Python 3 binaries

[Brandon Invergo]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> My understanding was that users (really: Python developers) would expect
> to get a ‘python3’ binary when they install the latest, and a ‘python’
> binary otherwise.

It depends.  I've grown used to having python(->python3) and python2
binaries in Parabola/Arch, where it is the policy to always have the
latest version as the default.

> Then that means we don’t really have to worry, and just document that
> the python-3.x package is an unmodified upstream package, with its
> binary is called ‘python3’.

I think that is a fine way to do it.  The most important part is
internal consistency.  It seems that the "unmodified upstream" strategy
is the path of least resistance, and it will fit with the expectations
of all of the Debian-based users out there.  

As for the shebangs, you may well still have to do some patching for
some packages, if they were written in python3 but the shebang is for
/usr/bin/python.

-brandon


-- 
Brandon Invergo
http://brandon.invergo.net

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: Python 3 binaries

[Andreas Enge]

Hello, time to give a quick update!

What I implemented in the python branch, following the discussions
on the list, is the following:

On Mon, Sep 02, 2013 at 08:24:48AM +0200, Brandon Invergo wrote:
> > Then that means we don’t really have to worry, and just document that
> > the python-3.x package is an unmodified upstream package, with its
> > binary is called ‘python3’.
> I think that is a fine way to do it.  The most important part is
> internal consistency.  It seems that the "unmodified upstream" strategy
> is the path of least resistance, and it will fit with the expectations
> of all of the Debian-based users out there.  

There are unmodified binaries for Python 2.7.5 and Python 3.3.2
(which should probably be updated to 3.4.0).

> As for the shebangs, you may well still have to do some patching for
> some packages, if they were written in python3 but the shebang is for
> /usr/bin/python.

Then there is a package called "python-wrapper", which simply adds the
following symlinks into the Python 3 bin/ directory:
idle -> idle3
pydoc -> pydoc3
python -> python3

It is used internally as an input, so that shebangs need not be rewritten;
but users may also install it if they wish Python 3 binaries known under
the name of "python" etc., without also installing Python 2.

Simple packages compile, and once a few problems with the build system are
solved, I should be able to add setuptools, which will pave the way for more
modules.

Andreas