the upcoming Great Python2 Purge™

the upcoming Great Python2 Purge™

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

We're now about a year out from the official EOL for python2 (Jan 1,
2020). So far we've been not adding python2 variants of packages that
are new unless they're actually needed for something. Do we want to
start removing python2 packages when updating other packages if they are
leaf packages?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: the upcoming Great Python2 Purge™

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

Efraim Flashner <efraim@flashner.co.il> writes:

> We're now about a year out from the official EOL for python2 (Jan 1,
> 2020). So far we've been not adding python2 variants of packages that
> are new unless they're actually needed for something. Do we want to
> start removing python2 packages when updating other packages if they are
> leaf packages?

I think it's okay to start removing "leaf" Python 2 packages.  In most
cases they were probably never used anyway, or the dependents have
transitioned to their Python 3 counterparts.

We'll probably break some channels, but I'm sure our users won't have
any difficulties adding them back to their own channels if need be.

\f

On a related note, we also have a number of [Python 3] packages that
have been failing to build for a long time.  Some of these are trivial,
i.e. what "guix import" produces.

It would be good to get rid of those as well, as the would-be user is
much better off starting from "guix import" instead of first getting
disappointed by the Guix package and then having to go through all the
trouble of submitting a patch.

Should we have some sort of policy or threshold for when to remove such
packages?  Maybe after 3-6 months?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

add DEPRECATION grace period: the upcoming Great Python2 Purge™

[Pjotr Prins]

On Wed, Dec 26, 2018 at 01:30:53PM +0100, Marius Bakke wrote:
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > We're now about a year out from the official EOL for python2 (Jan 1,
> > 2020). So far we've been not adding python2 variants of packages that
> > are new unless they're actually needed for something. Do we want to
> > start removing python2 packages when updating other packages if they are
> > leaf packages?
> 
> I think it's okay to start removing "leaf" Python 2 packages.  In most
> cases they were probably never used anyway, or the dependents have
> transitioned to their Python 3 counterparts.
> 
> We'll probably break some channels, but I'm sure our users won't have
> any difficulties adding them back to their own channels if need be.
> 
> On a related note, we also have a number of [Python 3] packages that
> have been failing to build for a long time.  Some of these are trivial,
> i.e. what "guix import" produces.
> 
> It would be good to get rid of those as well, as the would-be user is
> much better off starting from "guix import" instead of first getting
> disappointed by the Guix package and then having to go through all the
> trouble of submitting a patch.
> 
> Should we have some sort of policy or threshold for when to remove such
> packages?  Maybe after 3-6 months?

A lot of software outside Guix still depends on Python2, for better or
worse. I don't believe EOL means they are going to drop security
updates. Leaf packages may well be in use today.

Is there a way we mark packages as DEPRECATED? I think we should not
just remove packages without a grace period. Deprecate for, say, 3
months or even 6 months is the way to do this. A deprecation tag
should include a time stamp that gives the (planned) removal time.

Pj.

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

On 26/12/2018 10:38, Efraim Flashner wrote:
> We're now about a year out from the official EOL for python2 (Jan 1,
> 2020). So far we've been not adding python2 variants of packages that
> are new unless they're actually needed for something. Do we want to
> start removing python2 packages when updating other packages if they are
> leaf packages?

What would we gain by that? Plenty of people are still using Python 2. 
Many rather specialized packages still exist only for Python 2, and may 
well depend on packages that are leaves within Guix. Why make their life 
more difficult as long as there is no clear benefit in terms of Guix 
maintenance effort?


Konrad.

Re: the upcoming Great Python2 Purge™

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

On Wed, Dec 26, 2018 at 11:38:12AM +0200, Efraim Flashner wrote:
> We're now about a year out from the official EOL for python2 (Jan 1,
> 2020). So far we've been not adding python2 variants of packages that
> are new unless they're actually needed for something. Do we want to
> start removing python2 packages when updating other packages if they are
> leaf packages?

This was previously discussed here:

https://lists.gnu.org/archive/html/guix-devel/2018-06/msg00237.html

That discussion didn't go very far. As you mentioned, the consensus
seemed to be that we 1) relax the policy of always providing both Python
2 and 3 packages and 2) we'll act when we need to.

My opinion is that we should remove Python 2 packages after the upstream
EOL announcement if they are causing trouble somehow.

But I don't think we need to remove them en masse. We offer many other
packages that are basically abandoned upstream, so I think it will be
okay to keep the Python 2 packages as long as there are no bugs or if
they are maintained somehow.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: add DEPRECATION grace period: the upcoming Great Python2 Purge™

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]

On Wed, Dec 26, 2018 at 02:33:55PM +0100, Pjotr Prins wrote:
> A lot of software outside Guix still depends on Python2, for better or
> worse. I don't believe EOL means they are going to drop security
> updates. Leaf packages may well be in use today.

I do think it means that the current Python team at python.org will stop
issuing security updates for Python 2. [0]

Previously, Guido van Rossum said "The way I see the situation for 2.7
is that EOL is January 1st, 2020, and there will be no updates, not even
source-only security patches, after that date. Support (from the core
devs, the PSF, and python.org) stops completely on that date." [1]

Well, Guido is no longer involved with Python, so maybe the situation
has changed. In any case, I think we can expect third parties like Red
Hat to keep maintaining Python 2 for some years, and we can use their
work.

> Is there a way we mark packages as DEPRECATED? I think we should not
> just remove packages without a grace period. Deprecate for, say, 3
> months or even 6 months is the way to do this. A deprecation tag
> should include a time stamp that gives the (planned) removal time.

Not exactly, although there is a 'deprecated-package' procedure that
accepts a replacement package to supersede the deprecated package. It
doesn't do what you suggest.

[0] Already, the status of Python 2 is 'bugfix'. If it reaches "end
of life", the bugfixing activity will presumably cease, although they do
describe another 'security' status that seems lesser than 'bugfix':
https://devguide.python.org/#status-of-python-branches

[1]
https://mail.python.org/pipermail/python-dev/2018-March/152348.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: the upcoming Great Python2 Purge™

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On Wed, Dec 26, 2018 at 01:30:53PM +0100, Marius Bakke wrote:
> Efraim Flashner <efraim@flashner.co.il> writes:
> > We're now about a year out from the official EOL for python2 (Jan 1,
> > 2020). So far we've been not adding python2 variants of packages that
> > are new unless they're actually needed for something. Do we want to
> > start removing python2 packages when updating other packages if they are
> > leaf packages?
> 
> I think it's okay to start removing "leaf" Python 2 packages.  In most
> cases they were probably never used anyway, or the dependents have
> transitioned to their Python 3 counterparts.
> 
> We'll probably break some channels, but I'm sure our users won't have
> any difficulties adding them back to their own channels if need be.

If we were to start removing Python 2 packages, I like Pjotr's
suggestion that we offer a lengthy grace period to help people set up
some channels to support their work. We don't need to always do this
sort of thing but, in this case, it will be good practice for everyone
involved, and the change is large and well-publicized.

My opinion is that we don't need to start removing them before 2020 and,
even after that, we may choose to use a 3rd-party Python 2 distribution
from someone like Red Hat. Of course, as always, it depends on whether
or not any Guix developers are willing to do the work.

> On a related note, we also have a number of [Python 3] packages that
> have been failing to build for a long time.  Some of these are trivial,
> i.e. what "guix import" produces.
> 
> It would be good to get rid of those as well, as the would-be user is
> much better off starting from "guix import" instead of first getting
> disappointed by the Guix package and then having to go through all the
> trouble of submitting a patch.
> 
> Should we have some sort of policy or threshold for when to remove such
> packages?  Maybe after 3-6 months?

Yes, I agree, there must be some point where we remove packages that
simply don't work at all.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: the upcoming Great Python2 Purge™

[Brett Gilio]

Leo Famulari writes:

> On Wed, Dec 26, 2018 at 11:38:12AM +0200, Efraim Flashner wrote:
>> We're now about a year out from the official EOL for python2 (Jan 1,
>> 2020). So far we've been not adding python2 variants of packages that
>> are new unless they're actually needed for something. Do we want to
>> start removing python2 packages when updating other packages if they are
>> leaf packages?
>
> This was previously discussed here:
>
> https://lists.gnu.org/archive/html/guix-devel/2018-06/msg00237.html
>
> That discussion didn't go very far. As you mentioned, the consensus
> seemed to be that we 1) relax the policy of always providing both Python
> 2 and 3 packages and 2) we'll act when we need to.
>
> My opinion is that we should remove Python 2 packages after the upstream
> EOL announcement if they are causing trouble somehow.
>
> But I don't think we need to remove them en masse. We offer many other
> packages that are basically abandoned upstream, so I think it will be
> okay to keep the Python 2 packages as long as there are no bugs or if
> they are maintained somehow.

I think we could also just move them to a different Guix channel
entirely as a sort of legacy-support option. I am almost positive
somebody among us would be willing to maintain them. Could be better
than everybody wanting to maintain their own channel for it.

Also, on a side note, how would this work for the python importers?
Would we stop offering python2 substitutes on the build servers? There
are some other questions here that I think aren't getting addressed.

Brett Gilio

Re: add DEPRECATION grace period: the upcoming Great Python2 Purge™

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 3067 bytes --]

Hello everyone!

Leo Famulari <leo@famulari.name> writes:

> On Wed, Dec 26, 2018 at 02:33:55PM +0100, Pjotr Prins wrote:
>> A lot of software outside Guix still depends on Python2, for better or
>> worse. I don't believe EOL means they are going to drop security
>> updates. Leaf packages may well be in use today.
>
> I do think it means that the current Python team at python.org will stop
> issuing security updates for Python 2. [0]
>
> Previously, Guido van Rossum said "The way I see the situation for 2.7
> is that EOL is January 1st, 2020, and there will be no updates, not even
> source-only security patches, after that date. Support (from the core
> devs, the PSF, and python.org) stops completely on that date." [1]
>
> Well, Guido is no longer involved with Python, so maybe the situation
> has changed. In any case, I think we can expect third parties like Red
> Hat to keep maintaining Python 2 for some years, and we can use their
> work.
>
I suggest everyone to read these two LWN articles[0][1]. IMO, we should
start deprecating all python 2 packages which are already available in
python 3 and are not dependencies of python-2-only packages(*). Also, we
should not create python 2 definition for new python packages
anymore. Of course, we can make an exception if there's a demand for
it. This way, we can start warning everybody that python 2 is going EOL
and support is going to be dropped gradually. Right now,
'guix refresh -l python2' shows there're 1692 packages depending on
python 2.

For security updates, as Guido has mentioned python devs will no longer
provide security updates after 1/1/2020, which others seemed to
agree. You can read the whole thread here[2]. However, centos and debian
will still be supporting python 2 past that date. Centos will support
python 2 until 2024 and I suspect Debian will have to support it for
even longer since their next stable release (Debian 10 on 2020) will
include python 2.

Cheers,
Alex

(*): Should we make a new construct that does it? Also, I think we
should mention it in the guix blog, so others can learn about the
deprecation.

[0]: https://lwn.net/Articles/756628/
[1]: https://lwn.net/Articles/750833/
[2]: https://www.mail-archive.com/python-dev@python.org/msg100031.html

>> Is there a way we mark packages as DEPRECATED? I think we should not
>> just remove packages without a grace period. Deprecate for, say, 3
>> months or even 6 months is the way to do this. A deprecation tag
>> should include a time stamp that gives the (planned) removal time.
>
> Not exactly, although there is a 'deprecated-package' procedure that
> accepts a replacement package to supersede the deprecated package. It
> doesn't do what you suggest.
>

> [0] Already, the status of Python 2 is 'bugfix'. If it reaches "end
> of life", the bugfixing activity will presumably cease, although they do
> describe another 'security' status that seems lesser than 'bugfix':
> https://devguide.python.org/#status-of-python-branches
>
> [1]
> https://mail.python.org/pipermail/python-dev/2018-March/152348.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: the upcoming Great Python2 Purge™

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 676 bytes --]

I checked 'guix refresh -l python2' and I got a list of the python2-*
packages which are leaf packages. I'm not suggesting that we right now
get rid of them, but it seems to me something worth keeping an eye on.

'guix package -A ^python2- | wc -l' gave me 737
the number of leaf python2 packages I found was 222.

This is only a first round; I didn't check to see if we were to remove
these packages how many more leaf python2 packages we would have.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: python2-leaf-packages --]
[-- Type: text/plain, Size: 5310 bytes --]

python2-activepapers@0.2.2
python2-ansi2html@1.2.0
python2-apache-libcloud@2.4.0
python2-autograd@0.0.0-0.442205d
python2-autopep8@1.3.5
python2-axolotl@0.1.39
python2-behave-web-api@1.0.6
python2-betamax-matchers@0.4.0
python2-bigfloat@0.3.0
python2-binaryornot@0.4.4
python2-biom-format@2.1.7
python2-cachecontrol@0.11.6
python2-capstone@3.0.5
python2-carbon@1.0.2
python2-ccm@2.1.6
python2-celery@4.2.1
python2-clf@0.5.7
python2-conda@4.3.16
python2-consul@0.6.1
python2-cvxopt@1.2.1
python2-cypari2@2.0.3
python2-debian@0.1.28
python2-defusedxml@0.5.0
python2-discogs-client@2.2.1
python2-django-filter@1.1.0
python2-django-simple-math-captcha@1.0.7
python2-dns-lexicon@2.4.0
python2-empy@3.3
python2-enum@0.4.6
python2-eventlet@0.20.1
python2-exif-read@2.1.2
python2-falcon-cors@1.1.7
python2-feather-format@0.4.0
python2-feedgenerator@1.9
python2-file@5.33
python2-flasgger@0.6.3
python2-flask-htmlmin@1.2
python2-flask-httpauth@3.2.3
python2-flask-login@0.4.1
python2-flask-migrate@2.0.3
python2-flask-multistatic@1.0
python2-flask-principal@0.4.0
python2-flask-restful-swagger@0.19
python2-flask-wtf@0.13.1
python2-furl@0.5.6
python2-gdrivefs@0.14.9
python2-ghp-import@0.5.5
python2-git-review@1.27.0
python2-glances@3.0.2
python2-glob2@0.6
python2-gmpy2@2.0.8
python2-gnupg@0.4.3
python2-gpg@1.10.0
python2-grako@3.99.9
python2-graphene@0.10.2
python2-graphite-web@1.0.2
python2-gridmap@0.13.0
python2-gst@1.14.4
python2-guzzle-sphinx-theme@0.7.11
python2-gyp@0.0.0-0.5e2b3dd
python2-hdf4@0.9
python2-honcho@1.0.1
python2-httpretty@0.8.14
python2-hy@0.13.0
python2-i3-py@0.6.5
python2-internetarchive@1.7.4
python2-invoke@1.1.0
python2-ipy@0.83
python2-ipywidgets@5.2.2
python2-iso3166@0.9
python2-iso639@0.4.5
python2-isoweek@1.3.3
python2-jellyfish@0.5.6
python2-joblib@0.13.0
python2-josepy@1.1.0
python2-jsonpatch@1.16
python2-jsonrpclib@0.1.7
python2-jsonrpclib-pelix@0.3.2
python2-jupyter-console@5.2.0
python2-keepkey@6.0.2
python2-keystoneclient@1.8.1
python2-kitchen@1.2.5
python2-kivy@1.10.1
python2-ledgerblue@0.1.16
python2-libadalang@0.0.0-0.9b205e9
python2-libarchive-c@2.8
python2-libmpsse@1.3
python2-libvirt@4.10.0
python2-lirc@1.2.1-2.c28708b
python2-lit@0.5.1
python2-llfuse@1.3.5
python2-lmdb@0.94
python2-lzstring@1.0.4
python2-mapnik@3.0.16
python2-matplotlib-documentation@2.2.3
python2-mechanicalsoup@0.11.0
python2-mpd2@0.5.5
python2-munch@2.0.4
python2-munkres@1.0.8
python2-musicbrainzngs@0.6
python2-mutagen@1.38
python2-mwclient@0.8.4
python2-natsort@5.4.1
python2-nbxmpp@0.6.9
python2-ndg-httpsclient@0.5.1
python2-neo4j-driver@1.4.0
python2-netcdf4@1.4.2
python2-nltk@3.2.1
python2-nose2@0.6.5
python2-nose-timer@0.7.3
python2-numpy-documentation@1.15.4
python2-odfpy@1.3.3
python2-openid-cla@1.2
python2-openid-teams@1.1
python2-orator@0.9.7
python2-paramunittest@0.2
python2-parsedatetime@2.4
python2-parted@3.11.1
python2-passlib@1.7.1
python2-pastescript@2.0.2
python2-peewee@2.10.2
python2-phonenumbers@8.9.1
python2-pika@0.12.0
python2-plastid@0.4.8
python2-plotly@2.4.1
python2-ply@3.10
python2-polib@1.0.8
python2-publicsuffix2@2.20160818
python2-py2neo@3.1.2
python2-pyaes@1.6.1
python2-pyalsaaudio@0.8.4
python2-pyaudio@0.2.11
python2-pybigwig@0.3.12
python2-pybugz@0.6.11
python2-pycountry@18.5.26
python2-pycryptodome@3.7.3
python2-pydiff@0.2
python2-pydot@1.2.4
python2-pyechonest@9.0.0
python2-pygit2@0.27.3
python2-pykafka@2.4.0
python2-pykka@1.2.1
python2-pylast@2.0.0
python2-pyld@1.0.3
python2-pymysql@0.9.2
python2-pyodbc@4.0.24
python2-pyodbc-c@3.1.4
python2-pyrfc3339@1.1
python2-pystache@0.5.4
python2-pytest-flakes@1.0.1
python2-pytest-subtesthack@0.1.1
python2-pytest-xdist@1.25.0
python2-pyusb@1.0.2
python2-q@2.6
python2-qrcode@6.1
python2-quex@0.68.1
python2-rarfile@2.8
python2-ratelimiter@1.2.0
python2-rdflib@4.2.2
python2-readlike@0.1.3
python2-relatorio@0.8.0
python2-reparser@1.4.3
python2-reportlab@3.5.13
python2-roca-detect@1.0.8
python2-rope@0.11.0
python2-rpython@0.2.1
python2-ruamel.ordereddict@0.4.9
python2-s3cmd@1.6.1
python2-s3transfer@0.1.13
python2-sadisplay@0.4.8
python2-schedule@0.4.3
python2-schema@0.6.6
python2-schematics@1.1.1
python2-scikit-image@0.14.1
python2-screed@1.0
python2-scripttest@1.3
python2-setproctitle@1.1.10
python2-setuptools@40.0.0
python2-shapely@1.6.3
python2-shedskin@0.9.4
python2-slowaes@0.1a1
python2-sockjs-tornado@1.0.6
python2-socksipy-branch@1.01
python2-spectra@0.0.11
python2-sphinx-cloud-sptheme@1.8.0
python2-sphinxcontrib-programoutput@0.10
python2-sphinx-repoze-autointerface@0.8
python2-sql@0.9
python2-sqlalchemy-utils@0.32.21
python2-stdnum@1.8.1
python2-stem@1.7.0
python2-straight-plugin@1.4.1
python2-swagger-spec-validator@2.1.0
python2-sympy@1.1.1
python2-tables@3.4.4
python2-testlib@0.6.5
python2-texttable@0.8.7
python2-tlsh@3.4.5
python2-tmx@1.10
python2-translitcodec@0.4.0
python2-trezor@0.11.1
python2-trezor-agent@0.13.0
python2-trollius-redis@0.1.4
python2-twine@1.9.1
python2-url@0.2.0
python2-user-agents@1.1.0
python2-utils@2.1.0
python2-validictory@1.0.1
python2-verboselogs@1.7
python2-waf@2.0.11
python2-warpedlmm@0.21
python2-websocket-client@0.54.0
python2-whoosh@2.7.4
python2-wsgiproxy2@0.4.5
python2-xdo@0.3
python2-xenon@0.5.4
python2-xlrd@1.0.0
python2-xopen@0.3.3
python2-xsge@2018.02.26
python2-yapf@0.24.0
python2-zope-security@4.0.3
ptpython2@0.34

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Efraim Flashner <efraim@flashner.co.il> writes:

> I checked 'guix refresh -l python2' and I got a list of the python2-*
> packages which are leaf packages. I'm not suggesting that we right now
> get rid of them, but it seems to me something worth keeping an eye on.

Please be careful with "leaf packages". The leaves of my software
dependency tree are in my home directory, not in Guix. But I depend on
Guix to supply most of the dependencies of these leaves.

In the case of Python, any package that provides a Python library API
should not be considered a leaf package.

Konrad.

Re: the upcoming Great Python2 Purge™

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Efraim Flashner <efraim@flashner.co.il> writes:
>
>> I checked 'guix refresh -l python2' and I got a list of the python2-*
>> packages which are leaf packages. I'm not suggesting that we right now
>> get rid of them, but it seems to me something worth keeping an eye on.
>
> Please be careful with "leaf packages". The leaves of my software
> dependency tree are in my home directory, not in Guix. But I depend on
> Guix to supply most of the dependencies of these leaves.
>
> In the case of Python, any package that provides a Python library API
> should not be considered a leaf package.

Yes, in the presence of channels this can be tricky.  However, we are
not committed to supporting third party channels when this results in
problems on our end.  We reserve the freedom to break third party
channels.

Konrad, going forward it might be reasonable to keep copies of required
Python 2 packages in your channel.  We aren’t going to remove Python 2
packages now, but in the future we may not be able to fix unmaintained
packages and may have to remove them.

--
Ricardo

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Hi Ricardo,

> Konrad, going forward it might be reasonable to keep copies of required
> Python 2 packages in your channel.  We aren’t going to remove Python 2
> packages now, but in the future we may not be able to fix unmaintained
> packages and may have to remove them.

"My channel" doesn't exist (because I haven't yet found the time to
figure out how to set up and manage a channel, although it's been on my
to-do list for a while).

But... how about splitting off *all* of Python 2 and everything that
depends on it into a separate channel, which would then be maintained
by a separate team?

This would remove the Python 2 maintenance burden from the Guix team,
but still allow shared maintenance (expected to be low-effort) of
a coherent distribution of Python 2 packages.

If everybody starts a personal channel of whatever subset of those
packages they require, there's a lot of duplicated effort and no clear
location to turn to for end users. And if parts of Python 2 remain in
Guix but other parts move elsewhere, there's needless friction in
coordinating two projects that will end up having different policies.

Konrad.

Re: the upcoming Great Python2 Purge™

[Ricardo Wurmus]

Hi Konrad,

>> Konrad, going forward it might be reasonable to keep copies of required
>> Python 2 packages in your channel.  We aren’t going to remove Python 2
>> packages now, but in the future we may not be able to fix unmaintained
>> packages and may have to remove them.
>
> "My channel" doesn't exist (because I haven't yet found the time to
> figure out how to set up and manage a channel, although it's been on my
> to-do list for a while).

I’d be happy to assist.

> But... how about splitting off *all* of Python 2 and everything that
> depends on it into a separate channel, which would then be maintained
> by a separate team?

Currently this is not feasible, in my opinion, as a lot of packages in
Guix still depend on Python 2 for some reason or the other.  When Python
2 reaches EOL, however, I think this would be a reasonable thing to do.

--
Ricardo

Re: the upcoming Great Python2 Purge™

[zimoun]

Dear,

Maybe I am wrong and I miss one point.

If you have local packages that depends on Guix packages, then it is
not a big issue when the Guix packages are removed. Because you just
have to `guix pull --commit=<sha>' where <sha> points to a Guix commit
owning the required packages.
And if I have correct, you can put that in a manifest and then
instantiate an environment for your local packages.

Instead of splitting off all of Python 2 into a separate channel, why
not just put the python2 packages which need to be removed.
Then nothing breaks and it provides a picture of how many python2
packages necessary to Guix are still there.


Speaking about channel, instead of GUIX_PACKAGE_PATH=~/work/guix/bimbs
you create the file ~/.config/guix/channels.scm containing:

(cons* (channel
        (name 'bimsb)
        (url "file:///home/simon/work/guix/bimsb")
        (branch "master"))
      %default-channels)

Then:

guix pull
guix package -i <your-package>

Here the url file:///home/simon/work/guix/bimsb is a clone of
https://github.com/BIMSBbioinfo/guix-bimsb
But you can directly specify Git repo online etc.


Hope that helps


All the best,
simon

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Hi Ricardo,

>> "My channel" doesn't exist (because I haven't yet found the time to
>> figure out how to set up and manage a channel, although it's been on my
>> to-do list for a while).
>
> I’d be happy to assist.

Thanks! I might come back to that offer when I find at least enough time
to figure out precisely what information I might be lacking
(that's meta-lack-of-time).

>> But... how about splitting off *all* of Python 2 and everything that
>> depends on it into a separate channel, which would then be maintained
>> by a separate team?
>
> Currently this is not feasible, in my opinion, as a lot of packages in
> Guix still depend on Python 2 for some reason or the other.  When Python
> 2 reaches EOL, however, I think this would be a reasonable thing to do.

Sounds good, then the only remaining issue is defining a transition
protocol. What I'd like to avoid is that packages disappear one by one
from Guix and then have to be dug out one by one from Git history for
setting up a Python 2 channel.

More generally, I think it would be useful to collect in some place all
package definitions that are removed from Guix. For example, a list with
package names plus the last Guix commit that had the package.

Konrad.

Re: the upcoming Great Python2 Purge�?�

[ng0]

Konrad Hinsen transcribed 1.2K bytes:
> Hi Ricardo,
> 
> >> "My channel" doesn't exist (because I haven't yet found the time to
> >> figure out how to set up and manage a channel, although it's been on my
> >> to-do list for a while).
> >
> > I’d be happy to assist.
> 
> Thanks! I might come back to that offer when I find at least enough time
> to figure out precisely what information I might be lacking
> (that's meta-lack-of-time).
> 
> >> But... how about splitting off *all* of Python 2 and everything that
> >> depends on it into a separate channel, which would then be maintained
> >> by a separate team?
> >
> > Currently this is not feasible, in my opinion, as a lot of packages in
> > Guix still depend on Python 2 for some reason or the other.  When Python
> > 2 reaches EOL, however, I think this would be a reasonable thing to do.
> 
> Sounds good, then the only remaining issue is defining a transition
> protocol. What I'd like to avoid is that packages disappear one by one
> from Guix and then have to be dug out one by one from Git history for
> setting up a Python 2 channel.
> 
> More generally, I think it would be useful to collect in some place all
> package definitions that are removed from Guix. For example, a list with
> package names plus the last Guix commit that had the package.
> 
> Konrad.
> 
>

package number one which will stay on python2: getmail.
In exchanges with the maintainer on their mailinglist
it was made clear that Charles has no intention to
port it to python3, as except for a few minor additions
and fixes, getmail is in maintenance mode.. you get
releases every now and then.

Re: the upcoming Great Python2 Purge™

[Brett Gilio]

Efraim Flashner writes:

> I checked 'guix refresh -l python2' and I got a list of the python2-*
> packages which are leaf packages. I'm not suggesting that we right now
> get rid of them, but it seems to me something worth keeping an eye on.
>
> 'guix package -A ^python2- | wc -l' gave me 737
> the number of leaf python2 packages I found was 222.
>
> This is only a first round; I didn't check to see if we were to remove
> these packages how many more leaf python2 packages we would have.

There are a few leaf packages that are being used to generate the
python3 equivalent. So, maybe we should make a list of leaf packages
that are otherwise safe to be dismissed. Also, what is the status of
somebody deciding to move a lot of these to a channel?

Re: the upcoming Great Python2 Purge™

[zimoun]

Hi Konrad,


On Mon, 18 Feb 2019 at 17:31, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>
> Hi Ricardo,
>
> >> "My channel" doesn't exist (because I haven't yet found the time to
> >> figure out how to set up and manage a channel, although it's been on my
> >> to-do list for a while).
> >
> > I’d be happy to assist.
>
> Thanks! I might come back to that offer when I find at least enough time
> to figure out precisely what information I might be lacking
> (that's meta-lack-of-time).

From my understanding, it is simpler than expected ;-)

For example, instead of GUIX_PACKAGE_PATH=~/work/your-repo
you create the file ~/.config/guix/channels.scm containing:

(cons* (channel
        (name 'bimsb)
        (url "file:///home/simon/work/your-repo")
        (branch "master"))
      %default-channels)

Then:

guix pull
guix package -s <your-package>
guix package -i <your-other-package>

etc.


> >> But... how about splitting off *all* of Python 2 and everything that
> >> depends on it into a separate channel, which would then be maintained
> >> by a separate team?
> >
> > Currently this is not feasible, in my opinion, as a lot of packages in
> > Guix still depend on Python 2 for some reason or the other.  When Python
> > 2 reaches EOL, however, I think this would be a reasonable thing to do.
>
> Sounds good, then the only remaining issue is defining a transition
> protocol. What I'd like to avoid is that packages disappear one by one
> from Guix and then have to be dug out one by one from Git history for
> setting up a Python 2 channel.

First, the package will not disappear. Because now, they are in the
Guix tree, one can always reach them with `guix pull --commit' even if
they are not "visible" in gnu/packages.

Second, I am not sure it is that ugly. :-)
If one package "disappears", then you can quickly find the last commit
where it is defined with:

git rev-list --all | xargs git --no-pager grep 'public python2-stuff' | head -1

Then you can add the package to your channel (or a semi-official one),
with its dependencies if needed.

To me, this is the best transition protocol.
 a- the Guix tree is moving forward "reserving its freedom to break third party
channels"
 b- if no one moves the disappeared package, then it means the package
is really obsolete.


Last about scientific reproducibility, imagine you have your local
channel, containing your packages that rocks!
Today, you compute some stuff and so on.
You "just" have to store the output of `guix describe'. For example,
my current one is:

Generation 16   Feb 07 2019 15:47:37    (current)
  bimsb d1cba4a
    repository URL: file:///home/simon/work/guix/bimsb
    branch: master
    commit: d1cba4a2cba1eb6c6c33328fea511b75dfcffe39
  guix 89ea625
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 89ea6252b6849131ba35d141006e1bbf3a49594f

Months later, my "collaborators" want to reproduce my result. With
this information (the 2 commit hashes), they will be able to set again
all the dependency graph.
(however, substitutes should not be there so maybe they should need to
build a lot; another story)

The `guix pull --commit=' is not super friendly yet with multiple
channels. But, it is only UI gaps, isn't it? :-)

For example, it should be nice to run:
 guix pull -C bimsb --commit=d1cba4a -C guix --commit=89ea625
to restore the exact same tree.

Today, if I am correct, you can put this information in a manifest
file and then instantiate an environment for your computations. From
my point of view, it is the long-term practise to have the
"scientific" reproducibilty.


Please correct me if I am wrong. :-)


However, some whistle.
 1. Even if inferiors and other improve the travel in time, there is
no guarantee that nothing will be break in the future compared to the
past (if you see what I mean).
 2. Archivist issue: will the current material be still available?


> More generally, I think it would be useful to collect in some place all
> package definitions that are removed from Guix. For example, a list with
> package names plus the last Guix commit that had the package.

From my opinion, when we speak about reproducibility, remove a package
is similar to update it. And all this information is tracked by the
Git repo.
I agree that maybe it is not friendly to dive in and it is really
resource hungry.
For example, it is not "easy" to find when the package lispf4 have been removed.

And if we speak about scientific reproducibility, it appears to me
difficult to have the Q&A that you propose at the Guix commit level.
However, it should be easier to find from which commit the package is
installed. Somehow, improve the `guix package --list-installed' and/or
`guix package --list-generations' commands, i.e., also tracking the
channel and the commit.

As a user, if I run:
 guix package -i python-numpy
 guix pull # the tree is changed
 do other stuff (install remove pull again etc.)
 guix package -i python-scipy
and I forgot to store the commit hash of my first install.
Then, it requires some work to find it: I need  to correlate the `guix
pull -l' information to the `guix package -l' information.




All the best,
simon

Re: the upcoming Great Python2 Purge™

[Björn Höfling]

[-- Attachment #1.1: Type: text/plain, Size: 1111 bytes --]

On Mon, 18 Feb 2019 11:56:19 +0200
Efraim Flashner <efraim@flashner.co.il> wrote:

> I checked 'guix refresh -l python2' and I got a list of the python2-*
> packages which are leaf packages. I'm not suggesting that we right now
> get rid of them, but it seems to me something worth keeping an eye on.
> 
> 'guix package -A ^python2- | wc -l' gave me 737
> the number of leaf python2 packages I found was 222.
> 
> This is only a first round; I didn't check to see if we were to remove
> these packages how many more leaf python2 packages we would have.

I would also point attention to non-leaf packages, where python2 is
often a native-input which might or might not be eliminated.

Just out of curiosity I looked at the dependency-graph of icecat and
there are 24 edges going to python2 including from icecat itself, rust,
curl, clang, fontforge, zziplib, etc.

For example, I found out that zziplib uses python2 for creating
documents and there is a patch in the next release that will make it
work with python3.

I'm attaching that icecat->python2 subgraph as a dot-file.

Björn

[-- Attachment #1.2: icecat-python2.dot --]
[-- Type: application/msword-template, Size: 16419 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]