Python 2 end-of-life?

Python 2 end-of-life?

[zimoun]

Dear,

The Python 2 end-of-life is coming...

What is the schedule to deprecate the python2 packages?


All the best,
simon

Re: Python 2 end-of-life?

[zimoun]

Dear,

The end-of-life of Python 2 is coming...

https://www.python.org/doc/sunset-python-2/
https://www.python.org/dev/peps/pep-0373/

Projects pledge to drop their support of Python 2.

https://python3statement.org/


What do you do?
Do we deprecate the python2 packages? If yes, what would be the
schedule? If no, do we move all the python2 packages to a
python2-xyz.scm file?
Do we do nothing? Based on what rationale?


Thanks in advance for any comments, thoughts, etc. :-)


All the best,
simon

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Simon,

> What do you do?

Well, me, personally, I continue to do most of my research using Python 2
because I cannot afford to port everything to Python 3. And since I do
only number crunching, meaning nothing with security implications, I am
not particularly worried.

> Do we deprecate the python2 packages? If yes, what would be the
> schedule? If no, do we move all the python2 packages to a
> python2-xyz.scm file?
> Do we do nothing? Based on what rationale?

I'd say the very first thing we should do is look at all non-Python
packages that depend indirectly on Python 2. I remember during a recent
installation on a virtual machine that my very first package install,
which is always Emacs, downloaded Python 2 among many others, so there
remains some work to be done.

Once Python 2 lives in a largely isolated package sub-universe, I don't
see much harm keeping it in Guix for now. If security issues become
apparent, we might have to do something more drastic.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Alex Griffin]

On Thu, Nov 21, 2019, at 11:46 AM, zimoun wrote:
> What do you do?
> Do we deprecate the python2 packages? If yes, what would be the
> schedule? If no, do we move all the python2 packages to a
> python2-xyz.scm file?
> Do we do nothing? Based on what rationale?

I think a good place to start is by just filing bugs against packages that rely on Python 2, saying that they face removal once Python 2 is EOL. As Konrad pointed out though, not every package necessarily needs to support Python 3. I think there can probably be exceptions for packages that do not add much security risk or maintenance work.

Red Hat will continue providing security fixes to Python 2 for several more years. PyPy will also continue to support Python 2 indefinitely, so packaging that and making it an option in python-build-system may be an option for some packages.

-- 
Alex Griffin

Re: Python 2 end-of-life?

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> I'd say the very first thing we should do is look at all non-Python
> packages that depend indirectly on Python 2.

Here is an attempt at identifying them:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(use-modules (guix packages)
             (gnu packages)
             (srfi srfi-1)
             (ice-9 format))

(define (in-python2-ecosystem? package)
  (string-prefix? "python2-" (package-name package)))

(define python2-dependent-packages
  (fold-packages (lambda (package result)
                   (cond ((in-python2-ecosystem? package)
                          result)
                         ((any in-python2-ecosystem?
                               (filter package?
                                       (map second
                                            (package-direct-inputs package))))
                          (cons package result))
                         (else result)))
                 '()))

(for-each (lambda (package)
            (format #t "~a~%" (package-full-name package)))
          python2-dependent-packages)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

I find 313 packages (see list below). A few of them are still Python
stuff (has "python" in the name but not "python2"), but most of them
look like packages that are not themselves Python libraries.

Some of these packages are real heavyweights (llvm, texlive, qtwebkit,
...), so I'd say each package deserves a bug report of its own if we
agree that a dependency on Python 2 should be considered a bug for a
non-Python package.

Cheers,
  Konrad.


xcb-util-errors@1.0-1.5d660eb
xpra@2.5.3
arandr@0.1.9
keybinder@0.3.1
polybar@3.4.0
wicd@1.7.4
woof@2012-05-31
linkchecker@9.4.0
wabt@1.0.12
xen@4.11.1
criu@3.11
bubblewrap@0.3.3
ffmpeg@4.2.1
youtube-dl-gui@0.3.8
handbrake@1.2.2
ffmpeg@3.4.6
git@2.24.0
git-annex-remote-hubic@0.3.1
cvs-fast-export@1.45
miniupnpc@2.1.20190824
miniupnpc-monero@2.1-monero-0.12.3.0-0.6a63f99
tryton@4.6.2
tor@0.4.1.6
lyx@2.3.2-2
texlive-bin@20180414
rubber@1.1
ceph@13.2.6
pspp@1.2.0
sdcc@3.7.0
scribus@1.5.5
zn-poly@0.9.1
rust@1.28.0
rust@1.26.2
rust@1.21.0
rust@1.20.0
rust@1.29.2
rust@1.31.1
rust@1.30.1
rust@1.32.0
rust@1.35.0
rust@1.27.2
rust@1.23.0
rust@1.36.0
rust@1.24.1
rust@1.25.0
rust@1.34.1
rust@1.37.0
rust@1.33.0
rust@1.22.1
rrdtool@1.7.1
qtwebkit@5.212.0-alpha3
qtmultimedia@5.11.3
qt@4.8.7
qtbase@5.11.3
qtdeclarative@5.11.3
ptpython2@0.34
python-rope@0.11.0
bpython2@0.18
python-cookies@2.2.1
pulseaudio-dlna@0.5.2-1.4472928
pdfposter@0.6.0
impressive@0.12.0
stapler@0.3.2
patches@0.0-1.ef1b8a7
openbox@3.6.1
ocaml-dose3@5.0.1
gourmet@0.17.4
node@10.16.0
openvswitch@2.12.0
gtklick@0.6.4
solfege@3.22.2
mod-host@0.10.6-3.1726ad06b
beast@0.10.0
lilypond@2.19.80
mono@4.4.1.0
pidgin@2.13.0
hexchat@2.14.2
bitlbee@3.5.1
pybitmessage@0.6.3.2
petsc-complex-openmpi@3.11.2
flann@1.8.4
slepc-complex-openmpi@3.11.1
slepc-openmpi@3.11.1
petsc-openmpi@3.11.2
slepc@3.11.1
sundials@3.1.1
sundials-openmpi@3.1.1
mlucas@18
petsc-complex@3.11.2
slepc-complex@3.11.1
lapack@3.7.1
petsc@3.11.2
atril@1.22.0
mate-applets@1.22.0
pluma@1.22.0
mate-menus@1.22.0
hoedown@3.0.7
alot@0.5.1
offlineimap@7.2.4
claws-mail@3.17.4
opensmtpd-extras@5.7.1
postorius@1.0.3
ghmm@0.9-rc3-0.2341
kaldi-gstreamer-server@0-1.1735ba4
lci@0.11.2
clang-runtime@8.0.0
clang@3.9.1
clang@6.0.1
emacs-clang-format@8.0.0
clang-runtime@3.9.1
clang@8.0.0
clang-runtime@3.7.1
clang-runtime@3.8.1
clang@7.0.1
llvm@8.0.0
clang-runtime@3.6.2
clang@3.7.1
llvm@6.0.1
emacs-clang-rename@8.0.0
llvm@7.0.1
llvm@3.6.2
clang-runtime@3.5.2
clang-runtime@7.0.1
clang@3.8.1
clang-runtime@6.0.1
clang@3.6.2
llvm@3.8.1
llvm@3.9.1
clang@3.5.2
llvm-for-extempore@3.7.1
llvm@3.7.1
llvm@3.5.2
crda@3.18
libnl@3.5.0
perf@5.3.8
iotop@0.6
tegaki-wagomu-simplified-chinese@0.3
tegaki-wagomu-japanese-kyoiku@0.3
tegaki-wagomu-japanese@0.3
tegaki-zinnia-japanese-light@0.3
tegaki-zinnia-simplified-chinese@0.3
tegaki-wagomu-traditional-chinese@0.3
tegaki-zinnia-traditional-chinese@0.3
tegaki-zinnia-traditional-chinese-light@0.3
tegaki-wagomu-japanese-joyo@0.3
tegaki-zinnia-japanese@0.3
tegaki-zinnia-japanese-joyo@0.3
tegaki-zinnia-japanese-kyoiku@0.3
tegaki-zinnia-simplified-chinese-light@0.3
kodi@18.4
kodi-wayland@18.4
key-mon@1.17
kfilemetadata@5.55.0
julia@1.1.1
inkscape@0.92.4
vigra@1.11.1
mcomix@1.2.1
mia@2.4.6
ghc@8.0.2
ghc@7.10.2
chirp@20181205
graphene@1.6.0
libdbusmenu@16.04.0
dot2tex@2.9.0
openimageio@1.7.19
openimageio@1.8.17
rapicorn@16.0.0
icecat@68.2.0-guix0-preview3
mozjs@52.0-1.6507e63
conkeror@68.2.0-guix0-preview3
mozjs@24.2.0
mozjs@38.2.1.rc0
mozjs@60.2.3-2
mozjs@17.0.0
pius@2.2.7
gnurl@7.63.0
gnunet@0.10.1
totem@3.30.0
gcr@3.28.1
rhythmbox@3.4.3
gnome-doc-utils@0.20.10
terminator@1.91
bluefish@2.2.10
glade@3.22.1
evince@3.34.1
gnome-keyring@3.28.2
deja-dup@34.3
caribou@0.4.21
gnumeric@1.12.45
telepathy-glib@0.24.1
gimp@2.10.12
osm2pgsql@0.96.0
gnubackgammon@1.06.002
gnubg@1.06.002
golly@3.2
slingshot@0.9
kiki-the-nano-bot@1.0.2
freeorion@0.4.8
kiki@1.0.2
0ad@0.0.23b-alpha
renpy@7.3.5
telepathy-mission-control@5.16.5
telepathy-logger@0.8.2
telepathy-idle@0.2.0
arachne-pnr@0.0-2-840bdfdeb
nototools@20170925
seabios@1.12.1
ovmf-aarch64@20170116-1.13a50a6
ovmf-arm@20170116-1.13a50a6
ovmf@20170116-1.13a50a6
ledger@3.1.3
glusterfs@3.10.12
lekha@0.2.1
kicad@5.0.2
qucs@0.0.19-0.b4f27d9
volk@1.3
childsplay@3.4
gcompris-qt@0.96
calibre@3.42.0
asciidoc@8.6.10
pootle@2.8.2
lightdm@1.24.0
parted@3.3
dico@2.9
mongodb@3.4.10
rocksdb@5.18.3
4store@1.1.6
libpqxx@4.0.1
r-protviz@0.4.0
coq@8.9.1
zziplib@0.13.69
makeself-safeextract@0.0.0-1.1a95e12
cinnamon-desktop@3.4.2
ungoogled-chromium-wayland@68.2.0-guix0-preview3
ungoogled-chromium@68.2.0-guix0-preview3
avogadro@1.2.0
domainfinder@2.0.5
nmoldyn@3.0.11
googletest@1.8.1
cmdtest@0.32
dvdstyler@3.0.4
gn@0.0-1530.1ab6fa2
bam@0.5.1
u-boot-mx6cuboxi@2019.04
u-boot-novena@2019.04
vboot-utils@R63-10032.B
u-boot-a20-olinuxino-micro@2019.04
u-boot-nintendo-nes-classic-edition@2019.04
u-boot-pinebook@2019.04
u-boot-rockpro64-rk3399@2019.10
u-boot-firefly-rk3399@2019.10
u-boot-am335x-evm@2019.04
u-boot-cubieboard@2019.04
u-boot-rock64-rk3328@2019.10
syslinux@6.04-pre-1.bb41e93
u-boot-bananapi-m2-ultra@2019.04
u-boot-malta@2019.04
u-boot-wandboard@2019.04
dtc@1.5.1
u-boot-cubietruck@2019.04
u-boot-am335x-boneblack@2019.04
u-boot-a20-olinuxino-lime@2019.04
u-boot-vexpress-ca9x4@2019.04
u-boot-tools@2019.04
u-boot-a20-olinuxino-lime2@2019.04
u-boot-puma-rk3399@2019.04
u-boot-pine64-plus@2019.04
boost@1.70.0
deluge@1.3.15
libtorrent-rasterbar@1.1.13
bamm@1.7.3
freebayes@1.0.2-1.3ce827d
grit@2.0.5
pyicoteo@2.0.7
bedtools@2.18.0
poretools@0.6.0-1.e426b1f
libbigwig@0.4.4
pbtranscript-tofu@2.2.3.8f5467f
miso@0.5.4
gess@1.0
vcflib@0.0.0-1.5ac0913
clipper@1.2.1
bedtools@2.26.0
tadbit@0.2.0
codingquarry@2.0
macs@2.1.1.20160309
jellyfish@2.2.10
pepr@1.0.9
piranha@1.2.1-1.0466d364b
tetoolkit@2.0.3
crossmap@0.2.9
find-circ@1.2-1.8655dca
rseqc@2.6.1
taxtastic@0.8.5
proteinortho@5.16b
fraggenescan@1.30
bedtools@2.27.1
imp@2.6.2
tophat@2.1.1
ribodiff@0.2.2
filtlong@0.2.0-1.d1bb46d
express-beta-diversity@1.0.8
couger@1.8.2
fio@3.14
rdiff-backup@1.2.8
duplicity@0.7.19
lash@0.6.0-rc2
audacity@2.3.2
android-googletest@1.8.0
git-repo@1.12.37
singular@4.1.2p1
fabric@1.14.0
nmap@7.80

Re: Python 2 end-of-life?

[Hartmut Goebel]

Hi Konrad,

Am 26.11.19 um 17:51 schrieb Konrad Hinsen:
> I find 313 packages (see list below). A few of them are still Python
> stuff (has "python" in the name but not "python2"), but most of them
> look like packages that are not themselves Python libraries.

Thanks for having brought up this topic and for the list (enhancement
proposal: sort it :-)

I assume many of these package can be updated. (Indeed I just updated
pdfposter, which I'm maintaining.)

bypthon2 and ptpython2 are REPLs for python2 - bypthon and ptpython are
the resp. packages for python3.

Regarding the qt packages: I updated them on staging, but still have
Python 2 in there. I will take care of these.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Python 2 end-of-life?

[ng0]

getmail is still python2.7, but on the mailinglist of it
(not archived iirc) there are efforts to move it to python3-only,
however this will take time. I've looked into porting it before
and half-succeeded (only half due to the same problem as the
maintainer has: time).

I think i saw getmail on the list sent earlier.

Re: Python 2 end-of-life?

[Bengt Richter]

Hi Konrad,

On +2019-11-26 17:51:52 +0100, Konrad Hinsen wrote:
> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
> 
> > I'd say the very first thing we should do is look at all non-Python
> > packages that depend indirectly on Python 2.
> 
> Here is an attempt at identifying them:
> 
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> (use-modules (guix packages)
>              (gnu packages)
>              (srfi srfi-1)
>              (ice-9 format))
> 
> (define (in-python2-ecosystem? package)
>   (string-prefix? "python2-" (package-name package)))
> 
> (define python2-dependent-packages
>   (fold-packages (lambda (package result)
>                    (cond ((in-python2-ecosystem? package)
>                           result)
>                          ((any in-python2-ecosystem?
>                                (filter package?
>                                        (map second
>                                             (package-direct-inputs package))))
>                           (cons package result))
>                          (else result)))
>                  '()))
> 
> (for-each (lambda (package)
>             (format #t "~a~%" (package-full-name package)))
>           python2-dependent-packages)
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> 
> I find 313 packages (see list below). A few of them are still Python
> stuff (has "python" in the name but not "python2"), but most of them
> look like packages that are not themselves Python libraries.
> 
> Some of these packages are real heavyweights (llvm, texlive, qtwebkit,
> ...), so I'd say each package deserves a bug report of its own if we
> agree that a dependency on Python 2 should be considered a bug for a
> non-Python package.
> 
> Cheers,
>   Konrad.
> 
>

Pretty code :)
Dependencies, not so much ;-/

[ ... orig list snipped ]

I saved your post into file py2eol.txt and did:

egrep -oh '^[^@;]+' py2eol.txt|sort|uniq -c|sort -h|tail
--8<---------------cut here---------------start------------->8---
      1 zziplib
      2 ffmpeg
      2 ghc
      2 openimageio
      3 bedtools
      5 mozjs
      8 clang
      8 clang-runtime
      8 llvm
     18 rust
--8<---------------cut here---------------end--------------->8---

IOW, a bunch just differ by version -- I wonder how many of the
packages that drew in old versions could run fine with respective
latest versions of what they are dependent on?

It would be really interesting if you could tweak your py2-dependent-package
lister to show for each how many lines of py2 code are causing the py2 dependency!

It would be a shame if half a dozen lines of python were pulling in
all that weight if it could have been a few lines of guile or bash instead.

The time it takes to figure out whether/how a trivial dependency can be
eliminated is, ISTM, a major cause of gordian-knot dependency bloat -- which IMO
in turn is a kind of RYF threat: it's effectively a free-time pay-wall.

So, since memory is cheap and processors are fast, anything that "does the job"
gets incorporated into our hacks, and trustworthiness dies of obesity ;-/

-- 
Regards,
Bengt Richter

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Bengt,

> IOW, a bunch just differ by version -- I wonder how many of the
> packages that drew in old versions could run fine with respective
> latest versions of what they are dependent on?

That's a very good question!

> It would be really interesting if you could tweak your py2-dependent-package
> lister to show for each how many lines of py2 code are causing the py2 dependency!

That's a much harder job, and I doubt it can be automatized. What I
analyze is the package definitions in Guix. They don't provide any
information on how the package uses its dependencies. They merely say
that the packager has declared some dependencies.

BTW, the number of lines of Python 2 code doesn't really matter either.
What we need to know is (1) can the Python 3 dependencies be replaced
by equivalent Python 3 dependencies and (2) are these dependencies
essential, or do they merely enable some minor extra (such as a Python
API for a C library). In the second case, the package could be split
into two.

> It would be a shame if half a dozen lines of python were pulling in
> all that weight if it could have been a few lines of guile or bash instead.

That's yet another question: could we patch the upstream code to replace
Python by something else that's more convenient for Guix. That may
actually be a worthwhile approach to reduce software bloat in Guix,
but it also shifts some of the maintenance burden from upstream to Guix.

> The time it takes to figure out whether/how a trivial dependency can be
> eliminated is, ISTM, a major cause of gordian-knot dependency bloat -- which IMO
> in turn is a kind of RYF threat: it's effectively a free-time
> pay-wall.

That's certainly true but on the other hand there is "never break a
working system". So there is a trade-off between long-term
maintainability and short-term reliability.

Guix might well contribute to a long-term improvement of everyone's
habits in this respect, by providing the bird's-eyes view of software
systems that developers and maintainers of individual packages cannot
have. What you describe is just the application of well-known ideas from
software engineering (cleaning up code before it becomes unmaintainable)
to the systems level. Which is possible because Guix turns the systems
level into just another software layer.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Hartmut,

> I assume many of these package can be updated. (Indeed I just updated
> pdfposter, which I'm maintaining.)

Great. That's the kind of response I was hoping for!

> bypthon2 and ptpython2 are REPLs for python2 - bypthon and ptpython are
> the resp. packages for python3.

There are a few other packages in my list that are part of the Python 2
universe (my own nmoldyn for example), but I see no easy way to detect
them in an analysis of the package graph. Perhaps I could filter out
packages whose dependencies are *all* from the Python 2 universe.

> Regarding the qt packages: I updated them on staging, but still have
> Python 2 in there. I will take care of these.

Excellent. Lots of other packages depend on those, so that will be a big
step towards Python 2 independence.

LLVM/clang looks like another important building block to liberate from
Python 2. I hope someone can take a look at that.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[zimoun]

Hi,

On Tue, 26 Nov 2019 at 22:52, Bengt Richter <bokr@bokr.com> wrote:

> egrep -oh '^[^@;]+' py2eol.txt|sort|uniq -c|sort -h|tail
> --8<---------------cut here---------------start------------->8---
>       1 zziplib
>       2 ffmpeg
>       2 ghc
>       2 openimageio
>       3 bedtools
>       5 mozjs
>       8 clang
>       8 clang-runtime
>       8 llvm
>      18 rust
> --8<---------------cut here---------------end--------------->8---
>
> IOW, a bunch just differ by version -- I wonder how many of the
> packages that drew in old versions could run fine with respective
> latest versions of what they are dependent on?

For example, considering rust, it is about the bootstrappability. See [1].

[1] http://guix.gnu.org/blog/2018/bootstrapping-rust/

I am interesting to know about clang/clang-runtime/llvm. Do we support
8 versions? Or version n-1 is useful to build version n?


> It would be really interesting if you could tweak your py2-dependent-package
> lister to show for each how many lines of py2 code are causing the py2 dependency!

It is really hard -- nor impossible. And I am not convinced that the
tough work will pay off.


> It would be a shame if half a dozen lines of python were pulling in
> all that weight if it could have been a few lines of guile or bash instead.

Do you propose to patch each time? Because I am not convinced again
that upstream will change Python to Guile.


> The time it takes to figure out whether/how a trivial dependency can be
> eliminated is, ISTM, a major cause of gordian-knot dependency bloat -- which IMO
> in turn is a kind of RYF threat: it's effectively a free-time pay-wall.

To me, one path to remove unnecessary dependencies of Python2 is to
give a look package by package, try to replace the Python2 dependency
by the Python3 (if exist) and see what happens. If it does not build
because the package really uses Python2 features, figure out which
one, patch with the Python3 equivalent and submit the patch upstream.


All the best,
simon

Re: Python 2 end-of-life?

[zimoun]

Hi Konrad,

On Wed, 27 Nov 2019 at 08:56, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> > IOW, a bunch just differ by version -- I wonder how many of the
> > packages that drew in old versions could run fine with respective
> > latest versions of what they are dependent on?
>
> That's a very good question!

Is it not cheap to replace the python2 dependency by the python3 one
and try to locally build?


The snippet of code you sent helps to figure out which packages are
still using Python2.

As you said, some packages will not be updated to Python 3. Let
document it in description for example and let provide a rationale
explaining why that we put in the manual.


> That's yet another question: could we patch the upstream code to replace
> Python by something else that's more convenient for Guix. That may
> actually be a worthwhile approach to reduce software bloat in Guix,
> but it also shifts some of the maintenance burden from upstream to Guix.

It does not appear to me a reasonable approach.


> Guix might well contribute to a long-term improvement of everyone's
> habits in this respect, by providing the bird's-eyes view of software
> systems that developers and maintainers of individual packages cannot
> have. What you describe is just the application of well-known ideas from
> software engineering (cleaning up code before it becomes unmaintainable)
> to the systems level. Which is possible because Guix turns the systems
> level into just another software layer.

Agree! :-)


All the best,
simon

Re: Python 2 end-of-life?

[zimoun]

Hi,

On Wed, 27 Nov 2019 at 09:36, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> > I assume many of these package can be updated. (Indeed I just updated
> > pdfposter, which I'm maintaining.)
>
> Great. That's the kind of response I was hoping for!

Nice!

On lost time (building, boring meeting, etc.) I will pick one package
from the list and try to replace/update. :-)


> LLVM/clang looks like another important building block to liberate from
> Python 2. I hope someone can take a look at that.

Maybe ask to Mathieu or Carl?
They just submitted bunch of patches about LLVM/clang.


All the best,
simon

Re: Python 2 end-of-life?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

>> It would be a shame if half a dozen lines of python were pulling in
>> all that weight if it could have been a few lines of guile or bash instead.
>
> Do you propose to patch each time? Because I am not convinced again
> that upstream will change Python to Guile.

We have python-on-guile, which might just work as a replacement for
trivial uses of Python 2.

--
Ricardo

Re: Python 2 end-of-life?

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> I'd say the very first thing we should do is look at all non-Python
> packages that depend indirectly on Python 2.

Here comes an update of my Python-2-in-Guix analysis. Sorted output,
distinction between likely libraries and likely application packages.
Plus the list of Python 2 libraries that something else depends on, with
special attention on those who don't currently have a Python 3
equivalent in Guix. Those would be good targets for porting work as
well.

Cheers,
  Konrad.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(use-modules (guix packages)
             (gnu packages)
             (srfi srfi-1)
             (ice-9 format))

(define (library-in-python2-ecosystem? package)
  (string-prefix? "python2" (package-name package)))

(define (has-no-python3-equivalent? package)
  (define python3-name (string-replace (package-name package) "python" 0 7))
  (null? (find-packages-by-name python3-name)))

(define (dependencies package)
  (filter package?
          (map second
               (package-direct-inputs package))))

(define python2-dependent-packages
  (fold-packages
   (lambda (package result)
     (define deps (dependencies package))
     (cond ((library-in-python2-ecosystem? package)
            result)
           ((and (not (null? deps))
                 (every library-in-python2-ecosystem? deps))
            (list (cons package (first result))
                  (second result)
                  (lset-union equal? deps (third result))))
           ((any library-in-python2-ecosystem? deps)
            (list (first result)
                  (cons package (second result))
                  (lset-union equal? (third result))))
           (else result)))
   '(() () ())))

(define (package-< p1 p2)
  (or (string<? (package-name p1) (package-name p2))
      (and (string=? (package-name p1) (package-name p2))
           (string<? (package-version p1) (package-version p2)))))

(display "Application packages written in Python 2\n")
(display "(all dependencies are Python 2)\n")
(display "\n")
(for-each (lambda (package)
            (format #t "~a~%" (package-full-name package)))
          (sort (first python2-dependent-packages) package-<))

(display "\n")
(display "\n")
(display "Application packages depending on Python 2\n")
(display "(some Python 2 dependencies)\n")
(display "\n")
(for-each (lambda (package)
            (format #t "~a~%" (package-full-name package)))
          (sort (second python2-dependent-packages) package-<))

(display "\n")
(display "\n")
(display "Python 2 packages on which the above packages depend\n")
(display "\n")
(for-each (lambda (package)
            (format #t "~a~a~%"
                    (package-full-name package)
                    (if (has-no-python3-equivalent? package)
                        " (no Python 3 version in Guix)"
                        "")))
          (sort (third python2-dependent-packages) package-<))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;


Output on today's state of Guix:

Application packages written in Python 2
(all dependencies are Python 2)

alot@0.5.1
android-googletest@1.8.0
bpython2@0.18
chirp@20181205
clipper@1.2.1
cmdtest@0.32
domainfinder@2.0.5
express-beta-diversity@1.0.8
fabric@1.14.0
find-circ@1.2-1.8655dca
gess@1.0
git-annex-remote-hubic@0.3.1
googletest@1.8.1
grit@2.0.5
iotop@0.6
key-mon@1.17
lekha@0.2.1
macs@2.1.1.20160309
miniupnpc@2.1.20190824
mlucas@18
nototools@20170925
pagekite@1.0.0.190721
patches@0.0-1.ef1b8a7
pbtranscript-tofu@2.2.3.8f5467f
pepr@1.0.9
pootle@2.8.2
postorius@1.0.3
ptpython2@0.34
pulseaudio-dlna@0.5.2-1.4472928
pyicoteo@2.0.7
python-cookies@2.2.1
python-rope@0.11.0
ribodiff@0.2.2
seabios@1.12.1
slingshot@0.9
stapler@0.3.2
taxtastic@0.8.5
tegaki-wagomu-japanese@0.3
tegaki-wagomu-japanese-joyo@0.3
tegaki-wagomu-japanese-kyoiku@0.3
tegaki-wagomu-simplified-chinese@0.3
tegaki-wagomu-traditional-chinese@0.3
tegaki-zinnia-japanese@0.3
tegaki-zinnia-japanese-joyo@0.3
tegaki-zinnia-japanese-kyoiku@0.3
tegaki-zinnia-japanese-light@0.3
tegaki-zinnia-simplified-chinese@0.3
tegaki-zinnia-simplified-chinese-light@0.3
tegaki-zinnia-traditional-chinese@0.3
tegaki-zinnia-traditional-chinese-light@0.3
tryton@4.6.2
woof@2012-05-31


Application packages depending on Python 2
(some Python 2 dependencies)

0ad@0.0.23b-alpha
4store@1.1.6
arachne-pnr@0.0-2-840bdfdeb
arandr@0.1.9
asciidoc@8.6.10
atril@1.22.0
audacity@2.3.2
avogadro@1.2.0
bam@0.5.1
bamm@1.7.3
beast@0.10.0
bedtools@2.18.0
bedtools@2.26.0
bedtools@2.27.1
bitlbee@3.5.1
bluefish@2.2.10
boost@1.70.0
boost-static@1.70.0
bubblewrap@0.4.0
calibre@3.42.0
caribou@0.4.21
ceph@13.2.6
childsplay@3.4
cinnamon-desktop@3.4.2
clang@3.5.2
clang@3.6.2
clang@3.7.1
clang@3.8.1
clang@3.9.1
clang@6.0.1
clang@7.0.1
clang@8.0.0
clang@9.0.0
clang-runtime@3.5.2
clang-runtime@3.6.2
clang-runtime@3.7.1
clang-runtime@3.8.1
clang-runtime@3.9.1
clang-runtime@6.0.1
clang-runtime@7.0.1
clang-runtime@8.0.0
clang-runtime@9.0.0
claws-mail@3.17.4
codingquarry@2.0
conkeror@68.2.0-guix0-preview3
coq@8.9.1
couger@1.8.2
crda@3.18
criu@3.11
crossmap@0.2.9
cvs-fast-export@1.45
deja-dup@34.3
deluge@1.3.15
dico@2.9
dot2tex@2.9.0
dtc@1.5.1
duplicity@0.7.19
dvdstyler@3.0.4
emacs-clang-format@8.0.0
emacs-clang-rename@8.0.0
evince@3.34.1
ffmpeg@3.4.6
ffmpeg@4.2.1
filtlong@0.2.0-1.d1bb46d
fio@3.14
flann@1.8.4
fraggenescan@1.30
freebayes@1.0.2-1.3ce827d
freeorion@0.4.8
gcompris-qt@0.96
gcr@3.28.1
ghc@7.10.2
ghc@8.0.2
ghmm@0.9-rc3-0.2341
gimp@2.10.14
git@2.24.0
git-repo@1.12.37
glade@3.22.1
glusterfs@3.10.12
gn@0.0-1666.6e5ba2e
gnome-doc-utils@0.20.10
gnome-keyring@3.28.2
gnubackgammon@1.06.002
gnubg@1.06.002
gnumeric@1.12.46
gnurl@7.63.0
golly@3.2
gourmet@0.17.4
graphene@1.6.0
gtklick@0.6.4
handbrake@1.2.2
hexchat@2.14.2
hoedown@3.0.7
icecat@68.2.0-guix0-preview3
imp@2.6.2
impressive@0.12.0
inkscape@0.92.4
jellyfish@2.2.10
john-the-ripper-jumbo@1.9.0-1
julia@1.1.1
kaldi-gstreamer-server@0-1.1735ba4
keybinder@0.3.1
kfilemetadata@5.55.0
kiki@1.0.2
kiki-the-nano-bot@1.0.2
kodi@18.4
kodi-wayland@18.4
lapack@3.7.1
lash@0.6.0-rc2
lci@0.11.2
ledger@3.1.3
libbigwig@0.4.4
libdbusmenu@16.04.0
libnl@3.5.0
libpqxx@4.0.1
libtorrent-rasterbar@1.1.13
lightdm@1.24.0
lilypond@2.19.80
linkchecker@9.4.0
llvm@3.5.2
llvm@3.6.2
llvm@3.7.1
llvm@3.8.1
llvm@3.9.1
llvm@6.0.1
llvm@7.0.1
llvm@8.0.0
llvm@9.0.0
llvm-for-extempore@3.7.1
makeself-safeextract@0.0.0-1.1a95e12
mate-applets@1.22.0
mate-menus@1.22.0
mcomix@1.2.1
mia@2.4.6
miso@0.5.4
mod-host@0.10.6-3.1726ad06b
mongodb@3.4.10
mono@4.4.1.0
mozjs@17.0.0
mozjs@24.2.0
mozjs@38.2.1.rc0
mozjs@52.0-1.6507e63
mozjs@60.2.3-2
nmap@7.80
nmoldyn@3.0.11
node@10.16.0
ocaml-dose3@5.0.1
offlineimap@7.2.4
openbox@3.6.1
openimageio@1.7.19
openimageio@1.8.17
opensmtpd-extras@5.7.1
openvswitch@2.12.0
osm2pgsql@0.96.0
ovmf@20170116-1.13a50a6
ovmf-aarch64@20170116-1.13a50a6
ovmf-arm@20170116-1.13a50a6
parted@3.3
perf@5.3.13
petsc@3.11.2
petsc-complex@3.11.2
petsc-complex-openmpi@3.11.2
petsc-openmpi@3.11.2
pidgin@2.13.0
piranha@1.2.1-1.0466d364b
pius@2.2.7
pluma@1.22.0
polybar@3.4.1
poretools@0.6.0-1.e426b1f
proteinortho@5.16b
pspp@1.2.0
pybitmessage@0.6.3.2
qt@4.8.7
qtbase@5.11.3
qtdeclarative@5.11.3
qtmultimedia@5.11.3
qtwebkit@5.212.0-alpha3
qucs@0.0.19-0.b4f27d9
rapicorn@16.0.0
rdiff-backup@1.2.8
renpy@7.3.5
rhythmbox@3.4.3
rocksdb@5.18.3
rrdtool@1.7.1
rseqc@2.6.1
rubber@1.1
rust@1.20.0
rust@1.21.0
rust@1.22.1
rust@1.23.0
rust@1.24.1
rust@1.25.0
rust@1.26.2
rust@1.27.2
rust@1.28.0
rust@1.29.2
rust@1.30.1
rust@1.31.1
rust@1.32.0
rust@1.33.0
rust@1.34.1
rust@1.35.0
rust@1.36.0
rust@1.37.0
scribus@1.5.5
sdcc@3.7.0
singular@4.1.2p1
slepc@3.11.1
slepc-complex@3.11.1
slepc-complex-openmpi@3.11.1
slepc-openmpi@3.11.1
solfege@3.22.2
sundials@3.1.1
sundials-openmpi@3.1.1
syslinux@6.04-pre-1.bb41e93
tadbit@0.2.0
telepathy-glib@0.24.1
telepathy-idle@0.2.0
telepathy-logger@0.8.2
telepathy-mission-control@5.16.5
terminator@1.91
tetoolkit@2.0.3
texlive-bin@20180414
tophat@2.1.1
tor@0.4.1.6
totem@3.30.0
u-boot-a20-olinuxino-lime@2019.04
u-boot-a20-olinuxino-lime2@2019.04
u-boot-a20-olinuxino-micro@2019.04
u-boot-am335x-boneblack@2019.04
u-boot-am335x-evm@2019.04
u-boot-bananapi-m2-ultra@2019.04
u-boot-cubieboard@2019.04
u-boot-cubietruck@2019.04
u-boot-firefly-rk3399@2019.10
u-boot-malta@2019.04
u-boot-mx6cuboxi@2019.04
u-boot-nintendo-nes-classic-edition@2019.04
u-boot-novena@2019.04
u-boot-pine64-plus@2019.04
u-boot-pinebook@2019.04
u-boot-puma-rk3399@2019.04
u-boot-rock64-rk3328@2019.10
u-boot-rockpro64-rk3399@2019.10
u-boot-tools@2019.04
u-boot-vexpress-ca9x4@2019.04
u-boot-wandboard@2019.04
ungoogled-chromium@78.0.3904.108-0.8f06513
ungoogled-chromium-wayland@78.0.3904.108-0.8f06513
vboot-utils@R63-10032.B
vcflib@0.0.0-1.5ac0913
vigra@1.11.1
volk@1.3
wabt@1.0.12
wicd@1.7.4
xcb-util-errors@1.0-1.5d660eb
xen@4.11.1
xpra@2.5.3
youtube-dl-gui@0.3.8
zn-poly@0.9.1
zziplib@0.13.69


Python 2 packages on which the above packages depend

python2@2.7.16
python2-babel@2.7.0
python2-bcrypt@3.1.7
python2-biopython@1.70
python2-booleanoperations@0.7.1 (no Python 3 version in Guix)
python2-bx-python@0.8.2
python2-chardet@3.0.4
python2-cliapp@1.20180812.1 (no Python 3 version in Guix)
python2-configobj@5.0.6
python2-coverage-test-runner@1.15 (no Python 3 version in Guix)
python2-cssmin@0.2.0
python2-curtsies@0.2.11
python2-cython@0.29.13
python2-dateutil@2.8.0
python2-dbus@1.2.14
python2-decorator@4.3.0
python2-defcon@0.3.5 (no Python 3 version in Guix)
python2-dendropy@4.4.0
python2-diff-match-patch@20121119
python2-dirsync@2.2.3
python2-django@1.11.25
python2-django-allauth@0.39.1
python2-django-assets@0.12
python2-django-bulk-update@1.1.10
python2-django-contact-form@1.3
python2-django-contrib-comments@1.8.0
python2-django-jsonfield@1.0.3
python2-django-mailman3@1.1.0 (no Python 3 version in Guix)
python2-django-overextends@0.4.3
python2-django-redis@4.10.0
python2-django-rq@1.3.1
python2-django-sortedm2m@1.3.3
python2-django-statici18n@1.3.0
python2-docopt@0.6.2
python2-efl@1.23.0
python2-elasticsearch@6.1.1
python2-factory-boy@2.8.1
python2-fastalite@0.3
python2-fonttools@3.38.0
python2-fudge@0.9.6
python2-futures@3.2.0 (no Python 3 version in Guix)
python2-greenlet@0.4.15
python2-h5py@2.8.0
python2-htseq@0.9.1 (no Python 3 version in Guix)
python2-jedi@0.15.1
python2-jinja2@2.10.1
python2-levenshtein@0.12.0
python2-libxml2@2.9.9
python2-lxml@4.4.1
python2-magic@0.4.15
python2-mailmanclient@3.1.1
python2-markdown@3.1.1
python2-matplotlib@2.2.4
python2-mmtk@2.7.12 (no Python 3 version in Guix)
python2-mock@2.0.0
python2-mysqlclient@1.3.13
python2-netifaces@0.10.7
python2-networkx@2.2
python2-nose@1.3.7
python2-notify2@0.3.1
python2-notmuch@0.29.3
python2-numpy@1.16.5
python2-pandas@0.24.2
python2-paramiko@2.4.2
python2-pbcore@1.2.10 (no Python 3 version in Guix)
python2-pillow@6.1.0
python2-prompt-toolkit@2.0.7
python2-protobuf@3.10.0
python2-psutil@5.6.5
python2-psycopg2@2.7.7
python2-pybedtools@0.8.0
python2-pyclipper@1.1.0.post1
python2-pygame@1.9.4
python2-pygments@2.4.2
python2-pygobject@3.28.3
python2-pygpgme@0.3
python2-pygtk@2.24.0 (no Python 3 version in Guix)
python2-pynacl@1.3.0
python2-pypdf2@1.26.0
python2-pyroute2@0.5.6 (no Python 3 version in Guix)
python2-pysam@0.15.1
python2-pyserial@3.1.1
python2-pytest@4.4.2
python2-pytest-catchlog@1.2.2
python2-pytest-cov@2.6.1
python2-pytest-django@3.1.2
python2-pytz@2019.1
python2-pyxdg@0.25
python2-pyyaml@3.13
python2-rauth@0.7.3
python2-requests@2.22.0
python2-rq@0.13.0
python2-rsvg@2.32.0 (no Python 3 version in Guix)
python2-scandir@1.9.0
python2-scikit-learn@0.20.4
python2-scipy@1.2.2
python2-setproctitle@1.1.10
python2-six@1.12.0
python2-socksipychain@2.0.15
python2-sphinx@1.7.7
python2-sqlalchemy@1.3.3
python2-statsmodels@0.9.0
python2-stemming@1.0.1 (no Python 3 version in Guix)
python2-swiftclient@2.6.0
python2-tegaki-tools@0.3.1 (no Python 3 version in Guix)
python2-translate-toolkit@2.1.0
python2-ttystatus@0.36 (no Python 3 version in Guix)
python2-twisted@19.7.0
python2-ufolib@2.1.1 (no Python 3 version in Guix)
python2-unittest2@1.1.0
python2-urwid@2.0.1
python2-urwidtrees@1.0.2
python2-xlib@0.14 (no Python 3 version in Guix)
python2-zeroconf@0.19.1 (no Python 3 version in Guix)

Re: Python 2 end-of-life?

[Hartmut Goebel]

Am 28.11.19 um 15:40 schrieb Konrad Hinsen:
> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>> I'd say the very first thing we should do is look at all non-Python
>> packages that depend indirectly on Python 2.
> Here comes an update of my Python-2-in-Guix analysis. Sorted output,
> distinction between likely libraries and likely application packages.
> Plus the list of Python 2 libraries that something else depends on, with
> special attention on those who don't currently have a Python 3
> equivalent in Guix. Those would be good targets for porting work as
> well.

Great!

I've put this into a ehterpad, so we can collaboratively work on this,
without doing duplicate work, and add comments:

https://annuel.framapad.org/p/guix-python2-eol?lang=en

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Python 2 end-of-life?

[zimoun]

Hi Konrad and Hartmut,

This is great!


On Thu, 28 Nov 2019 at 18:24, Hartmut Goebel
<h.goebel@crazy-compilers.com> wrote:

> I've put this into a ehterpad, so we can collaboratively work on this,
> without doing duplicate work, and add comments:
>
> https://annuel.framapad.org/p/guix-python2-eol?lang=en

I am proposing to have all the discussion in this bug report [1]. It
will ease to follow the thread, IMO.
If you agree please remove guix-devel of CC and instead CC
38420@debbugs.gnu.org.

[1] http://issues.guix.gnu.org/issue/38420


Cheers,
simon

Re: Python 2 end-of-life?

[Bengt Richter]

Hi zimoun,

On +2019-11-27 18:35:42 +0100, zimoun wrote:
> Hi Konrad,
> 
> On Wed, 27 Nov 2019 at 08:56, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
> 
> > > IOW, a bunch just differ by version -- I wonder how many of the
> > > packages that drew in old versions could run fine with respective
> > > latest versions of what they are dependent on?
> >
> > That's a very good question!
> 
> Is it not cheap to replace the python2 dependency by the python3 one
> and try to locally build?
>

The first step is probably cheap, but if you just s/python2/python3/g
and it "works," what do you know? Can you even begin to trust, unless
automated tests were _really_^n well designed ??

Also -- are we talking about build-time dependency or run-time?
Is the latter a pure static image or dynamically linked?

And lots more questions ;-)

> 
> The snippet of code you sent helps to figure out which packages are
> still using Python2.
>

I think it represents more than that. It represents an instance of
a kind of system viewer/browser.

Konrad made a new version which is even nicer, as you have probably
already seen.

I think it will be worthwhile to have various viewing/browsing tools
that can help us do reasonable triage on where to put effort.
Something that will reveal the low-hanging fruit ;-)

Also, this is not just about python2 vs python3. There can be minor
dependencies on perl or tcl or lua or whatever, or the language can
be 95% of the source for the package.

How do we know the proportions or their significance without a tool?

Yeah, we can get the sources for the package manually and
look at them various ways, but how long before you make
yourself a viewer of some sort?

But everybody can't easily whip up a useful tool.

So  I think it would be good for guix if the top talent would
favor allocating time to making wizard tools that will empower
lesser mortals to see and operate on what they can't see
without the tools.


Of course, (IMO :) tools should never clobber anything valuable without
understandable warning, and explicit confirmation (and confirmation
of having safely bailed out, for comfort :-). Having /dev/sda1 clobbered
is a nuisance even if you have a backup image. Been there. Twice shy ;-)

> As you said, some packages will not be updated to Python 3. Let
> document it in description for example and let provide a rationale
> explaining why that we put in the manual.
> 
> 
> > That's yet another question: could we patch the upstream code to replace
> > Python by something else that's more convenient for Guix. That may
> > actually be a worthwhile approach to reduce software bloat in Guix,
> > but it also shifts some of the maintenance burden from upstream to Guix.
> 
> It does not appear to me a reasonable approach.

If it could be an automated patch substitution of a pure guix function that
will pass the same well-designed test suite as the python function it replaces,
then I think it is entirely reasonable. We are not asking upstream to do anything,
other than providing a well-designed test suite that will serve themselves well.

Then the need for updates on our side would be detected as testing errors.

> 
> 
> > Guix might well contribute to a long-term improvement of everyone's
> > habits in this respect, by providing the bird's-eyes view of software
> > systems that developers and maintainers of individual packages cannot
> > have. What you describe is just the application of well-known ideas from
> > software engineering (cleaning up code before it becomes unmaintainable)
> > to the systems level. Which is possible because Guix turns the systems
> > level into just another software layer.
> 
> Agree! :-)
>

Yes, +1 for more bird's-eyes viewers ;-)

> 
> All the best,
> simon

-- 
Regards,
Bengt Richter

Re: Python 2 end-of-life?

[Bengt Richter]

Hi zimoun,

On +2019-11-27 18:28:36 +0100, zimoun wrote:
> Hi,
> 
> On Tue, 26 Nov 2019 at 22:52, Bengt Richter <bokr@bokr.com> wrote:
> 
> > egrep -oh '^[^@;]+' py2eol.txt|sort|uniq -c|sort -h|tail
> > --8<---------------cut here---------------start------------->8---
> >       1 zziplib
> >       2 ffmpeg
> >       2 ghc
> >       2 openimageio
> >       3 bedtools
> >       5 mozjs
> >       8 clang
> >       8 clang-runtime
> >       8 llvm
> >      18 rust
> > --8<---------------cut here---------------end--------------->8---
> >
> > IOW, a bunch just differ by version -- I wonder how many of the
> > packages that drew in old versions could run fine with respective
> > latest versions of what they are dependent on?
> 
> For example, considering rust, it is about the bootstrappability. See [1].
> 
> [1] http://guix.gnu.org/blog/2018/bootstrapping-rust/
>

That looks horrible to me :)
Don't they have a "guild disasseble" kind of thing that they could use
to recover a mungeable source of the final product, and then
make a self-hoster from that (even if it takes serious hacking)?

> I am interesting to know about clang/clang-runtime/llvm. Do we support
> 8 versions? Or version n-1 is useful to build version n?
> 
> 
> > It would be really interesting if you could tweak your py2-dependent-package
> > lister to show for each how many lines of py2 code are causing the py2 dependency!
> 
> It is really hard -- nor impossible. And I am not convinced that the
> tough work will pay off.
>

Why so hard? Is not all the information available in sources?

If you wanted to do the job of preplacing a dependency with a guix/guile/bash
native equivalent for a particular package manually, couldn't that be done?
At least, couldn't one decide whether it was going to be easy or hard?

So what tool would help you get to the information you need for that decision quickly?
How would you write a wish-item for that?

That's the kind of tools I hope will emerge, if it proves
feasible to reduce external dependencies that way.

> 
> > It would be a shame if half a dozen lines of python were pulling in
> > all that weight if it could have been a few lines of guile or bash instead.
> 
> Do you propose to patch each time? Because I am not convinced again
> that upstream will change Python to Guile.
>

Yes, I would propose patching if it could be automated along the lines
I suggested in my other reply to you.

The trick is to find the packages for which this would be possible,
so I am hoping tools to look for the low-hanging fruit will emerge.

> 
> > The time it takes to figure out whether/how a trivial dependency can be
> > eliminated is, ISTM, a major cause of gordian-knot dependency bloat -- which IMO
> > in turn is a kind of RYF threat: it's effectively a free-time pay-wall.
> 
> To me, one path to remove unnecessary dependencies of Python2 is to
> give a look package by package, try to replace the Python2 dependency
> by the Python3 (if exist) and see what happens. If it does not build
> because the package really uses Python2 features, figure out which
> one, patch with the Python3 equivalent and submit the patch upstream.

That will likely be the best thing to do for a number of packages,
but I would rather plug in a guix/guile/bash equivalent that passes
the same well-designed (! :) test suite, where that is possible.

So I am hoping for tools to help find those packages where it _is_ possible ;-)

> 
> All the best,
> simon

-- 
Regards,
Bengt Richter

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Bengt and Simon,

Nice discussion, please go on. With Simon taking the pragmatic point of
view ("what can we do right now?") and Bengt the long-term idealist
perspective ("where should Guix be going?"), we might end up getting
both ends right.


Bengt Richter <bokr@bokr.com> writes:

> The first step is probably cheap, but if you just s/python2/python3/g
> and it "works," what do you know? Can you even begin to trust, unless
> automated tests were _really_^n well designed ??

That's every packager's problem. We see our main job as assembling
pieces into a whole, but we should ideally also check the quality
of our building blocks. Unfortunately, that kind of responsibility
has never been valued in the tech world.

Which means that, pragmatically speaking, we'll have to do what everyone
else does: delegate quality control to the end user. Otherwise we'd have
only a hundred packages in Guix and nobody would be interested.

> I think it represents more than that. It represents an instance of
> a kind of system viewer/browser.

That's exactly where I want to go in the long run. The kind of analysis
I have been doing with a Guile script should be integrated into a
decent user interface, emacs-guix being the obvious candidate.
Every software system should have a scriptable user interface,
in my opinion.

> Yeah, we can get the sources for the package manually and
> look at them various ways, but how long before you make
> yourself a viewer of some sort?

Unfortunately that's a hard problem, given the enormous diversity of
packages in Guix. But for specific cases, it looks doable. Much like we
have various build systems for frequent special cases, we could have
inspectors for them as well.

Back to Python, I could add a check for python-build-system to
distinguish between Python libraries/applications and non-Python
code that just contains small helper scripts.

> So  I think it would be good for guix if the top talent would
> favor allocating time to making wizard tools that will empower
> lesser mortals to see and operate on what they can't see
> without the tools.

While I disagree with your choice of terms (top vs. lesser), I do
agree with the principle of providing empowering tools for a wide
range of users (who are not "lesser mortals" for me, but people
with a different focus).

>> For example, considering rust, it is about the bootstrappability. See [1].
>> 
>> [1] http://guix.gnu.org/blog/2018/bootstrapping-rust/
>
> That looks horrible to me :)

Me too. Another long-term mission for Guixers (or perhaps better the
Reproducible Builds community) is education about bootstrapping.
I can understand the reasoning that led the Rust developers to their
approach, but it also shows that they are not aware that they are part
of a larger universe.

>> > It would be really interesting if you could tweak your py2-dependent-package
>> > lister to show for each how many lines of py2 code are causing the py2 dependency!
>> 
>> It is really hard -- nor impossible. And I am not convinced that the
>> tough work will pay off.
>
> Why so hard? Is not all the information available in sources?

Yes, but the structure of the code is not very standardized. Counting
the lines of Python code in a source package is straightforward, but
knowing how important it is is a hard problem. Consider a three-module
Python package with 15 example scripts.

> That's the kind of tools I hope will emerge, if it proves
> feasible to reduce external dependencies that way.

Indeed. And not just for Python. Everybody is living in dependency hell
these days. Guix could in the long run help to fix that.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[zimoun]

Hi Bengt,


On Fri, 29 Nov 2019 at 07:07, Bengt Richter <bokr@bokr.com> wrote:

> > Is it not cheap to replace the python2 dependency by the python3 one
> > and try to locally build?
> >
>
> The first step is probably cheap, but if you just s/python2/python3/g
> and it "works," what do you know? Can you even begin to trust, unless
> automated tests were _really_^n well designed ??

What do you know today? Nothing more?
I do not understand your argument.

Today, the usual rule is: if a package builds and passes its
test-suite then it is included. Next is, if user experiences troubles,
they have 2 choices: report to us and we inspect if it comes from our
side and if not report upstream; or report directly to upstream and
they inspect if it comes from their side and if not report to us.

Therefore, I do not understand what you are talking about. If the test
suite of any package is not well designed, hum?! it is not our problem
but the problem of upstream. We cannot do more; expect contribute
upstream to improve the situation.

I mean, if you want to change how we include one package, let talk
about it. But it is not related to python2 -> python3 switch, IMO.


> Also -- are we talking about build-time dependency or run-time?
> Is the latter a pure static image or dynamically linked?

It is clearly defined by the manual [1], see the part about inputs,
native-inputs, propagated-inputs.

I agree that we have some packages where it is not well set. (I have
in mind a recent complain by Mathieu cross-compiling.) But the usual
rule is: the community commits with good faith and I trust the work
already done.

Again, I do not understand what you are talking about.

[1] https://guix.gnu.org/manual/en/html_node/package-Reference.html#package-Reference


> And lots more questions ;-)

Please report them. :-)


> I think it will be worthwhile to have various viewing/browsing tools
> that can help us do reasonable triage on where to put effort.
> Something that will reveal the low-hanging fruit ;-)

I agree. :-)

Effort about which package is not reproducible and maybe why -- even
if it is really hard to automated that, AFAIK; and which package does
not have a clean bootstrappable path and maybe why -- easier because
generally we already how which packages lead to issue: compilers.



> How do we know the proportions or their significance without a tool?
>
> Yeah, we can get the sources for the package manually and
> look at them various ways, but how long before you make
> yourself a viewer of some sort?
>
> But everybody can't easily whip up a useful tool.
>
> So  I think it would be good for guix if the top talent would
> favor allocating time to making wizard tools that will empower
> lesser mortals to see and operate on what they can't see
> without the tools.

Can you describe what is the aim of such tool?


There is source code living on Internet. Then to include this upstream
source code, we package it which means: write pieces of code to fit
our architecture and correctly set the dependencies. This upstream
source code uses (lua|perl|python|name-it) language to be built and/or
run. And so, why it is interesting to know how many lines? What is the
final goal? Replace this <extra-language> by something else? Why?
Reproducibility? Bootstrappability?



> > > That's yet another question: could we patch the upstream code to replace
> > > Python by something else that's more convenient for Guix. That may
> > > actually be a worthwhile approach to reduce software bloat in Guix,
> > > but it also shifts some of the maintenance burden from upstream to Guix.
> >
> > It does not appear to me a reasonable approach.
>
> If it could be an automated patch substitution of a pure guix function that
> will pass the same well-designed test suite as the python function it replaces,
> then I think it is entirely reasonable. We are not asking upstream to do anything,
> other than providing a well-designed test suite that will serve themselves well.

Please show me the code and I will change my mind about the
"reasonable approach". :-)

And what do we win? More reproducibility? More bootstrappability?


All the best,
simon

Re: Python 2 end-of-life?

[zimoun]

Hi Bengt,

On Fri, 29 Nov 2019 at 07:54, Bengt Richter <bokr@bokr.com> wrote:

> > For example, considering rust, it is about the bootstrappability. See [1].
> >
> > [1] http://guix.gnu.org/blog/2018/bootstrapping-rust/
> >
>
> That looks horrible to me :)
> Don't they have a "guild disasseble" kind of thing that they could use
> to recover a mungeable source of the final product, and then
> make a self-hoster from that (even if it takes serious hacking)?

Do you have better to propose to bootstrap Rust? Does it work?



> > > It would be really interesting if you could tweak your py2-dependent-package
> > > lister to show for each how many lines of py2 code are causing the py2 dependency!
> >
> > It is really hard -- nor impossible. And I am not convinced that the
> > tough work will pay off.
>
> Why so hard? Is not all the information available in sources?

Again, go ahead if you evaluate that it deserves it.
My humble point of view is: it is an hard task and the work will not pay off.

It will not pay off because it is fully unclear what to do with this
information about "how many lines of py2".


My approach is: try to build and/or run with Python 3. If yes, cool!
update the package and remove from the list [1], else inspect why and
try to fix (patch) and report this patch upstream.

[1] http://issues.guix.gnu.org/issue/38420


> > To me, one path to remove unnecessary dependencies of Python2 is to
> > give a look package by package, try to replace the Python2 dependency
> > by the Python3 (if exist) and see what happens. If it does not build
> > because the package really uses Python2 features, figure out which
> > one, patch with the Python3 equivalent and submit the patch upstream.
>
> That will likely be the best thing to do for a number of packages,
> but I would rather plug in a guix/guile/bash equivalent that passes
> the same well-designed (! :) test suite, where that is possible.

What is the goal of such? What do we win? Larger than python 2 -> python 3.


All the best,
simon

Re: Python 2 end-of-life?

[zimoun]

Hi Konrad

On Fri, 29 Nov 2019 at 08:38, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Nice discussion, please go on. With Simon taking the pragmatic point of
> view ("what can we do right now?") and Bengt the long-term idealist
> perspective ("where should Guix be going?"), we might end up getting
> both ends right.

Thank you for this kind words. :-)


> > The first step is probably cheap, but if you just s/python2/python3/g
> > and it "works," what do you know? Can you even begin to trust, unless
> > automated tests were _really_^n well designed ??
>
> That's every packager's problem. We see our main job as assembling
> pieces into a whole, but we should ideally also check the quality
> of our building blocks. Unfortunately, that kind of responsibility
> has never been valued in the tech world.

I agree.


> Which means that, pragmatically speaking, we'll have to do what everyone
> else does: delegate quality control to the end user. Otherwise we'd have
> only a hundred packages in Guix and nobody would be interested.

Delegate the quality control first to the test suite and second to the end user.

Proposing an automated testing framework free of any underlining
language is impossible, IMHO.

Even if I am personally interested in how to efficiently test the code
-- the QuickCheck [2] approach is really interesting and the CakeML
[2] even more :-) -- I am not sure this goal fits the purpose of Guix.
I mean, Guix cannot fix the broken world. ;-) And it already fixes a
lot of things...


[1] http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html
[2] https://cakeml.org/



> > I think it represents more than that. It represents an instance of
> > a kind of system viewer/browser.
>
> That's exactly where I want to go in the long run. The kind of analysis
> I have been doing with a Guile script should be integrated into a
> decent user interface, emacs-guix being the obvious candidate.
> Every software system should have a scriptable user interface,
> in my opinion.

I agree that we are lacking of tools. And our tooling should be improved a lot!

What kind of metrics do you want with such "script"? And metrics to
inform about which goal?


> > That looks horrible to me :)
>
> Me too. Another long-term mission for Guixers (or perhaps better the
> Reproducible Builds community) is education about bootstrapping.
> I can understand the reasoning that led the Rust developers to their
> approach, but it also shows that they are not aware that they are part
> of a larger universe.

AFAIK it is worse in the Haskell world. ;-) Ricardo tried [3] and some
Haskellers spoke [4] about the issue but I have not seen any
initiative from the Haskell community.


[3] https://elephly.net/posts/2017-01-09-bootstrapping-haskell-part-1.html
[4] https://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC


> Indeed. And not just for Python. Everybody is living in dependency hell
> these days. Guix could in the long run help to fix that.

Guix fixes that! (overseeling? ;-)


Cheers,
simon

Re: Python 2 end-of-life?

[Bengt Richter]

Hi zimoun,

On +2019-11-29 12:41:43 +0100, zimoun wrote:
> Hi Bengt,
> 
[...]
> 
> > > > That's yet another question: could we patch the upstream code to replace
> > > > Python by something else that's more convenient for Guix. That may
> > > > actually be a worthwhile approach to reduce software bloat in Guix,
> > > > but it also shifts some of the maintenance burden from upstream to Guix.
> > >
> > > It does not appear to me a reasonable approach.
> >
> > If it could be an automated patch substitution of a pure guix function that
> > will pass the same well-designed test suite as the python function it replaces,
> > then I think it is entirely reasonable. We are not asking upstream to do anything,
> > other than providing a well-designed test suite that will serve themselves well.
> 
> Please show me the code and I will change my mind about the
> "reasonable approach". :-)
> 
> And what do we win? More reproducibility? More bootstrappability?
>

https://en.wikiquote.org/wiki/C._A._R._Hoare#The_Emperor's_Old_Clothes

The first quote in particular ;-)

> 
> All the best,
> simon

-- 
Regards,
Bengt Richter

Re: Python 2 end-of-life?

[zimoun]

Hi Bengt,

On Fri, 29 Nov 2019 at 14:42, Bengt Richter <bokr@bokr.com> wrote:

> > And what do we win? More reproducibility? More bootstrappability?
>
> https://en.wikiquote.org/wiki/C._A._R._Hoare#The_Emperor's_Old_Clothes
>
> The first quote in particular ;-)

I agree with the quote. :-)
But it does not answer to the question. Because your proposal appears
to me more complicated than simple. ;-)

Well, my answer would be: python -c 'import this' ;-)
[...]
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
--


All the best,
simon

Re: Python 2 end-of-life?

[Bengt Richter]

Hi simon,

On +2019-11-29 15:12:49 +0100, zimoun wrote:
> Hi Bengt,
> 
> On Fri, 29 Nov 2019 at 14:42, Bengt Richter <bokr@bokr.com> wrote:
> 
> > > And what do we win? More reproducibility? More bootstrappability?
> >
> > https://en.wikiquote.org/wiki/C._A._R._Hoare#The_Emperor's_Old_Clothes
> >
> > The first quote in particular ;-)
> 
> I agree with the quote. :-)
> But it does not answer to the question. Because your proposal appears
> to me more complicated than simple. ;-)
>

If you don't see that we win something by pursuing the goal of simplicity
that Tony Hoare describes:

    There are two ways of constructing a software design: One
    way is to make it so simple that there are obviously no
    deficiencies, and the other way is to make it so complicated
    that there are no obvious deficiencies. The first method is
    far more difficult. It demands the same skill, devotion,
    insight, and even inspiration as the discovery of the simple
    physical laws which underlie the complex phenomena of
    nature.

then perhaps you sensibly prefer to avoid having the "far more difficult"
goal making your life as a packager "more complicated than simple ;-)".

I'm afraid I am not so sensible, so I am drawn to challenges
that I mostly fail at. ( But mm, aha, yes!: the good moments! :)

Chacun à son goût -- or different strokes for different folks,
as the hippies used to say. Showing my age :)

> Well, my answer would be: python -c 'import this' ;-)
> [...]
> Now is better than never.
> Although never is often better than *right* now.
> If the implementation is hard to explain, it's a bad idea.
> If the implementation is easy to explain, it may be a good idea.
> --

Too many knobs is worse than no knobs, although according to Mae West,
"Too much of a good thing can be wonderful." ;-)

> 
> 
> All the best,
> simon

-- 
Regards,
Bengt Richter