Supporting upstream supported Python versions

Supporting upstream supported Python versions

[jgart]

Hi Guixers,

What is the opinion on supporting current upstream supported versions
of python?

The master branch only has 3.9 but I'd like to have substitutes available
for 3.7, 3.8 and 3.10:

https://endoflife.date/python

Today, I needed python 3.8 and I had to resort to using asdf
(https://asdf-vm.com/) to get a binary. I'd prefer to use Guix for
getting all my needed python versions as substitutes. I think that providing
this is important if we want Guix to be a competitive tool in the python
ecosystem.

Could we make all upstream supported versions of python available in master?

This is currently the experience for ruby having various supported
versions in Guix master. I'd like to have available all supported python
versions also.

I'm happy to work on this.

wdyt

Re: Supporting upstream supported Python versions

[zimoun]

Hi,

On Tue, 17 May 2022 at 00:38, jgart <jgart@dismail.de> wrote:

> What is the opinion on supporting current upstream supported versions
> of python?
>
> The master branch only has 3.9 but I'd like to have substitutes available
> for 3.7, 3.8 and 3.10:

By substitutes, do you mean python@x.y?  Or all python-* packages
compiled with python@x.y instead of python@3.9?


> Could we make all upstream supported versions of python available in
> master?

From my point of view, we can support several version of the CPython VM,
say python@3.10 (python-next), python@3.9 (current default), and various
older CPython interpreters as python@3.8, 3.7, say.

But only the substitutes for python-* packages compiled with the current
default (here python@3.9).  Somehow, we do not have the manpower (I do
not know about he build farm) for maintaining more.  For instance, give
a look at:

     <https://ci.guix.gnu.org/eval/304888/dashboard>

and type “python-”; well, it’s not so bad but it is not all green
though.  Somehow, we have the experience of maintaining 2 versions
(CPython@3 and CPython@2 ;-)) and it is a piece of work to maintain, so
it would be really similar when multiplying the number of CPython@3.x.

Note, however, it is easy to apply ’package-with-explicit-python’ from
’(guix build-system python).


Well, IMHO, supporting several version of CPython is good idea. :-)

Cheers,
simon

Re: Supporting upstream supported Python versions

[Tobias Geerinckx-Rice]

On 17 May 2022 05:38:48 UTC, jgart <jgart@dismail.de> wrote:
>What is the opinion on supporting current upstream supported versions
>of python?
>
>The master branch only has 3.9 but I'd like to have substitutes available
>for 3.7, 3.8 and 3.10:

Go for it if you're willing to maintain them.



Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Re: Supporting upstream supported Python versions

[Ricardo Wurmus]

Hi,

> From: jgart <jgart@dismail.de>
> To: Guix Devel <guix-devel@gnu.org>
> Subject: Supporting upstream supported Python versions
> Message-ID: <20220517003848.GB18763@gac>
[…]
> What is the opinion on supporting current upstream supported versions
> of python?
>
> The master branch only has 3.9 but I'd like to have substitutes available
> for 3.7, 3.8 and 3.10:

The Guix Past channel now also has python-3.8.

I recently needed to be able to build a profile with Python packages
that were built with Python 3.8, but without replacing each and every
instance of Python (because I don’t want to build librsvg, gtk+, etc).

Here’s the manifest I came up with:

--8<---------------cut here---------------start------------->8---
;;; Manifest to build the latest Python packages with Python 3.8.
(use-modules (guix packages)           ; for "package", "package-arguments"...
             (guix build-system python)
             (guix utils)
             (ice-9 match))

;; The list of Python packages (or rather specifications) that we want
;; to build with an older Python.
(define packages
  (list "python-numpy"
        "python-pytorch"
        "python-matplotlib"
        "python-scipy"
        "python-scikit-learn"
        "python-seaborn"))

\f
(define old-python
  (specification->package "python@3.8"))

(define old-python-wrapper
  ;; We use wrap-python3 to create a "python" executable.  Python
  ;; itself only comes with "python3".  Python 3.8 is available in the
  ;; guix-past channel.
  ((@@ (gnu packages python) wrap-python3) old-python))

(define (python-package? package)
  (or (eq? python-build-system
           (package-build-system package))
      ;; Special cases: packages that produce Python modules but don't
      ;; use the Python build system.
      (member (package-name package)
              (list "pybind11"))))

;; I'd love to just use modify-inputs here, but this python-wrapper vs
;; python complication forces me to do this manually.
(define (replace-python-in-inputs inputs python-wrapper python)
  "Some packages have an explicit Python in the inputs, or use a
separate output of the Python package.  We use PYTHON-WRAPPER wherever
the \"python\" executable may be needed and the plain PYTHON package
where we need selected outputs."
  (map (match-lambda
         (("python" package out) `("python" ,python ,out))
         (("python" package) `("python" ,python-wrapper))
         (anything anything))
       inputs))

(define (package-with-different-python pkg python-wrapper python)
  "Return a new package based on PKG that uses PYTHON during the build."
  (package/inherit pkg
    (arguments
     (if (eq? (package-build-system pkg)
              python-build-system)
         (ensure-keyword-arguments
          (package-arguments pkg)
          `(#:python ,python-wrapper
            #:tests? #false))   ;running tests is slow, so why bother?
         (package-arguments pkg)))
    (native-inputs
     (replace-python-in-inputs (package-native-inputs pkg)
                               python-wrapper python))
    (inputs
     (replace-python-in-inputs (package-inputs pkg)
                               python-wrapper python))
    (propagated-inputs
     (replace-python-in-inputs (package-propagated-inputs pkg)
                               python-wrapper python))))

;; This is a recursive package transformer.  When given a package
;; "pkg" it checks if it is a Python package by looking at its build
;; system; if that is the case, it will return a package variant that
;; is built with the old Python.  It does this recursively, so all
;; dependencies are also modified.
(define use-old-python
  (package-mapping
   (lambda (pkg)
     (if (python-package? pkg)
         (let ((modified (package-with-different-python pkg
                                                        old-python-wrapper
                                                        old-python)))
           (match (package-name pkg)
             ;; This package also needs a newer version of setuptools;
             ;; Python 3.8 comes with an older version of setuptools.
             ((or "python-importlib-metadata"
                  "python-ipython")
              (package/inherit modified
                (native-inputs
                 (modify-inputs (package-native-inputs modified)
                   (prepend (specification->package "python-setuptools"))))))
             ;; This package expects typing.py to export
             ;; _SpecialGenericAlias, but this version of Python does
             ;; not define it.
             ("python-typing-inspect"
              (package/inherit modified
                (arguments
                 (substitute-keyword-arguments (package-arguments modified)
                   ((#:phases phases '%standard-phases)
                    `(modify-phases ,phases
                       (add-after 'unpack 'do-not-import-SpecialGenericAlias
                         (lambda _
                           (substitute* "typing_inspect.py"
                             (("_SpecialGenericAlias")
                              "_GenericAlias"))))))))))
             ;; The other packages don't need special treatment.
             (else modified)))
         pkg))
   ;; Stop recursion when we hit a package that is not considered a
   ;; Python package.  This could be a package that uses Python or
   ;; Python modules as inputs but does not itself produce a Python
   ;; module.  This is to avoid needlessly rebuilding big things like
   ;; GTK+.
   (negate python-package?)
   #:deep? #false))

;; Apply the transformer to the list of packages.
(define python-packages-with-old-python
  (map (compose use-old-python specification->package)
       packages))

;; Build a manifest from the list of modified packages and the old
;; Python variant itself.
(packages->manifest
 (cons old-python
       python-packages-with-old-python))
--8<---------------cut here---------------end--------------->8---


-- 
Ricardo

Re: Supporting upstream supported Python versions

[zimoun]

Hi Ricardo,

I am late to the party. :-)

On Wed, 01 Jun 2022 at 09:50, Ricardo Wurmus <rekado@elephly.net> wrote:

> Here’s the manifest I came up with:

By curiosity, why does package-with-explicit-python from (guix
build-system python) not fit the bill?


Cheers,
simon