Using an alternative python to build python modules

Using an alternative python to build python modules

[Kyle Andrews]

Dear Guix,

I want to use the python-apted package in my manifest. That package is
not in Guix, but can readily be fetched from:

```
guix import pypi APTED > python-apted.scm
```

I just had to prefix the following modules to turn that into a package
definition as well as a manifest.

```
(use-modules (guix packages)
	     (guix download)
	     (guix licenses)
	     (guix profiles)
	     (gnu packages)
	     (guix build-system python))
```

During my original testing of my scientific workflow I discovered that
there was a bug introduced to the APTED package which was caused by a
change in how python worked under the hood. As a result, the APTED
package only advertises compatibility with python versions less than
3.7. So, I would like to use python 3.6 to be on the safe side.

I also want to include 60+ other packages in my manifest which are
"current" with the Guix repository and included within it. For
situations like this it would be really convenient if it were possible
to pass a version of python to python-build-system. Then, I hope all I
would have to do is pass another another defined variant of the python
package (e.g. called python-3.6) into that build-system argument.

For example:

```
(build-system python-build-system #:python python-3.6)
```

That would be quite convenient and in line with the level of complexity
I was faced with when I took the conda approach before trying to use
Guix. Unfortunately, this functionality doesn't seem to be provided out
of box and I am seeking help in the hopes that there is an easy way to
do it.

Am I thinking about this right?

Thanks,
Kyle

P.S.

APTED claims to also be able to run on top of python-2.7 so maybe that
fact could be used to simplify this specific issue. I just wanted to
discuss the general problem first because I really want Guix to gain
broad traction within the scientific community.

P.P.S.

Here is a pretend manifest where I would love guidance on how to make
python-apted get built in the context of python-2.7 or ideally
python-3.6 even though the latest version of python is python-3.9 (where
APTED doesn't always work).

```
(use-modules (guix packages)
	     (guix download)
	     (guix licenses)
	     (guix profiles)
	     (gnu packages)
	     (guix build-system python))

(define-public python-apted
  (package
    (name "python-apted")
    (version "1.0.3")
    (source (origin
              (method url-fetch)
              (uri (pypi-uri "apted" version))
              (sha256
               (base32
		"1sawf6s5c64fgnliwy5w5yxliq2fc215m6alisl7yiflwa0m3ymy"))))
    (build-system python-build-system)
    (arguments '(#:tests? #f))
    (home-page "https://github.com/JoaoFelipe/apted")
    (synopsis "APTED algorithm for the Tree Edit Distance")
    (description "APTED algorithm for the Tree Edit Distance")
    (license expat)))

(concatenate-manifests
 (list (specifications->manifest (list "r" "r-dplyr" "r-reticulate"))
       (packages->manifest (list python-apted))))
```

Re: Using an alternative python to build python modules

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 4348 bytes --]

Hi Kyle,

Have you looked at the (package-input-rewriting) procedure described
towards the bottom of "Defining Package Variants"[1]?

You might also like to look at the not-exported
(package-with-explicit-python) proc defined in
guix/build-system/python.scm[2]. It is used to produce python2 variants
of packages but I suppose it would also work for swapping the minor
versions of python3.

Good luck with your task :)

Wojtek

[1] https://guix.gnu.org/manual/en/html_node/Defining-Package-Variants.html
[2] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm?id=b544f460989a6189af111bb3ff6752cabdf23abc#n82

-- (sig_start)
website: https://koszko.org/koszko.html
PGP: https://koszko.org/key.gpg
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Fri, 17 Feb 2023 21:27:39 +0000
Kyle Andrews <kyle@posteo.net> wrote:

> Dear Guix,
> 
> I want to use the python-apted package in my manifest. That package is
> not in Guix, but can readily be fetched from:
> 
> ```
> guix import pypi APTED > python-apted.scm
> ```
> 
> I just had to prefix the following modules to turn that into a package
> definition as well as a manifest.
> 
> ```
> (use-modules (guix packages)
> 	     (guix download)
> 	     (guix licenses)
> 	     (guix profiles)
> 	     (gnu packages)
> 	     (guix build-system python))
> ```
> 
> During my original testing of my scientific workflow I discovered that
> there was a bug introduced to the APTED package which was caused by a
> change in how python worked under the hood. As a result, the APTED
> package only advertises compatibility with python versions less than
> 3.7. So, I would like to use python 3.6 to be on the safe side.
> 
> I also want to include 60+ other packages in my manifest which are
> "current" with the Guix repository and included within it. For
> situations like this it would be really convenient if it were possible
> to pass a version of python to python-build-system. Then, I hope all I
> would have to do is pass another another defined variant of the python
> package (e.g. called python-3.6) into that build-system argument.
> 
> For example:
> 
> ```
> (build-system python-build-system #:python python-3.6)
> ```
> 
> That would be quite convenient and in line with the level of complexity
> I was faced with when I took the conda approach before trying to use
> Guix. Unfortunately, this functionality doesn't seem to be provided out
> of box and I am seeking help in the hopes that there is an easy way to
> do it.
> 
> Am I thinking about this right?
> 
> Thanks,
> Kyle
> 
> P.S.
> 
> APTED claims to also be able to run on top of python-2.7 so maybe that
> fact could be used to simplify this specific issue. I just wanted to
> discuss the general problem first because I really want Guix to gain
> broad traction within the scientific community.
> 
> P.P.S.
> 
> Here is a pretend manifest where I would love guidance on how to make
> python-apted get built in the context of python-2.7 or ideally
> python-3.6 even though the latest version of python is python-3.9 (where
> APTED doesn't always work).
> 
> ```
> (use-modules (guix packages)
> 	     (guix download)
> 	     (guix licenses)
> 	     (guix profiles)
> 	     (gnu packages)
> 	     (guix build-system python))
> 
> (define-public python-apted
>   (package
>     (name "python-apted")
>     (version "1.0.3")
>     (source (origin
>               (method url-fetch)
>               (uri (pypi-uri "apted" version))
>               (sha256
>                (base32
> 		"1sawf6s5c64fgnliwy5w5yxliq2fc215m6alisl7yiflwa0m3ymy"))))
>     (build-system python-build-system)
>     (arguments '(#:tests? #f))
>     (home-page "https://github.com/JoaoFelipe/apted")
>     (synopsis "APTED algorithm for the Tree Edit Distance")
>     (description "APTED algorithm for the Tree Edit Distance")
>     (license expat)))
> 
> (concatenate-manifests
>  (list (specifications->manifest (list "r" "r-dplyr" "r-reticulate"))
>        (packages->manifest (list python-apted))))
> ```
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Using an alternative python to build python modules

[Edouard Klein]

As an example in this line of thought, a long time ago I submitted a
patch (since then reverted as it broke a lot of things) to use an
obsolete version of a python module, which was the only one that worked
with a specific module that hadn't updated its dependencies:

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=32ba87c14fd5e5b54d95211cd9a159d568ce7c67

Look at the following lines:

#+begin_src scheme
(define-public prompt-toolkit-2-instead-of-prompt-toolkit
  (package-input-rewriting/spec
   `(("python-prompt-toolkit" . ,(const python-prompt-toolkit-2)))))
#+end_src

If instead of replacing "python-prompt-toolkit" with
python-prompt-toolkit-2, you could replace "python" with the version of
python you want to use, package-input-rewriting would then handle the
work of changing the python package anywhere it appears in any
dependency of your manifest.

You can then end your manifest with
(python-3.6-instead-of-python python-apted)

Cheers,

Edouard.


Wojtek Kosior via <help-guix@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> Hi Kyle,
>
> Have you looked at the (package-input-rewriting) procedure described
> towards the bottom of "Defining Package Variants"[1]?
>
> You might also like to look at the not-exported
> (package-with-explicit-python) proc defined in
> guix/build-system/python.scm[2]. It is used to produce python2 variants
> of packages but I suppose it would also work for swapping the minor
> versions of python3.
>
> Good luck with your task :)
>
> Wojtek
>
> [1] https://guix.gnu.org/manual/en/html_node/Defining-Package-Variants.html
> [2] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm?id=b544f460989a6189af111bb3ff6752cabdf23abc#n82
>
> -- (sig_start)
> website: https://koszko.org/koszko.html
> PGP: https://koszko.org/key.gpg
> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
>
> ♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
> ✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
> -- (sig_end)
>
>
> On Fri, 17 Feb 2023 21:27:39 +0000
> Kyle Andrews <kyle@posteo.net> wrote:
>
>> Dear Guix,
>>
>> I want to use the python-apted package in my manifest. That package is
>> not in Guix, but can readily be fetched from:
>>
>> ```
>> guix import pypi APTED > python-apted.scm
>> ```
>>
>> I just had to prefix the following modules to turn that into a package
>> definition as well as a manifest.
>>
>> ```
>> (use-modules (guix packages)
>> 	     (guix download)
>> 	     (guix licenses)
>> 	     (guix profiles)
>> 	     (gnu packages)
>> 	     (guix build-system python))
>> ```
>>
>> During my original testing of my scientific workflow I discovered that
>> there was a bug introduced to the APTED package which was caused by a
>> change in how python worked under the hood. As a result, the APTED
>> package only advertises compatibility with python versions less than
>> 3.7. So, I would like to use python 3.6 to be on the safe side.
>>
>> I also want to include 60+ other packages in my manifest which are
>> "current" with the Guix repository and included within it. For
>> situations like this it would be really convenient if it were possible
>> to pass a version of python to python-build-system. Then, I hope all I
>> would have to do is pass another another defined variant of the python
>> package (e.g. called python-3.6) into that build-system argument.
>>
>> For example:
>>
>> ```
>> (build-system python-build-system #:python python-3.6)
>> ```
>>
>> That would be quite convenient and in line with the level of complexity
>> I was faced with when I took the conda approach before trying to use
>> Guix. Unfortunately, this functionality doesn't seem to be provided out
>> of box and I am seeking help in the hopes that there is an easy way to
>> do it.
>>
>> Am I thinking about this right?
>>
>> Thanks,
>> Kyle
>>
>> P.S.
>>
>> APTED claims to also be able to run on top of python-2.7 so maybe that
>> fact could be used to simplify this specific issue. I just wanted to
>> discuss the general problem first because I really want Guix to gain
>> broad traction within the scientific community.
>>
>> P.P.S.
>>
>> Here is a pretend manifest where I would love guidance on how to make
>> python-apted get built in the context of python-2.7 or ideally
>> python-3.6 even though the latest version of python is python-3.9 (where
>> APTED doesn't always work).
>>
>> ```
>> (use-modules (guix packages)
>> 	     (guix download)
>> 	     (guix licenses)
>> 	     (guix profiles)
>> 	     (gnu packages)
>> 	     (guix build-system python))
>>
>> (define-public python-apted
>>   (package
>>     (name "python-apted")
>>     (version "1.0.3")
>>     (source (origin
>>               (method url-fetch)
>>               (uri (pypi-uri "apted" version))
>>               (sha256
>>                (base32
>> 		"1sawf6s5c64fgnliwy5w5yxliq2fc215m6alisl7yiflwa0m3ymy"))))
>>     (build-system python-build-system)
>>     (arguments '(#:tests? #f))
>>     (home-page "https://github.com/JoaoFelipe/apted")
>>     (synopsis "APTED algorithm for the Tree Edit Distance")
>>     (description "APTED algorithm for the Tree Edit Distance")
>>     (license expat)))
>>
>> (concatenate-manifests
>>  (list (specifications->manifest (list "r" "r-dplyr" "r-reticulate"))
>>        (packages->manifest (list python-apted))))
>> ```
>>
>
>
> [[End of PGP Signed Part]]

Re: Using an alternative python to build python modules

[Kyle Andrews]

Dear Edouard and Wojtek,

Thank you very much for your generous suggestions!

I would like to give you an update to share the state of my still
incomplete understanding after thinking about them.

With regards to this sort of thing:

```
(package-input-rewriting/spec
 `(("python-prompt-toolkit" . ,(const python-prompt-toolkit-2)))))
```

I did not try this yet because I don't understand it yet. My confusion
stems from the fact that python is not in the transitive inputs of any
python package. 

```
(use-modules (gnu packages) (gnu packages python))
,pretty-print (package-transitive-inputs python)
```

How could modifying the graph have any effect?

Instead I tried to go down the package-with-python2 route since it APTED
claimed it was tested on python 2.7. Unfortunately, I ran into many
other errors where different python packages in the transitive inputs of
python-numpy failed because they were using various python 3 syntaxes.

Now I am trying to figure out how to best go about trying with python 3
instead. One issue I am not sure how to get around is the
python?-variant property convention. It is not clear to me how to write
one for an inferior python for use with
`package-with-explicit-python'.

I will keep you posted if I make any breakthroughs. I'm really eager to
be able to replace conda with guix.

Best Regards,
Kyle

Edouard Klein <edou@rdklein.fr> writes:

> As an example in this line of thought, a long time ago I submitted a
> patch (since then reverted as it broke a lot of things) to use an
> obsolete version of a python module, which was the only one that worked
> with a specific module that hadn't updated its dependencies:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=32ba87c14fd5e5b54d95211cd9a159d568ce7c67
>
> Look at the following lines:
>
> #+begin_src scheme
> (define-public prompt-toolkit-2-instead-of-prompt-toolkit
>   (package-input-rewriting/spec
>    `(("python-prompt-toolkit" . ,(const python-prompt-toolkit-2)))))
> #+end_src
>
> If instead of replacing "python-prompt-toolkit" with
> python-prompt-toolkit-2, you could replace "python" with the version of
> python you want to use, package-input-rewriting would then handle the
> work of changing the python package anywhere it appears in any
> dependency of your manifest.
>
> You can then end your manifest with
> (python-3.6-instead-of-python python-apted)
>
> Cheers,
>
> Edouard.
>
>
> Wojtek Kosior via <help-guix@gnu.org> writes:
>
>> [[PGP Signed Part:Undecided]]
>> Hi Kyle,
>>
>> Have you looked at the (package-input-rewriting) procedure described
>> towards the bottom of "Defining Package Variants"[1]?
>>
>> You might also like to look at the not-exported
>> (package-with-explicit-python) proc defined in
>> guix/build-system/python.scm[2]. It is used to produce python2 variants
>> of packages but I suppose it would also work for swapping the minor
>> versions of python3.
>>
>> Good luck with your task :)
>>
>> Wojtek
>>
>> [1] https://guix.gnu.org/manual/en/html_node/Defining-Package-Variants.html
>> [2] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm?id=b544f460989a6189af111bb3ff6752cabdf23abc#n82
>>
>> -- (sig_start)
>> website: https://koszko.org/koszko.html
>> PGP: https://koszko.org/key.gpg
>> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
>>
>> ♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
>> ✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
>> -- (sig_end)
>>
>>
>> On Fri, 17 Feb 2023 21:27:39 +0000
>> Kyle Andrews <kyle@posteo.net> wrote:
>>
>>> Dear Guix,
>>>
>>> I want to use the python-apted package in my manifest. That package is
>>> not in Guix, but can readily be fetched from:
>>>
>>> ```
>>> guix import pypi APTED > python-apted.scm
>>> ```
>>>
>>> I just had to prefix the following modules to turn that into a package
>>> definition as well as a manifest.
>>>
>>> ```
>>> (use-modules (guix packages)
>>> 	     (guix download)
>>> 	     (guix licenses)
>>> 	     (guix profiles)
>>> 	     (gnu packages)
>>> 	     (guix build-system python))
>>> ```
>>>
>>> During my original testing of my scientific workflow I discovered that
>>> there was a bug introduced to the APTED package which was caused by a
>>> change in how python worked under the hood. As a result, the APTED
>>> package only advertises compatibility with python versions less than
>>> 3.7. So, I would like to use python 3.6 to be on the safe side.
>>>
>>> I also want to include 60+ other packages in my manifest which are
>>> "current" with the Guix repository and included within it. For
>>> situations like this it would be really convenient if it were possible
>>> to pass a version of python to python-build-system. Then, I hope all I
>>> would have to do is pass another another defined variant of the python
>>> package (e.g. called python-3.6) into that build-system argument.
>>>
>>> For example:
>>>
>>> ```
>>> (build-system python-build-system #:python python-3.6)
>>> ```
>>>
>>> That would be quite convenient and in line with the level of complexity
>>> I was faced with when I took the conda approach before trying to use
>>> Guix. Unfortunately, this functionality doesn't seem to be provided out
>>> of box and I am seeking help in the hopes that there is an easy way to
>>> do it.
>>>
>>> Am I thinking about this right?
>>>
>>> Thanks,
>>> Kyle
>>>
>>> P.S.
>>>
>>> APTED claims to also be able to run on top of python-2.7 so maybe that
>>> fact could be used to simplify this specific issue. I just wanted to
>>> discuss the general problem first because I really want Guix to gain
>>> broad traction within the scientific community.
>>>
>>> P.P.S.
>>>
>>> Here is a pretend manifest where I would love guidance on how to make
>>> python-apted get built in the context of python-2.7 or ideally
>>> python-3.6 even though the latest version of python is python-3.9 (where
>>> APTED doesn't always work).
>>>
>>> ```
>>> (use-modules (guix packages)
>>> 	     (guix download)
>>> 	     (guix licenses)
>>> 	     (guix profiles)
>>> 	     (gnu packages)
>>> 	     (guix build-system python))
>>>
>>> (define-public python-apted
>>>   (package
>>>     (name "python-apted")
>>>     (version "1.0.3")
>>>     (source (origin
>>>               (method url-fetch)
>>>               (uri (pypi-uri "apted" version))
>>>               (sha256
>>>                (base32
>>> 		"1sawf6s5c64fgnliwy5w5yxliq2fc215m6alisl7yiflwa0m3ymy"))))
>>>     (build-system python-build-system)
>>>     (arguments '(#:tests? #f))
>>>     (home-page "https://github.com/JoaoFelipe/apted")
>>>     (synopsis "APTED algorithm for the Tree Edit Distance")
>>>     (description "APTED algorithm for the Tree Edit Distance")
>>>     (license expat)))
>>>
>>> (concatenate-manifests
>>>  (list (specifications->manifest (list "r" "r-dplyr" "r-reticulate"))
>>>        (packages->manifest (list python-apted))))
>>> ```
>>>
>>
>>
>> [[End of PGP Signed Part]]

Re: Using an alternative python to build python modules

[Edouard Klein]

Hi !

I am not an expert, and your question definitively is in somewhat new
territory to me, if a real expert comes by, listen to them instead. From
what I gather, transitive inputs are only directs inputs and then
propagated inputs (from the docstring of the package-transitive-inputs
func).

However, package-input-rewriting, as per the doc:

replaces its direct and indirect dependencies, including implicit inputs
when deep? is true, according to replacements

I'm not sure of the precise meaning of indirect dependencies, but I
would expect the Python interpreter to be one of them (and also the
compiler that would build this interpreter, for example).

Therefore, adding #:deep #t may suffice.

However, Simon's solution with package-with-explicit-python looks more
appropriate to me.

Reading its source, I see that python packages have a #:python package
argument, so in the package definition of python-apted, adding

    (arguments
     `(#:python ,your-old-python))

my be enough, but that is just a guess and if I were you I'd try simon's
snippet first.

Good luck,

Cheers,

Edouard


Kyle Andrews <kyle@posteo.net> writes:

> Dear Edouard and Wojtek,
>
> Thank you very much for your generous suggestions!
>
> I would like to give you an update to share the state of my still
> incomplete understanding after thinking about them.
>
> With regards to this sort of thing:
>
> ```
> (package-input-rewriting/spec
>  `(("python-prompt-toolkit" . ,(const python-prompt-toolkit-2)))))
> ```
>
>
> I did not try this yet because I don't understand it yet. My confusion
> stems from the fact that python is not in the transitive inputs of any
> python package.
>
> ```
> (use-modules (gnu packages) (gnu packages python))
> ,pretty-print (package-transitive-inputs python)
> ```
>
> How could modifying the graph have any effect?
>
> Instead I tried to go down the package-with-python2 route since it APTED
> claimed it was tested on python 2.7. Unfortunately, I ran into many
> other errors where different python packages in the transitive inputs of
> python-numpy failed because they were using various python 3 syntaxes.
>
> Now I am trying to figure out how to best go about trying with python 3
> instead. One issue I am not sure how to get around is the
> python?-variant property convention. It is not clear to me how to write
> one for an inferior python for use with
> `package-with-explicit-python'.
>
> I will keep you posted if I make any breakthroughs. I'm really eager to
> be able to replace conda with guix.
>
> Best Regards,
> Kyle
>
> Edouard Klein <edou@rdklein.fr> writes:
>
>> As an example in this line of thought, a long time ago I submitted a
>> patch (since then reverted as it broke a lot of things) to use an
>> obsolete version of a python module, which was the only one that worked
>> with a specific module that hadn't updated its dependencies:
>>
>> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=32ba87c14fd5e5b54d95211cd9a159d568ce7c67
>>
>> Look at the following lines:
>>
>> #+begin_src scheme
>> (define-public prompt-toolkit-2-instead-of-prompt-toolkit
>>   (package-input-rewriting/spec
>>    `(("python-prompt-toolkit" . ,(const python-prompt-toolkit-2)))))
>> #+end_src
>>
>> If instead of replacing "python-prompt-toolkit" with
>> python-prompt-toolkit-2, you could replace "python" with the version of
>> python you want to use, package-input-rewriting would then handle the
>> work of changing the python package anywhere it appears in any
>> dependency of your manifest.
>>
>> You can then end your manifest with
>> (python-3.6-instead-of-python python-apted)
>>
>> Cheers,
>>
>> Edouard.
>>
>>
>> Wojtek Kosior via <help-guix@gnu.org> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> Hi Kyle,
>>>
>>> Have you looked at the (package-input-rewriting) procedure described
>>> towards the bottom of "Defining Package Variants"[1]?
>>>
>>> You might also like to look at the not-exported
>>> (package-with-explicit-python) proc defined in
>>> guix/build-system/python.scm[2]. It is used to produce python2 variants
>>> of packages but I suppose it would also work for swapping the minor
>>> versions of python3.
>>>
>>> Good luck with your task :)
>>>
>>> Wojtek
>>>
>>> [1] https://guix.gnu.org/manual/en/html_node/Defining-Package-Variants.html
>>> [2] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/python.scm?id=b544f460989a6189af111bb3ff6752cabdf23abc#n82
>>>
>>> -- (sig_start)
>>> website: https://koszko.org/koszko.html
>>> PGP: https://koszko.org/key.gpg
>>> fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
>>>
>>> ♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
>>> ✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
>>> -- (sig_end)
>>>
>>>
>>> On Fri, 17 Feb 2023 21:27:39 +0000
>>> Kyle Andrews <kyle@posteo.net> wrote:
>>>
>>>> Dear Guix,
>>>>
>>>> I want to use the python-apted package in my manifest. That package is
>>>> not in Guix, but can readily be fetched from:
>>>>
>>>> ```
>>>> guix import pypi APTED > python-apted.scm
>>>> ```
>>>>
>>>> I just had to prefix the following modules to turn that into a package
>>>> definition as well as a manifest.
>>>>
>>>> ```
>>>> (use-modules (guix packages)
>>>> 	     (guix download)
>>>> 	     (guix licenses)
>>>> 	     (guix profiles)
>>>> 	     (gnu packages)
>>>> 	     (guix build-system python))
>>>> ```
>>>>
>>>> During my original testing of my scientific workflow I discovered that
>>>> there was a bug introduced to the APTED package which was caused by a
>>>> change in how python worked under the hood. As a result, the APTED
>>>> package only advertises compatibility with python versions less than
>>>> 3.7. So, I would like to use python 3.6 to be on the safe side.
>>>>
>>>> I also want to include 60+ other packages in my manifest which are
>>>> "current" with the Guix repository and included within it. For
>>>> situations like this it would be really convenient if it were possible
>>>> to pass a version of python to python-build-system. Then, I hope all I
>>>> would have to do is pass another another defined variant of the python
>>>> package (e.g. called python-3.6) into that build-system argument.
>>>>
>>>> For example:
>>>>
>>>> ```
>>>> (build-system python-build-system #:python python-3.6)
>>>> ```
>>>>
>>>> That would be quite convenient and in line with the level of complexity
>>>> I was faced with when I took the conda approach before trying to use
>>>> Guix. Unfortunately, this functionality doesn't seem to be provided out
>>>> of box and I am seeking help in the hopes that there is an easy way to
>>>> do it.
>>>>
>>>> Am I thinking about this right?
>>>>
>>>> Thanks,
>>>> Kyle
>>>>
>>>> P.S.
>>>>
>>>> APTED claims to also be able to run on top of python-2.7 so maybe that
>>>> fact could be used to simplify this specific issue. I just wanted to
>>>> discuss the general problem first because I really want Guix to gain
>>>> broad traction within the scientific community.
>>>>
>>>> P.P.S.
>>>>
>>>> Here is a pretend manifest where I would love guidance on how to make
>>>> python-apted get built in the context of python-2.7 or ideally
>>>> python-3.6 even though the latest version of python is python-3.9 (where
>>>> APTED doesn't always work).
>>>>
>>>> ```
>>>> (use-modules (guix packages)
>>>> 	     (guix download)
>>>> 	     (guix licenses)
>>>> 	     (guix profiles)
>>>> 	     (gnu packages)
>>>> 	     (guix build-system python))
>>>>
>>>> (define-public python-apted
>>>>   (package
>>>>     (name "python-apted")
>>>>     (version "1.0.3")
>>>>     (source (origin
>>>>               (method url-fetch)
>>>>               (uri (pypi-uri "apted" version))
>>>>               (sha256
>>>>                (base32
>>>> 		"1sawf6s5c64fgnliwy5w5yxliq2fc215m6alisl7yiflwa0m3ymy"))))
>>>>     (build-system python-build-system)
>>>>     (arguments '(#:tests? #f))
>>>>     (home-page "https://github.com/JoaoFelipe/apted")
>>>>     (synopsis "APTED algorithm for the Tree Edit Distance")
>>>>     (description "APTED algorithm for the Tree Edit Distance")
>>>>     (license expat)))
>>>>
>>>> (concatenate-manifests
>>>>  (list (specifications->manifest (list "r" "r-dplyr" "r-reticulate"))
>>>>        (packages->manifest (list python-apted))))
>>>> ```
>>>>
>>>
>>>
>>> [[End of PGP Signed Part]]

Re: Using an alternative python to build python modules

[Simon Tournier]

Hi,

On Fri, 17 Feb 2023 at 21:27, Kyle Andrews <kyle@posteo.net> wrote:

> For example:
>
> ```
> (build-system python-build-system #:python python-3.6)
> ```
>
> That would be quite convenient and in line with the level of complexity
> I was faced with when I took the conda approach before trying to use
> Guix. Unfortunately, this functionality doesn't seem to be provided out
> of box and I am seeking help in the hopes that there is an easy way to
> do it.

You might be interested by the unexported procedure from (guix
build-system python):

--8<---------------cut here---------------start------------->8---
(define* (package-with-explicit-python python old-prefix new-prefix
                                       #:key variant-property)
  "Return a procedure of one argument, P.  The procedure creates a package with
the same fields as P, which is assumed to use PYTHON-BUILD-SYSTEM, such that
it is compiled with PYTHON instead.  The inputs are changed recursively
accordingly.  If the name of P starts with OLD-PREFIX, this is replaced by
NEW-PREFIX; otherwise, NEW-PREFIX is prepended to the name.

When VARIANT-PROPERTY is present, it is used as a key to search for
pre-defined variants of this transformation recorded in the 'properties' field
of packages.  The property value must be the promise of a package.  This is a
convenient way for package writers to force the transformation to use
pre-defined variants."
--8<---------------cut here---------------end--------------->8---

For example, it was intensively used for maintaining both Python 3 and
Python 2 variants.

For instance, if you have a variant of the Python interpreter say
variant-python, then can have a manifest that apply the transformation,
something like:

--8<---------------cut here---------------start------------->8---
(map (lambda (pkg)
       ((package-with-explicit-python variant-python
       “python-” “python-” #:variant-property my-python)
       pkg))
  (list
    python-one python-two python-three etc))
--8<---------------cut here---------------end--------------->8---
  

Cheers,
simon

Re: Using an alternative python to build python modules

[Kyle Andrews]

Simon Tournier <zimon.toutoune@gmail.com> writes:

>
> You might be interested by the unexported procedure from (guix
> build-system python)

Thanks, Simon. Wojtek already suggested this procedure to me and I have
been trying to understand it. Unfortunately, I am stuck earlier than I
originally thought. I can't figure out how to call it at all. I was
hoping it could be accessed via th `@@` procedure which I could think of
as analogous to the `:::` procedure in R which supposedly performs a
similar action.

```
repl> (define pwep (@@ (guix build-system python) package-with-explicit-python))
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
error: package-with-explicit-python: unbound variable
```

I am starting the REPL via the `guix repl --listen=tcp:37146` command
and am able to execute `,use (guix build python)` inside of the guile
running in geiser.

There seems to be some secret Guile incantation I am overlooking. Could
you give me a hint of what this could be?

Thanks,
Kyle

Re: Using an alternative python to build python modules

[Simon Tournier]

Hi,

On Tue, 21 Feb 2023 at 20:26, Kyle Andrews <kyle@posteo.net> wrote:
> ```
> repl> (define pwep (@@ (guix build-system python) package-with-explicit-python))
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> error: package-with-explicit-python: unbound variable
> ```

[...]

> There seems to be some secret Guile incantation I am overlooking. Could
> you give me a hint of what this could be?

It is about the Guile module system and sometime I am also puzzled why
the @@ is not working as expected.

Well, sorry I do not know and I cannot help.  Even, I would also like
having a clue about how to fix this annoyance.

Cheers,
simon

Re: Using an alternative python to build python modules

[Simon Tournier]

Hi,

On Tue, 21 Feb 2023 at 22:10, Simon Tournier <zimon.toutoune@gmail.com> wrote:

> It is about the Guile module system and sometime I am also puzzled why
> the @@ is not working as expected.

Probably because, quoting Tobias [1]:

        could this be due to declarative modules? The definition's not
        used anywhere else and *may* be inlined when Guile sees fit

1: <https://logs.guix.gnu.org/guix/2023-02-21.log#221358>


Cheers,
simon

Re: Using an alternative python to build python modules

[Kyle Andrews]

Kyle Andrews <kyle@posteo.net> writes:

> There seems to be some secret Guile incantation I am overlooking. Could
> you give me a hint of what this could be?

Tobias pointed out to me that the Guile developers could be to blame
here. They appear a bit too preoccupied with prematurely optimizing the
speed of code with the result that it has already been made inaccessible
before users have a fighting chance to interactively study it to
understand how it works.

The workflow described in the Guile manual seems like it would be
perfect. It suggests I should be able to just modify the source code and
reload it into the running environment. Unfortunately, I don't know how
to associate my running environment via the Guix REPL with my git
checkout.

Re: Using an alternative python to build python modules

[Kyle Andrews]

Kyle Andrews <kyle@posteo.net> writes:

> Kyle Andrews <kyle@posteo.net> writes:
>
>> There seems to be some secret Guile incantation I am overlooking. Could
>> you give me a hint of what this could be?
>
> Tobias pointed out to me that the Guile developers could be to blame
> here. They appear a bit too preoccupied with prematurely optimizing the
> speed of code with the result that it has already been made inaccessible
> before users have a fighting chance to interactively study it to
> understand how it works.
>
> The workflow described in the Guile manual seems like it would be
> perfect. It suggests I should be able to just modify the source code and
> reload it into the running environment. Unfortunately, I don't know how
> to associate my running environment via the Guix REPL with my git
> checkout.

Reading a few hours later I apologize that I came off a bit harshly. I
am grateful to guile writers for sharing their work. I appreciate that
it is a challenging task making a computing system which is approachable
while also being fast when it needs to be. There will always be some
room left for additional improvement.

I am starting to feel intense pressure to move on to my next project. My
colleagues just don't get it. I do, and I really really want to show
them that reproducible research can be done without much trouble even
for scientific workflows mixing R and python packages and scripts, which
are probably the most complex workflows "in the wild" in my neck of the
woods. So, even if they don't get it yet, they might in a few months
when their projects break as the software ecosystem moves on to the next
fad. It always does.

From a broader perspective, sometimes the only way for people to get
things is if they see a lot of pressure from other peers in their field
to do it. If it is perceived as a "go do" and not as an arduous journey
with a realistic prospect of failure, then it will get done. I want to
make sure it's the former. To that end, I want there to be an entry in
the cookbook for this. I just sent a patch outlining my ideas for what
the scope should be.

Of course, since I can't actually figure it out yet, it's more of a
draft at present. Honestly, I couldn't get my poor ancient laptop to
even finish compiling the inferior - much less try using that inferior
with your package-with-explicit-python. I'm just wishing that this
approach will work in the near future. So, please don't let anyone
include it yet :)

Re: Using an alternative python to build python modules

[Simon Tournier]

Hi,

There is no free lunch. ;-)

On Wed, 22 Feb 2023 at 04:23, Kyle Andrews <kyle@posteo.net> wrote:

>                   Honestly, I couldn't get my poor ancient laptop to
> even finish compiling the inferior - much less try using that inferior
> with your package-with-explicit-python. I'm just wishing that this
> approach will work in the near future. So, please don't let anyone
> include it yet :)

Well, ’package-with-explicit-python’ traverses all the graph and can go
deep; basically you might end with a world rebuild since the dependency
of Python is often deep.

(I let aside the inferior part that you do not need, IMHO.)

The computational environment you create with Conda is totally
inconsistent in regard to the Python version label.  The Conda
resolver works using the version label.  Consider this scenario:

A library X version x.y builds with Python interpreter version p.q and
you are interested by library A depending on library X version x.y and
also by some Python libraries and the Python interpreter version b.d,
then library X is not rebuilt with Python interpreter version b.d.
Somehow, Conda exploits some binary compatibility, roughly what Guix
names grafts. :-)

Other said, Conda is not building and then serving all the matrix of
possible combinations.  Somehow Conda cheats when Guix not.  Well, I am
not saying that cheating is not practical, instead I think that by using
this kind of cheat, the reproducibility becomes harder by design because
it lacks a fine control of the complete DAG.

This lead to the other point…

>                                         I'm just wishing that this
> approach will work in the near future. So, please don't let anyone
> include it yet :)

…rewrite on the fly the DAG is cheap with Guix but build it is not. :-)

I agree that a similar feature as Conda where you can specify the Python
version and a list of Python library, yeah this feature would help
people from Conda in convincing them that Guix is more suitable.

However, for most of the scientific applications I know, the version of
Python is not really something pertinent – other said, that’s a fun
feature but useless in practise, IMHO, just a dumb collective practise
inherited from the poor Python ecosystem, and for sure it does not help
for reproducing the computational environment (end of my grumbles :-)).

Well, I do not know if the rebuild you are experimenting is avoidable or
desirable.  Maybe, it would be possible to cut the graph earlier or to
avoid to walk too deeply.  I do not know.


Cheers,
simon

Re: Using an alternative python to build python modules

[Kyle Andrews]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> (I let aside the inferior part that you do not need, IMHO.)

I hope you are right. I will keep thinking about it and will try to
revisit some of my assumptions.

> The computational environment you create with Conda is totally
> inconsistent in regard to the Python version label.  The Conda
> resolver works using the version label.  Consider this scenario:
>
> A library X version x.y builds with Python interpreter version p.q and
> you are interested by library A depending on library X version x.y and
> also by some Python libraries and the Python interpreter version b.d,
> then library X is not rebuilt with Python interpreter version b.d.
> Somehow, Conda exploits some binary compatibility, roughly what Guix
> names grafts. :-)
>
> Other said, Conda is not building and then serving all the matrix of
> possible combinations.  Somehow Conda cheats when Guix not.  Well, I am
> not saying that cheating is not practical, instead I think that by using
> this kind of cheat, the reproducibility becomes harder by design because
> it lacks a fine control of the complete DAG.

That makes sense. However, I see scientists using conda as a way to get
up and running quickly with any kind of development environment at all
for the packages that interest them. I just wish there was a low
complexity path to go from that chaos to a reproducible specification
once the scientist has established their proof of concept was a success.

That said, I am very impressed by how much complex scientific software
is already available in Guix. For example, during this email I was going
to make an example of rdkit only to find that it was already
packaged!. I had struggled for a long time to get that installed on
centos7 outside of a conda environment before I became comfortable with
them. Of course, there are still some big gaps in the chemistry
packages, especially for widely used for Density Functional Theory-based
calculations like ABINIT and Quantum Espresso. I suppose it should only
take one person rolling up their sleeves.

=> https://www.abinit.org/
=> https://www.quantum-espresso.org/

> This lead to the other point…
>
>>                                         I'm just wishing that this
>> approach will work in the near future. So, please don't let anyone
>> include it yet :)
>
> …rewrite on the fly the DAG is cheap with Guix but build it is not. :-)

That makes sense.

> I agree that a similar feature as Conda where you can specify the Python
> version and a list of Python library, yeah this feature would help
> people from Conda in convincing them that Guix is more suitable.

> However, for most of the scientific applications I know, the version of
> Python is not really something pertinent – other said, that’s a fun
> feature but useless in practise, IMHO, just a dumb collective practise
> inherited from the poor Python ecosystem, and for sure it does not help
> for reproducing the computational environment (end of my grumbles :-)).

Again, I hope you are right. I really appreciate all your guidance.