Replicable environment with Python packages missing from Guix

Replicable environment with Python packages missing from Guix

[Lasse Schlör]

Hello,

I am in the process of setting up a replicable environment (OS & software versions) for a research project using Guix. This includes an installation of the Python package PsychoPy.

As far as I understand, the ideal way to do this is to use a `manifest.scm` file that includes a `python-psychopy` package together with a `channels.scm` to pin/freeze the exact software versions.

Currently, no PsychoPy package exists for Guix, and so I attempted to write my own Guix packages for PsychoPy and those of its dependencies that don't exist as Guix packages either.

However, while I was able to successfully wrap a few Python packages as Guix packages, the number of build and test dependencies turns out to be quite large, and I am not sure I have the knowledge and time to write Guix packages for all of them.

As a workaround, I have instead written a manifest file that provides a fixed version of Python, Pip, and some of PsychoPy's dependencies. I can use this to start a container with `guix time-machine […] -- shell --container --share=path/to/persistent/.local=/home/username/.local […]`, which has a persistent `~/.local` directory, into which the correct version of PsychoPy can be installed via `python3 -m pip install -r path/to/requirements.txt`. Making `~/.local` persistent allows to start new instances of this container without having to re-install PsychoPy every time.

Of course, this doesn't feel as elegant as the purely functional Guix way of doing things. I am sure others before me have had similar situations. Thus, my question is: Is there a known/preferred/elegant way of creating replicable Guix environments with fixed versions of Python packages when these packages do not exist as Guix packages?

I'd appreciate any insight on this.

Thanks and best regards,
Lasse Schlör

Re: Replicable environment with Python packages missing from Guix

[wolf]

[-- Attachment #1: Type: text/plain, Size: 2649 bytes --]

On 2023-10-05 15:00:13 +0200, Lasse Schlör wrote:
> Hello,
> 
> I am in the process of setting up a replicable environment (OS & software versions) for a research project using Guix. This includes an installation of the Python package PsychoPy.
> 
> As far as I understand, the ideal way to do this is to use a `manifest.scm` file that includes a `python-psychopy` package together with a `channels.scm` to pin/freeze the exact software versions.
> 
> Currently, no PsychoPy package exists for Guix, and so I attempted to write my own Guix packages for PsychoPy and those of its dependencies that don't exist as Guix packages either.
> 
> However, while I was able to successfully wrap a few Python packages as Guix packages, the number of build and test dependencies turns out to be quite large, and I am not sure I have the knowledge and time to write Guix packages for all of them.

You could try to use output of `guix import'[0] as a base.

0: https://guix.gnu.org/manual/en/manual/devel/en/html_node/Invoking-guix-import.html

> 
> As a workaround, I have instead written a manifest file that provides a fixed version of Python, Pip, and some of PsychoPy's dependencies. I can use this to start a container with `guix time-machine […] -- shell --container --share=path/to/persistent/.local=/home/username/.local […]`, which has a persistent `~/.local` directory, into which the correct version of PsychoPy can be installed via `python3 -m pip install -r path/to/requirements.txt`. Making `~/.local` persistent allows to start new instances of this container without having to re-install PsychoPy every time.
> 
> Of course, this doesn't feel as elegant as the purely functional Guix way of doing things. I am sure others before me have had similar situations. Thus, my question is: Is there a known/preferred/elegant way of creating replicable Guix environments with fixed versions of Python packages when these packages do not exist as Guix packages?

Maybe someone will correct me and provide a better way, but I am afraid the
correct approach here is to indeed create the packages in the manifest file.

Hm, maybe there is an ugly solution, which however might work.  You could do the
pip install and then archive and package the resulting directory.  That will be
messy, but it might work.

However, I would recommend just trying to make the packages, possibly using guix
import.

> 
> I'd appreciate any insight on this.
> 
> Thanks and best regards,
> Lasse Schlör
> 
>

W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Replicable environment with Python packages missing from Guix

[Simon Tournier]

Hi,

On Thu, 05 Oct 2023 at 15:00, Lasse Schlör <lasse.schloer@uni-tuebingen.de> wrote:

> Currently, no PsychoPy package exists for Guix, and so I attempted to
> write my own Guix packages for PsychoPy and those of its dependencies
> that don't exist as Guix packages either.

Well, sadly Guix cannot fix the world. :-) Here, PsychoPy is poorly
packaged in PyPI, thus there is no easy way to write Guix recipe for it.

--8<---------------cut here---------------start------------->8---
$ guix import pypi PsychoPy -r

Starting download of /tmp/guix-file.6lTxL2
From https://files.pythonhosted.org/packages/16/17/a73ebae6cfcbd6f4cf094fba59b17f1ca1b03b6a8c0afdaa11468bc3fe62/psychopy-2023.2.3.tar.gz...
 …3.2.3.tar.gz  34.3MiB               550KiB/s 01:04 ▕██████████████████▏ 100.0%
guix import: warning: Cannot guess requirements from source archive: no requires.txt file found.
(define-public python-psychopy
  (package
    (name "python-psychopy")
    (version "2023.2.3")
    (source (origin
              (method url-fetch)
              (uri (pypi-uri "psychopy" version))
              (sha256
               (base32
                "0mf3gas4892108c7y0n8sl7phkl6w2z4lgymb4n0d6h9p6g58mpq"))))
    (build-system pyproject-build-system)
    (home-page "")
    (synopsis
     "PsychoPy provides easy, precise, flexible experiments in behavioural sciences")
    (description
     "@code{PsychoPy} provides easy, precise, flexible experiments in behavioural
sciences")
    (license #f)))
--8<---------------cut here---------------end--------------->8---

All, and I do not see that packages in any of extra scientific channels:

    https://hpc.guix.info/channels/

but I can have overlooked.

> As a workaround, I have instead written a manifest file that provides
> a fixed version of Python, Pip, and some of PsychoPy's dependencies. I
> can use this to start a container with `guix time-machine […] -- shell
> --container --share=path/to/persistent/.local=/home/username/.local
> […]`, which has a persistent `~/.local` directory, into which the
> correct version of PsychoPy can be installed via `python3 -m pip
> install -r path/to/requirements.txt`. Making `~/.local` persistent
> allows to start new instances of this container without having to
> re-install PsychoPy every time.

If I understand correctly, here the only thing you will be able to
reproduce from one machine to another is ’python3’.  But you have no
guarantee that you will be able to reproduce ~/.local elsewhere.  Well,
based on my poor knowledge of ’pip’and PyPI.

However, that’s already good.  Because it means that if your
computations work in an isolated environment created with “guix shell
--container” then it means there is no leak to system-wide libraries.
And, to my knowledge, this kind of packages are easier to package. ;-)


> Of course, this doesn't feel as elegant as the purely functional Guix
> way of doing things. I am sure others before me have had similar
> situations. Thus, my question is: Is there a known/preferred/elegant
> way of creating replicable Guix environments with fixed versions of
> Python packages when these packages do not exist as Guix packages?

The only way to have replicable Guix environment is to fully control all
the chain, and thus, it means to have a Guix recipe for the packages.

However, as you are pointing, when packages are missing, I accept the
trade-off to work with language-specific package manager knowing that I
loose some guarantees for reproducing.  Trade-off. :-)



Cheers,
simon