Setting up a python environment

Setting up a python environment

[Roy Lemmon]

Hi,

I would like to ask about the general philosophy of setting up a python
environment under guix.

For other linux systems I have used pip to manage python packages and
libraries.

In guix is the idea that guix replaces pip for managing packages and
libraries ? So I would create a python-package for any missing python
packages ? In the guix package lists, I can find many of the python
libraries but there are a few missing that I would like, eg. astropy - an
astronomy analysis library. So should I be creating a guix package for that
rather than installing with pip ?

Thanks
Roy.

Re: Setting up a python environment

[sirgazil]

 ---- On Mon, 11 May 2020 07:34:09 -0500 Roy Lemmon <roy@roylemmon.com> wrote ----
 > Hi,
 > 
 > I would like to ask about the general philosophy of setting up a python
 > environment under guix.
 > 
 > For other linux systems I have used pip to manage python packages and
 > libraries.
 > 
 > In guix is the idea that guix replaces pip for managing packages and
 > libraries ?

I think so, yes, ideally.

 > So I would create a python-package for any missing python
 > packages ?

Yes.

 > In the guix package lists, I can find many of the python
 > libraries but there are a few missing that I would like, eg. astropy - an
 > astronomy analysis library. So should I be creating a guix package for that
 > rather than installing with pip ?
 
That's the idea, yes: to have all Python packages available in Guix. That way, if your projects depend on Python packages and non-Python packages, you can specify all the requirements of your projects conveniently in a Guix manifest file (instead of having a requirements file for pip and another requirements file for some OS package manager, and possibly, another requirements file for another programmnig language used in a project).

In practice, though, your projects could require many packages that are not yet in Guix. For example, astropy could depend on other packages that are not available, and those packages depend on other packages that are not available, and so on. To package them all would be a job in itself, and you would not have time to work on your project.  So, depending on your specific case, you might still need to use Python virtual environments and pip to get things done. For the latter you can use Python's venv, or poetry, or other tools that are already available in Guix for managing Python projects and environments.

I recommend to watch the https://guix.gnu.org/videos/ for an introduction to packaging.




---
https://sirgazil.bitbucket.io/

Re: Setting up a python environment

[Roy Lemmon]

Thanks sirgazil. That is very helpful.

Yes, it might be that astropy has many dependencies.

I will take a look at that video and also get a better handle on packaging.

Cheers
Roy.


On Mon, 11 May 2020, 14:56 sirgazil, <sirgazil@zoho.com> wrote:

>  ---- On Mon, 11 May 2020 07:34:09 -0500 Roy Lemmon <roy@roylemmon.com>
> wrote ----
>  > Hi,
>  >
>  > I would like to ask about the general philosophy of setting up a python
>  > environment under guix.
>  >
>  > For other linux systems I have used pip to manage python packages and
>  > libraries.
>  >
>  > In guix is the idea that guix replaces pip for managing packages and
>  > libraries ?
>
> I think so, yes, ideally.
>
>  > So I would create a python-package for any missing python
>  > packages ?
>
> Yes.
>
>  > In the guix package lists, I can find many of the python
>  > libraries but there are a few missing that I would like, eg. astropy -
> an
>  > astronomy analysis library. So should I be creating a guix package for
> that
>  > rather than installing with pip ?
>
> That's the idea, yes: to have all Python packages available in Guix. That
> way, if your projects depend on Python packages and non-Python packages,
> you can specify all the requirements of your projects conveniently in a
> Guix manifest file (instead of having a requirements file for pip and
> another requirements file for some OS package manager, and possibly,
> another requirements file for another programmnig language used in a
> project).
>
> In practice, though, your projects could require many packages that are
> not yet in Guix. For example, astropy could depend on other packages that
> are not available, and those packages depend on other packages that are not
> available, and so on. To package them all would be a job in itself, and you
> would not have time to work on your project.  So, depending on your
> specific case, you might still need to use Python virtual environments and
> pip to get things done. For the latter you can use Python's venv, or
> poetry, or other tools that are already available in Guix for managing
> Python projects and environments.
>
> I recommend to watch the https://guix.gnu.org/videos/ for an introduction
> to packaging.
>
>
>
>
> ---
> https://sirgazil.bitbucket.io/
>
>
>
>

Re: Setting up a python environment

[Julien Lepiller]

Le 11 mai 2020 10:09:40 GMT-04:00, Roy Lemmon <roy@roylemmon.com> a écrit :
>Thanks sirgazil. That is very helpful.
>
>Yes, it might be that astropy has many dependencies.
>
>I will take a look at that video and also get a better handle on
>packaging.
>
>Cheers
>Roy.
>
>
>On Mon, 11 May 2020, 14:56 sirgazil, <sirgazil@zoho.com> wrote:
>
>>  ---- On Mon, 11 May 2020 07:34:09 -0500 Roy Lemmon
><roy@roylemmon.com>
>> wrote ----
>>  > Hi,
>>  >
>>  > I would like to ask about the general philosophy of setting up a
>python
>>  > environment under guix.
>>  >
>>  > For other linux systems I have used pip to manage python packages
>and
>>  > libraries.
>>  >
>>  > In guix is the idea that guix replaces pip for managing packages
>and
>>  > libraries ?
>>
>> I think so, yes, ideally.
>>
>>  > So I would create a python-package for any missing python
>>  > packages ?
>>
>> Yes.
>>
>>  > In the guix package lists, I can find many of the python
>>  > libraries but there are a few missing that I would like, eg.
>astropy -
>> an
>>  > astronomy analysis library. So should I be creating a guix package
>for
>> that
>>  > rather than installing with pip ?
>>
>> That's the idea, yes: to have all Python packages available in Guix.
>That
>> way, if your projects depend on Python packages and non-Python
>packages,
>> you can specify all the requirements of your projects conveniently in
>a
>> Guix manifest file (instead of having a requirements file for pip and
>> another requirements file for some OS package manager, and possibly,
>> another requirements file for another programmnig language used in a
>> project).
>>
>> In practice, though, your projects could require many packages that
>are
>> not yet in Guix. For example, astropy could depend on other packages
>that
>> are not available, and those packages depend on other packages that
>are not
>> available, and so on. To package them all would be a job in itself,
>and you
>> would not have time to work on your project.  So, depending on your
>> specific case, you might still need to use Python virtual
>environments and
>> pip to get things done. For the latter you can use Python's venv, or
>> poetry, or other tools that are already available in Guix for
>managing
>> Python projects and environments.
>>
>> I recommend to watch the https://guix.gnu.org/videos/ for an
>introduction
>> to packaging.
>>
>>
>>
>>
>> ---
>> https://sirgazil.bitbucket.io/
>>
>>
>>
>>

Guix also has a cool import feature that might help you package missing paekages. You thould try 'guix import pypi -r astropy'.

Re: Setting up a python environment

[Roy Lemmon]

Thanks. I will try that.

Cheers
Roy.

On Mon, 11 May 2020, 15:31 Julien Lepiller, <julien@lepiller.eu> wrote:

> Le 11 mai 2020 10:09:40 GMT-04:00, Roy Lemmon <roy@roylemmon.com> a écrit
> :
> >Thanks sirgazil. That is very helpful.
> >
> >Yes, it might be that astropy has many dependencies.
> >
> >I will take a look at that video and also get a better handle on
> >packaging.
> >
> >Cheers
> >Roy.
> >
> >
> >On Mon, 11 May 2020, 14:56 sirgazil, <sirgazil@zoho.com> wrote:
> >
> >>  ---- On Mon, 11 May 2020 07:34:09 -0500 Roy Lemmon
> ><roy@roylemmon.com>
> >> wrote ----
> >>  > Hi,
> >>  >
> >>  > I would like to ask about the general philosophy of setting up a
> >python
> >>  > environment under guix.
> >>  >
> >>  > For other linux systems I have used pip to manage python packages
> >and
> >>  > libraries.
> >>  >
> >>  > In guix is the idea that guix replaces pip for managing packages
> >and
> >>  > libraries ?
> >>
> >> I think so, yes, ideally.
> >>
> >>  > So I would create a python-package for any missing python
> >>  > packages ?
> >>
> >> Yes.
> >>
> >>  > In the guix package lists, I can find many of the python
> >>  > libraries but there are a few missing that I would like, eg.
> >astropy -
> >> an
> >>  > astronomy analysis library. So should I be creating a guix package
> >for
> >> that
> >>  > rather than installing with pip ?
> >>
> >> That's the idea, yes: to have all Python packages available in Guix.
> >That
> >> way, if your projects depend on Python packages and non-Python
> >packages,
> >> you can specify all the requirements of your projects conveniently in
> >a
> >> Guix manifest file (instead of having a requirements file for pip and
> >> another requirements file for some OS package manager, and possibly,
> >> another requirements file for another programmnig language used in a
> >> project).
> >>
> >> In practice, though, your projects could require many packages that
> >are
> >> not yet in Guix. For example, astropy could depend on other packages
> >that
> >> are not available, and those packages depend on other packages that
> >are not
> >> available, and so on. To package them all would be a job in itself,
> >and you
> >> would not have time to work on your project.  So, depending on your
> >> specific case, you might still need to use Python virtual
> >environments and
> >> pip to get things done. For the latter you can use Python's venv, or
> >> poetry, or other tools that are already available in Guix for
> >managing
> >> Python projects and environments.
> >>
> >> I recommend to watch the https://guix.gnu.org/videos/ for an
> >introduction
> >> to packaging.
> >>
> >>
> >>
> >>
> >> ---
> >> https://sirgazil.bitbucket.io/
> >>
> >>
> >>
> >>
>
> Guix also has a cool import feature that might help you package missing
> paekages. You thould try 'guix import pypi -r astropy'.
>

Re: Setting up a python environment

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

Hi Roy,

On Mon, 11 May 2020 at 14:49, Roy Lemmon <roy@roylemmon.com> wrote:

> I would like to ask about the general philosophy of setting up a python
> environment under guix.

Well, let me describe my workflow -- and I do not know if it the correct one.

I use 'manifest', 'profile' and "guix environment".  For each project
I am working on, I have dedicated folder containing the code, doc and
other files useful for the very project.  In this folder, I add a
manifest file [1] tracking the tools that the project needs.  From
this manifest, I instantiate a profile in the project folder.  And I
track the channel too to be able to reproduce elsewhere and elsetime.
Time to time, I spawn an environment with the manifest to do some
checks.  Well, basically, the CLI looks like:

   cd /path/to/project
   edit manifest.scm
   guix package -m manifest.scm -p profile
   guix describe -f channels > channels.scm
   git add manifest.scm channels.scm
   git commit -am

   eval $(guix package --search-paths=prefix -p profile)
   ipython

   guix environment -m manifest --ad-hoc python-other-tools

   guix time-machine -C channels.scm -- environment -m manifest.scm


Last, when a package is missing, I use "guix import pypi -r name".  If
the package is correctly packaged on the PyPI side, it is almost done.
;-)
To be concrete, again in the project folder:

   guix import pypi -r astropy > pkgs.scm
   edit pkgs.scm # add define-module etc.
   guix show -L . python-astropy
   edit pkgs.scm # add missing import
   guix show -L . python-astropy # works!

   guix build -L . python-astropy

Well, and then the "hard" packaging job starts. :-)
Find attached 'pkgs.scm' still failing to build but maybe a starting point.


Hope that helps,
simon

[-- Attachment #2: pkgs.scm --]
[-- Type: text/x-scheme, Size: 8475 bytes --]

(define-module (pkgs)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix build-system python)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (gnu packages python-xyz)
  #:use-module (gnu packages check)
  )

(define-public python-skyfield
  (package
    (name "python-skyfield")
    (version "1.20")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "skyfield" version))
        (sha256
          (base32
            "0lm4j6zdavvw0wgfv5g2rdk77n3dh2b096z44halfcjcf4k4dg2x"))))
    (build-system python-build-system)
    (home-page
      "http://github.com/brandon-rhodes/python-skyfield/")
    (synopsis "Elegant astronomy for Python")
    (description "Elegant astronomy for Python")
    (license license:expat)))

(define-public python-pytest-mpl
  (package
    (name "python-pytest-mpl")
    (version "0.11")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-mpl" version))
        (sha256
          (base32
            "1km202c1s5kcn52fx0266p06qb34va3warcby594dh6vixxa9i96"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-matplotlib" ,python-matplotlib)
        ("python-nose" ,python-nose)
        ("python-pillow" ,python-pillow)
        ("python-pytest" ,python-pytest)))
    (home-page
      "https://github.com/matplotlib/pytest-mpl")
    (synopsis
      "pytest plugin to help with testing figures output from Matplotlib")
    (description
      "pytest plugin to help with testing figures output from Matplotlib")
    (license license:bsd-3)))

(define-public python-pytest-filter-subpackage
  (package
    (name "python-pytest-filter-subpackage")
    (version "0.1.1")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-filter-subpackage" version))
        (sha256
          (base32
            "1s4s2kd31yc65rfvl4xhy8xx806xhy59kc7668h6b6wq88xgrn5p"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-pytest" ,python-pytest)))
    (native-inputs
      `(("python-pytest" ,python-pytest)
        ("python-pytest-cov" ,python-pytest-cov)
        ("python-pytest-doctestplus"
         ,python-pytest-doctestplus)))
    (home-page "https://astropy.org")
    (synopsis
      "Pytest plugin for filtering based on sub-packages")
    (description
      "Pytest plugin for filtering based on sub-packages")
    (license license:bsd-3)))

(define-public python-pytest-arraydiff
  (package
    (name "python-pytest-arraydiff")
    (version "0.3")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-arraydiff" version))
        (sha256
          (base32
            "05bcvhh2ycxa35znl8b3l9vkcmx7vwm5c3fpakbpw46c7vsn4bfy"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-numpy" ,python-numpy)
        ("python-pytest" ,python-pytest)
        ("python-six" ,python-six)))
    (home-page
      "https://github.com/astrofrog/pytest-arraydiff")
    (synopsis
      "pytest plugin to help with comparing array output from tests")
    (description
      "pytest plugin to help with comparing array output from tests")
    (license license:bsd-3)))

(define-public python-pytest-astropy-header
  (package
    (name "python-pytest-astropy-header")
    (version "0.1.2")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-astropy-header" version))
        (sha256
          (base32
            "1y87agr324p6x5gvhziymxjlw54pyn4gqnd49papbl941djpkp5g"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-pytest" ,python-pytest)))
    (native-inputs
      `(("python-astropy" ,python-astropy)
        ("python-codecov" ,python-codecov)
        ("python-coverage" ,python-coverage)
        ("python-numpy" ,python-numpy)
        ("python-pytest" ,python-pytest)
        ("python-pytest-cov" ,python-pytest-cov)))
    (home-page "http://astropy.org")
    (synopsis
      "pytest plugin to add diagnostic information to the header of the test output")
    (description
      "pytest plugin to add diagnostic information to the header of the test output")
    (license #f)))

(define-public python-pytest-openfiles
  (package
    (name "python-pytest-openfiles")
    (version "0.5.0")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-openfiles" version))
        (sha256
          (base32
            "0n0a7fdc9m86360y96l23fvdmd6rw04bl6h5xqgl9qxfv08jk70p"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-psutil" ,python-psutil)
        ("python-pytest" ,python-pytest)))
    (home-page
      "https://github.com/astropy/pytest-openfiles")
    (synopsis
      "Pytest plugin for detecting inadvertent open file handles")
    (description
      "Pytest plugin for detecting inadvertent open file handles")
    (license license:bsd-3)))

(define-public python-pytest-remotedata
  (package
    (name "python-pytest-remotedata")
    (version "0.3.2")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-remotedata" version))
        (sha256
          (base32
            "1h6g6shib6z07azf12rnsa053470ggbd7hy3bnbw8nf3nza5h372"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-pytest" ,python-pytest)
        ("python-six" ,python-six)))
    (home-page "https://astropy.org")
    (synopsis
      "Pytest plugin for controlling remote data access.")
    (description
      "Pytest plugin for controlling remote data access.")
    (license license:bsd-3)))

(define-public python-pytest-doctestplus
  (package
    (name "python-pytest-doctestplus")
    (version "0.6.1")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-doctestplus" version))
        (sha256
          (base32
            "11f1gjj975aapannwc7xfqc6ijckgqn9w2pbk0n4jn8qljv77jvv"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-pip" ,python-pip)
        ("python-pytest" ,python-pytest)))
    (home-page "https://astropy.org")
    (synopsis
      "Pytest plugin with advanced doctest features.")
    (description
      "Pytest plugin with advanced doctest features.")
    (license license:bsd-3)))

(define-public python-pytest-astropy
  (package
    (name "python-pytest-astropy")
    (version "0.8.0")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "pytest-astropy" version))
        (sha256
          (base32
            "18j6z6y2fvykmcs5z0mldhhaxxn6wzpnhlm2ps7m8r5z5kmh1631"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-hypothesis" ,python-hypothesis)
        ("python-pytest" ,python-pytest)
        ("python-pytest-arraydiff"
         ,python-pytest-arraydiff)
        ("python-pytest-astropy-header"
         ,python-pytest-astropy-header)
        ("python-pytest-cov" ,python-pytest-cov)
        ("python-pytest-doctestplus"
         ,python-pytest-doctestplus)
        ("python-pytest-filter-subpackage"
         ,python-pytest-filter-subpackage)
        ("python-pytest-openfiles"
         ,python-pytest-openfiles)
        ("python-pytest-remotedata"
         ,python-pytest-remotedata)))
    (home-page
      "https://github.com/astropy/pytest-astropy")
    (synopsis
      "Meta-package containing dependencies for testing")
    (description
      "Meta-package containing dependencies for testing")
    (license license:bsd-3)))

(define-public python-astropy
  (package
    (name "python-astropy")
    (version "4.0.1.post1")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "astropy" version))
        (sha256
          (base32
            "1da4xj793ldck29aajyb514wpz330cml26f3gdp45jj531n4lc2w"))))
    (build-system python-build-system)
    (propagated-inputs
      `(("python-numpy" ,python-numpy)))

    (native-inputs
     `(
       ("python-coverage" ,python-coverage)
          ("python-ipython" ,python-ipython)
          ("python-objgraph" ,python-objgraph)
     ;;     ("python-pytest-astropy" ,python-pytest-astropy)
          ("python-pytest-mpl" ,python-pytest-mpl)
          ("python-pytest-xdist" ,python-pytest-xdist)
          ("python-skyfield" ,python-skyfield)
     ))
    (home-page "http://astropy.org")
    (synopsis
      "Community-developed python astronomy tools")
    (description
      "Community-developed python astronomy tools")
    (license #f)))

Re: Setting up a python environment

[Roy Lemmon]

Hi Simon,

That is very helpful indeed thanks. I will work through that.

Cheers
Roy.

On Mon, 11 May 2020, 15:51 zimoun, <zimon.toutoune@gmail.com> wrote:

> Hi Roy,
>
> On Mon, 11 May 2020 at 14:49, Roy Lemmon <roy@roylemmon.com> wrote:
>
> > I would like to ask about the general philosophy of setting up a python
> > environment under guix.
>
> Well, let me describe my workflow -- and I do not know if it the correct
> one.
>
> I use 'manifest', 'profile' and "guix environment".  For each project
> I am working on, I have dedicated folder containing the code, doc and
> other files useful for the very project.  In this folder, I add a
> manifest file [1] tracking the tools that the project needs.  From
> this manifest, I instantiate a profile in the project folder.  And I
> track the channel too to be able to reproduce elsewhere and elsetime.
> Time to time, I spawn an environment with the manifest to do some
> checks.  Well, basically, the CLI looks like:
>
>    cd /path/to/project
>    edit manifest.scm
>    guix package -m manifest.scm -p profile
>    guix describe -f channels > channels.scm
>    git add manifest.scm channels.scm
>    git commit -am
>
>    eval $(guix package --search-paths=prefix -p profile)
>    ipython
>
>    guix environment -m manifest --ad-hoc python-other-tools
>
>    guix time-machine -C channels.scm -- environment -m manifest.scm
>
>
> Last, when a package is missing, I use "guix import pypi -r name".  If
> the package is correctly packaged on the PyPI side, it is almost done.
> ;-)
> To be concrete, again in the project folder:
>
>    guix import pypi -r astropy > pkgs.scm
>    edit pkgs.scm # add define-module etc.
>    guix show -L . python-astropy
>    edit pkgs.scm # add missing import
>    guix show -L . python-astropy # works!
>
>    guix build -L . python-astropy
>
> Well, and then the "hard" packaging job starts. :-)
> Find attached 'pkgs.scm' still failing to build but maybe a starting point.
>
>
> Hope that helps,
> simon
>