usage of pypi-uri in the pypi importer

usage of pypi-uri in the pypi importer

[Malte Frank Gerdes]

Hi,

i noticed that the python pypi importer generates packages definitions
which rely on the pypi-uri function declared in the python build
system. I observed that the python build system itself doesn't use this
function at all, so why is it defined there and not in the python
importer, which seems to be the only piece of software using it?

Also: I tried to package pyinstrument-cext and noticed that the importer
generated function isn't able to find the source package - only
404's. The importer uses PyPI's json API to get information about a
given package (this includes it's url - called source-url), so is the
pypi-uri function needed at all? the necessary information is available
even without it. Or are there any specific reasons why things are as
they are?


Malte

Re: usage of pypi-uri in the pypi importer

[zimoun]

Dear,

On Sat, 17 Oct 2020 at 21:17, Malte Frank Gerdes <malte.f.gerdes@gmail.com> wrote:

> i noticed that the python pypi importer generates packages definitions
> which rely on the pypi-uri function declared in the python build
> system. I observed that the python build system itself doesn't use this
> function at all, so why is it defined there and not in the python
> importer, which seems to be the only piece of software using it?

Because, “pypi-uri” is used by *a lot* of Python packages – packages
using ’python-build-system’.  For example,
gnu/packages/python-science.scm:

--8<---------------cut here---------------start------------->8---
[..]
  #:use-module (guix build-system python))

(define-public python-scipy
  (package
    (name "python-scipy")
    (version "1.3.2")
    (source
     (origin
       (method url-fetch)
       (uri (pypi-uri "scipy" version))
       (sha256
        (base32 "192d8dsybvhv19igkrsafbdafx198nz7pibkjgrqjhlr66s3jfd0"))))
    (build-system python-build-system)
[..]    
--8<---------------cut here---------------end--------------->8---

In what you are proposing, if I read correctly, the
gnu/packages/python-science.scm should have to also import ’(guix import
pypi)’.



> Also: I tried to package pyinstrument-cext and noticed that the importer
> generated function isn't able to find the source package - only
> 404's. The importer uses PyPI's json API to get information about a
> given package (this includes it's url - called source-url), so is the
> pypi-uri function needed at all? the necessary information is available
> even without it. Or are there any specific reasons why things are as
> they are?

I am not sure to understand.  Using Guix 353bdae, I get:

--8<---------------cut here---------------start------------->8---
$ guix import pypi pyinstrument-cext

Starting download of /tmp/guix-file.5tmgen
From https://files.pythonhosted.org/packages/b6/95/0a391322dd7f2f97f553373659a787b7428dfe15fc2a8e01844c8768f45e/pyinstrument_cext-0.2.2.tar.gz...
 …xt-0.2.2.tar.gz  5KiB               704KiB/s 00:00 [##################] 100.0%

Starting download of /tmp/guix-file.cZDsRo
From https://files.pythonhosted.org/packages/13/2e/d548d3727aa3c5b77eb79f667e7212de1acf93087ddce5274887abbe217d/pyinstrument_cext-0.2.2-cp27-cp27m-macosx_10_6_intel.whl...
 …-cp27m-macosx_10_6_intel.whl  7KiB  679KiB/s 00:00 [##################] 100.0%
(package
  (name "python-pyinstrument-cext")
  (version "0.2.2")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "pyinstrument-cext" version))
      (sha256
        (base32
          "0nycf7nhx2xzba49m8208agx5xghzxkma3iim5f43h3l3pvjb7pj"))))
  (build-system python-build-system)
  (home-page
    "https://github.com/joerick/pyinstrument_cext")
  (synopsis
    "A CPython extension supporting pyinstrument")
  (description
    "A CPython extension supporting pyinstrument")
  (license #f))
--8<---------------cut here---------------end--------------->8---


BTW, ’pypi-uri’ eases the management of all the packages coming from
PyPI.  If the URL of PyPI change or is mirrored, it is easy to switch or
add the mirror; otherwise one would have to go package by package.
Well, if I understand correctly what you mean.


All the best,
simon

Re: usage of pypi-uri in the pypi importer

[Malte Frank Gerdes]

TL;DR: Now that my initial problem is solved anyway. No change is needed
at all.

zimoun <zimon.toutoune@gmail.com> writes:

> Dear,
>
> On Sat, 17 Oct 2020 at 21:17, Malte Frank Gerdes <malte.f.gerdes@gmail.com> wrote:
>
>> i noticed that the python pypi importer generates packages definitions
>> which rely on the pypi-uri function declared in the python build
>> system. I observed that the python build system itself doesn't use this
>> function at all, so why is it defined there and not in the python
>> importer, which seems to be the only piece of software using it?
>
> Because, “pypi-uri” is used by *a lot* of Python packages – packages
> using ’python-build-system’.  For example,
> gnu/packages/python-science.scm:
>
> [..]
>   #:use-module (guix build-system python))
>
> (define-public python-scipy
>   (package
>     (name "python-scipy")
>     (version "1.3.2")
>     (source
>      (origin
>        (method url-fetch)
>        (uri (pypi-uri "scipy" version))
>        (sha256
>         (base32 "192d8dsybvhv19igkrsafbdafx198nz7pibkjgrqjhlr66s3jfd0"))))
>     (build-system python-build-system)
> [..]    
>
>
> In what you are proposing, if I read correctly, the
> gnu/packages/python-science.scm should have to also import ’(guix import
> pypi)’.

Yes, that was exactly what i meant. The thing is: Every time i tried to
build a python package that has it's uri specified vie `pypi-uri' i see
a redirect to the actual place where the package is located. I don't
know if this is to shorten the URLs or if it's a temperary solution. In
the former case there is no problem and nothing needs to be done, in the
latter case it might make sense to not use it. I'm not sure about this -
the particular problem that lead me to this was: (please read on)

>> Also: I tried to package pyinstrument-cext and noticed that the importer
>> generated function isn't able to find the source package - only
>> 404's. The importer uses PyPI's json API to get information about a
>> given package (this includes it's url - called source-url), so is the
>> pypi-uri function needed at all? the necessary information is available
>> even without it. Or are there any specific reasons why things are as
>> they are?
>
> I am not sure to understand.  Using Guix 353bdae, I get:
>
> $ guix import pypi pyinstrument-cext
>
> Starting download of /tmp/guix-file.5tmgen
> From https://files.pythonhosted.org/packages/b6/95/0a391322dd7f2f97f553373659a787b7428dfe15fc2a8e01844c8768f45e/pyinstrument_cext-0.2.2.tar.gz...
>  …xt-0.2.2.tar.gz  5KiB               704KiB/s 00:00 [##################] 100.0%
>
> Starting download of /tmp/guix-file.cZDsRo
> From https://files.pythonhosted.org/packages/13/2e/d548d3727aa3c5b77eb79f667e7212de1acf93087ddce5274887abbe217d/pyinstrument_cext-0.2.2-cp27-cp27m-macosx_10_6_intel.whl...
>  …-cp27m-macosx_10_6_intel.whl  7KiB  679KiB/s 00:00 [##################] 100.0%
> (package
>   (name "python-pyinstrument-cext")
>   (version "0.2.2")
>   (source
>     (origin
>       (method url-fetch)
>       (uri (pypi-uri "pyinstrument-cext" version))
>       (sha256
>         (base32
>           "0nycf7nhx2xzba49m8208agx5xghzxkma3iim5f43h3l3pvjb7pj"))))
>   (build-system python-build-system)
>   (home-page
>     "https://github.com/joerick/pyinstrument_cext")
>   (synopsis
>     "A CPython extension supporting pyinstrument")
>   (description
>     "A CPython extension supporting pyinstrument")
>   (license #f))

The problem here is: if you take that generated package definition as-is -
and add it to guix - then it will fail to download the sources because
the actual package name is `pyinstrument_cext' (notice the
underscore!). Efraim suggested to me tha ti should change that. This
works. 

> BTW, ’pypi-uri’ eases the management of all the packages coming from
> PyPI.  If the URL of PyPI change or is mirrored, it is easy to switch or
> add the mirror; otherwise one would have to go package by package.
> Well, if I understand correctly what you mean.

I see that. I was just wondering why this function exists, because the
importer itself knows the correct link anyway (it's just ugly because
it's long, the link i mean :D). But i might be misunderstanding how PyPI
works here.


Malte

Re: usage of pypi-uri in the pypi importer

[zimoun]

Dear,

On Tue, 27 Oct 2020 at 18:04, Malte Frank Gerdes <malte.f.gerdes@gmail.com> wrote:

>> (package
>>   (name "python-pyinstrument-cext")
>>   (version "0.2.2")
>>   (source
>>     (origin
>>       (method url-fetch)
>>       (uri (pypi-uri "pyinstrument-cext" version))
>>       (sha256
>>         (base32
>>           "0nycf7nhx2xzba49m8208agx5xghzxkma3iim5f43h3l3pvjb7pj"))))
>>   (build-system python-build-system)
>>   (home-page
>>     "https://github.com/joerick/pyinstrument_cext")
>>   (synopsis
>>     "A CPython extension supporting pyinstrument")
>>   (description
>>     "A CPython extension supporting pyinstrument")
>>   (license #f))
>
> The problem here is: if you take that generated package definition as-is -
> and add it to guix - then it will fail to download the sources because
> the actual package name is `pyinstrument_cext' (notice the
> underscore!). Efraim suggested to me tha ti should change that. This
> works. 

Yes.  It happens all the time.  I do not see what is the problem.  You
just have to do:

       (uri (pypi-uri "pyinstrument_cext" version))


For example, the upstream names of the CRAN or Bioconductor packages are
a big mess and do not fit –at all– the names in Guix.  That’s fine.

Let take an example:

    (name "r-matrix-utils")
[..]    
       (method url-fetch)
       (uri (cran-uri "Matrix.utils" version))
[..]
    (properties `((upstream-name . "Matrix.utils")))

And note the extra “upstream-name” property eases stuff, e.g., comment
in the ’r-build-system’:

         ;; R package names are case-sensitive and cannot be derived from the
         ;; Guix package name.  The exact package name is required as an
         ;; argument to ‘tools::testInstalledPackage’, which runs the tests
         ;; for a package given its name and the path to the “library” (a
         ;; location for a collection of R packages) containing it.


>> BTW, ’pypi-uri’ eases the management of all the packages coming from
>> PyPI.  If the URL of PyPI change or is mirrored, it is easy to switch or
>> add the mirror; otherwise one would have to go package by package.
>> Well, if I understand correctly what you mean.
>
> I see that. I was just wondering why this function exists, because the
> importer itself knows the correct link anyway (it's just ugly because
> it's long, the link i mean :D). But i might be misunderstanding how PyPI
> works here.

It is not only about PyPI and pypy-uri.  And maybe it is not the good
example.  Instead give a look at ’cran-uri’ defined in
’guix/build-system/r.scm’.  It returns the list:

   mirror://cran/src/contrib/<name>_<version>.tar.gz
   mirror://cran/src/contrib/Archive/<name>_<version>.tar.gz

and then ’%mirrors’ in ’guix/download.scm’ provides how the mirrors are
expanded:

      (cran
       ;; Arbitrary mirrors from http://cran.r-project.org/mirrors.html
       ;; This one automatically redirects to servers worldwide
       "http://cran.r-project.org/"
       "http://cran.rstudio.com/"
       "http://cran.univ-lyon1.fr/"
       "http://cran.ism.ac.jp/"
       "http://cran.stat.auckland.ac.nz/"
       "http://cran.mirror.ac.za/"
       "http://cran.csie.ntu.edu.tw/")

using ’maybe-expand-mirrors’ in ’guix/build/download.scm’.  Therefore,
if the R project decides to change their API to:

  source/contribution/ARCHIVE/<version>-<name>.tar.bz2

then it is only one change about ’cran-uri’ and not replace all the
URLs of all packages in ’gnu/packages/cran.scm’.  And idem if mirrors
are added or removed.

The same logic is applied to PyPI even if the upstream source is only
one URL with no mirror. :-)


Hope that helps,
simon