bug#47746: pypi importer generates wrong pypi-uri without underscore

bug#47746: pypi importer generates wrong pypi-uri without underscore

[Ricardo Wurmus]

When importing a package from Pypi whose name includes an 
underscore, the importer will generate a pypi-uri expression where 
the underscore has been replaced with a dash.

-- 
Ricardo

bug#47746: pypi importer generates wrong pypi-uri without underscore

[Xinglu Chen]

On Tue, Apr 13 2021, Ricardo Wurmus wrote:

> When importing a package from Pypi whose name includes an 
> underscore, the importer will generate a pypi-uri expression where 
> the underscore has been replaced with a dash.

Hmm, I just grabbed a random package from PyPi, and the URI seems fine
to me.

--8<---------------cut here---------------start------------->8---
~/src/guix $ ./pre-inst-env guix import pypi angreal_data_science_project
Starting download of /tmp/guix-file.4MH43L
From
https://files.pythonhosted.org/packages/28/5a/6c77f8f5c2333ab2d3e83c27615358a0badcfeb6513a36079ba0fee78524/angreal_data_science_project-0.5.5.tar.gz...
 …_project-0.5.5.tar.gz  15KiB        378KiB/s 00:00 [##################] 100.0%

Starting download of /tmp/guix-file.hrC4cL
From
https://files.pythonhosted.org/packages/f4/18/e4747e2f3232ca1fbbd1286455fe57f3390a94b14c26a37dba72eb829523/angreal_data_science_project-0.5.5-py3-none-any.whl...
 …ect-0.5.5-py3-none-any.whl  28KiB   244KiB/s 00:00 [##################] 100.0%
(package
  (name "python-angreal-data-science-project")
  (version "0.5.5")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "angreal_data_science_project" version))
      (sha256
        (base32
          "1krvhwmlmv77j8w3sxaxlkmhd0xzmr24f4n873p00631lacwmpck"))))
  (build-system python-build-system)
  (home-page
    "https://gitlab.com/angreal/data-science-project")
  (synopsis
    "An angreal template for data science projects.")
  (description
    "An angreal template for data science projects.")
  (license #f))
--8<---------------cut here---------------end--------------->8---

How would I reproduce the bug?

bug#47746: pypi importer generates wrong pypi-uri without underscore

[Ricardo Wurmus]

Xinglu Chen <public@yoctocell.xyz> writes:

> On Tue, Apr 13 2021, Ricardo Wurmus wrote:
>
>> When importing a package from Pypi whose name includes an 
>> underscore, the importer will generate a pypi-uri expression 
>> where 
>> the underscore has been replaced with a dash.
>
> Hmm, I just grabbed a random package from PyPi, and the URI 
> seems fine
> to me. […]
> How would I reproduce the bug?

You’re right, this only affects some packages.  Here are two 
examples:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi -r jupyterlab_widgets
following redirection to 
`https://pypi.org/pypi/jupyterlab-widgets/json'...
[…] (uri (pypi-uri "jupyterlab-widgets" version)) […]
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi -r jupyter_packaging
following redirection to 
`https://pypi.org/pypi/jupyter-packaging/json'...
[…] (uri (pypi-uri "jupyter-packaging" version)) […]
--8<---------------cut here---------------end--------------->8---

They both have a redirection in common, but the pypi-uri is 
incorrect as it needs to have the underscore.

-- 
Ricardo

bug#47746: pypi importer generates wrong pypi-uri without underscore

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Tue, Apr 13 2021, Ricardo Wurmus wrote:

> You’re right, this only affects some packages.  Here are two 
> examples:
>
> [...]
>
> They both have a redirection in common, but the pypi-uri is 
> incorrect as it needs to have the underscore.

Looking at the JSON object at
<https://pypi.org/pypi/jupyterlab-widgets/json>, it specifies
“jupyterlabs-widgets” as ‘info.downloads.name’, which is what the PyPi
importer reads to determine the PyPi URI.

Would it make sense to just make it read the name that the user
specifies in the shell, i.e. ‘./pre-inst-env guix import pypi
PACKAGE-NAME’ (see the patch below)?  Or will this potentially mess up
other PyPi URIs?


[-- Attachment #2: fix-pypi-uri --]
[-- Type: text/x-patch, Size: 1918 bytes --]

From ec5612e9d4c2026541874c6a3eaf7c9009ce4a31 Mon Sep 17 00:00:00 2001
Message-Id: <ec5612e9d4c2026541874c6a3eaf7c9009ce4a31.1618404153.git.public@yoctocell.xyz>
From: Xinglu Chen <public@yoctocell.xyz>
Date: Wed, 14 Apr 2021 14:37:01 +0200
Subject: [PATCH] import: pypi: Use package name from the shell to construct
 the URI.

Some packages will list an incorrect package name in the JSON object,
resulting in PyPi URIs that lead to nowhere.

* guix/import/pypi.scm (pypi->guix-package): Use the package name specified in
the shell (guix import pypi PACKAGE-NAME) to construct the PyPi URI.

Fixes <https://issues.guix.gnu.org/47746>.
Reported by Ricardo Wurmus <rekado@elephly.net>.
---
 guix/import/pypi.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index bf4dc50138..be9470ad26 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2020 Lars-Dominik Braun <ldb@leibniz-psychology.org>
 ;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -483,7 +484,7 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                          (leave (G_ "no source release for pypi package ~a ~a~%")
                                 (project-info-name info)
                                 (project-info-version info)))))
-              (make-pypi-sexp (project-info-name info)
+              (make-pypi-sexp package-name
                               (project-info-version info)
                               (and=> (latest-source-release project)
                                      distribution-url)

base-commit: 82543e9649da2da9a5285ede4ec4f718fd740fcb
-- 
2.31.1

bug#47746: pypi importer generates wrong pypi-uri without underscore

[Maxim Cournoyer]

Hi,

Xinglu Chen <public@yoctocell.xyz> writes:

> On Tue, Apr 13 2021, Ricardo Wurmus wrote:
>
>> You’re right, this only affects some packages.  Here are two 
>> examples:
>>
>> [...]
>>
>> They both have a redirection in common, but the pypi-uri is 
>> incorrect as it needs to have the underscore.
>
> Looking at the JSON object at
> <https://pypi.org/pypi/jupyterlab-widgets/json>, it specifies
> “jupyterlabs-widgets” as ‘info.downloads.name’, which is what the PyPi
> importer reads to determine the PyPi URI.
>
> Would it make sense to just make it read the name that the user
> specifies in the shell, i.e. ‘./pre-inst-env guix import pypi
> PACKAGE-NAME’ (see the patch below)?  Or will this potentially mess up
> other PyPi URIs?
>
>>From ec5612e9d4c2026541874c6a3eaf7c9009ce4a31 Mon Sep 17 00:00:00 2001
> Message-Id: <ec5612e9d4c2026541874c6a3eaf7c9009ce4a31.1618404153.git.public@yoctocell.xyz>
> From: Xinglu Chen <public@yoctocell.xyz>
> Date: Wed, 14 Apr 2021 14:37:01 +0200
> Subject: [PATCH] import: pypi: Use package name from the shell to construct
>  the URI.
>
> Some packages will list an incorrect package name in the JSON object,
> resulting in PyPi URIs that lead to nowhere.

In my opinion, if a package contains wrong values in their metadata,
it's a problem that should be reported to that package.  As with other
things, garbage in -> garbage out; Guix cannot do much here.  I seem to
recall that some older PyPI mirror had that problem.  It may be that
their metadata name -> URI is not consistent.  We could try to find what
that 2nd scheme is and attempt both, although that's not very elegant
and it probably points to packages being very outdated to start with.

Thanks,

Maxim