bug#44030: guix import pypi foo@1.2.3 breaks

bug#44030: guix import pypi foo@1.2.3 breaks

[zimoun]

Dear,

The syntax across the different importers are not uniform.  Especially,
compare:

   guix import hackage mtl@2.1.3.1
   guix import pypi itsdangerous@1.1.0

and worse, the option ’--recursive’ leads to an error for the latter.
Note that instead:

   guix import pypi itsdangerous/1.1.0

perfectly works, even the option ’--recursive’.

All the information is there, and the UI has to be fixed.  It is a
perfect first contribution fixes.


All the best,
simon

PS:
Reported by Zelphir Kaltstahl.
<https://lists.gnu.org/archive/html/help-guix/2020-10/msg00055.html>

bug#44030: [PATCH] guix: import: Add versioning syntax to pypi importer.

[Lulu]

Use @ as the unified versioning syntax, as per crate and
hackage importers, plus minor spacing fixes.


diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..add2a7ed7a 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,13 +118,15 @@
 
 (define (pypi-fetch name)
   "Return a <pypi-project> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
-         json->pypi-project))
+  ;; Convert @ in package name to / to access the correct URL.
+  (let ((versioned-name (string-join (string-split name #\@) "/")))
+    (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
+           json->pypi-project)))
 
 ;; For packages found on PyPI that lack a source distribution.
 (define-condition-type &missing-source-error &error
   missing-source-error?
-  (package  missing-source-error-package))
+  (package missing-source-error-package))
 
 (define (latest-source-release pypi-package)
   "Return the latest source release for PYPI-PACKAGE."
@@ -371,7 +373,7 @@ be extracted in a temporary directory."
                  (invoke "tar" "xf" archive "-C" dir)))
            (let ((requires.txt-files
                   (find-files dir (lambda (abs-file-name _)
-		                    (string-match "\\.egg-info/requires.txt$"
+                                    (string-match "\\.egg-info/requires.txt$"
                                                   abs-file-name)))))
              (match requires.txt-files
                (()

bug#44030: [PATCH] guix: import: Add versioning syntax to pypi importer.

[Lulu]

Ah, heck, I used an old revision of the file when I generated the diff.
Sorry about that. Here's the properly working patch:


diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..1ec1ecbfa1 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,13 +118,15 @@
 
 (define (pypi-fetch name)
   "Return a <pypi-project> record for package NAME, or #f on failure."
-  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
-         json->pypi-project))
+  ;; Convert @ in package name to / to access the correct URL.
+  (let ((versioned-name (string-join (string-split name #\@) "/")))
+    (and=> (json-fetch (string-append "https://pypi.org/pypi/" versioned-name "/json"))
+           json->pypi-project)))
 
 ;; For packages found on PyPI that lack a source distribution.
 (define-condition-type &missing-source-error &error
   missing-source-error?
-  (package  missing-source-error-package))
+  (package missing-source-error-package))
 
 (define (latest-source-release pypi-package)
   "Return the latest source release for PYPI-PACKAGE."
@@ -371,7 +373,7 @@ be extracted in a temporary directory."
                  (invoke "tar" "xf" archive "-C" dir)))
            (let ((requires.txt-files
                   (find-files dir (lambda (abs-file-name _)
-		                    (string-match "\\.egg-info/requires.txt$"
+                                    (string-match "\\.egg-info/requires.txt$"
                                                   abs-file-name)))))
              (match requires.txt-files
                (()

bug#44030: guix import pypi foo@1.2.3 breaks

[Lulu]

I just sent in a patch to fix this for the pypi importer, although
ideally we'd want versioning support for all importers with a uniform
syntax.

I poked and prodded at the RubyGems API to see if it can do what PyPI
can. It's unfortunately much more limited: API v1 provides a method
for querying all versions of a gem [1] and v2 provides a method for
querying specific versions of a gem [2], although neither of them
provides information about dependency versions, so we can't rely on it
for recursive imports (since it would try to import latest versions of
all dependencies). There's a method that fetches dependency and
versioning info for all versions of all given gems but (probably due
to the verbosity of JSON/YAML) it's in binary Ruby serialisation
format [3]. I can try to hack a parser that surgically extracts
dependency info from a given gem version. What do you think?

ELPA runs into a similar problem in that it provides tarballs/files
for older versions but dependency info is only provided in the repo
file. (MELPA tries to directly peg all packages to their respective
Git repos' trunks.)

I presume it's much simpler with the gnu importer, as it's only a
matter of pointing the FTP fetch in the right direction, although I
couldn't confirm it, as the gnu importer doesn't work for me since my
ISP blocks PGP keyserver ports.

I need to take a closer look at CRAN, CPAN, TeXLive and opam.

--
Lulu

[1]: https://guides.rubygems.org/rubygems-org-api/#gem-version-methods
[2]: https://guides.rubygems.org/rubygems-org-api-v2/
[3]: https://guides.rubygems.org/rubygems-org-api/#misc-methods

bug#44030: [PATCH] guix: import: Add versioning syntax to pypi importer.

[zimoun]

Hi,

Thank you for working on this.

On Sun, 25 Oct 2020 at 23:04, Lulu <me@erkin.party> wrote:
> Ah, heck, I used an old revision of the file when I generated the diff.
> Sorry about that. Here's the properly working patch:

That’s fine.  Usually to ease the reading, it is a good habit to
increase the version of the patch, (see the option ’reroll-count’ of
git-format-patch). 


However, the 2 patches are not proper commit, right?  You can locally
commit your changes and send them.  Basically, it means the same thing
but including the commit message.  Here it looks like:

--8<---------------cut here---------------start------------->8---
import: pypi: Allow ’@’ for specifying the version.

Fixes <https://bugs.gnu.org/44030>.

* guix/import/pypi.json (pypi-fetch): Allow ’@’ for specifying the
  version.
--8<---------------cut here---------------end--------------->8---

Well, I let you find more inspiration in previous commit messages. ;-)


> diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
> index 15116e349d..1ec1ecbfa1 100644
> --- a/guix/import/pypi.scm
> +++ b/guix/import/pypi.scm
> @@ -118,13 +118,15 @@
>  
>  (define (pypi-fetch name)
>    "Return a <pypi-project> record for package NAME, or #f on failure."
> -  (and=> (json-fetch (string-append "https://pypi.org/pypi/" name "/json"))
> -         json->pypi-project))
> +  ;; Convert @ in package name to / to access the correct URL.
> +  (let ((versioned-name (string-join (string-split name #\@) "/")))
> +    (and=> (json-fetch (string-append "https://pypi.org/pypi/" versioned-name "/json"))
> +           json->pypi-project)))

If you feel adventurous, you could try to add a test in the file
“tests/pypi.scm”. :-)

>  ;; For packages found on PyPI that lack a source distribution.
>  (define-condition-type &missing-source-error &error
>    missing-source-error?
> -  (package  missing-source-error-package))
> +  (package missing-source-error-package))

Why is this line modified?


>  (define (latest-source-release pypi-package)
>    "Return the latest source release for PYPI-PACKAGE."
> @@ -371,7 +373,7 @@ be extracted in a temporary directory."
>                   (invoke "tar" "xf" archive "-C" dir)))
>             (let ((requires.txt-files
>                    (find-files dir (lambda (abs-file-name _)
> -		                    (string-match "\\.egg-info/requires.txt$"
> +                                    (string-match "\\.egg-info/requires.txt$"

Idem.


All the best,
simon

bug#44030: guix import pypi foo@1.2.3 breaks

[zimoun]

Hi,

On Sun, 25 Oct 2020 at 23:50, Lulu <me@erkin.party> wrote:
> I just sent in a patch to fix this for the pypi importer, although
> ideally we'd want versioning support for all importers with a uniform
> syntax.

The uniform syntax is ’@’.  However, it is importer by importer (case by
case) since the external API is different.  AFAIU.


Speaking about “guix import pypi”, the “--recursive” options still
ungracefully fails; even with you patch.


> I poked and prodded at the RubyGems API to see if it can do what PyPI
> can. It's unfortunately much more limited: API v1 provides a method
> for querying all versions of a gem [1] and v2 provides a method for
> querying specific versions of a gem [2], although neither of them
> provides information about dependency versions, so we can't rely on it
> for recursive imports (since it would try to import latest versions of
> all dependencies). There's a method that fetches dependency and
> versioning info for all versions of all given gems but (probably due
> to the verbosity of JSON/YAML) it's in binary Ruby serialisation
> format [3]. I can try to hack a parser that surgically extracts
> dependency info from a given gem version. What do you think?
>
> ELPA runs into a similar problem in that it provides tarballs/files
> for older versions but dependency info is only provided in the repo
> file. (MELPA tries to directly peg all packages to their respective
> Git repos' trunks.)
>
> I presume it's much simpler with the gnu importer, as it's only a
> matter of pointing the FTP fetch in the right direction, although I
> couldn't confirm it, as the gnu importer doesn't work for me since my
> ISP blocks PGP keyserver ports.
>
> I need to take a closer look at CRAN, CPAN, TeXLive and opam.

Hum?  You are discussing this bug [1], right?  So please could you
discuss overthere.

1: <http://issues.guix.gnu.org/issue/44115>


All the best,
simon

bug#44030: [PATCH] import: pypi: Add '@' syntax for specifying the package version.

[Lulu]

* guix/import/pypi.scm (pypi-fetch): Add ’@’ syntax for specifying the package version.
---
  guix/import/pypi.scm | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 15116e349d..559be4a98b 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -118,8 +118,10 @@
  
  (define (pypi-fetch name)
     "Return a <pypi-project>  record for package NAME, or #f on failure."
-   (and=>  (json-fetch (string-append "https://pypi.org/pypi/"  name "/json"))
-             json->pypi-project))
+   ;; Convert @ in package name to / to access the correct URL.
+   (let ((versioned-name (string-join (string-split name #\@) "/")))
+      (and=>  (json-fetch (string-append "https://pypi.org/pypi/"  versioned-name "/json"))
+                json->pypi-project)))
  
  ;; For packages found on PyPI that lack a source distribution.
  (define-condition-type &missing-source-error &error
-- 
2.29.1

bug#44030: guix import pypi foo@1.2.3 breaks

[Lulu]

Okay, I tested this a bit and it seems to work fine. I'm going to write formal
tests for it (and other importers, if possible) later.

--
Lulu

bug#44030: guix import pypi foo@1.2.3 breaks

[zimoun]

Hi,

On Fri, 30 Oct 2020 at 01:09, Lulu <me@erkin.party> wrote:
> Okay, I tested this a bit and it seems to work fine. I'm going to write formal
> tests for it (and other importers, if possible) later.

Usually, we try to use the option ’--reroll-count’ of git-format-patch
for the second (or third or more) version.  It helps the review to know
– from the subject – which is the email to read, etc.  Well, next
time. :-)


What do you mean by write “formal tests”?


All the best,
simon

bug#44030: guix import pypi foo@1.2.3 breaks

[Maxim Cournoyer]

Hi,

zimoun <zimon.toutoune@gmail.com> writes:

> Dear,
>
> The syntax across the different importers are not uniform.  Especially,
> compare:
>
>    guix import hackage mtl@2.1.3.1
>    guix import pypi itsdangerous@1.1.0
>
> and worse, the option ’--recursive’ leads to an error for the latter.
> Note that instead:
>
>    guix import pypi itsdangerous/1.1.0
>
> perfectly works, even the option ’--recursive’.
>
> All the information is there, and the UI has to be fixed.  It is a
> perfect first contribution fixes.
>
>
> All the best,
> simon

This works now:

$ ./pre-inst-env guix import pypi ansible@4.4.0
Starting download of /tmp/guix-file.mtcWzL
From https://files.pythonhosted.org/packages/b5/01/dd6bf3cb7d834c7493d10fa1f0720c34c7703fc9bf12c93f294def465bb0/ansible-4.4.0.tar.gz...
 …4.0.tar.gz  33.7MiB                49.2MiB/s 00:01 [##################] 100.0%
(package
  (name "python-ansible")
  (version "4.4.0")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "ansible" version))
      (sha256
        (base32 "031n22j0lsmh69x6i6gkva81j68b4yzh1pbg3q2h4bknl85q46ag"))))
  (build-system python-build-system)
  (propagated-inputs (list python-ansible-core))
  (home-page "https://ansible.com/")
  (synopsis "Radically simple IT automation")
  (description "Radically simple IT automation")
  (license #f))

$ ./pre-inst-env guix import pypi ansible@4.4.1
guix import: error: no source release for pypi package ansible 4.4.1

Closing!

Maxim

bug#44030: guix import pypi foo@1.2.3 breaks

[zimoun]

Hi Maxim,

On Sun, 21 Nov 2021 at 07:14, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> The syntax across the different importers are not uniform.  Especially,
>> compare:
>>
>>    guix import hackage mtl@2.1.3.1
>>    guix import pypi itsdangerous@1.1.0
>>
>> and worse, the option ’--recursive’ leads to an error for the latter.

Yes, this is indeed fixed.


> > Note that instead:
> >
> >    guix import pypi itsdangerous/1.1.0
> >
> > perfectly works, even the option ’--recursive’.

Not this one.  Well, except if we consider it is a feature. ;-)



Cheers,
simon