pypi URL change

pypi URL change

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Hi,

Today I tried using pypi-url to autogenerate a package that was only 
released today. Unfortunately, it seems that the URL scheme has changed 
and I was unable to download, though I believe this only affects newly 
released package versions. I found this on the nix mailing list:
http://comments.gmane.org/gmane.linux.distributions.nixos/20158

>    So, previously PyPI used URLs like :
>//packages//{python version}/{name[0]}/{name}/{filename}
>
>    Now it uses:
>//packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>    Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()

Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.

ben


[-- Attachment #2: Type: text/html, Size: 1145 bytes --]

Re: pypi URL change

[Cyril Roelandt]

On 04/24/2016 09:27 AM, Ben Woodcroft wrote:
> Hi,
> 
> Today I tried using pypi-url to autogenerate a package that was only 
> released today. Unfortunately, it seems that the URL scheme has changed 
> and I was unable to download, though I believe this only affects newly 
> released package versions. I found this on the nix mailing list:
> http://comments.gmane.org/gmane.linux.distributions.nixos/20158
> 
>>    So, previously PyPI used URLs like :
>> //packages//{python version}/{name[0]}/{name}/{filename}
>>
>>    Now it uses:
>> //packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>>    Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()
> 
> Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.
> 

The URL should always be correct, because the pypi importer can detect
whether the URL build by pypi-uri is wrong (see make-pypi-sexp).

Now, in order to handle this new pattern, maybe pypi-uri should:

1) try to compute the "new URI"
2) issue a "curl --head $uri" using some Guile magic
3) return the URI or compute the "old URI" depending on 2)

WDYT?

Cyril.

Re: pypi URL change

[Ben Woodcroft]

On 25/04/16 08:26, Cyril Roelandt wrote:
> On 04/24/2016 09:27 AM, Ben Woodcroft wrote:
>> Hi,
>>
>> Today I tried using pypi-url to autogenerate a package that was only
>> released today. Unfortunately, it seems that the URL scheme has changed
>> and I was unable to download, though I believe this only affects newly
>> released package versions. I found this on the nix mailing list:
>> http://comments.gmane.org/gmane.linux.distributions.nixos/20158
>>
>>>     So, previously PyPI used URLs like :
>>> //packages//{python version}/{name[0]}/{name}/{filename}
>>>
>>>     Now it uses:
>>> //packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>>>     Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()
>> Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.
>>
> The URL should always be correct, because the pypi importer can detect
> whether the URL build by pypi-uri is wrong (see make-pypi-sexp).
>
> Now, in order to handle this new pattern, maybe pypi-uri should:
>
> 1) try to compute the "new URI"
> 2) issue a "curl --head $uri" using some Guile magic
> 3) return the URI or compute the "old URI" depending on 2)
>
> WDYT?

Are suggesting that pypi-uri queries pypi each time it is called, even 
during a build? That seems a little wasteful.

I think there is actually 2 problems here.

1) 'guix refresh -u' updates the package with 'pypi-uri', not the URL it 
couldn't guess.
2) The new URLs are even uglier than the old one and a working pypi-uri 
would hide them from our eyes.

Fixing (1) should be easier and guards against future URL changes too, 
but fixing (2) would be good too.

I'm not familiar with Guile bindings, but there is a C library for 
blake2b so perhaps could simply bind to that?
https://github.com/BLAKE2/libb2

Thoughts?

Thanks,
ben

Re: pypi URL change

[Cyril Roelandt]

On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
> 2) The new URLs are even uglier than the old one and a working pypi-uri 
> would hide them from our eyes.

Yeah but it is my understanding that the new pattern will only be
applied to new releases, so how will pypi-uri know what pattern to apply?

Cyril.

Re: pypi URL change

[Ben Woodcroft]

On 25/04/16 10:13, Cyril Roelandt wrote:
> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>> 2) The new URLs are even uglier than the old one and a working pypi-uri
>> would hide them from our eyes.
> Yeah but it is my understanding that the new pattern will only be
> applied to new releases, so how will pypi-uri know what pattern to apply?

Good point. How about making a new function, perhaps pypi-url/hash or 
similar? Import/refresh will know what to do. Or, add an :#format flag 
to pypi-url which defaults to the new style, and we patch all the 
current python packages.
ben

Re: pypi URL change

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>> 2) The new URLs are even uglier than the old one and a working pypi-uri 
>> would hide them from our eyes.
>
> Yeah but it is my understanding that the new pattern will only be
> applied to new releases, so how will pypi-uri know what pattern to apply?

How about changing ‘pypi-uri’ to return two URIs, the new one and the
old one, in this order?

That way, attempts to download will first try the first URI, then the
second.

Alternately, we could define a “pypi” mirror in (guix download) that
would contain at least these two URIs, and have ‘pypi-uri’ return
“mirror://pypi/…”.  That only works if the two URIs share the same
suffix, though.

WDYT?

Ludo’.

Re: pypi URL change

[Ben Woodcroft]

On 25/04/16 17:54, Ludovic Courtès wrote:
> Cyril Roelandt <tipecaml@gmail.com> skribis:
>
>> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>>> 2) The new URLs are even uglier than the old one and a working pypi-uri
>>> would hide them from our eyes.
>> Yeah but it is my understanding that the new pattern will only be
>> applied to new releases, so how will pypi-uri know what pattern to apply?
> How about changing ‘pypi-uri’ to return two URIs, the new one and the
> old one, in this order?
>
> That way, attempts to download will first try the first URI, then the
> second.
>
> Alternately, we could define a “pypi” mirror in (guix download) that
> would contain at least these two URIs, and have ‘pypi-uri’ return
> “mirror://pypi/…”.  That only works if the two URIs share the same
> suffix, though.
>
> WDYT?

Hmm. It just dawned on me that it is impossible to implement the 
pypi-uri so that it returns the new format because the new URI is 
generated from the content of the package, not just the name and 
version. So I think we must either write the hash or the entire URL into 
the package, or grab the json from pypi each time we download.

ben

Re: pypi URL change

[Ludovic Courtès]

Ben Woodcroft <donttrustben@gmail.com> skribis:

> Hmm. It just dawned on me that it is impossible to implement the
> pypi-uri so that it returns the new format because the new URI is
> generated from the content of the package, not just the name and
> version.

Oh indeed.

> So I think we must either write the hash or the entire URL into the
> package, or grab the json from pypi each time we download.

The former, I guess.

I guess the hash is SHA1, right?

Maybe we could make something like:

  (define-syntax-rule (pypi-origin package sha1 sha256-string)
    (origin
      (method url-fetch)
      (uri (string-append … sha1 …))
      (sha256 (base32 sha256-string))))

and then use it like:

  (package
    (name "python-foo")
    (source (pypi-origin "foo" "cabba9e" "somelongstring")))

?

Ludo’.

Re: pypi URL change

[Ben Woodcroft]

>> So I think we must either write the hash or the entire URL into the
>> package, or grab the json from pypi each time we download.

>The former, I guess.
>
>I guess the hash is SHA1, right?
>
>Maybe we could make something like:
>
>  (define-syntax-rule (pypi-origin package sha1 sha256-string)
>    (origin
>      (method url-fetch)
>      (uri (string-append … sha1 …))
>      (sha256 (base32 sha256-string))))
>
>and then use it like:
>
>  (package
>    (name "python-foo")
>    (source (pypi-origin "foo" "cabba9e" "somelongstring")))


Thanks for the specific advice. I've attached some patches, the first of which
makes this change, the others using it. However, it wasn't clear to me how to
include patches or other parts of 'origin' here. Alos, Having to define
".tar.gz" each time seems a bit inelegant.

Do you have any suggestions on how to improve this approach?

Thanks,
ben

[PATCH 1/3] build-system/python: Add pypi-origin.

[Ben Woodcroft]

* guix/build-system/python.scm (pypi-origin): New macro.
---
 guix/build-system/python.scm | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index c3d6c62..c40fd03 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
+;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -35,7 +36,8 @@
             strip-python2-variant
             python-build
             python-build-system
-            pypi-uri))
+            pypi-uri
+            pypi-origin))
 
 ;; Commentary:
 ;;
@@ -52,6 +54,25 @@ extension, such as '.tar.gz'."
                  (string-take name 1) "/" name "/"
                  name "-" version extension))
 
+(define-syntax-rule
+  (pypi-origin name version extension blake2 sha256-string ...)
+  (origin
+    (method url-fetch)
+    (uri (string-append
+          "https://pypi.python.org/packages/"
+          (substring blake2 0 2)
+          "/"
+          (substring blake2 2 4)
+          "/"
+          (substring blake2 4)
+          "/"
+          name
+          "-"
+          version
+          extension))
+    (sha256 (base32 sha256-string))
+    ...))
+
 (define %python-build-system-modules
   ;; Build-side modules imported by default.
   `((guix build python-build-system)
-- 
2.7.4

[PATCH 2/3] gnu: Add python-screed.

[Ben Woodcroft]

* gnu/packages/bioinformatics.scm (python-screed, python2-screed):
New variables.
---
 gnu/packages/bioinformatics.scm | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 04ed769..76a6ebc 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3359,6 +3359,42 @@ optimize the sequencing depth, or to screen multiple libraries to avoid low
 complexity samples.")
     (license license:gpl3+)))
 
+(define-public python-screed
+  (package
+    (name "python-screed")
+    (version "0.9")
+    (source
+     (pypi-origin
+      "screed" version ".tar.gz"
+      "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
+      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda _
+             (setenv "PYTHONPATH"
+                     (string-append (getenv "PYTHONPATH") ":."))
+             (zero? (system* "nosetests" "--attr" "!known_failing")))))))
+    (native-inputs
+     `(("python-nose" ,python-nose)))
+    (inputs
+     `(("python-bz2file" ,python-bz2file)))
+    (home-page "http://github.com/dib-lab/screed/")
+    (synopsis "Short read sequence database utilities")
+    (description "Screed parses FASTA and FASTQ files and generates databases.
+Values such as sequence name, sequence description, sequence quality and the
+sequence itself can be retrieved from these databases.")
+    (license license:bsd-3)))
+
+(define-public python2-screed
+  (let ((base (package-with-python2 (strip-python2-variant python-screed))))
+    (package
+      (inherit base)
+      (native-inputs `(("python2-setuptools" ,python2-setuptools)
+                       ,@(package-native-inputs base))))))
+
 (define-public sra-tools
   (package
     (name "sra-tools")
-- 
2.7.4

[PATCH 3/3] not a real patch

[Ben Woodcroft]

---
 gnu/packages/bioinformatics.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 76a6ebc..01fd1a1 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3367,7 +3367,8 @@ complexity samples.")
      (pypi-origin
       "screed" version ".tar.gz"
       "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
-      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))
+      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"
+      (patches (search-patches "matplotlib-setupext-tk.patch"))))
     (build-system python-build-system)
     (arguments
      `(#:phases
-- 
2.7.4

Re: [PATCH 2/3] gnu: Add python-screed.

[Leo Famulari]

On Sat, Jun 18, 2016 at 08:33:31AM +1000, Ben Woodcroft wrote:
> * gnu/packages/bioinformatics.scm (python-screed, python2-screed):

I didn't build it, but it LGTM assuming it works for you.

> +    (source
> +     (pypi-origin
> +      "screed" version ".tar.gz"
> +      "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
> +      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))

It looks like we are going to try using the pypi.io URLs instead:

http://lists.gnu.org/archive/html/guix-devel/2016-06/msg00876.html