all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* pypi URL change
@ 2016-04-24  7:27 Ben Woodcroft
  2016-04-24 22:26 ` Cyril Roelandt
  0 siblings, 1 reply; 13+ messages in thread
From: Ben Woodcroft @ 2016-04-24  7:27 UTC (permalink / raw)
  To: guix-devel@gnu.org

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Hi,

Today I tried using pypi-url to autogenerate a package that was only 
released today. Unfortunately, it seems that the URL scheme has changed 
and I was unable to download, though I believe this only affects newly 
released package versions. I found this on the nix mailing list:
http://comments.gmane.org/gmane.linux.distributions.nixos/20158

>    So, previously PyPI used URLs like :
>//packages//{python version}/{name[0]}/{name}/{filename}
>
>    Now it uses:
>//packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>    Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()

Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.

ben


[-- Attachment #2: Type: text/html, Size: 1145 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-24  7:27 pypi URL change Ben Woodcroft
@ 2016-04-24 22:26 ` Cyril Roelandt
  2016-04-24 23:21   ` Ben Woodcroft
  0 siblings, 1 reply; 13+ messages in thread
From: Cyril Roelandt @ 2016-04-24 22:26 UTC (permalink / raw)
  To: guix-devel

On 04/24/2016 09:27 AM, Ben Woodcroft wrote:
> Hi,
> 
> Today I tried using pypi-url to autogenerate a package that was only 
> released today. Unfortunately, it seems that the URL scheme has changed 
> and I was unable to download, though I believe this only affects newly 
> released package versions. I found this on the nix mailing list:
> http://comments.gmane.org/gmane.linux.distributions.nixos/20158
> 
>>    So, previously PyPI used URLs like :
>> //packages//{python version}/{name[0]}/{name}/{filename}
>>
>>    Now it uses:
>> //packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>>    Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()
> 
> Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.
> 

The URL should always be correct, because the pypi importer can detect
whether the URL build by pypi-uri is wrong (see make-pypi-sexp).

Now, in order to handle this new pattern, maybe pypi-uri should:

1) try to compute the "new URI"
2) issue a "curl --head $uri" using some Guile magic
3) return the URI or compute the "old URI" depending on 2)

WDYT?

Cyril.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-24 22:26 ` Cyril Roelandt
@ 2016-04-24 23:21   ` Ben Woodcroft
  2016-04-25  0:13     ` Cyril Roelandt
  0 siblings, 1 reply; 13+ messages in thread
From: Ben Woodcroft @ 2016-04-24 23:21 UTC (permalink / raw)
  To: Cyril Roelandt, guix-devel



On 25/04/16 08:26, Cyril Roelandt wrote:
> On 04/24/2016 09:27 AM, Ben Woodcroft wrote:
>> Hi,
>>
>> Today I tried using pypi-url to autogenerate a package that was only
>> released today. Unfortunately, it seems that the URL scheme has changed
>> and I was unable to download, though I believe this only affects newly
>> released package versions. I found this on the nix mailing list:
>> http://comments.gmane.org/gmane.linux.distributions.nixos/20158
>>
>>>     So, previously PyPI used URLs like :
>>> //packages//{python version}/{name[0]}/{name}/{filename}
>>>
>>>     Now it uses:
>>> //packages//{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>>>     Where hash is blake2b(file_content, digest_size=32).hexdigest().lower()
>> Is there a blake2b implemented in Guile? In the meantime, the URL will have to be specified manually: 'guix import pypi ...' provides the correct and updated URL.
>>
> The URL should always be correct, because the pypi importer can detect
> whether the URL build by pypi-uri is wrong (see make-pypi-sexp).
>
> Now, in order to handle this new pattern, maybe pypi-uri should:
>
> 1) try to compute the "new URI"
> 2) issue a "curl --head $uri" using some Guile magic
> 3) return the URI or compute the "old URI" depending on 2)
>
> WDYT?

Are suggesting that pypi-uri queries pypi each time it is called, even 
during a build? That seems a little wasteful.

I think there is actually 2 problems here.

1) 'guix refresh -u' updates the package with 'pypi-uri', not the URL it 
couldn't guess.
2) The new URLs are even uglier than the old one and a working pypi-uri 
would hide them from our eyes.

Fixing (1) should be easier and guards against future URL changes too, 
but fixing (2) would be good too.

I'm not familiar with Guile bindings, but there is a C library for 
blake2b so perhaps could simply bind to that?
https://github.com/BLAKE2/libb2

Thoughts?

Thanks,
ben

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-24 23:21   ` Ben Woodcroft
@ 2016-04-25  0:13     ` Cyril Roelandt
  2016-04-25  1:44       ` Ben Woodcroft
  2016-04-25  7:54       ` Ludovic Courtès
  0 siblings, 2 replies; 13+ messages in thread
From: Cyril Roelandt @ 2016-04-25  0:13 UTC (permalink / raw)
  To: Ben Woodcroft, guix-devel

On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
> 2) The new URLs are even uglier than the old one and a working pypi-uri 
> would hide them from our eyes.

Yeah but it is my understanding that the new pattern will only be
applied to new releases, so how will pypi-uri know what pattern to apply?

Cyril.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-25  0:13     ` Cyril Roelandt
@ 2016-04-25  1:44       ` Ben Woodcroft
  2016-04-25  7:54       ` Ludovic Courtès
  1 sibling, 0 replies; 13+ messages in thread
From: Ben Woodcroft @ 2016-04-25  1:44 UTC (permalink / raw)
  To: Cyril Roelandt, Ben Woodcroft, guix-devel



On 25/04/16 10:13, Cyril Roelandt wrote:
> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>> 2) The new URLs are even uglier than the old one and a working pypi-uri
>> would hide them from our eyes.
> Yeah but it is my understanding that the new pattern will only be
> applied to new releases, so how will pypi-uri know what pattern to apply?

Good point. How about making a new function, perhaps pypi-url/hash or 
similar? Import/refresh will know what to do. Or, add an :#format flag 
to pypi-url which defaults to the new style, and we patch all the 
current python packages.
ben

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-25  0:13     ` Cyril Roelandt
  2016-04-25  1:44       ` Ben Woodcroft
@ 2016-04-25  7:54       ` Ludovic Courtès
  2016-04-26 11:46         ` Ben Woodcroft
  1 sibling, 1 reply; 13+ messages in thread
From: Ludovic Courtès @ 2016-04-25  7:54 UTC (permalink / raw)
  To: Cyril Roelandt; +Cc: guix-devel

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>> 2) The new URLs are even uglier than the old one and a working pypi-uri 
>> would hide them from our eyes.
>
> Yeah but it is my understanding that the new pattern will only be
> applied to new releases, so how will pypi-uri know what pattern to apply?

How about changing ‘pypi-uri’ to return two URIs, the new one and the
old one, in this order?

That way, attempts to download will first try the first URI, then the
second.

Alternately, we could define a “pypi” mirror in (guix download) that
would contain at least these two URIs, and have ‘pypi-uri’ return
“mirror://pypi/…”.  That only works if the two URIs share the same
suffix, though.

WDYT?

Ludo’.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-25  7:54       ` Ludovic Courtès
@ 2016-04-26 11:46         ` Ben Woodcroft
  2016-04-27 12:25           ` Ludovic Courtès
  0 siblings, 1 reply; 13+ messages in thread
From: Ben Woodcroft @ 2016-04-26 11:46 UTC (permalink / raw)
  To: Ludovic Courtès, Cyril Roelandt; +Cc: guix-devel



On 25/04/16 17:54, Ludovic Courtès wrote:
> Cyril Roelandt <tipecaml@gmail.com> skribis:
>
>> On 04/25/2016 01:21 AM, Ben Woodcroft wrote:
>>> 2) The new URLs are even uglier than the old one and a working pypi-uri
>>> would hide them from our eyes.
>> Yeah but it is my understanding that the new pattern will only be
>> applied to new releases, so how will pypi-uri know what pattern to apply?
> How about changing ‘pypi-uri’ to return two URIs, the new one and the
> old one, in this order?
>
> That way, attempts to download will first try the first URI, then the
> second.
>
> Alternately, we could define a “pypi” mirror in (guix download) that
> would contain at least these two URIs, and have ‘pypi-uri’ return
> “mirror://pypi/…”.  That only works if the two URIs share the same
> suffix, though.
>
> WDYT?

Hmm. It just dawned on me that it is impossible to implement the 
pypi-uri so that it returns the new format because the new URI is 
generated from the content of the package, not just the name and 
version. So I think we must either write the hash or the entire URL into 
the package, or grab the json from pypi each time we download.

ben

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-26 11:46         ` Ben Woodcroft
@ 2016-04-27 12:25           ` Ludovic Courtès
  2016-06-17 22:33             ` Ben Woodcroft
  0 siblings, 1 reply; 13+ messages in thread
From: Ludovic Courtès @ 2016-04-27 12:25 UTC (permalink / raw)
  To: Ben Woodcroft; +Cc: guix-devel

Ben Woodcroft <donttrustben@gmail.com> skribis:

> Hmm. It just dawned on me that it is impossible to implement the
> pypi-uri so that it returns the new format because the new URI is
> generated from the content of the package, not just the name and
> version.

Oh indeed.

> So I think we must either write the hash or the entire URL into the
> package, or grab the json from pypi each time we download.

The former, I guess.

I guess the hash is SHA1, right?

Maybe we could make something like:

  (define-syntax-rule (pypi-origin package sha1 sha256-string)
    (origin
      (method url-fetch)
      (uri (string-append … sha1 …))
      (sha256 (base32 sha256-string))))

and then use it like:

  (package
    (name "python-foo")
    (source (pypi-origin "foo" "cabba9e" "somelongstring")))

?

Ludo’.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: pypi URL change
  2016-04-27 12:25           ` Ludovic Courtès
@ 2016-06-17 22:33             ` Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 1/3] build-system/python: Add pypi-origin Ben Woodcroft
                                 ` (2 more replies)
  0 siblings, 3 replies; 13+ messages in thread
From: Ben Woodcroft @ 2016-06-17 22:33 UTC (permalink / raw)
  To: ludo, guix-devel

>> So I think we must either write the hash or the entire URL into the
>> package, or grab the json from pypi each time we download.

>The former, I guess.
>
>I guess the hash is SHA1, right?
>
>Maybe we could make something like:
>
>  (define-syntax-rule (pypi-origin package sha1 sha256-string)
>    (origin
>      (method url-fetch)
>      (uri (string-append … sha1 …))
>      (sha256 (base32 sha256-string))))
>
>and then use it like:
>
>  (package
>    (name "python-foo")
>    (source (pypi-origin "foo" "cabba9e" "somelongstring")))


Thanks for the specific advice. I've attached some patches, the first of which
makes this change, the others using it. However, it wasn't clear to me how to
include patches or other parts of 'origin' here. Alos, Having to define
".tar.gz" each time seems a bit inelegant.

Do you have any suggestions on how to improve this approach?

Thanks,
ben

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH 1/3] build-system/python: Add pypi-origin.
  2016-06-17 22:33             ` Ben Woodcroft
@ 2016-06-17 22:33               ` Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 2/3] gnu: Add python-screed Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 3/3] not a real patch Ben Woodcroft
  2 siblings, 0 replies; 13+ messages in thread
From: Ben Woodcroft @ 2016-06-17 22:33 UTC (permalink / raw)
  To: ludo, guix-devel

* guix/build-system/python.scm (pypi-origin): New macro.
---
 guix/build-system/python.scm | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index c3d6c62..c40fd03 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
+;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -35,7 +36,8 @@
             strip-python2-variant
             python-build
             python-build-system
-            pypi-uri))
+            pypi-uri
+            pypi-origin))
 
 ;; Commentary:
 ;;
@@ -52,6 +54,25 @@ extension, such as '.tar.gz'."
                  (string-take name 1) "/" name "/"
                  name "-" version extension))
 
+(define-syntax-rule
+  (pypi-origin name version extension blake2 sha256-string ...)
+  (origin
+    (method url-fetch)
+    (uri (string-append
+          "https://pypi.python.org/packages/"
+          (substring blake2 0 2)
+          "/"
+          (substring blake2 2 4)
+          "/"
+          (substring blake2 4)
+          "/"
+          name
+          "-"
+          version
+          extension))
+    (sha256 (base32 sha256-string))
+    ...))
+
 (define %python-build-system-modules
   ;; Build-side modules imported by default.
   `((guix build python-build-system)
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH 2/3] gnu: Add python-screed.
  2016-06-17 22:33             ` Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 1/3] build-system/python: Add pypi-origin Ben Woodcroft
@ 2016-06-17 22:33               ` Ben Woodcroft
  2016-06-25 17:44                 ` Leo Famulari
  2016-06-17 22:33               ` [PATCH 3/3] not a real patch Ben Woodcroft
  2 siblings, 1 reply; 13+ messages in thread
From: Ben Woodcroft @ 2016-06-17 22:33 UTC (permalink / raw)
  To: ludo, guix-devel

* gnu/packages/bioinformatics.scm (python-screed, python2-screed):
New variables.
---
 gnu/packages/bioinformatics.scm | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 04ed769..76a6ebc 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3359,6 +3359,42 @@ optimize the sequencing depth, or to screen multiple libraries to avoid low
 complexity samples.")
     (license license:gpl3+)))
 
+(define-public python-screed
+  (package
+    (name "python-screed")
+    (version "0.9")
+    (source
+     (pypi-origin
+      "screed" version ".tar.gz"
+      "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
+      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda _
+             (setenv "PYTHONPATH"
+                     (string-append (getenv "PYTHONPATH") ":."))
+             (zero? (system* "nosetests" "--attr" "!known_failing")))))))
+    (native-inputs
+     `(("python-nose" ,python-nose)))
+    (inputs
+     `(("python-bz2file" ,python-bz2file)))
+    (home-page "http://github.com/dib-lab/screed/")
+    (synopsis "Short read sequence database utilities")
+    (description "Screed parses FASTA and FASTQ files and generates databases.
+Values such as sequence name, sequence description, sequence quality and the
+sequence itself can be retrieved from these databases.")
+    (license license:bsd-3)))
+
+(define-public python2-screed
+  (let ((base (package-with-python2 (strip-python2-variant python-screed))))
+    (package
+      (inherit base)
+      (native-inputs `(("python2-setuptools" ,python2-setuptools)
+                       ,@(package-native-inputs base))))))
+
 (define-public sra-tools
   (package
     (name "sra-tools")
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH 3/3] not a real patch
  2016-06-17 22:33             ` Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 1/3] build-system/python: Add pypi-origin Ben Woodcroft
  2016-06-17 22:33               ` [PATCH 2/3] gnu: Add python-screed Ben Woodcroft
@ 2016-06-17 22:33               ` Ben Woodcroft
  2 siblings, 0 replies; 13+ messages in thread
From: Ben Woodcroft @ 2016-06-17 22:33 UTC (permalink / raw)
  To: ludo, guix-devel

---
 gnu/packages/bioinformatics.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 76a6ebc..01fd1a1 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3367,7 +3367,8 @@ complexity samples.")
      (pypi-origin
       "screed" version ".tar.gz"
       "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
-      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))
+      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"
+      (patches (search-patches "matplotlib-setupext-tk.patch"))))
     (build-system python-build-system)
     (arguments
      `(#:phases
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [PATCH 2/3] gnu: Add python-screed.
  2016-06-17 22:33               ` [PATCH 2/3] gnu: Add python-screed Ben Woodcroft
@ 2016-06-25 17:44                 ` Leo Famulari
  0 siblings, 0 replies; 13+ messages in thread
From: Leo Famulari @ 2016-06-25 17:44 UTC (permalink / raw)
  To: Ben Woodcroft; +Cc: guix-devel

On Sat, Jun 18, 2016 at 08:33:31AM +1000, Ben Woodcroft wrote:
> * gnu/packages/bioinformatics.scm (python-screed, python2-screed):

I didn't build it, but it LGTM assuming it works for you.

> +    (source
> +     (pypi-origin
> +      "screed" version ".tar.gz"
> +      "a06ba90c7ab7b0ad1eab211053c85c1242a81379f34c5f5933392079c9b36858"
> +      "18czszp9fkx3j6jr7y5kp6dfialscgddk05mw1zkhh2zhn0jd8i0"))

It looks like we are going to try using the pypi.io URLs instead:

http://lists.gnu.org/archive/html/guix-devel/2016-06/msg00876.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2016-06-25 17:44 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-24  7:27 pypi URL change Ben Woodcroft
2016-04-24 22:26 ` Cyril Roelandt
2016-04-24 23:21   ` Ben Woodcroft
2016-04-25  0:13     ` Cyril Roelandt
2016-04-25  1:44       ` Ben Woodcroft
2016-04-25  7:54       ` Ludovic Courtès
2016-04-26 11:46         ` Ben Woodcroft
2016-04-27 12:25           ` Ludovic Courtès
2016-06-17 22:33             ` Ben Woodcroft
2016-06-17 22:33               ` [PATCH 1/3] build-system/python: Add pypi-origin Ben Woodcroft
2016-06-17 22:33               ` [PATCH 2/3] gnu: Add python-screed Ben Woodcroft
2016-06-25 17:44                 ` Leo Famulari
2016-06-17 22:33               ` [PATCH 3/3] not a real patch Ben Woodcroft

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.