[PATCH 00/12] Tons of patches to run "guix-tox" on python-keystoneclient!

[PATCH 00/12] Tons of patches to run "guix-tox" on python-keystoneclient!

[Cyril Roelandt]

The following 12 patches should be enough to be able to run "guix-tox" on
python-keystoneclient, which is nice!

Cyril.
--
Cyril Roelandt (12):
  guix: download: properly detect https when mirror:// is used.
  guix: Add a "pypi-uri" helper method.
  import: pypi: Use "pypi-uri" instead of building the URL manually.
  gnu: Update python-requests.
  gnu: Add python-mccabe 0.2.1
  gnu: Add python-flake8-2.2.4
  gnu: Add python-hacking.
  gnu: Add python-oslosphinx.
  gnu: Add python-os-testr.
  gnu: paramiko: Move python-pycrypto to the propagated inputs.
  gnu: Add python-oslo.log.
  gnu: Add python-tempest-lib.

 gnu/packages/openstack.scm   | 182 +++++++++++++++++++++++++++++++++++++++++++
 gnu/packages/python.scm      |  46 ++++++++++-
 guix/build-system/python.scm |  10 ++-
 guix/download.scm            |  17 ++--
 guix/import/pypi.scm         |   2 +-
 tests/pypi.scm               |   3 +-
 6 files changed, 244 insertions(+), 16 deletions(-)

-- 
2.1.4

[PATCH 01/12] guix: download: properly detect https when mirror:// is used.

[Cyril Roelandt]

* guix/download.scm (url-fetch): fix need-gnutls? which always returned #f when
  a URL with "mirror://" scheme was used.
---
 guix/download.scm | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/guix/download.scm b/guix/download.scm
index 204cfc0..8ec47ce 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -235,13 +235,12 @@ in the store."
        (basename url))))
 
   (define need-gnutls?
-    ;; True if any of the URLs need TLS support.
-    (let ((https? (cut string-prefix? "https://" <>)))
-      (match url
-        ((? string?)
-         (https? url))
-        ((url ...)
-         (any https? url)))))
+    (let ((https? (lambda (uri)
+                    (eq? 'https (uri-scheme uri)))))
+      (any https? (append-map (cut build:maybe-expand-mirrors <> %mirrors)
+                  (match url
+                    ((_ ...) (map string->uri url))
+                    (_       (list (string->uri url))))))))
 
   (define builder
     #~(begin
-- 
2.1.4

[PATCH 02/12] guix: Add a "pypi-uri" helper method.

[Cyril Roelandt]

* guix/download.scm (mirrors): New "pypi" mirror.
* guix/build-system/python.scm (pypi-uri): New method.
---
 guix/build-system/python.scm | 10 +++++++++-
 guix/download.scm            |  4 +++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index e9fffcc..01525f4 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -31,7 +31,8 @@
   #:export (%python-build-system-modules
             package-with-python2
             python-build
-            python-build-system))
+            python-build-system
+            pypi-uri))
 
 ;; Commentary:
 ;;
@@ -40,6 +41,13 @@
 ;;
 ;; Code:
 
+(define (pypi-uri name version)
+  "Return a URI string for the Python package hosted on the Python Package
+Index (PyPI) corresponding to NAME and VERSION."
+  (string-append "mirror://pypi/packages/source/"
+                 (string-take name 1) "/" name "/"
+                 name "-" version ".tar.gz"))
+
 (define %python-build-system-modules
   ;; Build-side modules imported by default.
   `((guix build python-build-system)
diff --git a/guix/download.scm b/guix/download.scm
index 8ec47ce..53b28be 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -200,7 +200,9 @@
       (debian
        "http://ftp.de.debian.org/debian/"
        "http://ftp.fr.debian.org/debian/"
-       "http://ftp.debian.org/debian/"))))
+       "http://ftp.debian.org/debian/")
+      (pypi
+       "https://pypi.python.org/"))))
 
 (define %mirror-file
   ;; Copy of the list of mirrors to a file.  This allows us to keep a single
-- 
2.1.4

[PATCH 03/12] import: pypi: Use "pypi-uri" instead of building the URL manually.

[Cyril Roelandt]

* guix/import/pypi.scm (make-pypi-sexp): Use "pypi-uri".
* tests/pypi.scm: Update the tests accordingly.
---
 guix/import/pypi.scm | 2 +-
 tests/pypi.scm       | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index d04a685..647ef61 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -165,7 +165,7 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
              (version ,version)
              (source (origin
                        (method url-fetch)
-                       (uri (string-append ,@(factorize-uri source-url version)))
+                       (uri (pypi-uri ,name version))
                        (sha256
                         (base32
                          ,(guix-hash-url temp)))))
diff --git a/tests/pypi.scm b/tests/pypi.scm
index c772474..960b8cd 100644
--- a/tests/pypi.scm
+++ b/tests/pypi.scm
@@ -84,8 +84,7 @@ baz > 13.37")
          ('version "1.0.0")
          ('source ('origin
                     ('method 'url-fetch)
-                    ('uri ('string-append "https://example.com/foo-"
-                                          'version ".tar.gz"))
+                    ('uri (pypi-uri "foo" version))
                     ('sha256
                      ('base32
                       (? string? hash)))))
-- 
2.1.4

[PATCH 04/12] gnu: Update python-requests.

[Cyril Roelandt]

* gnu/packages/python.scm (python-requests, python2-requests): Bump to 2.8.0.
---
 gnu/packages/python.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index fc4fad6..44e3e6b 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -2071,7 +2071,7 @@ with sensible defaults out of the box.")
 (define-public python-requests
   (package
     (name "python-requests")
-    (version "2.4.0")
+    (version "2.8.0")
     (source (origin
              (method url-fetch)
              (uri
@@ -2080,7 +2080,7 @@ with sensible defaults out of the box.")
                version ".tar.gz"))
              (sha256
               (base32
-               "0gknlfx1wakrrm1zi8gi03x2lzj4dsns0vjw0nsmgqvkphyf01vh"))))
+               "0yrvj8hfnabrdxds59w6d6887sn5j0jlgpmcq04lk4k0kdc07w5j"))))
     (build-system python-build-system)
     (inputs
      `(("python-setuptools" ,python-setuptools)
-- 
2.1.4

[PATCH 05/12] gnu: Add python-mccabe 0.2.1

[Cyril Roelandt]

* gnu/packages/python.scm (python-mccabe-0.2.1,
  python2-mccabe-0.2.1): New variables.
---
 gnu/packages/python.scm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 44e3e6b..e8d8a02 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -4834,6 +4834,20 @@ complexity of Python source code.")
 (define-public python2-mccabe
   (package-with-python2 python-mccabe))
 
+(define-public python-mccabe-0.2.1
+  (package (inherit python-mccabe)
+    (version "0.2.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "mccabe" version))
+        (sha256
+          (base32
+            "0fi4a81kr5bcv5p4xgibqr595hyj5dafkqgsmfk96mfy8w71fajs"))))))
+
+(define-public python2-mccabe-0.2.1
+  (package-with-python2 python-mccabe-0.2.1))
+
 ;; Flake8 2.4.1 requires an older version of pep8.
 ;; This should be removed ASAP.
 (define-public python-pep8-1.5.7
-- 
2.1.4

[PATCH 06/12] gnu: Add python-flake8-2.2.4

[Cyril Roelandt]

* gnu/packages/python.scm (python-flake8-2.2.4,
  python2-flake8-2.2.4): New variables.
---
 gnu/packages/python.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index e8d8a02..d88a3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -4918,6 +4918,29 @@ complexity of Python source code.")
 (define-public python2-flake8
   (package-with-python2 python-flake8))
 
+;; This will only be needed by the python-hacking package and will not be
+;; necessary once python-hacking > 0.10.2 is released.
+(define-public python-flake8-2.2.4
+  (package (inherit python-flake8)
+    (inputs
+      `(("python-setuptools" ,python-setuptools)
+        ("python-pep8" ,python-pep8-1.5.7)
+        ("python-pyflakes" ,python-pyflakes-0.8.1)
+        ("python-mccabe" ,python-mccabe-0.2.1)
+        ("python-mock" ,python-mock)
+        ("python-nose" ,python-nose)))
+    (version "2.2.4")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "flake8" version))
+        (sha256
+          (base32
+            "1r9wsry4va45h1rck5hxd3vzsg2q3y6lnl6pym1bxvz8ry19jwx8"))))))
+
+(define-public python2-flake8-2.2.4
+  (package-with-python2 python-flake8-2.2.4))
+
 (define-public python-mistune
   (package
     (name "python-mistune")
-- 
2.1.4

[PATCH 07/12] gnu: Add python-hacking.

[Cyril Roelandt]

* gnu/packages/openstack.scm (python-hacking,
  python2-hacking): New variables.
---
 gnu/packages/openstack.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/openstack.scm b/gnu/packages/openstack.scm
index 4fd1c80..d6f9ec4 100644
--- a/gnu/packages/openstack.scm
+++ b/gnu/packages/openstack.scm
@@ -100,6 +100,39 @@ manner.")
 (define-public python2-debtcollector
   (package-with-python2 python-debtcollector))
 
+(define-public python-hacking
+  (package
+    (name "python-hacking")
+    (version "0.10.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "hacking" version))
+       (sha256
+        (base32
+         "1a310k3dv04jg7zvmk37h2ql7y9kf4hvdxb74bjlwdxgmy6h4wap"))))
+    (build-system python-build-system)
+    (propagated-inputs
+      `(("python-flake8-2.2.4" ,python-flake8-2.2.4)
+        ("python-mccabe-0.2.1" ,python-mccabe-0.2.1)
+        ("python-pbr" ,python-pbr)
+        ("python-pep8-1.5.7" ,python-pep8-1.5.7)
+        ("python-pyflakes-0.8.1" ,python-pyflakes-0.8.1)
+        ("python-six" ,python-six)))
+    (inputs
+      `(("python-setuptools" ,python-setuptools)
+        ;; Tests
+        ("python-testscenarios" ,python-testscenarios)))
+    (home-page "http://github.com/openstack-dev/hacking")
+    (synopsis "OpenStack Hacking Guideline Enforcement")
+    (description
+      "A set of flake8 plugins that test and enforce the OpenStack Style
+  Guidelines.")
+    (license asl2.0)))
+
+(define-public python2-hacking
+  (package-with-python2 python-hacking))
+
 (define-public python-mox3
   (package
     (name "python-mox3")
-- 
2.1.4

[PATCH 08/12] gnu: Add python-oslosphinx.

[Cyril Roelandt]

* gnu/packages/openstack.scm (python-oslosphinx,
  python2-oslosphinx): New variables.
---
 gnu/packages/openstack.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/openstack.scm b/gnu/packages/openstack.scm
index d6f9ec4..0fa105e 100644
--- a/gnu/packages/openstack.scm
+++ b/gnu/packages/openstack.scm
@@ -464,6 +464,36 @@ in transmittable and storable formats, such as JSON and MessagePack.")
 (define-public python2-oslo.serialization
   (package-with-python2 python-oslo.serialization))
 
+(define-public python-oslosphinx
+  (package
+    (name "python-oslosphinx")
+    (version "3.1.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "oslosphinx" version))
+       (sha256
+        (base32
+         "0zcshdc9s1f7hnvg0fm2ps5rak3dpnm8kqg4i21lknhmsvb7p5cb"))))
+    (build-system python-build-system)
+    (propagated-inputs
+      `(("python-requests" ,python-requests)))
+    (inputs
+      `(("python-pbr" ,python-pbr)
+        ("python-docutils" ,python-docutils)
+        ("python-hacking" ,python-hacking)
+        ("python-setuptools" ,python-setuptools)
+        ("python-sphinx" ,python-sphinx)))
+    (home-page "http://www.openstack.org/")
+    (synopsis "OpenStack Sphinx Extensions and Theme")
+    (description
+      "Theme and extension support for Sphinx documentation from the OpenStack
+  project.")
+    (license asl2.0)))
+
+(define-public python2-oslosphinx
+  (package-with-python2 python-oslosphinx))
+
 (define-public python-oslotest
   (package
     (name "python-oslotest")
-- 
2.1.4

[PATCH 09/12] gnu: Add python-os-testr.

[Cyril Roelandt]

* gnu/packages/openstack.scm (python-ostestr,
  python2-ostestr): New variables.
---
 gnu/packages/openstack.scm | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/gnu/packages/openstack.scm b/gnu/packages/openstack.scm
index 0fa105e..912fab3 100644
--- a/gnu/packages/openstack.scm
+++ b/gnu/packages/openstack.scm
@@ -205,6 +205,40 @@ tested on Python version 3.2, 2.7 and 2.6.")
 (define-public python2-mox3
   (package-with-python2 python-mox3))
 
+(define-public python-os-testr
+  (package
+    (name "python-os-testr")
+    (version "0.4.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "os-testr" version))
+       (sha256
+        (base32
+         "0474z0mxb7y3vfk4s097wf1mzji5d135vh27cvlh9q17rq3x9r3w"))))
+    (build-system python-build-system)
+    (arguments
+     ;; os-testr uses itself to run the tests. It seems like pbr writes the
+     ;; exectuable in the virtualenv when using tox. Not sure how to do this
+     ;; when building the package. Skip the tests for now.
+     `(#:tests? #f))
+    (propagated-inputs
+     `(("python-pbr" ,python-pbr)
+       ("python-subunit" ,python-subunit)
+       ("python-testtools" ,python-testtools)))
+    (inputs
+      `(("python-babel" ,python-babel)
+        ("python-setuptools" ,python-setuptools)))
+    (home-page "http://www.openstack.org/")
+    (synopsis "Testr wrapper to provide functionality for OpenStack projects")
+    (description
+      "Os-testr provides developers with a testr wrapper and an output filter
+  for subunit.")
+    (license asl2.0)))
+
+(define-public python2-os-testr
+  (package-with-python2 python-os-testr))
+
 (define-public python-pbr
   (package
     (name "python-pbr")
-- 
2.1.4

[PATCH 10/12] gnu: paramiko: Move python-pycrypto to the propagated inputs.

[Cyril Roelandt]

* gnu/packages/python.scm (python-paramiko): Move python-pycrypto to the
  propagated-inputs.
---
 gnu/packages/python.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d88a3c6..6cecaac 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -411,9 +411,10 @@ John the Ripper).")
     (build-system python-build-system)
     (native-inputs
      `(("python-setuptools" ,python-setuptools)))
+    (propagated-inputs
+     `(("python-pycrypto" ,python-pycrypto)))
     (inputs
-     `(("python-ecdsa" ,python-ecdsa)
-       ("python-pycrypto" ,python-pycrypto)))
+     `(("python-ecdsa" ,python-ecdsa)))
     (home-page "http://www.paramiko.org/")
     (synopsis "SSHv2 protocol library")
     (description "Paramiko is a python implementation of the SSHv2 protocol,
-- 
2.1.4

[PATCH 11/12] gnu: Add python-oslo.log.

[Cyril Roelandt]

* gnu/packages/openstack.scm (python-oslo.log,
  python2-oslo.log): New variables.
---
 gnu/packages/openstack.scm | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/gnu/packages/openstack.scm b/gnu/packages/openstack.scm
index 912fab3..8bc1c68 100644
--- a/gnu/packages/openstack.scm
+++ b/gnu/packages/openstack.scm
@@ -459,6 +459,47 @@ in an application or library.")
 (define-public python2-oslo.i18n
   (package-with-python2 python-oslo.i18n))
 
+(define-public python-oslo.log
+  (package
+  (name "python-oslo.log")
+  (version "1.6.0")
+  (source
+    (origin
+      (method url-fetch)
+      (uri (string-append
+             "https://pypi.python.org/packages/source/o/oslo.log/oslo.log-"
+             version
+             ".tar.gz"))
+      (sha256
+        (base32
+          "1fhy6yvbd565nv4x4i3ppyrlbmz3yy9d0xsvw5nkqsa7g43nmf8z"))))
+  (build-system python-build-system)
+  (propagated-inputs
+   `(("python-debtcollector" ,python-debtcollector)
+     ("python-oslo.config" ,python-oslo.config)
+     ("python-oslo.context" ,python-oslo.context)
+     ("python-oslo.i18n" ,python-oslo.i18n)
+     ("python-oslo.utils" ,python-oslo.utils)
+     ("python-oslo.serialization" ,python-oslo.serialization)
+     ("python-six" ,python-six)))
+  (inputs
+    `(("python-babel" ,python-babel)
+      ("python-iso8601" ,python-iso8601)
+      ("python-mock" ,python-mock)
+      ("python-oslotest" ,python-oslotest)
+      ("python-pbr" ,python-pbr)
+      ("python-setuptools" ,python-setuptools)))
+  (home-page "http://launchpad.net/oslo")
+  (synopsis "Oslo logging API")
+  (description
+    "The oslo.log (logging) configuration library provides standardized
+configuration for all openstack projects. It also provides custom formatters,
+handlers and support for context specific logging (like resource id’s etc).")
+  (license asl2.0)))
+
+(define-public python2-oslo.log
+  (package-with-python2 python-oslo.log))
+
 (define-public python-oslo.serialization
   (package
     (name "python-oslo.serialization")
-- 
2.1.4

[PATCH 12/12] gnu: Add python-tempest-lib.

[Cyril Roelandt]

* gnu/packages/openstack.scm (python-tempest-lib,
  python2-tempest-lib): New variables.
---
 gnu/packages/openstack.scm | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/gnu/packages/openstack.scm b/gnu/packages/openstack.scm
index 8bc1c68..a7ebb25 100644
--- a/gnu/packages/openstack.scm
+++ b/gnu/packages/openstack.scm
@@ -353,6 +353,50 @@ extensions.")
 (define-public python2-stevedore
   (package-with-python2 python-stevedore))
 
+(define-public python-tempest-lib
+  (package
+    (name "python-tempest-lib")
+    (version "0.9.0")
+    (source
+     (origin
+      (method url-fetch)
+      (uri (pypi-uri "tempest-lib" version))
+      (sha256
+       (base32
+        "1s32rpxw86p41ip9nr7zbqxd60mw1cqz2isirby36rh4vl68bfhx"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before
+          'check 'pre-check
+          (lambda _
+            (substitute* "tempest_lib/tests/cli/test_execute.py"
+              (("/bin/ls") (which "ls"))))))))
+    (inputs
+      `(("python-babel" ,python-babel)
+        ("python-fixtures" ,python-fixtures)
+        ("python-httplib2" ,python-httplib2)
+        ("python-iso8601" ,python-iso8601)
+        ("python-jsonschema" ,python-jsonschema)
+        ("python-mock" ,python-mock)
+        ("python-mox3" ,python-mox3)
+        ("python-os-testr" ,python-os-testr)
+        ("python-oslo.log" ,python-oslo.log)
+        ("python-oslotest" ,python-oslotest)
+        ("python-paramiko" ,python-paramiko)
+        ("python-pbr" ,python-pbr)
+        ("python-setuptools" ,python-setuptools)
+        ("python-six" ,python-six)))
+    (home-page "http://www.openstack.org/")
+    (synopsis "OpenStack Functional Testing Library")
+    (description
+      "OpenStack Functional Testing Library")
+    (license asl2.0)))
+
+(define-public python2-tempest-lib
+  (package-with-python2 python-tempest-lib))
+
 ;; Packages from the Oslo library
 (define-public python-oslo.config
   (package
-- 
2.1.4

Re: [PATCH 02/12] guix: Add a "pypi-uri" helper method.

[Ricardo Wurmus]

Cyril Roelandt <tipecaml@gmail.com> writes:

> * guix/download.scm (mirrors): New "pypi" mirror.
> * guix/build-system/python.scm (pypi-uri): New method.

It’s a good idea to provide “pypi-uri”!  But why is a mirror entry
required when “pypi.python.org” is the only source for packages?

~~ Ricardo

Re: [PATCH 02/12] guix: Add a "pypi-uri" helper method.

[Thompson, David]

On Tue, Oct 13, 2015 at 3:47 AM, Ricardo Wurmus
<ricardo.wurmus@mdc-berlin.de> wrote:
>
> Cyril Roelandt <tipecaml@gmail.com> writes:
>
>> * guix/download.scm (mirrors): New "pypi" mirror.
>> * guix/build-system/python.scm (pypi-uri): New method.
>
> It’s a good idea to provide “pypi-uri”!  But why is a mirror entry
> required when “pypi.python.org” is the only source for packages?

Yeah, get rid of the mirror thing and this patch looks good to me.

- Dave

Re: [PATCH 02/12] guix: Add a "pypi-uri" helper method.

[Cyril Roelandt]

On 10/13/2015 03:51 PM, Thompson, David wrote:
> Yeah, get rid of the mirror thing and this patch looks good to me.

Yeah, will do. I thought we might add mirrors, but apparently there are
no official mirrors any more, so...

Cyril.

Re: [PATCH 01/12] guix: download: properly detect https when mirror:// is used.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1826 bytes --]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/download.scm (url-fetch): fix need-gnutls? which always returned #f when
>   a URL with "mirror://" scheme was used.

[...]

>    (define need-gnutls?
> -    ;; True if any of the URLs need TLS support.

Please keep/adjust the comment.

> -    (let ((https? (cut string-prefix? "https://" <>)))
> -      (match url
> -        ((? string?)
> -         (https? url))
> -        ((url ...)
> -         (any https? url)))))
> +    (let ((https? (lambda (uri)
> +                    (eq? 'https (uri-scheme uri)))))
> +      (any https? (append-map (cut build:maybe-expand-mirrors <> %mirrors)
> +                  (match url
> +                    ((_ ...) (map string->uri url))
> +                    (_       (list (string->uri url))))))))

This looks like a good idea, but it might raise bootstrapping issues.

For instance, what if mirror://gnu includes HTTPS URLs?  Try the
following:

  guix gc -d /gnu/store/*-glibc-2.22.tar.xz
  ./pre-inst-env guix build -S \
     -e '(@@ (gnu packages commencement) glibc-final)' \
     --no-substitutes

If mirror://gnu contains HTTPS URLs, this will create a circular
dependency (glibc’s source depends on GnuTLS, which depends on glibc,
which depends on glibc’s source), leading to a stack overflow and
maximum user unhappiness.

So address that, I modified the patch as shown in the attached file.  It
solves the bootstrapping case.

But that still doesn’t handle the more general problem of creating a
circular dependency between GnuTLS and source downloads.  That could
actually happen anywhere in the package graph.  So all in all, I’d
rather take the conservative approach and avoid that.

Is there a mirror for which that is a serious issue?

Thanks,
Ludo’.


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 3617 bytes --]

From ae4e4168aefd04b001ba1dd368fb08ae0c5af433 Mon Sep 17 00:00:00 2001
From: Cyril Roelandt <tipecaml@gmail.com>
Date: Mon, 12 Oct 2015 23:40:57 +0200
Subject: [PATCH] download: Detect https when mirror:// is used.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* guix/download.scm (url-fetch): Add #:allow-tls? parameter and honor
  it.
  [https?]: New local procedure.
  [need-gnutls?]: Check whether any of the mirrors requires HTTPS.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
---
 gnu/packages/bootstrap.scm |  5 ++++-
 guix/download.scm          | 25 ++++++++++++++++---------
 2 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/bootstrap.scm b/gnu/packages/bootstrap.scm
index f5bf069..8ab9713 100644
--- a/gnu/packages/bootstrap.scm
+++ b/gnu/packages/bootstrap.scm
@@ -63,7 +63,10 @@
               #:optional name #:key system)
       (fetch url hash-algo hash
              #:guile %bootstrap-guile
-             #:system system)))
+             #:system system
+
+             ;; Make sure we don't introduce a dependency on GnuTLS.
+             #:allow-tls? #f)))
 
   (define %bootstrap-patch-inputs
     ;; Packages used when an <origin> has a non-empty 'patches' field.
diff --git a/guix/download.scm b/guix/download.scm
index 204cfc0..2780f4c 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -45,6 +45,7 @@
           '(;; This one redirects to a (supposedly) nearby and (supposedly)
             ;; up-to-date mirror.
             "http://ftpmirror.gnu.org/"
+            "https://ftpmirror.gnu.org/"
 
             "ftp://ftp.cs.tu-berlin.de/pub/gnu/"
             "ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/"
@@ -216,7 +217,8 @@
 (define* (url-fetch url hash-algo hash
                     #:optional name
                     #:key (system (%current-system))
-                    (guile (default-guile)))
+                    (guile (default-guile))
+                    (allow-tls? #t))
   "Return a fixed-output derivation that fetches URL (a string, or a list of
 strings denoting alternate URLs), which is expected to have hash HASH of type
 HASH-ALGO (a symbol).  By default, the file name is the base name of URL;
@@ -226,7 +228,10 @@ When one of the URL starts with mirror://, then its host part is
 interpreted as the name of a mirror scheme, taken from %MIRROR-FILE.
 
 Alternately, when URL starts with file://, return the corresponding file name
-in the store."
+in the store.
+
+ALLOW-TLS? determines whether to allow TLS for downloads, which entails adding
+a dependency on GnuTLS."
   (define file-name
     (match url
       ((head _ ...)
@@ -234,18 +239,20 @@ in the store."
       (_
        (basename url))))
 
+  (define (https? uri)
+    (eq? 'https (uri-scheme uri)))
+
   (define need-gnutls?
     ;; True if any of the URLs need TLS support.
-    (let ((https? (cut string-prefix? "https://" <>)))
-      (match url
-        ((? string?)
-         (https? url))
-        ((url ...)
-         (any https? url)))))
+    (any https?
+         (append-map (cut build:maybe-expand-mirrors <> %mirrors)
+                     (match url
+                       ((_ ...) (map string->uri url))
+                       (_       (list (string->uri url)))))))
 
   (define builder
     #~(begin
-        #+(if need-gnutls?
+        #+(if (and allow-tls? need-gnutls?)
 
               ;; Add GnuTLS to the inputs and to the load path.
               #~(eval-when (load expand eval)
-- 
2.5.0

Re: [PATCH 04/12] gnu: Update python-requests.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/python.scm (python-requests, python2-requests): Bump to 2.8.0.

OK

Re: [PATCH 05/12] gnu: Add python-mccabe 0.2.1

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/python.scm (python-mccabe-0.2.1,
>   python2-mccabe-0.2.1): New variables.

OK

Re: [PATCH 06/12] gnu: Add python-flake8-2.2.4

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/python.scm (python-flake8-2.2.4,
>   python2-flake8-2.2.4): New variables.

OK

Re: [PATCH 07/12] gnu: Add python-hacking.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/openstack.scm (python-hacking,
>   python2-hacking): New variables.

[...]

> +    (home-page "http://github.com/openstack-dev/hacking")
> +    (synopsis "OpenStack Hacking Guideline Enforcement")

Lowercase please, except for “OpenStack.”

> +    (description
> +      "A set of flake8 plugins that test and enforce the OpenStack Style
> +  Guidelines.")

Rather:

  Python-hacking is a set of flake8 plugins that test and enforce the
  @uref{http://docs.openstack.org/developer/hacking/, OpenStack style
  guidelines}.

OK with these changes.

Thanks,
Ludo’.

Re: [PATCH 08/12] gnu: Add python-oslosphinx.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/openstack.scm (python-oslosphinx,
>   python2-oslosphinx): New variables.

[...]

> +    (synopsis "OpenStack Sphinx Extensions and Theme")

Lowercase.

> +    (description
> +      "Theme and extension support for Sphinx documentation from the OpenStack
> +  project.")

Rather “This package provides themes and extensions for Sphinx
documentation…”

Otherwise LGTM.

Ludo’.

Re: [PATCH 09/12] gnu: Add python-os-testr.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/openstack.scm (python-ostestr,
>   python2-ostestr): New variables.

OK

Re: [PATCH 10/12] gnu: paramiko: Move python-pycrypto to the propagated inputs.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/python.scm (python-paramiko): Move python-pycrypto to the
>   propagated-inputs.

OK

Re: [PATCH 11/12] gnu: Add python-oslo.log.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/openstack.scm (python-oslo.log,
>   python2-oslo.log): New variables.


[...]

> +  (home-page "http://launchpad.net/oslo")
> +  (synopsis "Oslo logging API")

What about “Python logging library of the Oslo project”?

> +  (description
> +    "The oslo.log (logging) configuration library provides standardized
> +configuration for all openstack projects. It also provides custom formatters,

“OpenStack” (capitalization) + two spaces after period.

OK with this change.

Ludo’.

Re: [PATCH 12/12] gnu: Add python-tempest-lib.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * gnu/packages/openstack.scm (python-tempest-lib,
>   python2-tempest-lib): New variables.

[...]

> +    (inputs
> +      `(("python-babel" ,python-babel)
> +        ("python-fixtures" ,python-fixtures)
> +        ("python-httplib2" ,python-httplib2)
> +        ("python-iso8601" ,python-iso8601)
> +        ("python-jsonschema" ,python-jsonschema)
> +        ("python-mock" ,python-mock)
> +        ("python-mox3" ,python-mox3)
> +        ("python-os-testr" ,python-os-testr)
> +        ("python-oslo.log" ,python-oslo.log)
> +        ("python-oslotest" ,python-oslotest)
> +        ("python-paramiko" ,python-paramiko)
> +        ("python-pbr" ,python-pbr)
> +        ("python-setuptools" ,python-setuptools)
> +        ("python-six" ,python-six)))

Shouldn’t most of these be propagated?

> +    (synopsis "OpenStack Functional Testing Library")

Lowercase.

> +    (description
> +      "OpenStack Functional Testing Library")

Please expound a bit.

Ludo’.

Re: [PATCH 01/12] guix: download: properly detect https when mirror:// is used.

[Cyril Roelandt]

On 10/16/2015 12:14 PM, Ludovic Courtès wrote:
> Is there a mirror for which that is a serious issue?

I wanted to use mirrors for PyPI, but I ended up dropping the patch (see
patch 2). So I'll drop this one as well.

Cyril.

Re: [PATCH 01/12] guix: download: properly detect https when mirror:// is used.

[Cyril Roelandt]

On 10/20/2015 05:57 PM, Cyril Roelandt wrote:
> On 10/16/2015 12:14 PM, Ludovic Courtès wrote:
>> Is there a mirror for which that is a serious issue?
> 
> I wanted to use mirrors for PyPI, but I ended up dropping the patch (see
> patch 2). So I'll drop this one as well.
> 

Oops. I meant that I dropped the use of "mirror://" from patch 2, not
that I dropped the whole patch.

Cyril.

Re: [PATCH 03/12] import: pypi: Use "pypi-uri" instead of building the URL manually.

[Cyril Roelandt]

On 10/12/2015 11:40 PM, Cyril Roelandt wrote:
> * guix/import/pypi.scm (make-pypi-sexp): Use "pypi-uri".
> * tests/pypi.scm: Update the tests accordingly.

This patch seems trivial to me, but it would be nice to get some
feedback before I push the whole patch series :)

WDYT?

Cyril.

Re: [PATCH 03/12] import: pypi: Use "pypi-uri" instead of building the URL manually.

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (make-pypi-sexp): Use "pypi-uri".
> * tests/pypi.scm: Update the tests accordingly.

LGTM, provided ‘pypi-uri’ does not rely on mirror://, as discussed
earlier:

  https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00347.html

Sorry for the delay!

Ludo’.

Re: [PATCH 07/12] gnu: Add python-hacking.

[Leo Famulari]

On Mon, Oct 12, 2015 at 11:41:03PM +0200, Cyril Roelandt wrote:
> * gnu/packages/openstack.scm (python-hacking,
>   python2-hacking): New variables.

This package is failing on core-updates, because several of the
"special" old versions of its dependencies are failing to build:

https://hydra.gnu.org/build/1550461

Would anyone like to look into it?