Problem with pkgconfig source https redirect

Problem with pkgconfig source https redirect

[Christopher W Carpenter]

[-- Attachment #1: Type: text/plain, Size: 7730 bytes --]


Talked about this on IRC davexunit, but there appears to be a circular dependency
issue with gnutls and pkg-config now that pkg-config needs to be
downloaded via https.

If I do a

guix build pkg-config --check --no-substitutes

on master right now i get:

Starting download of /gnu/store/87mxipb93cxhg4wy13wa8wk9mjsxd937-pkg-config-0.29.tar.gz
From http://pkgconfig.freedesktop.org/releases/pkg-config-0.29.tar.gz...
following redirection to `https://pkg-config.freedesktop.org/releases/pkg-config-0.29.tar.gz'...
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
ERROR: In procedure module-lookup: Unbound variable: make-session
failed to download "/gnu/store/87mxipb93cxhg4wy13wa8wk9mjsxd937-pkg-config-0.29.tar.gz" from "http://pkgconfig.freedesktop.org/releases/pkg-config-0.29.tar.gz"
builder for `/gnu/store/yp6432nfcwblxwvmm3kmsg6zdgqfyc9p-pkg-config-0.29.tar.gz.drv' failed to produce output path `/gnu/store/87mxipb93cxhg4wy13wa8wk9mjsxd937-pkg-config-0.29.tar.gz'
@ build-failed /gnu/store/yp6432nfcwblxwvmm3kmsg6zdgqfyc9p-pkg-config-0.29.tar.gz.drv - 1 builder for `/gnu/store/yp6432nfcwblxwvmm3kmsg6zdgqfyc9p-pkg-config-0.29.tar.gz.drv' failed to produce output path `/gnu/store/87mxipb93cxhg4wy13wa8wk9mjsxd937-pkg-config-0.29.tar.gz'
cannot build derivation `/gnu/store/ns04f833g7r72gbaxlcakjb4n69iw2fw-pkg-config-0.29.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/ns04f833g7r72gbaxlcakjb4n69iw2fw-pkg-config-0.29.drv' failed


If I change the url to the redirection url (see attached patch) I get:

Backtrace:
In guix/gexp.scm:
 402: 19 [#<procedure 3fdbd00 at guix/gexp.scm:402:4 (state)> #]
 405: 18 [#<procedure 3b44bd0 at guix/gexp.scm:405:22 (state)> #]
In guix/store.scm:
1001: 17 [#<procedure 3b44c00 at guix/store.scm:1000:28 (store)> #]
In guix/packages.scm:
 733: 16 [cache # # #<procedure thunk ()>]
 978: 15 [thunk]
 910: 14 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 13 [map #<procedure 3b44b10 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 12 [expand-input # # # ...]
In guix/store.scm:
1061: 11 [run-with-store # ...]
In guix/download.scm:
 271: 10 [#<procedure 38a4840 at guix/download.scm:271:8 (state)> #]
In guix/store.scm:
1001: 9 [#<procedure 3b44990 at guix/store.scm:1000:28 (store)> #]
In guix/packages.scm:
 733: 8 [cache # # #<procedure thunk ()>]
 978: 7 [thunk]
 910: 6 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 5 [map #<procedure 3b44900 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 4 [expand-input # # # ...]
In guix/store.scm:
1008: 3 [#<procedure f73140 at guix/store.scm:1007:26 (store . args)> # # ...]
In guix/packages.scm:
1080: 2 [origin->derivation # "x86_64-linux"]
In guix/download.scm:
 239: 1 [url-fetch "mirror://gnu/guile/guile-2.0.11.tar.xz" sha256 ...]
In unknown file:
   ?: 0 [string-prefix? "https://" "mirror://gnu/guile/guile-2.0.11.tar.xz" ...]

ERROR: In procedure string-prefix?:
ERROR: Throw to key `vm-error' with args `(vm-run "VM: Stack overflow" ())'.
Backtrace:
In guix/store.scm:
1001: 19 [#<procedure 3b44c00 at guix/store.scm:1000:28 (store)> #]
In guix/packages.scm:
 733: 18 [cache # # #<procedure thunk ()>]
 978: 17 [thunk]
 910: 16 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 15 [map #<procedure 3b44b10 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 14 [expand-input # # # ...]
In guix/store.scm:
1061: 13 [run-with-store # ...]
In guix/download.scm:
 271: 12 [#<procedure 38a4840 at guix/download.scm:271:8 (state)> #]
In guix/store.scm:
1001: 11 [#<procedure 3b44990 at guix/store.scm:1000:28 (store)> #]
In guix/packages.scm:
 733: 10 [cache # # #<procedure thunk ()>]
 978: 9 [thunk]
 910: 8 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 7 [map #<procedure 3b44900 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 6 [expand-input # # # ...]
In guix/store.scm:
1008: 5 [#<procedure f73140 at guix/store.scm:1007:26 (store . args)> # # ...]
In guix/packages.scm:
1080: 4 [origin->derivation # "x86_64-linux"]
In guix/download.scm:
 239: 3 [url-fetch "mirror://gnu/guile/guile-2.0.11.tar.xz" sha256 ...]
In unknown file:
   ?: 2 [string-prefix? "https://" "mirror://gnu/guile/guile-2.0.11.tar.xz" ...]
In ice-9/boot-9.scm:
 105: 1 [#<procedure 83b180 at ice-9/boot-9.scm:97:6 (thrown-k . args)> vm-error ...]
  65: 0 [abort-to-prompt catch7 vm-error vm-run "VM: Stack overflow" ()]

ice-9/boot-9.scm:65:2: In procedure abort-to-prompt:
ice-9/boot-9.scm:65:2: Throw to key `vm-error' with args `(vm-run "VM: Stack overflow" ())'.
Backtrace:
In guix/packages.scm:
 978: 19 [thunk]
 910: 18 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 17 [map #<procedure 3b44b10 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 16 [expand-input # # # ...]
In guix/store.scm:
1061: 15 [run-with-store # ...]
In guix/download.scm:
 271: 14 [#<procedure 38a4840 at guix/download.scm:271:8 (state)> #]
In guix/store.scm:
1001: 13 [#<procedure 3b44990 at guix/store.scm:1000:28 (store)> #]
In guix/packages.scm:
 733: 12 [cache # # #<procedure thunk ()>]
 978: 11 [thunk]
 910: 10 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 9 [map #<procedure 3b44900 at guix/packages.scm:912:30 (t-887996)> #]
In guix/packages.scm:
 790: 8 [expand-input # # # ...]
In guix/store.scm:
1008: 7 [#<procedure f73140 at guix/store.scm:1007:26 (store . args)> # # ...]
In guix/packages.scm:
1080: 6 [origin->derivation # "x86_64-linux"]
In guix/download.scm:
 239: 5 [url-fetch "mirror://gnu/guile/guile-2.0.11.tar.xz" sha256 ...]
In unknown file:
   ?: 4 [string-prefix? "https://" "mirror://gnu/guile/guile-2.0.11.tar.xz" ...]
In ice-9/boot-9.scm:
 105: 3 [#<procedure 83b180 at ice-9/boot-9.scm:97:6 (thrown-k . args)> vm-error ...]
  65: 2 [abort-to-prompt catch7 vm-error vm-run "VM: Stack overflow" ()]
 105: 1 [#<procedure 83b180 at ice-9/boot-9.scm:97:6 (thrown-k . args)> vm-error ...]
  65: 0 [abort-to-prompt catch7 vm-error vm-run "VM: Stack overflow" ()]

ice-9/boot-9.scm:65:2: In procedure abort-to-prompt:
ice-9/boot-9.scm:65:2: Throw to key `vm-error' with args `(vm-run "VM: Stack overflow" ())'.

davexunit on #guix mentioned that this may need to be solved using a
minimal build of gnutls that doesn't require pkg-config for https
downloads but i'm a bit out of my depth here so I'm just reporting :).

I am happy to do some grunt work once the project decides what direction
to go.

Thanks,
Christopher Carpenter


===File
/home/mordocai/git_repos/guix/0001-Fix-pkg-config-source-url-for-redirection.patch===
From ec3fd8fecac1d76fe429915f85556be937bc6ec8 Mon Sep 17 00:00:00 2001
From: "Christopher W. Carpenter (https://mordocai.net/gpg.html)"
 <mordocai@mordocai.net>
Date: Tue, 9 Feb 2016 20:52:38 -0600
Subject: [PATCH] Fix pkg-config source url for redirection

---
 gnu/packages/pkg-config.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/pkg-config.scm b/gnu/packages/pkg-config.scm
index 5923395..a6be228 100644
--- a/gnu/packages/pkg-config.scm
+++ b/gnu/packages/pkg-config.scm
@@ -34,7 +34,7 @@
    (source (origin
             (method url-fetch)
             (uri (string-append
-                  "http://pkgconfig.freedesktop.org/releases/pkg-config-"
+                  "https://pkg-config.freedesktop.org/releases/pkg-config-"
                   version ".tar.gz"))
             (sha256
              (base32
-- 
2.7.0

============================================================

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Problem with pkgconfig source https redirect

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Tue, 09 Feb 2016 20:56:59 -0600
Christopher W Carpenter <mordocai@mordocai.net> wrote:

> Talked about this on IRC davexunit, but there appears to be a circular dependency
> issue with gnutls and pkg-config now that pkg-config needs to be
> downloaded via https.
> 
[snip]
> 
> davexunit on #guix mentioned that this may need to be solved using a
> minimal build of gnutls that doesn't require pkg-config for https
> downloads but i'm a bit out of my depth here so I'm just reporting :).
> 
> I am happy to do some grunt work once the project decides what direction
> to go.
> 
> Thanks,
> Christopher Carpenter
> 
 
Another option is to try to build gnutls without pkg-config and as part of
the configure phase manually tell it where to find the dependencies. IIRC we
do that with some of the less complicated lower level packages.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Problem with pkgconfig source https redirect

[Thompson, David]

On Wed, Feb 10, 2016 at 2:43 AM, Efraim Flashner <efraim@flashner.co.il> wrote:
> On Tue, 09 Feb 2016 20:56:59 -0600
> Christopher W Carpenter <mordocai@mordocai.net> wrote:
>
>> Talked about this on IRC davexunit, but there appears to be a circular dependency
>> issue with gnutls and pkg-config now that pkg-config needs to be
>> downloaded via https.
>>
> [snip]
>>
>> davexunit on #guix mentioned that this may need to be solved using a
>> minimal build of gnutls that doesn't require pkg-config for https
>> downloads but i'm a bit out of my depth here so I'm just reporting :).
>>
>> I am happy to do some grunt work once the project decides what direction
>> to go.
>>
>> Thanks,
>> Christopher Carpenter
>>
>
> Another option is to try to build gnutls without pkg-config and as part of
> the configure phase manually tell it where to find the dependencies. IIRC we
> do that with some of the less complicated lower level packages.

The bigger problem to be aware of is this:  No package in the gnutls
dependency graph may have its source code downloaded over HTTPS.  Even
if we hack around this for pkg-config, I'm sure it will bite us again
when another upstream starts enforcing HTTPS.

So, what can we do here?

- Dave

Re: Problem with pkgconfig source https redirect

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> The bigger problem to be aware of is this:  No package in the gnutls
> dependency graph may have its source code downloaded over HTTPS.  Even
> if we hack around this for pkg-config, I'm sure it will bite us again
> when another upstream starts enforcing HTTPS.
>
> So, what can we do here?

Nix recently added a ‘fetchurl’ primitive to the Nix language, in part
to address this problem.

The equivalent for us is to simply perform the download on the “host
side” rather than on the “build side”, thus entirely side-stepping the
issue.  Moving code from one side to the other is obviously easy for us.

One way to do that is by adding a new origin method, along the lines of
this incomplete patch:


[-- Attachment #2: Type: text/x-patch, Size: 2059 bytes --]

diff --git a/gnu/packages/pkg-config.scm b/gnu/packages/pkg-config.scm
index 5923395..299c7c8 100644
--- a/gnu/packages/pkg-config.scm
+++ b/gnu/packages/pkg-config.scm
@@ -32,7 +32,7 @@
    (name "pkg-config")
    (version "0.29")
    (source (origin
-            (method url-fetch)
+            (method host-url-fetch)
             (uri (string-append
                   "http://pkgconfig.freedesktop.org/releases/pkg-config-"
                   version ".tar.gz"))
diff --git a/guix/download.scm b/guix/download.scm
index 204cfc0..32b5e4d 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2015 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013, 2014, 2015 Andreas Enge <andreas@enge.fr>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -31,6 +31,7 @@
   #:use-module (srfi srfi-26)
   #:export (%mirrors
             url-fetch
+            host-url-fetch
             download-to-store))
 
 ;;; Commentary:
@@ -294,6 +295,12 @@ in the store."
                             ;; <https://bugs.gnu.org/18747>.)
                             #:local-build? #t)))))
 
+(define* (host-url-fetch url hash-algo hash
+                         #:optional name)
+  ;; FIXME: Check HASH, and cache downloaded stuff in ~/.cache/guix, similar
+  ;; to what 'http-fetch/cached' does.  See 'downloadFileCached' in Nix.
+  (download-to-store* url name))
+
 (define* (download-to-store store url #:optional (name (basename url))
                             #:key (log (current-error-port)) recursive?)
   "Download from URL to STORE, either under NAME or URL's basename if
@@ -314,4 +321,7 @@ the same-named parameter of 'add-to-store'."
            (and result
                 (add-to-store store name recursive? "sha256" temp)))))))
 
+(define download-to-store*
+  (store-lift download-to-store))
+
 ;;; download.scm ends here

[-- Attachment #3: Type: text/plain, Size: 147 bytes --]


Some care is needed to get performance right and to make sure we never
needlessly re-download stuff, but it’s definitely doable.

Ludo’.