bug#20402: guix cannot download via an http proxy

bug#20402: guix cannot download via an http proxy

[Joshua Randall]

[-- Attachment #1: Type: text/plain, Size: 2984 bytes --]

Package: guix
Version: 0.8.1

I am attempting to use guix from within a network that does not allow
outbound http connections except via an http proxy. I am using Guile
v2.0.11, which supports http proxies, so my expectation would be that since
I have http_proxy and https_proxy environment variables set, guix would use
the specified proxy for outbound http connections, but instead it appears
to ignore the proxy settings and attempts to contact the http server
directly, which results in a timeout.

For example, when doing a `guix pull` I got the following:
$ guix pull
starting download of `/tmp/guix-file.l1zwZ7' from `
http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz'...
ERROR: In procedure connect: Connection timed out
failed to download "/tmp/guix-file.l1zwZ7" from "
http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz"
guix pull: error: failed to download up-to-date source, exiting

It appears that Guile has had support for http proxies in the web client
package since v2.0.10, and although guix is using the http-get method from
Guile, it isn't using the open-socket-for-uri method, which is the one that
implements making a proxy connection. Instead, guix seems to have copied
and modified the code from an older version of open-socket-for-uri
into open-connection-for-uri
(
http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/download.scm?id=v0.8.1#n153)
and uses that instead. I suspect what has happened is that the Guile
version of open-socket-for-uri has added proxy support since the code was
copied into open-connection-for-uri. One fix would be to port over the
changes to open-socket-for-uri that were made in Guile 2.0.10.

However, it appears from the code comment that that the reason
open-connection-for-uri copies the functionality of open-socket-for-uri is
to avoid NSS lookups for symbolic port arguments, and it looks to me that
since version 2.0.7 of Guile, its open-socket-for-uri can be convinced not
to do NSS lookups as long as (uri-port uri) is not #f (see
http://git.savannah.gnu.org/cgit/guile.git/tree/module/web/client.scm?id=v2.0.7#n53
).

Rather than porting the new code from Guile's open-socket-for-uri, it might
make more sense to just call open-socket-for-uri with a uri that always has
a port (i.e. implement the same hard-coding for http and https in the
http-fetch function to make sure that uri has the default port set - I
notice for some reason Guile's string->uri parser does not set the port for
http and https even though it has the default ports for both set in the
code. I suppose one could use the existing post-2.0.7? test to keep calling
open-connection-for-uri for backwards compatibility with old versions of
Guile (which in any case don't have proxy support, so for my use case it
doesn't matter).

I can try to put together a patch that implements this fix, although I
haven't written scheme in quite a while, so someone else may be better
suited for it.

Cheers,

Josh.

[-- Attachment #2: Type: text/html, Size: 4367 bytes --]

bug#20402: guix cannot download via an http proxy

[Joshua Randall]

[-- Attachment #1.1: Type: text/plain, Size: 1604 bytes --]

Tags: patch

Please find a proposed patch that should fix this issue. I haven't written
scheme in many years, so please excuse me if I have failed to follow
convention in any way - I won't be offended at all if you want to refactor
it. This patch should apply cleanly on the current git master branch and
also on the v0.8.1 release.

This patch modifies http-fetch (in build/download.scm) such that it calls
Guile's open-socket-for-uri after fixing up the uri so that it always has a
port specified. I'm not sure how to test the bootstrapping NSS issue that
required open-connection-for-uri, but my expectation based on reading the
source is that this alternative should work for Guile > 2.0.7, and I've
left the original open-connection-for-uri in there for backwards
compatibility with Guile < 2.0.7. If someone can test this against a
situation known to have needed the NSS workaround, that would be great.

I've also changed the only other call to open-connection-for-uri, which is
in the probe-uri function in scripts/lint.scm - my suspicion is that won't
be an issue because I'm guessing the lint scripts are not used while
bootstrapping, so the open-socket-for-uri function will probably be fine
for the purpose of probing whether a URL is valid.

After applying this patch, guix should use the http proxy support built in
to Guile >= 2.0.10. This appears to be working for me - I've just started a
bootstrap and it has now started to successfully download packages - so
far, so good.

Please let me know if there is anything else I can do to assist with
getting this issue fixed.

Cheers,

Josh.

[-- Attachment #1.2: Type: text/html, Size: 1864 bytes --]

[-- Attachment #2: guix-git-dd3a42e6-http-proxy-fix.patch --]
[-- Type: application/octet-stream, Size: 2986 bytes --]

diff --git a/guix/build/download.scm b/guix/build/download.scm
index a3105ad..ce16092 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -349,7 +349,30 @@ Return the resulting target URI."
       (Accept . "*/*")))
 
   (let*-values (((connection)
-                 (open-connection-for-uri uri))
+                 (if post-2.0.7?
+                     ;; open-socket-for-uri in Guile > 2.0.7 won't do an
+                     ;; NSS lookup if we specify a port in the uri, so we
+                     ;; can force the port to the uri-scheme default here
+                     ;; and then call open-socket-for-uri, which as of
+                     ;; Guile 2.0.10 also supports proxies
+                     (let ((uri (build-uri
+                                 (uri-scheme uri)
+                                 #:userinfo (uri-userinfo uri)
+                                 #:host (uri-host uri)
+                                 #:port (or (uri-port uri)
+                                            (case (uri-scheme uri)
+                                              ((http) 80)
+                                              ((https) 443)
+                                              (else
+                                               (error "unsupported URI scheme"
+                                                      (uri-scheme uri)))))
+                                 #:path (uri-path uri)
+                                 #:query (uri-query uri)
+                                 #:fragment (uri-fragment uri))))
+                       (open-socket-for-uri uri))
+                     ;; Guile older than 2.0.7 still needs our modified
+                     ;; open-connection-for-uri for bootstrapping
+                     (open-connection-for-uri uri)))
                 ((resp bv-or-port)
                  ;; XXX: `http-get*' was introduced in 2.0.7, and replaced by
                  ;; #:streaming? in 2.0.8.  We know we're using it within the
diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm
index cced1bd..601b6c6 100644
--- a/guix/scripts/lint.scm
+++ b/guix/scripts/lint.scm
@@ -36,8 +36,9 @@
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
   #:use-module ((guix build download)
-                #:select (maybe-expand-mirrors
-                          open-connection-for-uri))
+                #:select (maybe-expand-mirrors))
+  #:use-module ((web client)
+                #:select (open-socket-for-uri))
   #:use-module (web request)
   #:use-module (web response)
   #:use-module (srfi srfi-1)
@@ -232,7 +233,7 @@ response from URI, and additional details, such as the actual HTTP response."
       ((or 'http 'https)
        (catch #t
          (lambda ()
-           (let ((port    (open-connection-for-uri uri))
+           (let ((port    (open-socket-for-uri uri))
                  (request (build-request uri #:headers headers)))
              (define response
                (dynamic-wind

bug#20402: guix cannot download via an http proxy

[Joshua Randall]

[-- Attachment #1: Type: text/plain, Size: 2516 bytes --]

After more testing this morning, I see that although the core functionality
is working after applying the proposed patch (i.e. the guix daemon itself
can download things), the derivations are still failing because the
environment is being squashed and so the http_proxy and https_proxy
environment variables are not available to the build. It looks like nix has
a way around this involving setting imputeEnvVars = "http_proxy
https_proxy" so that those variables get copied into the build environment
(for fixed-output derivations only, which makes sense to me).

I'll try to come up with a patch that inserts the imputeEnvVars, but I'm
getting farther from the code I understand so I will probably need some
help in getting it in the right place.

Cheers,

Josh.


On Wed, Apr 22, 2015 at 2:15 AM, Joshua Randall <jcrandall@alum.mit.edu>
wrote:

> Tags: patch
>
> Please find a proposed patch that should fix this issue. I haven't written
> scheme in many years, so please excuse me if I have failed to follow
> convention in any way - I won't be offended at all if you want to refactor
> it. This patch should apply cleanly on the current git master branch and
> also on the v0.8.1 release.
>
> This patch modifies http-fetch (in build/download.scm) such that it calls
> Guile's open-socket-for-uri after fixing up the uri so that it always has a
> port specified. I'm not sure how to test the bootstrapping NSS issue that
> required open-connection-for-uri, but my expectation based on reading the
> source is that this alternative should work for Guile > 2.0.7, and I've
> left the original open-connection-for-uri in there for backwards
> compatibility with Guile < 2.0.7. If someone can test this against a
> situation known to have needed the NSS workaround, that would be great.
>
> I've also changed the only other call to open-connection-for-uri, which is
> in the probe-uri function in scripts/lint.scm - my suspicion is that won't
> be an issue because I'm guessing the lint scripts are not used while
> bootstrapping, so the open-socket-for-uri function will probably be fine
> for the purpose of probing whether a URL is valid.
>
> After applying this patch, guix should use the http proxy support built in
> to Guile >= 2.0.10. This appears to be working for me - I've just started a
> bootstrap and it has now started to successfully download packages - so
> far, so good.
>
> Please let me know if there is anything else I can do to assist with
> getting this issue fixed.
>
> Cheers,
>
> Josh.
>
>

[-- Attachment #2: Type: text/html, Size: 3116 bytes --]

bug#20402: updated patch to pass http_proxy and https_proxy env vars to derivation builds

[Joshua Randall]

[-- Attachment #1.1: Type: text/plain, Size: 511 bytes --]

Tags: patch

I've updated my patch to also modify guix/derivations.scm to add a setting
for nix's impureEnvVars, such that it will pass http_proxy and https_proxy
variables from the guix daemon to the build processes (which nix will do
only for fixed-output derivations).

I believe this is working, but I've now run into a third problem: the
bootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxy
support. What would I need to do to use a later version of bootstrap guile?

Cheers,

Josh.

[-- Attachment #1.2: Type: text/html, Size: 660 bytes --]

[-- Attachment #2: guix-git-dd3a42e6-http-proxy-impure-fix.patch --]
[-- Type: application/octet-stream, Size: 3551 bytes --]

diff --git a/guix/build/download.scm b/guix/build/download.scm
index a3105ad..ce16092 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -349,7 +349,30 @@ Return the resulting target URI."
       (Accept . "*/*")))
 
   (let*-values (((connection)
-                 (open-connection-for-uri uri))
+                 (if post-2.0.7?
+                     ;; open-socket-for-uri in Guile > 2.0.7 won't do an
+                     ;; NSS lookup if we specify a port in the uri, so we
+                     ;; can force the port to the uri-scheme default here
+                     ;; and then call open-socket-for-uri, which as of
+                     ;; Guile 2.0.10 also supports proxies
+                     (let ((uri (build-uri
+                                 (uri-scheme uri)
+                                 #:userinfo (uri-userinfo uri)
+                                 #:host (uri-host uri)
+                                 #:port (or (uri-port uri)
+                                            (case (uri-scheme uri)
+                                              ((http) 80)
+                                              ((https) 443)
+                                              (else
+                                               (error "unsupported URI scheme"
+                                                      (uri-scheme uri)))))
+                                 #:path (uri-path uri)
+                                 #:query (uri-query uri)
+                                 #:fragment (uri-fragment uri))))
+                       (open-socket-for-uri uri))
+                     ;; Guile older than 2.0.7 still needs our modified
+                     ;; open-connection-for-uri for bootstrapping
+                     (open-connection-for-uri uri)))
                 ((resp bv-or-port)
                  ;; XXX: `http-get*' was introduced in 2.0.7, and replaced by
                  ;; #:streaming? in 2.0.8.  We know we're using it within the
diff --git a/guix/derivations.scm b/guix/derivations.scm
index 7737e39..74f4e52 100644
--- a/guix/derivations.scm
+++ b/guix/derivations.scm
@@ -751,6 +751,7 @@ derivations where the costs of data transfers would outweigh the benefits."
                             `(("allowedReferences"
                                . ,(string-join allowed-references)))
                             '())
+                      ,@`(("impureEnvVars" . "http_proxy https_proxy"))
                       ,@env-vars)))
       (match references-graphs
         (((file . path) ...)
diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm
index cced1bd..601b6c6 100644
--- a/guix/scripts/lint.scm
+++ b/guix/scripts/lint.scm
@@ -36,8 +36,9 @@
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
   #:use-module ((guix build download)
-                #:select (maybe-expand-mirrors
-                          open-connection-for-uri))
+                #:select (maybe-expand-mirrors))
+  #:use-module ((web client)
+                #:select (open-socket-for-uri))
   #:use-module (web request)
   #:use-module (web response)
   #:use-module (srfi srfi-1)
@@ -232,7 +233,7 @@ response from URI, and additional details, such as the actual HTTP response."
       ((or 'http 'https)
        (catch #t
          (lambda ()
-           (let ((port    (open-connection-for-uri uri))
+           (let ((port    (open-socket-for-uri uri))
                  (request (build-request uri #:headers headers)))
              (define response
                (dynamic-wind

bug#20402: guix cannot download via an http proxy

[Ludovic Courtès]

Hi,

Sorry for the delay, and thanks for the investigation and patch!

Joshua Randall <jcrandall@alum.mit.edu> skribis:

> This patch modifies http-fetch (in build/download.scm) such that it calls
> Guile's open-socket-for-uri after fixing up the uri so that it always has a
> port specified. I'm not sure how to test the bootstrapping NSS issue that
> required open-connection-for-uri, but my expectation based on reading the
> source is that this alternative should work for Guile > 2.0.7, and I've
> left the original open-connection-for-uri in there for backwards
> compatibility with Guile < 2.0.7. If someone can test this against a
> situation known to have needed the NSS workaround, that would be great.

To name lookup with the bootstrap Guile, one way is to run this:

  $ guix gc -d $(guix build -S -e '(@@ (gnu packages commencement) glibc-final)')
  $ ./pre-inst-env guix build -S \
     -e '(@@ (gnu packages commencement) glibc-final)' --no-substitutes

The second command here uses the bootstrap Guile.

Another approach is this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(gnu packages bootstrap)
scheme@(guile-user)> ,enter-store-monad
store-monad@(guile-user) [1]> (mlet %store-monad ((guile (package->derivation %bootstrap-guile)))
				(gexp->derivation "foo"
						  #~(begin
						      (mkdir #$output)
						      (pk (getaddrinfo "www.gnu.org" "http")))
						  #:hash-algo 'sha256
						  #:hash (base32 "0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73")
						  #:guile-for-build guile))
$5 = #<derivation /gnu/store/5prfiblj6ddziavk9nz31bkgy8jzaghx-foo.drv => /gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo 2ae7960>
store-monad@(guile-user) [1]> (built-derivations (list $5))
building path(s) `/gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo'

;;; ((#(0 2 1 6 #(2 3497454484 80) #f) #(0 2 2 17 #(2 3497454484 80) #f) #(0 10 1 6 #(10 42541952298791455573290623124440612874 80 0 0) #f) #(0 10 2 17 #(10 42541952298791455573290623124440612874 80 0 0) #f)))
--8<---------------cut here---------------end--------------->8---

Here ‘built-derivations’ fails but the build log shows that
‘getaddrinfo’ succeeded.

Lastly, one can extract
gnu/packages/bootstrap/x86_64-linux/guile-2.0.9.tar.xz and run:

  strace -o log ./bin/guile -c '(getaddrinfo "www.gnu.org" "http")'

Here the log shows that /etc/nsswitch.conf, /etc/services, and
/etc/hosts are accessed and things just work; it does not attempt to
connect to the nscd.

A bit of archeology shows the following timeline:

  1. d14ecda introduces the ‘open-connection-for-uri’ hack (Oct. 2012).

  2. d3b5972 changes libc used to make bootstrap tarballs to use static
     NSS modules (Jan. 2013).

  3. 0621349 updates the bootstrap guile-2.0.9.tar.xz tarballs
     (Nov. 2013), meaning that our current bootstrap Guile indeed uses
     static NSS modules and doesn’t attempt to talk to nscd.

In other words, the hack is no longer needed.

Thus, ‘open-connection-for-uri’ is almost (see below) unneeded now,
which simplifies the solution to the problem you raise.

> I've also changed the only other call to open-connection-for-uri, which is
> in the probe-uri function in scripts/lint.scm - my suspicion is that won't
> be an issue because I'm guessing the lint scripts are not used while
> bootstrapping, so the open-socket-for-uri function will probably be fine
> for the purpose of probing whether a URL is valid.

‘open-connection-for-uri’ also handles TLS connections, which are also
useful for ‘guix lint’, so we cannot completely get rid of it.

Commit d17551d simplifies it so that it is just a wrapper around
‘open-socket-for-uri’.  After that, ‘guix download’ honors $http_proxy.

$https_proxy is not handled yet because that requires more work, and I
do not fully understand how that is supposed to work.  Patch welcome,
though.  :-)

Thanks!

Ludo’.

bug#20402: updated patch to pass http_proxy and https_proxy env vars to derivation builds

[Ludovic Courtès]

Joshua Randall <jcrandall@alum.mit.edu> skribis:

> I've updated my patch to also modify guix/derivations.scm to add a setting
> for nix's impureEnvVars, such that it will pass http_proxy and https_proxy
> variables from the guix daemon to the build processes (which nix will do
> only for fixed-output derivations).

I took a slightly different approach in commit c046815 (and also not
passing “impureEnvVars” by default since that would lead to a full
rebuild.)

With commit 0d88313, $http_proxy is honored, but note that it’s the
$http_proxy value in the daemon’s environment that is used, not in the
client’s environment.

> I believe this is working, but I've now run into a third problem: the
> bootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxy
> support. What would I need to do to use a later version of bootstrap guile?

We will need to update the bootstrap Guile, but that won’t happen until
a later full-rebuild cycle.

You should be able to use substitutes as a workaround in the meantime.
How does that sound?

Thank you!

Ludo’.

bug#20402: guix cannot download via an http proxy

[Claes Wallin]

Would support for $ftp_proxy as well require the implementation of new
functionality, or would it be a case of just letting the environment
variable through?

bug#20402: guix cannot download via an http proxy

[Mark H Weaver]

Claes Wallin (韋嘉誠) <gnu@clacke.user.lysator.liu.se> writes:

> Would support for $ftp_proxy as well require the implementation of new
> functionality, or would it be a case of just letting the environment
> variable through?

It would require new functionality.  Our FTP client would have to be
enhanced to support proxies.

      Mark