bug#20402: HTTP and FTP proxies for guix package / guix-daemon

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Ludovic Courtès]

(Please always keep the bug email address Cc’d.)

"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:

> I suppose if I have a freshly built guix with "guix package -i guix"
> or "guix package -u guix" it should be using guile 2.0.11 then?
> Actually looking at the head of .../bin/guix it's referring to a
> guile-2.0.11.

Yes, so that should be OK.  And it’s Guix > 0.8.2, right?

> But guix-daemon still doesn't seem to be honoring http_proxy.

As a first test, could you check whether ‘guix download’ honors it?

A second test would be:

  1. Define ‘http_proxy’ in your user shell.
  2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.

Does that honor ‘http_proxy’?

And a third test is for substitutes: Are they also downloaded via the
proxy?  Here you need to:

  1. Define ‘http_proxy’ in a root shell.
  2. Run ‘guix-daemon’ from this shell.
  3. From a user shell, run ‘guix build gcc’, say, which should normally
     download a substitute from hydra.gnu.org rather than trying to build
     things.

Does the substitute download honor ‘http_proxy’?

Thanks in advance!

Ludo’.

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Claes Wallin]

Finally got around to this.

On Sat, Jun 13, 2015 at 7:44 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> "Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
>
>> I suppose if I have a freshly built guix with "guix package -i guix"
>> or "guix package -u guix" it should be using guile 2.0.11 then?
>> Actually looking at the head of .../bin/guix it's referring to a
>> guile-2.0.11.
>
> Yes, so that should be OK.  And it’s Guix > 0.8.2, right?

# /root/.guix-profile/bin/guix --version
guix (GNU Guix) 0.8.3

# ls -l /root/.guix-profile/bin/guix
lrwxrwxrwx 3 root 999 71 Jan  1  1970 /root/.guix-profile/bin/guix ->
/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f/bin/guix


>> But guix-daemon still doesn't seem to be honoring http_proxy.
>
> As a first test, could you check whether ‘guix download’ honors it?

It does.

> A second test would be:
>
>   1. Define ‘http_proxy’ in your user shell.
>   2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.
>
> Does that honor ‘http_proxy’?

http_proxy is set (and exported) in user shell and in the daemon process.

No.

> And a third test is for substitutes: Are they also downloaded via the
> proxy?  Here you need to:
>
>   1. Define ‘http_proxy’ in a root shell.
>   2. Run ‘guix-daemon’ from this shell.
>   3. From a user shell, run ‘guix build gcc’, say, which should normally
>      download a substitute from hydra.gnu.org rather than trying to build
>      things.
>
> Does the substitute download honor ‘http_proxy’?

No. Whenever the daemon is doing the downloading, http_proxy is
ignored and I get "Temporary failure in name resolution" because the
VM doesn't have an internet DNS.

As I said earlier, guix has guile-2.0.11 in the hashbang. But
guix-daemon seems to be making use of guile-bootstrap.

If I provoke it a bit I get this:

# find /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/
-name '*.scm' -exec touch {} +
# guix package -i texinfo
 . . .
;;; note: source file
/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/eval.scm
;;;       newer than compiled
/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/lib/guile/2.0/ccache/ice-9/eval.go
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
. . .
;;; it seems /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/receive.scm
;;; is part of the compiler; skipping auto-compilation
;;; WARNING: compilation of
/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scm
failed:
;;; ERROR: failed to create path for auto-compiled file
"/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scm"
starting download of
`/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'
from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...
ERROR: In procedure getaddrinfo: Temporary failure in name resolution

... so it's pretty clear that *something* in there is still
referencing the bootstrap guile. But maybe that's expected? Are the
guile packages built using only the bootstrap and not themselves?

/gnu/store/0aq1kq37b6kr562330xz61rg08l6y5yg-guix-0.8.2.a43b55f.drv

has:
Derive([("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f","","")]
. . . ("/gnu/store/4lkrvcnh2y923la3ivsbyv2m2nv3xpjy-guile-2.0.9.tar.xz.drv",["out"])
. . . ("/gnu/store/d3x2skfxhvqj21w39f6bckmxni1q3bbp-guile-2.0.11.tar.xz.drv",["out"])
. . . ("/gnu/store/hvl3gwa57p7k1qnliyy6ymbmz2509azh-guile-2.0.11.drv",["out"])
. . . ("/gnu/store/p4i1inysxx5ra0hvhcrd9liw3a5skcbx-guile-2.0.9.tar.xz.drv",["out"])
. . . ("/gnu/store/wlivhx78f1a05dx0blqy209cb1nqwgf5-guile-2.0.9.tar.xz.drv",["out"])
. . . ["/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],"x86_64-linux","/gnu/store/cnqmkmj40jmssnx6fkf9n0n3bqj5x426-guile-2.0.11/bin/guile",["--no-auto-compile","-L","/gnu/store/x2nfxrqhdsvir8l65x19bhnrwd4h41rs-module-import","/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],[("GUILE_LOAD_COMPILED_PATH","/gnu/store/07n184fk33b7gqkyrbn6b2ckdfisp09i-module-import-compiled"),("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f")])

Why so many different guile drivers? Is that expected?

> Thanks in advance!

No problem!

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Claes Wallin]

On Wed, Jun 17, 2015 at 6:46 PM, Claes Wallin (韋嘉誠)
<gnu@clacke.user.lysator.liu.se> wrote:

> Why so many different guile drivers? Is that expected?


... derivations. Sorry.

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Ludovic Courtès]

"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:

>>> But guix-daemon still doesn't seem to be honoring http_proxy.
>>
>> As a first test, could you check whether ‘guix download’ honors it?
>
> It does.
>
>> A second test would be:
>>
>>   1. Define ‘http_proxy’ in your user shell.
>>   2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say.
>>
>> Does that honor ‘http_proxy’?
>
> http_proxy is set (and exported) in user shell and in the daemon process.

Oops, my instructions were incorrect: ‘http_proxy’ needs to be defined
in the daemon process.

To test, I first run the daemon like this:

  sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'

and then I run:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build -S gcc --no-substitutes
accepted connection from pid 14399, user ludo
La jenaj derivoj estos konstruataj:
   /gnu/store/s32s498y88sidfnrppsn33nfmhlacpzx-gcc-5.1.0.tar.xz.drv
   /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv
@ build-started /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv - x86_64-linux /var/log/guix/drvs/z1//n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv.bz2
starting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `http://ftpmirror.gnu.org/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'...
ERROR: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
starting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `ftp://ftp.cs.tu-berlin.de/pub/gnu/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'...
ftp://ftp.cs.tu-berlin.de/.../gcc-5.1.0.tar.bz2	  3.9% of 92728.9 KiB (610. KiB/s)  C-c C-cSIGPOLL
--8<---------------cut here---------------end--------------->8---

Here the error message comes from the fact that $http_proxy is a
malformed URL, showing that it *is* honored.  Since the http URL fails,
the download logic falls back to the next URL, which happens to be an
ftp URL, which is unaffected by ‘http_proxy’.

So it’s working as expected here.

>> And a third test is for substitutes: Are they also downloaded via the
>> proxy?  Here you need to:
>>
>>   1. Define ‘http_proxy’ in a root shell.
>>   2. Run ‘guix-daemon’ from this shell.
>>   3. From a user shell, run ‘guix build gcc’, say, which should normally
>>      download a substitute from hydra.gnu.org rather than trying to build
>>      things.
>>
>> Does the substitute download honor ‘http_proxy’?

This test also works as expected here:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build -S inkscape
accepted connection from pid 15118, user ludo
La jena derivo estos konstruata:
   /gnu/store/fa6ss1ljwb97ns75332nshskj2bv0dh4-inkscape-0.91.tar.gz.drv
La jenaj derivoj estos elŝutataj:
   /gnu/store/acmzy3nkms38g6vrphh07bczf5s5mbjd-gnutls-3.4.1
   /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1
@ substituter-started /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 /home/ludo/src/guix/nix/scripts/substitute
found valid signature for '/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1', from 'http://hydra.gnu.org/nar/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1'
downloading `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' (1.7 MiB installed)...

[...]

In guix/scripts/substitute.scm:
 182: 2 [#<procedure 1523210 at guix/scripts/substitute.scm:165:9 ()>]
In guix/build/download.scm:
 212: 1 [open-socket-for-uri #]
In web/client.scm:
  73: 0 [open-socket-for-uri #]

web/client.scm:73:16: In procedure open-socket-for-uri:
web/client.scm:73:16: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
@ substituter-failed /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 256 fetching path `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' failed with exit code 1
guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/2v6lls3f8dj1rkpram4jnl8ckqs77r6m-gnutls-3.4.1.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
--8<---------------cut here---------------end--------------->8---

The backtrace again stems from the bogus $http_proxy that I chose (not
nice, but shows that $http_proxy is honored.)

Please let me know if you have evidence that something works differently
for you.

> No. Whenever the daemon is doing the downloading, http_proxy is
> ignored and I get "Temporary failure in name resolution" because the
> VM doesn't have an internet DNS.

[...]

> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'
> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...
> ERROR: In procedure getaddrinfo: Temporary failure in name resolution

That’s another, unrelated issue, probably in your network or on your
machine.

Thanks,
Ludo’.

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Claes Wallin]

On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> To test, I first run the daemon like this:
>
>   sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'

Could there be a difference because you are testing with the
checked-out, semi-manually built guix and I was checking with a
freshly packaged (pull, package -i guix) guix? Like I said in
http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html ,
guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe
that only happens due to guix rewriting hashbangs etc, and on a build
outside guix (even when using guix environment) it uses whatever guile
happens to be in the paths?

When I have the time again I will check it exactly the way you are doing it.


> Here the error message comes from the fact that $http_proxy is a
> malformed URL, showing that it *is* honored.  Since the http URL fails,
> the download logic falls back to the next URL, which happens to be an
> ftp URL, which is unaffected by ‘http_proxy’.

Ok, that's maybe a clearer way of testing it.


> Please let me know if you have evidence that something works differently
> for you.

I would say that both of us have evidence that it's working and not
working. The conclusion must be that we are not testing the exact same
program.


>> No. Whenever the daemon is doing the downloading, http_proxy is
>> ignored and I get "Temporary failure in name resolution" because the
>> VM doesn't have an internet DNS.
>
> [...]
>
>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'
>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...
>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution
>
> That’s another, unrelated issue, probably in your network or on your
> machine.

Yes. This is why the proxy is required. There is no internet DNS for
the VM, because it is on an isolated lab network.

When it does go through the proxy, like when running guix download, no
name lookup is done and everything works.

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Ludovic Courtès]

"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:

> On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> To test, I first run the daemon like this:
>>
>>   sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild'
>
> Could there be a difference because you are testing with the
> checked-out, semi-manually built guix and I was checking with a
> freshly packaged (pull, package -i guix) guix? Like I said in
> http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html ,
> guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe
> that only happens due to guix rewriting hashbangs etc, and on a build
> outside guix (even when using guix environment) it uses whatever guile
> happens to be in the paths?

Could you send the store file name of Guix that is being used?
For instance, I have:

--8<---------------cut here---------------start------------->8---
$ sudo deco status guix-daemon
Pasvorto:
Status of guix-daemon:
  It is started.
  Running value is 31316.
  It is enabled.
  Provides (guix-daemon).
  Requires (user-processes).
  Conflicts with ().
  Will be respawned.
$ ps 31316
  PID TTY      STAT   TIME COMMAND
31316 ?        Ss     0:00 /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon --build-users-group guixbu
--8<---------------cut here---------------end--------------->8---

That means I’m using commit 684bf7c of Guix, which has the fix, which
came up after the http_proxy commit (commit 0d88313.)

>>> No. Whenever the daemon is doing the downloading, http_proxy is
>>> ignored and I get "Temporary failure in name resolution" because the
>>> VM doesn't have an internet DNS.
>>
>> [...]
>>
>>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'
>>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...
>>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution
>>
>> That’s another, unrelated issue, probably in your network or on your
>> machine.
>
> Yes. This is why the proxy is required. There is no internet DNS for
> the VM, because it is on an isolated lab network.

Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to
the proxy (so its name must be available in DNS lookup), not to the
target server.  The code in (web client) goes like this:

--8<---------------cut here---------------start------------->8---
(define (open-socket-for-uri uri-or-string)
  "Return an open input/output port for a connection to URI."
  (define http-proxy (current-http-proxy))
  (define uri (ensure-uri (or http-proxy uri-or-string)))  ;← proxy
  (define addresses
    (let ((port (uri-port uri)))
      (delete-duplicates
       (getaddrinfo (uri-host uri)
                    (cond (port => number->string)
                          (else (symbol->string (uri-scheme uri))))
                    (if port
                        AI_NUMERICSERV
                        0))
       ;; ...
--8<---------------cut here---------------end--------------->8---

Do you see the same DNS issue when running:

  http_proxy=http://... guix download http://...

?

Thanks,
Ludo’.

bug#20402: HTTP and FTP proxies for guix package / guix-daemon

[Claes Wallin]

[-- Attachment #1: Type: text/plain, Size: 4340 bytes --]

Ludovic, sorry for duplicate mail again.

On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:

> > Yes. This is why the proxy is required. There is no internet DNS for
> > the VM, because it is on an isolated lab network.
>
> Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to
> the proxy (so its name must be available in DNS lookup), not to the
> target server.  The code in (web client) goes like this:

The proxy is in the DNS.

> Do you see the same DNS issue when running:
>
>   http_proxy=http://... guix download http://...

Nope. guix download works. It successfully looks up and connects to the
proxy.

When I next have the chance I will report:

1) Results with guix-compiled guix-daemon
2) Results with dev-compiled guix-daemon
3) /gnu/store hash of 1)
On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:

> "Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
>
> > On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> To test, I first run the daemon like this:
> >>
> >>   sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon
> --build-users-group=guixbuild'
> >
> > Could there be a difference because you are testing with the
> > checked-out, semi-manually built guix and I was checking with a
> > freshly packaged (pull, package -i guix) guix? Like I said in
> > http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html ,
> > guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe
> > that only happens due to guix rewriting hashbangs etc, and on a build
> > outside guix (even when using guix environment) it uses whatever guile
> > happens to be in the paths?
>
> Could you send the store file name of Guix that is being used?
> For instance, I have:
>
> --8<---------------cut here---------------start------------->8---
> $ sudo deco status guix-daemon
> Pasvorto:
> Status of guix-daemon:
>   It is started.
>   Running value is 31316.
>   It is enabled.
>   Provides (guix-daemon).
>   Requires (user-processes).
>   Conflicts with ().
>   Will be respawned.
> $ ps 31316
>   PID TTY      STAT   TIME COMMAND
> 31316 ?        Ss     0:00
> /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon
> --build-users-group guixbu
> --8<---------------cut here---------------end--------------->8---
>
> That means I’m using commit 684bf7c of Guix, which has the fix, which
> came up after the http_proxy commit (commit 0d88313.)
>
> >>> No. Whenever the daemon is doing the downloading, http_proxy is
> >>> ignored and I get "Temporary failure in name resolution" because the
> >>> VM doesn't have an internet DNS.
> >>
> >> [...]
> >>
> >>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2'
> >>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'...
> >>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution
> >>
> >> That’s another, unrelated issue, probably in your network or on your
> >> machine.
> >
> > Yes. This is why the proxy is required. There is no internet DNS for
> > the VM, because it is on an isolated lab network.
>
> Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to
> the proxy (so its name must be available in DNS lookup), not to the
> target server.  The code in (web client) goes like this:
>
> --8<---------------cut here---------------start------------->8---
> (define (open-socket-for-uri uri-or-string)
>   "Return an open input/output port for a connection to URI."
>   (define http-proxy (current-http-proxy))
>   (define uri (ensure-uri (or http-proxy uri-or-string)))  ;← proxy
>   (define addresses
>     (let ((port (uri-port uri)))
>       (delete-duplicates
>        (getaddrinfo (uri-host uri)
>                     (cond (port => number->string)
>                           (else (symbol->string (uri-scheme uri))))
>                     (if port
>                         AI_NUMERICSERV
>                         0))
>        ;; ...
> --8<---------------cut here---------------end--------------->8---
>
> Do you see the same DNS issue when running:
>
>   http_proxy=http://... guix download http://...
>
> ?
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 5622 bytes --]