Multiple substitute servers question

Multiple substitute servers question

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 3266 bytes --]

Hello Guix!

I'm using both the default substitute server (mirrors.hydra.gnu.org) as
well as the recently added bayfront.guixsd.org[1], and today when I tried to
issue 'guix environment guix' I got:

--8<---------------cut here---------------start------------->8---

[...]

;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/tmux.go
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
;;;       newer than compiled /gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/share/guile/site/2.0/gnu/packages/tmux.go
Downloading https://bayfront.guixsd.org/nar/c64n03yy6kk66ah7x7pghpx2zc62j8d0-subversion-1.8.17.tar.bz2 (6.6MiB installed)...
guix substitute: error: connect: No route to host
killing process 7274
guix environment: error: build failed: some substitutes for the outputs of derivation `/gnu/store/2yl7ck1dzi32jb4aiqr2hhqxx90qkfxh-subversion-1.8.17.tar.bz2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
--8<---------------cut here---------------end--------------->8---

Given that issuing 'guix environment guix
--substitute-urls=mirrors.hydra.gnu.org' produces something different
(but unfortunately equally broken):

--8<---------------cut here---------------start------------->8---
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/tmux.go
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
;;;       newer than compiled /gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/share/guile/site/2.0/gnu/packages/tmux.go
substitute: Backtrace:
substitute: In ice-9/boot-9.scm:
substitute:  160: 9 [catch #t #<catch-closure 268ab40> ...]
substitute: In unknown file:
substitute:    ?: 8 [apply-smob/1 #<catch-closure 268ab40>]
substitute: In ice-9/boot-9.scm:
substitute:   66: 7 [call-with-prompt prompt0 ...]
substitute: In ice-9/eval.scm:
substitute:  432: 6 [eval # #]
substitute: In ice-9/boot-9.scm:
substitute: 2404: 5 [save-module-excursion #<procedure 26aa900 at ice-9/boot-9.scm:4051:3 ()>]
substitute: 4056: 4 [#<procedure 26aa900 at ice-9/boot-9.scm:4051:3 ()>]
substitute: 1727: 3 [%start-stack load-stack ...]
substitute: 1732: 2 [#<procedure 26beb70 ()>]
substitute: In unknown file:
substitute:    ?: 1 [primitive-load "/gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/bin/.guix-real"]
substitute: In guix/ui.scm:
substitute: 1228: 0 [run-guix-command substitute "--query"]
substitute: 
substitute: guix/ui.scm:1228:8: In procedure run-guix-command:
substitute: guix/ui.scm:1228:8: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
guix environment: error: corrupt input while restoring archive from #<closed: file 0>
--8<---------------cut here---------------end--------------->8---

Do I understand that upon failing with bayfront the daemon doesn't
proceed (fallback) to the regular mirrors.hydra.gnu.org substitute
server?

Maxim

[1]: https://lists.gnu.org/archive/html/guix-devel/2017-01/msg01746.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Multiple substitute servers question

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 2366 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Correction below:

> Hello Guix!
>
> I'm using both the default substitute server (mirrors.hydra.gnu.org) as
> well as the recently added bayfront.guixsd.org[1], and today when I tried to
> issue 'guix environment guix' I got:
>
>
> [...]
>
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
> ;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/tmux.go
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/tmux.scm
> ;;;       newer than compiled /gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/share/guile/site/2.0/gnu/packages/tmux.go
> Downloading https://bayfront.guixsd.org/nar/c64n03yy6kk66ah7x7pghpx2zc62j8d0-subversion-1.8.17.tar.bz2 (6.6MiB installed)...
> guix substitute: error: connect: No route to host
> killing process 7274
> guix environment: error: build failed: some substitutes for the outputs of derivation `/gnu/store/2yl7ck1dzi32jb4aiqr2hhqxx90qkfxh-subversion-1.8.17.tar.bz2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
>
>
> Given that issuing 'guix environment guix
> --substitute-urls=mirrors.hydra.gnu.org' produces something different
> (but unfortunately equally broken):

The substitute URI should be "https://mirrors.hydra.gnu.org". When using
such URI it works without an error.

My original question still holds though (why is an unresolvable
substitute-url able to bring the guix command down when at least one
working alternative substitute server exists? -- it should just warn
about it and push through).

> substitute: In unknown file:
> substitute:    ?: 1 [primitive-load "/gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/bin/.guix-real"]
> substitute: In guix/ui.scm:
> substitute: 1228: 0 [run-guix-command substitute "--query"]
> substitute: 
> substitute: guix/ui.scm:1228:8: In procedure run-guix-command:
> substitute: guix/ui.scm:1228:8: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
> guix environment: error: corrupt input while restoring archive from #<closed: file 0>
>

Also, this error message is very cryptic; it didn't provided much that
would have helped me to realize about my error (wrong URI) earlier.

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Multiple substitute servers question

[Ludovic Courtès]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> The substitute URI should be "https://mirrors.hydra.gnu.org". When using
> such URI it works without an error.
>
> My original question still holds though (why is an unresolvable
> substitute-url able to bring the guix command down when at least one
> working alternative substitute server exists? -- it should just warn
> about it and push through).

That’s what it does, except for URL syntax errors.

That is, if you pass:

  --substitute-urls="http://example.org/does-not-work https://mirror.hydra.gnu.org"

then the first one is ignored.

Does that make sense?

>> substitute: In unknown file:
>> substitute:    ?: 1 [primitive-load "/gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/bin/.guix-real"]
>> substitute: In guix/ui.scm:
>> substitute: 1228: 0 [run-guix-command substitute "--query"]
>> substitute: 
>> substitute: guix/ui.scm:1228:8: In procedure run-guix-command:
>> substitute: guix/ui.scm:1228:8: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
>> guix environment: error: corrupt input while restoring archive from #<closed: file 0>
>>
>
> Also, this error message is very cryptic; it didn't provided much that
> would have helped me to realize about my error (wrong URI) earlier.

Indeed.  Fixed in 8a2105078b5845e31768da06b360676352b1f3e9 (you won’t
get the fix until you run an updated guix-daemon.)

Thanks!

Ludo’.

Re: Multiple substitute servers question

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 2251 bytes --]

Hi Ludovic!

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>> The substitute URI should be "https://mirror.hydra.gnu.org". When using
>> such URI it works without an error.
>>
>> My original question still holds though (why is an unresolvable
>> substitute-url able to bring the guix command down when at least one
>> working alternative substitute server exists? -- it should just warn
>> about it and push through).
>
> That’s what it does, except for URL syntax errors.
>
>
>   --substitute-urls="http://example.org/does-not-work https://mirror.hydra.gnu.org"
>
> then the first one is ignored.
>
> Does that make sense?

What do you mean by "URL syntax errors"? Like, wrong protocol prefix or
non-ascii characters?  Otherwise, I don't see the difference between a "syntax"
error in a URL and a URL pointing to a dead server :).

Otherwise yes, that makes sense; that's exactly what I would expect to
happen.

>
>>> substitute: In unknown file:
>>> substitute:    ?: 1 [primitive-load "/gnu/store/v83285dvjy923ikq1dddncixb6kfba0k-guix-0.12.0-5.1162/bin/.guix-real"]
>>> substitute: In guix/ui.scm:
>>> substitute: 1228: 0 [run-guix-command substitute "--query"]
>>> substitute: 
>>> substitute: guix/ui.scm:1228:8: In procedure run-guix-command:
>>> substitute: guix/ui.scm:1228:8: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
>>> guix environment: error: corrupt input while restoring archive from #<closed: file 0>
>>>
>>
>> Also, this error message is very cryptic; it didn't provided much that
>> would have helped me to realize about my error (wrong URI) earlier.
>
> Indeed.  Fixed in 8a2105078b5845e31768da06b360676352b1f3e9 (you won’t
> get the fix until you run an updated guix-daemon.)
>
> Thanks!
>
> Ludo’.

Thanks for fixing it!

I'll keep this thread around for the next time a 'guix package' command
fails in an unexpected way. It seems to come everytime bayfront is in a
broken/semi-broken state (my regular url-substitutes list is made up of
1st bayfront, 2nd mirror.hydra.gnu.org).

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Multiple substitute servers question

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello!
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>> [...]
>>
>>> The substitute URI should be "https://mirror.hydra.gnu.org". When using
>>> such URI it works without an error.
>>>
>>> My original question still holds though (why is an unresolvable
>>> substitute-url able to bring the guix command down when at least one
>>> working alternative substitute server exists? -- it should just warn
>>> about it and push through).
>>
>> That’s what it does, except for URL syntax errors.
>>
>>
>>   --substitute-urls="http://example.org/does-not-work https://mirror.hydra.gnu.org"
>>
>> then the first one is ignored.
>>
>> Does that make sense?
>
> What do you mean by "URL syntax errors"? Like, wrong protocol prefix or
> non-ascii characters?

I mean any string for which ‘string->uri’ returns #f, such as
“foo.example.org”.

> I'll keep this thread around for the next time a 'guix package' command
> fails in an unexpected way. It seems to come everytime bayfront is in a
> broken/semi-broken state (my regular url-substitutes list is made up of
> 1st bayfront, 2nd mirror.hydra.gnu.org).

I do the same, but what I see is just that sometimes bayfront doesn’t
show up in the substitute messages, that’s all.

Thanks,
Ludo’.

Re: Multiple substitute servers question

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Ludovic!
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello!
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>> [...]
>>
>>> The substitute URI should be "https://mirror.hydra.gnu.org". When using
>>> such URI it works without an error.
>>>
>>> My original question still holds though (why is an unresolvable
>>> substitute-url able to bring the guix command down when at least one
>>> working alternative substitute server exists? -- it should just warn
>>> about it and push through).
>>
>> That’s what it does, except for URL syntax errors.
>>
>>
>>   --substitute-urls="http://example.org/does-not-work https://mirror.hydra.gnu.org"
>>
>> then the first one is ignored.
>>
>> Does that make sense?
>
>[...]
>
> I'll keep this thread around for the next time a 'guix package' command
> fails in an unexpected way. It seems to come everytime bayfront is in a
> broken/semi-broken state (my regular url-substitutes list is made up of
> 1st bayfront, 2nd mirror.hydra.gnu.org).

I had this problem again now.

--8<---------------cut here---------------start------------->8---
guix package -i emacs-dvc
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/emacs.go
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
;;;       newer than compiled /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
;;;       newer than compiled /run/current-system/profile/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
;;;       newer than compiled /home/maxim/.cache/guile/ccache/2.0-LE-8-2.0/home/maxim/src/guix/gnu/packages/emacs.scm.go
The following package will be installed:
   emacs-dvc    trunk-1.591     /gnu/store/sraxmg5qz9i4338s4ks7asgy4v68dgqs-emacs-dvc-trunk-1.591

substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
Downloading https://bayfront.guixsd.org/nar/gzip/6h7ym07plwxfn4zq53ld8zfpbx3a09al-at-spi2-core-2.22.0 (1.1MiB installed)...
guix substitute: error: connect: No route to host
killing process 13896
killing process 13896: No such process

^C
--8<---------------cut here---------------end--------------->8---

Q1: Why did it not fallback automatically to mirror.hydra.gnu.org, which
pings, and instead aborted (no, hung) attempting to kill an
(inexistant?) process? :)

Here are the ping checks:

--8<---------------cut here---------------start------------->8---
ping bayfront.guixsd.org
PING bayfront.guixsd.org (141.255.128.56): 56 data bytes
36 bytes from aquilenet.demarc.cogentco.com (149.6.44.18): Destination Host Unreachable
36 bytes from aquilenet.demarc.cogentco.com (149.6.44.18): Destination Host Unreachable
36 bytes from aquilenet.demarc.cogentco.com (149.6.44.18): Destination Host Unreachable
36 bytes from aquilenet.demarc.cogentco.com (149.6.44.18): Destination Host Unreachable
^C--- bayfront.guixsd.org ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

ping mirror.hydra.gnu.org
PING hydra.net.in.tum.de (131.159.14.26): 56 data bytes
64 bytes from 131.159.14.26: icmp_seq=0 ttl=52 time=164.453 ms
64 bytes from 131.159.14.26: icmp_seq=1 ttl=52 time=175.964 ms
64 bytes from 131.159.14.26: icmp_seq=2 ttl=52 time=167.195 ms
^C--- hydra.net.in.tum.de ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 164.453/169.204/175.964/4.909 ms
--8<---------------cut here---------------end--------------->8---

And some more information regarding which guix-daemon process(es) were
running at that time:

--8<---------------cut here---------------start------------->8---
ps aux | grep guix-daemon
root       254  0.0  0.0  34584  3436 ?        Ss   May01   0:00 /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/bin/guix-daemon --build-users-group guixbuild --substitute-urls https://bayfront.guixsd.org https://mirror.hydra.gnu.org
root     14347  3.0  0.1  37212  7568 ?        Ss   00:20   0:00 /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/bin/guix-daemon 14344               guixbuild --substitute-urls https://bayfront.guixsd.org https://mirror.hydra.gnu.org
maxim    14368  0.0  0.0   8168  1636 pts/4    S+   00:20   0:00 grep guix-daemon
root     30454  0.0  0.1  37056  7384 ?        Ss   May04   0:00 /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/bin/guix-daemon 30400               guixbuild --substitute-urls https://bayfront.guixsd.org https://mirror.hydra.gnu.org
--8<---------------cut here---------------end--------------->8---

Re: Multiple substitute servers question

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I had this problem again now.
>
> guix package -i emacs-dvc
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
> ;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/emacs.go
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
> ;;;       newer than compiled /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
> ;;;       newer than compiled /run/current-system/profile/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
> ;;;       newer than compiled /home/maxim/.cache/guile/ccache/2.0-LE-8-2.0/home/maxim/src/guix/gnu/packages/emacs.scm.go
> The following package will be installed:
>    emacs-dvc    trunk-1.591     /gnu/store/sraxmg5qz9i4338s4ks7asgy4v68dgqs-emacs-dvc-trunk-1.591
>
> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
> Downloading https://bayfront.guixsd.org/nar/gzip/6h7ym07plwxfn4zq53ld8zfpbx3a09al-at-spi2-core-2.22.0 (1.1MiB installed)...
> guix substitute: error: connect: No route to host
> killing process 13896
> killing process 13896: No such process

Oh I see.  ‘guix substitute --query’ (aka. “updating list of
substitutes”) correctly ignores the faulty machine, but ‘guix substitute
--substitute’ does not.  Could you email this issue to bug-guix?

> Q1: Why did it not fallback automatically to mirror.hydra.gnu.org, which
> pings, and instead aborted (no, hung) attempting to kill an
> (inexistant?) process? :)

I think the message is referring the ‘guix substitute’ process, but that
process exited after printing “error: connect”, and thus guix-daemon had
no process left to kill.

Thanks,
Ludo’.

Re: Multiple substitute servers question

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 2259 bytes --]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi!
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> I had this problem again now.
>>
>> guix package -i emacs-dvc
>> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
>> ;;;       newer than compiled /home/maxim/.config/guix/latest/gnu/packages/emacs.go
>> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
>> ;;;       newer than compiled /gnu/store/nqy9m6hhnkkfwr5wyq5bac96v9s9hc9i-guix-0.12.0-9.25a4/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
>> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
>> ;;;       newer than compiled /run/current-system/profile/lib/guile/2.0/site-ccache/gnu/packages/emacs.go
>> ;;; note: source file /home/maxim/.config/guix/latest/gnu/packages/emacs.scm
>> ;;;       newer than compiled /home/maxim/.cache/guile/ccache/2.0-LE-8-2.0/home/maxim/src/guix/gnu/packages/emacs.scm.go
>> The following package will be installed:
>>    emacs-dvc    trunk-1.591     /gnu/store/sraxmg5qz9i4338s4ks7asgy4v68dgqs-emacs-dvc-trunk-1.591
>>
>> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
>> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
>> Downloading https://bayfront.guixsd.org/nar/gzip/6h7ym07plwxfn4zq53ld8zfpbx3a09al-at-spi2-core-2.22.0 (1.1MiB installed)...
>> guix substitute: error: connect: No route to host
>> killing process 13896
>> killing process 13896: No such process
>
> Oh I see.  ‘guix substitute --query’ (aka. “updating list of
> substitutes”) correctly ignores the faulty machine, but ‘guix substitute
> --substitute’ does not.  Could you email this issue to bug-guix?
>
>> Q1: Why did it not fallback automatically to mirror.hydra.gnu.org, which
>> pings, and instead aborted (no, hung) attempting to kill an
>> (inexistant?) process? :)
>
> I think the message is referring the ‘guix substitute’ process, but that
> process exited after printing “error: connect”, and thus guix-daemon had
> no process left to kill.
>

I've opened an issue here:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=26833

Thanks!

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]