* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
@ 2022-01-31 18:37 Simon Streit
2022-01-31 20:17 ` Guillaume Le Vaillant
2022-02-01 2:06 ` Maxim Cournoyer
0 siblings, 2 replies; 11+ messages in thread
From: Simon Streit @ 2022-01-31 18:37 UTC (permalink / raw)
To: 53668
Hello,
quite often, and quite randomly I run into this situation that whenever
Guix tries to rebuild a profile, and sometimes while downloading from
local Guix hosts sharing their store items, the process will crash with
the following error:
--8<---------------cut here---------------start------------->8---
~ $1 reconfigure
substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In unknown file:
substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
substitute: In ice-9/boot-9.scm:
substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
substitute: In ice-9/eval.scm:
substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
substitute: In guix/ui.scm:
substitute: 2206:7 13 (run-guix . _)
substitute: 2169:10 12 (run-guix-command _ . _)
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In guix/scripts/substitute.scm:
substitute: 757:18 9 (_)
substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
substitute: In guix/substitutes.scm:
substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
substitute: In ice-9/boot-9.scm:
substitute: 1685:16 4 (raise-exception _ #:continuable? _)
substitute: 1685:16 3 (raise-exception _ #:continuable? _)
substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
substitute: 1685:16 1 (raise-exception _ #:continuable? _)
substitute: 1685:16 0 (raise-exception _ #:continuable? _)
substitute:
substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
substitute: Wrong type (expecting exact integer): #f
guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
--8<---------------cut here---------------end--------------->8---
Unfortunately this crash happens at random. Other times it goes
through. Current checkout is ff14bc60e56fb0c6636da1da9a850b7b04abb367,
which isn't the most current, I know. I've been observing this behavior
since some time now, and haven't figured out what the reason is behind
it yet. The error message looks similar to [1].
The way this error appears is, that I usually have one host that I
upgrade first, and then share the checkout and the store between hosts
to speed up the upgrading process locally. Unfortunately the updater
will crash randomly whenever the host starts scanning other hosts that
are found through mDNS. Sometimes this happens while fetching new
packages into a profile.
I've set up publishing:
--8<---------------cut here---------------start------------->8---
(service guix-publish-service-type
(guix-publish-configuration (host "0.0.0.0")
(port 3000)
(ttl #f)
(advertise? #t)))
--8<---------------cut here---------------end--------------->8---
and of course host discovery in guix-service-type too.
Kind regards
Simon
[1] https://issues.guix.gnu.org/52464
^ permalink raw reply [flat|nested] 11+ messages in thread* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-01-31 18:37 bug#53668: Updating substitutes on LAN hosts dies unexpectedly Simon Streit
@ 2022-01-31 20:17 ` Guillaume Le Vaillant
2022-01-31 21:38 ` Simon Streit
2022-02-01 2:06 ` Maxim Cournoyer
1 sibling, 1 reply; 11+ messages in thread
From: Guillaume Le Vaillant @ 2022-01-31 20:17 UTC (permalink / raw)
To: Simon Streit; +Cc: 53668
[-- Attachment #1: Type: text/plain, Size: 3661 bytes --]
Simon Streit <simon@netpanic.org> skribis:
> Hello,
>
> quite often, and quite randomly I run into this situation that whenever
> Guix tries to rebuild a profile, and sometimes while downloading from
> local Guix hosts sharing their store items, the process will crash with
> the following error:
>
> ~ $1 reconfigure
> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In unknown file:
> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
> substitute: In ice-9/boot-9.scm:
> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> substitute: In ice-9/eval.scm:
> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
> substitute: In guix/ui.scm:
> substitute: 2206:7 13 (run-guix . _)
> substitute: 2169:10 12 (run-guix-command _ . _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In guix/scripts/substitute.scm:
> substitute: 757:18 9 (_)
> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
> substitute: In guix/substitutes.scm:
> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
> substitute:
> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> substitute: Wrong type (expecting exact integer): #f
> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
>
>
> Unfortunately this crash happens at random. Other times it goes
> through. Current checkout is ff14bc60e56fb0c6636da1da9a850b7b04abb367,
> which isn't the most current, I know. I've been observing this behavior
> since some time now, and haven't figured out what the reason is behind
> it yet. The error message looks similar to [1].
>
> The way this error appears is, that I usually have one host that I
> upgrade first, and then share the checkout and the store between hosts
> to speed up the upgrading process locally. Unfortunately the updater
> will crash randomly whenever the host starts scanning other hosts that
> are found through mDNS. Sometimes this happens while fetching new
> packages into a profile.
>
> I've set up publishing:
>
> (service guix-publish-service-type
> (guix-publish-configuration (host "0.0.0.0")
> (port 3000)
> (ttl #f)
> (advertise? #t)))
>
> and of course host discovery in guix-service-type too.
>
>
> Kind regards
> Simon
>
>
> [1] https://issues.guix.gnu.org/52464
Hi,
I have also seen this kind of crashes on my LAN.
I don't know what causes this issue, but I noticed that it has happened
less often since I activated the cache for the guix-publish service.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-01-31 18:37 bug#53668: Updating substitutes on LAN hosts dies unexpectedly Simon Streit
2022-01-31 20:17 ` Guillaume Le Vaillant
@ 2022-02-01 2:06 ` Maxim Cournoyer
2022-05-11 14:58 ` Ludovic Courtès
1 sibling, 1 reply; 11+ messages in thread
From: Maxim Cournoyer @ 2022-02-01 2:06 UTC (permalink / raw)
To: Simon Streit; +Cc: 53668
Hi Simon,
Simon Streit <simon@netpanic.org> writes:
> Hello,
>
> quite often, and quite randomly I run into this situation that whenever
> Guix tries to rebuild a profile, and sometimes while downloading from
> local Guix hosts sharing their store items, the process will crash with
> the following error:
>
> ~ $1 reconfigure
> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In unknown file:
> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
> substitute: In ice-9/boot-9.scm:
> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> substitute: In ice-9/eval.scm:
> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
> substitute: In guix/ui.scm:
> substitute: 2206:7 13 (run-guix . _)
> substitute: 2169:10 12 (run-guix-command _ . _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In guix/scripts/substitute.scm:
> substitute: 757:18 9 (_)
> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
> substitute: In guix/substitutes.scm:
> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
> substitute:
> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> substitute: Wrong type (expecting exact integer): #f
> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
I suffer from this routinely on my slow HDD machine. I attribute the
failure to some timeout that isn't reported correctly by guile-ssh.
Perhaps related to
https://github.com/artyom-poptsov/guile-ssh/issues/29; there's a new
update of guile-ssh to try that Artyom has put up on guix-patches; we
should test it.
Thanks,
Maxim
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-02-01 2:06 ` Maxim Cournoyer
@ 2022-05-11 14:58 ` Ludovic Courtès
2022-05-11 15:02 ` Maxime Devos
2022-05-11 15:06 ` Maxim Cournoyer
0 siblings, 2 replies; 11+ messages in thread
From: Ludovic Courtès @ 2022-05-11 14:58 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: Simon Streit, 53668
Hello!
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> Simon Streit <simon@netpanic.org> writes:
>
>> Hello,
>>
>> quite often, and quite randomly I run into this situation that whenever
>> Guix tries to rebuild a profile, and sometimes while downloading from
>> local Guix hosts sharing their store items, the process will crash with
>> the following error:
>>
>> ~ $1 reconfigure
>> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: In unknown file:
>> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
>> substitute: In ice-9/eval.scm:
>> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
>> substitute: In guix/ui.scm:
>> substitute: 2206:7 13 (run-guix . _)
>> substitute: 2169:10 12 (run-guix-command _ . _)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: In guix/scripts/substitute.scm:
>> substitute: 757:18 9 (_)
>> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
>> substitute: In guix/substitutes.scm:
>> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
>> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
>> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
>> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
>> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
>> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
>> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
>> substitute:
>> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> substitute: Wrong type (expecting exact integer): #f
>> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
>
> I suffer from this routinely on my slow HDD machine. I attribute the
> failure to some timeout that isn't reported correctly by guile-ssh.
Guile-SSH is not involved here though.
However, this may well be this ‘guix publish’ bug that was recently
fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
replies:
https://issues.guix.gnu.org/54723
Could you confirm?
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-05-11 14:58 ` Ludovic Courtès
@ 2022-05-11 15:02 ` Maxime Devos
2022-05-12 1:17 ` Maxim Cournoyer
2022-05-11 15:06 ` Maxim Cournoyer
1 sibling, 1 reply; 11+ messages in thread
From: Maxime Devos @ 2022-05-11 15:02 UTC (permalink / raw)
To: Ludovic Courtès, Maxim Cournoyer; +Cc: Simon Streit, 53668
[-- Attachment #1: Type: text/plain, Size: 493 bytes --]
Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
> However, this may well be this ‘guix publish’ bug that was recently
> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
> replies:
>
> https://issues.guix.gnu.org/54723
>
> Could you confirm?
>
> Ludo’.
Even then, the error message says ‘wrong type error ...’, shouldn't it
say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
Greetings,
Maxime.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-05-11 15:02 ` Maxime Devos
@ 2022-05-12 1:17 ` Maxim Cournoyer
2022-05-12 8:21 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Maxim Cournoyer @ 2022-05-12 1:17 UTC (permalink / raw)
To: Maxime Devos; +Cc: Ludovic Courtès, Simon Streit, 53668
Hi Maxime,
Maxime Devos <maximedevos@telenet.be> writes:
> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>> However, this may well be this ‘guix publish’ bug that was recently
>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>> replies:
>>
>> https://issues.guix.gnu.org/54723
>>
>> Could you confirm?
>>
>> Ludo’.
>
> Even then, the error message says ‘wrong type error ...’, shouldn't it
> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
I agree that a more accurate error message would have been useful. If
you have the bandwidth for a patch, I'd be more than happy reviewing it
:-).
Thanks,
Maxim
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-05-12 1:17 ` Maxim Cournoyer
@ 2022-05-12 8:21 ` Ludovic Courtès
2022-05-12 13:47 ` Maxim Cournoyer
0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2022-05-12 8:21 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: Simon Streit, Maxime Devos, 53668
Hi,
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> Maxime Devos <maximedevos@telenet.be> writes:
>
>> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>>> However, this may well be this ‘guix publish’ bug that was recently
>>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>>> replies:
>>>
>>> https://issues.guix.gnu.org/54723
>>>
>>> Could you confirm?
>>>
>>> Ludo’.
>>
>> Even then, the error message says ‘wrong type error ...’, shouldn't it
>> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
>
> I agree that a more accurate error message would have been useful. If
> you have the bandwidth for a patch, I'd be more than happy reviewing it
> :-).
HTTP response parsing happens in (web response) though, not in Guix, so
this is where things should be patch.
Currently that code pretty much expects it’s talking to a well-behaved
HTTP server. Changing it to gracefully deal with any kind of garbage
might turn out to be quite an effort.
Food for thought!
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-05-12 8:21 ` Ludovic Courtès
@ 2022-05-12 13:47 ` Maxim Cournoyer
0 siblings, 0 replies; 11+ messages in thread
From: Maxim Cournoyer @ 2022-05-12 13:47 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Simon Streit, Maxime Devos, 53668
Hello,
Ludovic Courtès <ludo@gnu.org> writes:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Maxime Devos <maximedevos@telenet.be> writes:
>>
>>> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>>>> However, this may well be this ‘guix publish’ bug that was recently
>>>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>>>> replies:
>>>>
>>>> https://issues.guix.gnu.org/54723
>>>>
>>>> Could you confirm?
>>>>
>>>> Ludo’.
>>>
>>> Even then, the error message says ‘wrong type error ...’, shouldn't it
>>> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
>>
>> I agree that a more accurate error message would have been useful. If
>> you have the bandwidth for a patch, I'd be more than happy reviewing it
>> :-).
>
> HTTP response parsing happens in (web response) though, not in Guix, so
> this is where things should be patch.
>
> Currently that code pretty much expects it’s talking to a well-behaved
> HTTP server. Changing it to gracefully deal with any kind of garbage
> might turn out to be quite an effort.
Hmm. I see. Maxime, perhaps an issue can be logged against Guile to at
least remember it exists, to tackle by anyone with the
interest/bandwidth. I think it'd be valuable in the long run.
Thank you!
Maxim
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#53668: Updating substitutes on LAN hosts dies unexpectedly
2022-05-11 14:58 ` Ludovic Courtès
2022-05-11 15:02 ` Maxime Devos
@ 2022-05-11 15:06 ` Maxim Cournoyer
1 sibling, 0 replies; 11+ messages in thread
From: Maxim Cournoyer @ 2022-05-11 15:06 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 53668-done, Simon Streit
Hi Ludovic,
Ludovic Courtès <ludo@gnu.org> writes:
[...]
> However, this may well be this ‘guix publish’ bug that was recently
> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
> replies:
>
> https://issues.guix.gnu.org/54723
>
> Could you confirm?
I've reconfigured my offload machine and haven't seen the issue yet.
Tentatively closing.
Thank you for investigating and fixing the issue!
Maxim
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-05-12 13:48 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-31 18:37 bug#53668: Updating substitutes on LAN hosts dies unexpectedly Simon Streit
2022-01-31 20:17 ` Guillaume Le Vaillant
2022-01-31 21:38 ` Simon Streit
2022-01-31 21:55 ` Simon Streit
2022-02-01 2:06 ` Maxim Cournoyer
2022-05-11 14:58 ` Ludovic Courtès
2022-05-11 15:02 ` Maxime Devos
2022-05-12 1:17 ` Maxim Cournoyer
2022-05-12 8:21 ` Ludovic Courtès
2022-05-12 13:47 ` Maxim Cournoyer
2022-05-11 15:06 ` Maxim Cournoyer
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).