bug#26705: guix publish daemon on Hydra became dysfunctional; needed restart

[ludo@gnu.org (Ludovic Courtès)]

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> While trying to update my GuixSD system in the last hour, I found that
> every attempt by the substituter to download NARs resulted in a 500
> "Internal Server Error":

[...]

> Bypassing the mirror didn't help:

Did you try talking directly to ‘guix publish’ (localhost:9999 on
hydra.gnu.org)?

> Here's what I found in the recent output of the 'guix publish' server on
> Hydra (running in root's screen session):
>
> GET /74ch6nvjfkj3i56nygwijnaghlpi01d4.narinfo
> In guix/scripts/publish.scm:
>     393:2  2 (render-narinfo/cached #<build-daemon 256.96 211f7c0> ...)
> In guix/store.scm:
>     663:9  1 (query-path-from-hash-part #<build-daemon 256.96 2...> #)
> In unknown file:
>            0 (put-bytevector #<input-output: socket 12> #vu8(# ...) ...)
> ERROR: In procedure fport_write: Broken pipe
> GET /guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz
> In guix/scripts/publish.scm:
>     491:8  2 (render-nar #<build-daemon 256.96 211f7c0> #<<requ...> ...)
> In guix/store.scm:
>     648:0  1 (valid-path? #<build-daemon 256.96 211f7c0> "/gnu/sto...")
> In unknown file:
>            0 (put-bytevector #<input-output: socket 12> #vu8(1 ...) ...)
> ERROR: In procedure fport_write: Broken pipe

Ooh, the connection to the daemon was broken, hence this error.

Currently ‘guix publish’ assumes the connection opened in the
‘guix-publish’ procedure remains valid all along.  That’s normally the
case unless (1) the daemon is restarted, or (2) there’s a protocol error
somewhere that leads the daemon to close the connection.

#2 is not supposed to happen, but could happen if, for instance, an
exception is thrown before we have read the RPC reply sent by the
daemon.

Perhaps ‘guix publish’ could catch EPIPE when talking to the daemon?
(Currently EPIPEs are swallowed by ‘handle-request’ in (web server) and
converted to 500.)

That said, if there really is a situation where ‘guix publish’ can
violate the daemon protocol, that’s what should be fixed.

Thoughts?

Thanks for handling this quickly!

Ludo’.