* bug#33410: Offloaded builds can get stuck indefinitely due to network issues
@ 2018-11-17 4:08 Mark H Weaver
2018-11-17 14:21 ` Ludovic Courtès
0 siblings, 1 reply; 2+ messages in thread
From: Mark H Weaver @ 2018-11-17 4:08 UTC (permalink / raw)
To: 33410
I just discovered that 4 out of 5 armhf build slots on Hydra have been
stuck for 24 hours, apparently after the network connections to the
build slaves were lost, possibly due to a temporary network outage.
I've seen this kind of thing happen periodically since we switched to
using guile-ssh for offloaded builds.
On Hydra I can monitor the builds and investigate when a given build
seems to be taking far too long, and I can kill those jobs to free up
the build slots. There's no way to kill the builds from Hydra's web
interface, but I can kill them manually by logging into Hydra.
This might become a more serious problem on Berlin, as we add ARM build
slaves that are not on the same local network as Berlin itself, until
the web interface allows for this kind of monitoring and intervention.
Mark
^ permalink raw reply [flat|nested] 2+ messages in thread* bug#33410: Offloaded builds can get stuck indefinitely due to network issues
2018-11-17 4:08 bug#33410: Offloaded builds can get stuck indefinitely due to network issues Mark H Weaver
@ 2018-11-17 14:21 ` Ludovic Courtès
0 siblings, 0 replies; 2+ messages in thread
From: Ludovic Courtès @ 2018-11-17 14:21 UTC (permalink / raw)
To: Mark H Weaver; +Cc: 33410
Hello,
Mark H Weaver <mhw@netris.org> skribis:
> I just discovered that 4 out of 5 armhf build slots on Hydra have been
> stuck for 24 hours, apparently after the network connections to the
> build slaves were lost, possibly due to a temporary network outage.
>
> I've seen this kind of thing happen periodically since we switched to
> using guile-ssh for offloaded builds.
Which guix-daemon version is hydra running?
Commit a708de151c255712071e42e5c8284756b51768cd adds a safeguard to make
sure timeouts are honored, though there might be some cases where it
doesn’t quite work as expected (I suspect libssh handles EINTR
internally by looping, in which case our signal handling async doesn’t
get a chance to run.)
> On Hydra I can monitor the builds and investigate when a given build
> seems to be taking far too long, and I can kill those jobs to free up
> the build slots. There's no way to kill the builds from Hydra's web
> interface, but I can kill them manually by logging into Hydra.
>
> This might become a more serious problem on Berlin, as we add ARM build
> slaves that are not on the same local network as Berlin itself, until
> the web interface allows for this kind of monitoring and intervention.
The current situation on berlin is suboptimal: I run ‘guix processes’
when I suspect something is wrong, and that’s how I found about
<https://issues.guix.info/issue/33239>.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2018-11-17 14:22 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-17 4:08 bug#33410: Offloaded builds can get stuck indefinitely due to network issues Mark H Weaver
2018-11-17 14:21 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).