bug#58926: Shepherd becomes unresponsive after an interrupt

bug#58926: Shepherd becomes unresponsive after an interrupt

[Mathieu Othacehe]

Hello,

When running the following command:

--8<---------------cut here---------------start------------->8---
sudo herd restart service-that-hangs-upon-restart
--8<---------------cut here---------------end--------------->8---

then hitting C-c, Shepherd becomes totally unresponsive:

--8<---------------cut here---------------start------------->8---
sudo herd status
--8<---------------cut here---------------end--------------->8---

and all further Shpeherd commands hang forever. I was able to reproduce
it in two different configurations:

1. On my laptop with a Wireguard service trying to reach a non-existing
DNS server.

--8<---------------cut here---------------start------------->8---
            (service wireguard-service-type
                     (wireguard-configuration
                      (addresses (list "10.0.0.2/24"))
                      (dns '("10.0.0.50")) #does not exit
--8<---------------cut here---------------end--------------->8---

2. On Berlin, while trying to restart nginx.

In both situations, the "reboot" command was also hanging.

Thanks,

Mathieu

bug#58926: Shepherd becomes unresponsive after an interrupt

[Ludovic Courtès]

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

> sudo herd restart service-that-hangs-upon-restart
>
>
> then hitting C-c, Shepherd becomes totally unresponsive:
>
> sudo herd status
>
>
> and all further Shpeherd commands hang forever. I was able to reproduce
> it in two different configurations:
>
> 1. On my laptop with a Wireguard service trying to reach a non-existing
> DNS server.
>
>             (service wireguard-service-type
>                      (wireguard-configuration
>                       (addresses (list "10.0.0.2/24"))
>                       (dns '("10.0.0.50")) #does not exit
>
> 2. On Berlin, while trying to restart nginx.

I experienced case #2: in that case ‘strace -p1’ showed that shepherd
was stuck on waitpid of the nginx process, which was not terminating.
Killing that process would unlock shepherd.

This might be <https://issues.guix.gnu.org/56674>.

Would be good to see what’s up with WireGuard.

Ludo’.

bug#58926: Shepherd becomes unresponsive after an interrupt

[Ludovic Courtès]

Mathieu Othacehe <othacehe@gnu.org> skribis:

> 1. On my laptop with a Wireguard service trying to reach a non-existing
> DNS server.
>
>             (service wireguard-service-type
>                      (wireguard-configuration
>                       (addresses (list "10.0.0.2/24"))
>                       (dns '("10.0.0.50")) #does not exit

This one is similar to:

  https://issues.guix.gnu.org/53225
  https://issues.guix.gnu.org/53381

It has to do with the fact that “wg-quick up” blocks until it succeeds
and that ‘invoke’ gets stuck on ‘waitpid’ until the “wg-quick” process
terminates.

The solution will be to use something non-blocking instead of ‘invoke’;
I’m looking into it.

Ludo’.

bug#53225: bug#58926: Shepherd becomes unresponsive after an interrupt

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>> 1. On my laptop with a Wireguard service trying to reach a non-existing
>> DNS server.
>>
>>             (service wireguard-service-type
>>                      (wireguard-configuration
>>                       (addresses (list "10.0.0.2/24"))
>>                       (dns '("10.0.0.50")) #does not exit
>
> This one is similar to:
>
>   https://issues.guix.gnu.org/53225
>   https://issues.guix.gnu.org/53381
>
> It has to do with the fact that “wg-quick up” blocks until it succeeds
> and that ‘invoke’ gets stuck on ‘waitpid’ until the “wg-quick” process
> terminates.
>
> The solution will be to use something non-blocking instead of ‘invoke’;
> I’m looking into it.

This is fixed in the Shepherd 0.9.3, which landed in Guix commit
283d7318c5b312d7129adb6dbeea6ad205ce89d1.

As I wrote, I’m not sure whether it fixes the nginx situation since I
could not reproduce it.  I’m closing and let’s open a new issue
specifically for nginx if it comes up again with 0.9.3.

Thanks,
Ludo’.

bug#58926: Shepherd becomes unresponsive after an interrupt

[Ludovic Courtès]

Mathieu Othacehe <othacehe@gnu.org> skribis:

> then hitting C-c, Shepherd becomes totally unresponsive:
>
> sudo herd status
>
>
> and all further Shpeherd commands hang forever. I was able to reproduce
> it in two different configurations:

[...]

> 2. On Berlin, while trying to restart nginx.

I can’t reproduce it in a VM.

Before I try it on a production system :-), does anyone have a tip on
how to reproduce it?  Or perhaps strace output from a system that
exhibits this bug?

TIA!

Ludo’.