bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

[Hilton Chain via Bug reports for GNU Guix]

Hello!

I have experienced many instances of Shepherd hanging through my use
of Guix, though I don't have a clear record of when it first happened.

These days I have seen a few reports on the subject.  With a quick
search of recent bug reports, I can't find any related, only to find
this thread [1] on help-guix.  So I'll start a bug report here, but I
don't know how to debug Shepherd and I haven't found a way to
reproduce it stably.

I'm not sure if Shepherd hangs at usual, but most of the time I find
it already hanging is when doing a reconfiguration.  The
reconfiguration becomes unresponsive and it won't accept a ^C, herd
actions also hang.  This usually happens with home reconfiguration,
but I can remember once with system reconfiguration when adding and
deleting some services in the configuration file.

I'm not sure how Shepherd hangs either, because in the latter case
(system one) I can still see logs indicating that it's trying to
respawn a process I killed manually, even though that's just a output
and no processes are actually spawned.

And as shown in [1], there are also cases where Shepherd hangs at some
point in the halting process, usually after syslogd has been
terminated but before term-tty*.

(The termination message indicates that Shepherd is still functional,
and no logs after that point shows that that's a real action, but
because of this I can't know anything happened further either.  After
that I'm still able to switch ttys so I assume term-tty* are alive.)

Although I don't know how they are related, I have linked my
configurations below:
<https://codeberg.org/hako/Testament/src/branch/trunk/dorphine-home.scm>
<https://codeberg.org/hako/Testament/src/branch/trunk/dorphine-system.scm>

Thanks

[1]:
<https://lists.gnu.org/archive/html/help-guix/2023-07/msg00021.html>
(public-inbox mirror on yhetil.org)
<https://yhetil.org/guix/NZXMeM4--3-9@tutanota.com/t/#u>

bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

[Hilton Chain via Bug reports for GNU Guix]

On Wed, 09 Aug 2023 20:41:44 +0800,
Hilton Chain wrote:
> I'm not sure if Shepherd hangs at usual, but most of the time I find
> it already hanging is when doing a reconfiguration.  The
> reconfiguration becomes unresponsive and it won't accept a ^C, herd
> actions also hang.  This usually happens with home reconfiguration,

Today I encountered the home reconfiguration issue.  The behavior is
similar to <https://issues.guix.gnu.org/54919>.

Ending part of output for the hanging reconfiguration:
--8<---------------cut here---------------start------------->8---
[...]
Symlinking /home/hako/.config/fontconfig/fonts.conf -> /gnu/store/fvvqbma1xxgisfcq7rrwihbw7jwnyliv-fonts.conf... done
Symlinking /home/hako/.gnupg/gpg-agent.conf -> /gnu/store/kfaz4zrxmfz6p72x47c7qrqvb873gbyi-gpg-agent.conf... done
Symlinking /home/hako/.ssh/config -> /gnu/store/xb6f584pwclg48fr28wl21v1mxplqp6f-ssh.conf... done
Symlinking /home/hako/.icons/default/index.theme -> /gnu/store/3sraq69nrs04ii0fjgk36aw2c57q6z27-icons.theme... done
 done
Finished updating symlinks.


--8<---------------cut here---------------end--------------->8---

And `herd status' also hangs:
--8<---------------cut here---------------start------------->8---
$ herd status

--8<---------------cut here---------------end--------------->8---

bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 13 Aug 2023 23:25:59 +0800,
Hilton Chain wrote:
>
> Today I encountered the home reconfiguration issue.  The behavior is
> similar to <https://issues.guix.gnu.org/54919>.

And today Shepherd hung after starting a service [1], the service
itself started successfully (process started, logs available):
--8<---------------cut here---------------start------------->8---
$ sudo herd enable cloudflare-tunnel && sudo herd start cloudflare-tunnel
Enabled service cloudflare-tunnel.

--8<---------------cut here---------------end--------------->8---

[1]: <https://codeberg.org/hako/Rosenthal/src/commit/c7dc95c2932d7362673c28cdc2f52e6bb8357c18/rosenthal/services/child-error.scm#L151>

bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

[Ludovic Courtès]

Hi!

Hilton Chain <hako@ultrarare.space> scribes:

> On Sun, 13 Aug 2023 23:25:59 +0800,
> Hilton Chain wrote:
>>
>> Today I encountered the home reconfiguration issue.  The behavior is
>> similar to <https://issues.guix.gnu.org/54919>.
>
> And today Shepherd hung after starting a service [1], the service
> itself started successfully (process started, logs available):

I’m assuming this is shepherd 0.10.2, right?

> $ sudo herd enable cloudflare-tunnel && sudo herd start cloudflare-tunnel
> Enabled service cloudflare-tunnel.
>
> [1]: <https://codeberg.org/hako/Rosenthal/src/commit/c7dc95c2932d7362673c28cdc2f52e6bb8357c18/rosenthal/services/child-error.scm#L151>

Is any of the services you’re using doing “non-standard things” such as
using constructors/destructors other than those provided by shepherd
(‘make-forkexec-constructor’ et al.)?

Is it reproducible, and do you think you could come up with a reduce
test case (for example by removing services from the config until you
reach the minimum)?

Thanks,
Ludo’.

bug#65178: Shepherd hangs (was: Getting Guix to shutdown my laptop properly with Sway and no DE)

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 03 Sep 2023 04:49:35 +0800,
Ludovic Courtès wrote:
>
> Hi!
>
> Hilton Chain <hako@ultrarare.space> scribes:
>
> > On Sun, 13 Aug 2023 23:25:59 +0800,
> > Hilton Chain wrote:
> >>
> >> Today I encountered the home reconfiguration issue.  The behavior is
> >> similar to <https://issues.guix.gnu.org/54919>.
> >
> > And today Shepherd hung after starting a service [1], the service
> > itself started successfully (process started, logs available):
>
> I’m assuming this is shepherd 0.10.2, right?


Yes!


>
> > $ sudo herd enable cloudflare-tunnel && sudo herd start cloudflare-tunnel
> > Enabled service cloudflare-tunnel.
> >
> > [1]: <https://codeberg.org/hako/Rosenthal/src/commit/c7dc95c2932d7362673c28cdc2f52e6bb8357c18/rosenthal/services/child-error.scm#L151>
>
> Is any of the services you’re using doing “non-standard things” such as
> using constructors/destructors other than those provided by shepherd
> (‘make-forkexec-constructor’ et al.)?


No, I'm unaware of such things.


> Is it reproducible, and do you think you could come up with a reduce
> test case (for example by removing services from the config until you
> reach the minimum)?


I still don't know which condition triggers it, so I can't make a test
case.

It's unreproducible.  And I don't think it's really related to the
config, since Shepherd won't hang when rebooting to a system
generation which made it hanging at reconfiguration before.

It might be related to bug#65419 ([Shepherd] Non-reponding service
control fiber) you have reported, since there's similar behavior that
`herd status nscd' still works when Shepherd hangs.