shepherd: failing test: should `herd stop` stop a respawning process?

shepherd: failing test: should `herd stop` stop a respawning process?

[Attila Lendvai]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

i have a daemon process that quits soon after it starts (when it has some issue with its configurations).

then i usually fix the config, and do a guix system reconfigure. but i have noticed from the logs that this process often remains in a resawn loop, even if i herd stop and herd disable it after the reconfigure (i.e. a shepherd service upgrade).

i have attached a respawn2.sh that when put under tests/ reproduces the issue in a shepherd checkout (see the TODO notes):

$ guix shell
$ make check TESTS="tests/respawn2.sh"

what's wrong?
 - is it my expectation that herd stop should stop the respawning loop?

 - do i have a bug in my test.sh?

 - is this a shepherd bug? if so, then shall i finish up this test case as a proper patch for shepherd?

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“In all history there is no war which was not hatched by the governments, the governments alone, independent of the interests of the people, to whom war is always pernicious even when successful.”
	— Leo Tolstoy (1828–1910), 'On Patriotism' (1894)

[-- Attachment #2: respawn2.sh --]
[-- Type: application/x-shellscript, Size: 4298 bytes --]

Re: shepherd: failing test: should `herd stop` stop a respawning process?

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Attila,

On Sun, Oct 06 2024, Attila Lendvai wrote:

>  - my expectation that herd stop should stop the respawning loop?

My sense is that the Shepherd and systemd always restart services.  I
don't think they know whether a failure relates to configuration or to
something else, but they could probably measure the rate of failure and
stop at some point.

You can mark services as "one-shot."  They are not restarted.  Maybe
that setting is helpful until your configuration is working.

Kind regards
Felix

Re: shepherd: failing test: should `herd stop` stop a respawning process?

[Ludovic Courtès]

Hi,

Attila Lendvai <attila@lendvai.name> skribis:

> i have a daemon process that quits soon after it starts (when it has some issue with its configurations).
>
> then i usually fix the config, and do a guix system reconfigure. but i have noticed from the logs that this process often remains in a resawn loop, even if i herd stop and herd disable it after the reconfigure (i.e. a shepherd service upgrade).

If it’s in a respawn loop, the problem is that ‘herd stop’ may or may
not happen at the right moment, because the service oscillates between
the stopped/starting/running/stopping statuses.

However, ‘herd disable’ should prevent it from being respawned.
(Respawning calls ‘start-service’, which cannot start a service marked
as disabled.)

But I don’t know, there could be a bug.  Could you come up with a
reduced test case (I looked at the one attached but I’m not sure which
part to focus on), or do you have logs of the problem?

Thanks,
Ludo’.