herd stop hangs

herd stop hangs

[Vladilen Kozin]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

Hello.

I wrote a shepherd service that runs a java application in a semi standard
fashion

(start
  #~(make-forkexec-constructor ...
    #:user ..
    #:group ..
    #:environment-variables ...
(stop
  #~(make-kill-destructor)

$ sudo herd start my-service
starts without a hitch, but

$ sudo stop my-service
simply hangs for the default 5sec until Shepherd SIGKILLs it (as per
/var/log/messages) and even after that console where I run the herd stop
command doesn't get released until I manually CTRL-C it.

Been driving me nuts, cause Java handles SIGTERM but just in case I even
added shutdownHook to my app that should explicitly execute on SIGTERM and
other such signals. Testing it on OSX e.g. works just fine. But stop
wouldn't work on guix. I then decided to try everything manually:

$ start my java app from the console

In another console
$ sudo pkill -TERM pid

and nada ... it's like signal isn't even delivered.
$ sudo pkill -KILL pid

Eh ... still nothing
$ sudo kill -s SIGTERM pid
gets delivered

Ha? kill vs pkill ... ha? Could this give a clue as to why sudo herd stop
won't work?
What is going on here? Any ideas? I'm completely lost at this point.
-- 
Best regards
Vlad Kozin

[-- Attachment #2: Type: text/html, Size: 3952 bytes --]

Re: herd stop hangs

[Attila Lendvai]

> What is going on here? Any ideas? I'm completely lost at this point.

a random stab: with custom service code, it's possible to put shepherd into a state where it doesn't respond to herd remote requests anymore (nor shutdown/reboot requests). try if your shepherd works fine otherwise, e.g. with other services. if not, then the issue might be in the code that starts your service.

unfortunately i can't remember anymore what caused this for me, and shepherd has also seen a few fixes since then.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The penalty for laughing in a courtroom is six months in jail; if it were not for this penalty, the jury would never hear the evidence.”
	— H.L. Mencken (1880–1956)

Re: herd stop hangs

[Vlad Kozin]

Indeed reboot / shutdown no longer work … wat :) 

Don’t have any bright ideas about possible next steps tbh. 

Best,
Vlad

> On Apr 12, 2023, at 9:05 PM, Attila Lendvai <attila@lendvai.name> wrote:
> 
> 
>> 
>> What is going on here? Any ideas? I'm completely lost at this point.
> 
> a random stab: with custom service code, it's possible to put shepherd into a state where it doesn't respond to herd remote requests anymore (nor shutdown/reboot requests). try if your shepherd works fine otherwise, e.g. with other services. if not, then the issue might be in the code that starts your service.
> 
> unfortunately i can't remember anymore what caused this for me, and shepherd has also seen a few fixes since then.
> 
> -- 
> • attila lendvai
> • PGP: 963F 5D5F 45C7 DFCD 0A39
> --
> “The penalty for laughing in a courtroom is six months in jail; if it were not for this penalty, the jury would never hear the evidence.”
>    — H.L. Mencken (1880–1956)
> 

Re: herd stop hangs

[Ivan Sokolov]

Vladilen Kozin <vladilen.kozin@gmail.com> writes:

> Been driving me nuts, cause Java handles SIGTERM but just in case I even added
> shutdownHook to my app that should explicitly execute on SIGTERM and other such signals.
> Testing it on OSX e.g. works just fine. But stop wouldn't work on guix. I then decided to try
> everything manually:
>
> $ start my java app from the console
>
> In another console
> $ sudo pkill -TERM pid
>
> and nada ... it's like signal isn't even delivered.
> $ sudo pkill -KILL pid
>
> Eh ... still nothing
> $ sudo kill -s SIGTERM pid
> gets delivered
>
> Ha? kill vs pkill ... ha? Could this give a clue as to why sudo herd stop won't work?
> What is going on here? Any ideas? I'm completely lost at this point. 

I don't know about Shepherd, but I've never seen pkill used with a pid,
and pkill(1) does not mention it either.

Re: herd stop hangs

[Ludovic Courtès]

Hi,

Vladilen Kozin <vladilen.kozin@gmail.com> skribis:

> $ sudo herd start my-service
> starts without a hitch, but
>
> $ sudo stop my-service
> simply hangs for the default 5sec until Shepherd SIGKILLs it (as per
> /var/log/messages) and even after that console where I run the herd stop
> command doesn't get released until I manually CTRL-C it.

(I guess the second command is ‘sudo herd stop my-service’.)  If you
interrupt the ‘herd’ command, does ‘sudo herd status’ and similar
commands still respond?

If not, could you share the relevant /var/log/messages excerpt?

Thanks for reporting the issue,
Ludo’.

Re: herd stop hangs

[Vlad Kozin]

> (I guess the second command is ‘sudo herd stop my-service’.)  If you
> interrupt the ‘herd’ command, does ‘sudo herd status’ and similar
> commands still respond?

‘sudo herd stop myservice’ exactly. Sorry about that.
Herd continues to function. Appears that somehow SIGTERM has no effect or it never gets sent or it is in fact delivered and acted upon but shepherd fails to notice, so /var/log/messages shows subsequent SIGKILL. Whichever it is, herd status will show the service has been stopped. That’s one issue. Another is that the console where ‘sudo herd stop’ was typed required a CTRL C to get the prompt.

Could it be some weird sequence of what appears in the :running slot or smth? I’m using standard process constructors destructors mimicking other services

Re: herd stop hangs

[Ludovic Courtès]

Hi,

Vlad Kozin <vladilen.kozin@gmail.com> skribis:

>> (I guess the second command is ‘sudo herd stop my-service’.)  If you
>> interrupt the ‘herd’ command, does ‘sudo herd status’ and similar
>> commands still respond?
>
> ‘sudo herd stop myservice’ exactly. Sorry about that.
> Herd continues to function. Appears that somehow SIGTERM has no effect
> or it never gets sent or it is in fact delivered and acted upon but
> shepherd fails to notice, so /var/log/messages shows subsequent
> SIGKILL. Whichever it is, herd status will show the service has been
> stopped. That’s one issue.

Could you send the relevant /var/log/messages excerpt?

How frequent is it?  If you have a recipe to reproduce the issue, I’m
most interested!

> Another is that the console where ‘sudo herd stop’ was typed required
> a CTRL C to get the prompt.

OK.

> Could it be some weird sequence of what appears in the :running slot or smth? I’m using standard process constructors destructors mimicking other services

If this is a service that’s not in Guix proper, could you share the
code?  That way we’ll have everything we need to investigate.

Thanks,
Ludo’.