Shepherd does not recycle zombie processes

Shepherd does not recycle zombie processes

[Dale Mellor]

I'm running shepherd stand-alone in a Debian system.  But I am seeing
zombie processes which have been kicked off by shepherd, and they do not
get re-spawned.

Any suggestions?

Re: Shepherd does not recycle zombie processes

[Carlo Zancanaro]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

I've had problems with Shepherd and its daemonize action. If I run
daemonize (as the first thing when Shepherd starts) then it fails to handle
signals from child processes.

I've only been running without the daemonize call for a day or so, but it
seems to properly handle the child processes now.

On 06/11/2016 9:52 am, "Dale Mellor" <dale@rdmp.org> wrote:

I'm running shepherd stand-alone in a Debian system.  But I am seeing
zombie processes which have been kicked off by shepherd, and they do not
get re-spawned.

Any suggestions?

[-- Attachment #2: Type: text/html, Size: 880 bytes --]

Re: Shepherd does not recycle zombie processes

[Ludovic Courtès]

Hi Dale,

Dale Mellor <dale@rdmp.org> skribis:

> I'm running shepherd stand-alone in a Debian system.  But I am seeing
> zombie processes which have been kicked off by shepherd, and they do not
> get re-spawned.

This may have to do with unreliable signal handling (SIGCHLD in this
case) in Guile:

  https://lists.gnu.org/archive/html/guile-devel/2013-07/msg00004.html

shepherd.scm has a hack to mostly work around it, but it’s not always
sufficient as evidenced by occasionally failures of the respawn test:

  http://bugs.gnu.org/23811

Does it frequently fail to respawn?

Ludo’.

Re: Shepherd does not recycle zombie processes

[Ludovic Courtès]

Hi,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

> I've had problems with Shepherd and its daemonize action. If I run
> daemonize (as the first thing when Shepherd starts) then it fails to handle
> signals from child processes.

Could it be that you invoke the ‘daemonize’ action after respawnable
processes have been started?  The manual has this caveat (info
"(shepherd) The root and unknown services"):

  ‘daemonize’
       Fork and go into the background.  This should be called before
       respawnable services are started, as otherwise we would not get the
       ‘SIGCHLD’ signals when they terminate.

HTH,
Ludo’.

Re: Shepherd does not recycle zombie processes

[Carlo Zancanaro]

[-- Attachment #1.1: Type: text/plain, Size: 1312 bytes --]


Hey Ludo!

On Mon, Nov 07 2016, Ludovic Courtès wrote
> Could it be that you invoke the ‘daemonize’ action after 
> respawnable processes have been started?  The manual has this 
> caveat (info "(shepherd) The root and unknown services"):
>
>   ‘daemonize’ 
>        Fork and go into the background.  This should be called 
>        before respawnable services are started, as otherwise we 
>        would not get the ‘SIGCHLD’ signals when they terminate.

Yeah, I saw that note in the documentation. I used to have

    (action 'shepherd 'daemonize)

as the first line in ~/.config/shepherd/init.scm. Is there some 
other way that I was supposed to do that?

With that line in place, Shepherd will leave behind a process 
every time I stop/start a service.

I have attached an example init.scm that does this for me. If I 
start: 

    shepherd -c init.scm

and then run:

    herd stop sleep
    herd start sleep
    herd stop sleep
    herd start sleep
    herd stop sleep

then I will have three zombie sleep processes underneath my 
Shepherd process. (If the service were respawnable then it also 
would fail to restart the service.)

I assume this behaviour is wrong, but if I'm doing something wrong 
then please let me know what it is.

Carlo


[-- Attachment #1.2: init.scm --]
[-- Type: application/octet-stream, Size: 221 bytes --]

(action 'shepherd 'daemonize)

(define sleep
  (make <service>
    #:provides '(sleep)
    #:start (make-forkexec-constructor '("sleep" "100"))
    #:stop (make-kill-destructor)))

(register-services sleep)
(start sleep)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: Shepherd does not recycle zombie processes

[Ludovic Courtès]

Hi,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

> Yeah, I saw that note in the documentation. I used to have
>
>    (action 'shepherd 'daemonize)
>
> as the first line in ~/.config/shepherd/init.scm. Is there some other
> way that I was supposed to do that?

No, I think that should work.

> With that line in place, Shepherd will leave behind a process every
> time I stop/start a service.
>
> I have attached an example init.scm that does this for me. If I start: 
>
>    shepherd -c init.scm
>
> and then run:
>
>    herd stop sleep
>    herd start sleep
>    herd stop sleep
>    herd start sleep
>    herd stop sleep
>
> then I will have three zombie sleep processes underneath my Shepherd
> process. (If the service were respawnable then it also would fail to
> restart the service.)

Could you run shepherd in “strace -f” and see where the SIGCHLD signals
go?

Thanks,
Ludo’.

Re: Shepherd does not recycle zombie processes

[Carlo Zancanaro]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]


On Wed, Nov 09 2016, Ludovic Courtès wrote
> Could you run shepherd in “strace -f” and see where the SIGCHLD 
> signals go?

I don't really know how to read strace's output (and there's a lot 
of it), but sometimes it gives a line like this:

     --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, 
     si_pid=10496, si_status=SIGTERM, si_utime=142, si_stime=22} 
     ---

and sometimes it gives a line like this:

    [pid 10465] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, 
    si_pid=10556, si_status=SIGTERM, si_utime=84, si_stime=16} ---

If it has the [pid xxxxx] prefix then the signal is handled 
properly and the child process is reaped. If there's no prefix 
then a zombie process is left behind. (The pid in the prefix is 
the pid of the forked daemon process.)

Using my example init.scm file I have found that if I run `herd 
restart sleep` repeatedly and quickly (< 1 second) then it will 
consistently reap the processes. If I delay for more than about a 
second between runs then it will fail to reap the old process. 
(With strace attached this time is extended to a few seconds.)

Carlo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: Shepherd does not recycle zombie processes

[Dale Mellor]

On Mon, 2016-11-07 at 09:53 +0100, Ludovic Courtès wrote:
> Hi,
> 
> Carlo Zancanaro <carlo@zancanaro.id.au> skribis:
> 
> > I've had problems with Shepherd and its daemonize action. If I run
> > daemonize (as the first thing when Shepherd starts) then it fails to handle
> > signals from child processes.
> 
> Could it be that you invoke the ‘daemonize’ action after respawnable
> processes have been started?  The manual has this caveat (info
> "(shepherd) The root and unknown services"):
> 
>   ‘daemonize’
>        Fork and go into the background.  This should be called before
>        respawnable services are started, as otherwise we would not get the
>        ‘SIGCHLD’ signals when they terminate.
> 
> HTH,
> Ludo’.

  Update: I'm no longer making any use of the daemonize method, instead
simply running as a detached process: `( shepherd & )' at a bash
command-line.  It works perfectly well now, reaping (and restarting)
dead children as necessary.

  Another problem I see though is that if there are jobs which depend on
one which has died, those jobs are not recycled.  Probably this has not
been considered yet, and I'm thinking that a bit more thought needs to
go into the overall design of this thing... (I guess this also ties in
with the concepts of run-levels and hot-re-configuring the shepherd
core).  Wish I had time to look into it.

Tuppence,
/Dale

Re: Shepherd does not recycle zombie processes

[Ludovic Courtès]

Dale Mellor <no-reply@rdmp.org> skribis:

> On Mon, 2016-11-07 at 09:53 +0100, Ludovic Courtès wrote:
>> Hi,
>> 
>> Carlo Zancanaro <carlo@zancanaro.id.au> skribis:
>> 
>> > I've had problems with Shepherd and its daemonize action. If I run
>> > daemonize (as the first thing when Shepherd starts) then it fails to handle
>> > signals from child processes.
>> 
>> Could it be that you invoke the ‘daemonize’ action after respawnable
>> processes have been started?  The manual has this caveat (info
>> "(shepherd) The root and unknown services"):
>> 
>>   ‘daemonize’
>>        Fork and go into the background.  This should be called before
>>        respawnable services are started, as otherwise we would not get the
>>        ‘SIGCHLD’ signals when they terminate.
>> 
>> HTH,
>> Ludo’.
>
>   Update: I'm no longer making any use of the daemonize method, instead
> simply running as a detached process: `( shepherd & )' at a bash
> command-line.  It works perfectly well now, reaping (and restarting)
> dead children as necessary.

OK.  (That’s also what I do for my user Shepherd.)

>   Another problem I see though is that if there are jobs which depend on
> one which has died, those jobs are not recycled.  Probably this has not
> been considered yet, and I'm thinking that a bit more thought needs to
> go into the overall design of this thing... (I guess this also ties in
> with the concepts of run-levels and hot-re-configuring the shepherd
> core).  Wish I had time to look into it.

Certainly, help welcome!  :-)

Thanks,
Ludo’.