bug#74912: Shepherd: Growing number of user shepherds when relogging

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Jake]

[-- Attachment #1: Type: text/plain, Size: 3205 bytes --]

Hi

I think I'm experiencing a bug in Shepherd since version 1.0.
Whenever I log out and log back in again, my user shepherd from the
previous login session is still present, and a new user shepherd spawns for
the current login session.
So relogging N times results in N+1 user shepherds.

For example, I have relogged 5 times since I last rebooted:

$ herd status root
Status of root:
  It is running since 00:30:02 (10 minutes ago).
  Main PID: 23450
  Command:
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
...

$ pgrep shepherd
1
9891
10777
16417
18510
21960
23450

$  ps aux | grep shepherd
root         1  0.0  0.9 222872 74456 ?        Sl   Dec15   0:08
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--config /gnu/store/p7al8wd1inwk8f5di2q4llcpd64mjn5q-shepherd.conf
jake      9891  0.0  0.2  75816 23624 ?        Ss   Dec15   0:04
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     10777  0.0  0.3  76224 24752 ?        Ss   Dec16   0:03
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     16417  0.0  0.3  75752 24004 ?        Ss   Dec16   0:02
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     18510  0.0  0.2  75752 23760 ?        Ss   Dec16   0:01
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     21960  0.0  0.2 114608 22124 ?        Ss   Dec16   0:00
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     23450  0.0  0.2 114204 21328 ?        Ss   00:30   0:00
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9/bin/guile
--no-auto-compile
/gnu/store/nl0w5c7pxxdczqiv4r9iq44al7nd5y5g-shepherd-1.0.0/bin/shepherd
--silent --config /gnu/store/w3l6dmap815mm3qzx77xdazky853adda-shepherd.conf
jake     23672  0.0  0.0   6636  2552 pts/1    S+   00:32   0:00 grep
--color=auto shepherd

In addition, any daemons managed by the zombie shepherds also persist!

I'm experiencing this on both of my Guix System machines. One is running
GDM and XFCE. The other is running GDM and CWM.
Please let me know if I can provide more information.

Thanks
Jake

[-- Attachment #2: Type: text/html, Size: 3846 bytes --]

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Ludovic Courtès]

Hello,

Jake <jforst.mailman@gmail.com> skribis:

> I think I'm experiencing a bug in Shepherd since version 1.0.
> Whenever I log out and log back in again, my user shepherd from the
> previous login session is still present, and a new user shepherd spawns for
> the current login session.
> So relogging N times results in N+1 user shepherds.

I have a user shepherd via Guix Home and I experience the same problem
(though because I rarely log out it’s not really annoying :-)).

I suspect the problem has to do with how Guix Home determines whether or
not it should launch shepherd, but I haven’t checked yet.

Thanks for reporting the issue,
Ludo’.

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Jake <jforst.mailman@gmail.com> skribis:
>
>> I think I'm experiencing a bug in Shepherd since version 1.0.
>> Whenever I log out and log back in again, my user shepherd from the
>> previous login session is still present, and a new user shepherd spawns for
>> the current login session.
>> So relogging N times results in N+1 user shepherds.
>
> I have a user shepherd via Guix Home and I experience the same problem
> (though because I rarely log out it’s not really annoying :-)).
>
> I suspect the problem has to do with how Guix Home determines whether or
> not it should launch shepherd, but I haven’t checked yet.

When you have another login session active when you log out and in
again, new shepherd is *not* spawned.  I am guessing here but probably
last log out causes XDG_RUNTIME_DIR to be removed (by elogind in my
case), so on log in there is no /run/user/$UID/on-first-login-executed,
so it runs again and starts the shepherd.

But even if that would be solved, since the runtime directory was nuked,
there is no shepherd socket around anymore, so the (still running)
shepherd from previous login session cannot be contacted by herd.

Of the top of my head I can think of two possible solutions:

1. Stop the shepherd on log out.  So as we have on-first-login, we would
have on-last-logout.  I have no idea how to implement that.  Maybe we
could use ~/.bash_logout?  Or some PAM thing?

2. Shepherd could shutdown gracefully when the control socket is deleted
from the file system.  It is arguable how useful running shepherd is
without the socket anyway.

Any other ideas?

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Ludovic Courtès]

Hi!

Tomas Volf <~@wolfsden.cz> skribis:

> When you have another login session active when you log out and in
> again, new shepherd is *not* spawned.  I am guessing here but probably
> last log out causes XDG_RUNTIME_DIR to be removed (by elogind in my
> case), so on log in there is no /run/user/$UID/on-first-login-executed,
> so it runs again and starts the shepherd.
>
> But even if that would be solved, since the runtime directory was nuked,
> there is no shepherd socket around anymore, so the (still running)
> shepherd from previous login session cannot be contacted by herd.

Hmm, when is /run/user/UID deleted?

> Of the top of my head I can think of two possible solutions:
>
> 1. Stop the shepherd on log out.  So as we have on-first-login, we would
> have on-last-logout.  I have no idea how to implement that.  Maybe we
> could use ~/.bash_logout?  Or some PAM thing?

Or some elogind thing, rather?

But then, how do we make it work on other distros?  Maybe on systemd
distros shepherd receives SIGTERM or something, in which case it
terminates properly.

> 2. Shepherd could shutdown gracefully when the control socket is deleted
> from the file system.  It is arguable how useful running shepherd is
> without the socket anyway.

I don’t think that’s workable: you’d need to poll/inotify for the
existence of that socket, but even if it exists on the file system, you
cannot tell whether it matches the socket you’re accepting on.

Ludo’.

bug#74912: Shepherd: Growing number of user shepherds when relogging

[bokr]

On +2024-12-26 11:50:00 +0100, Ludovic Courtès wrote:
> Hi!
> 
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > When you have another login session active when you log out and in
> > again, new shepherd is *not* spawned.  I am guessing here but probably
> > last log out causes XDG_RUNTIME_DIR to be removed (by elogind in my
> > case), so on log in there is no /run/user/$UID/on-first-login-executed,
> > so it runs again and starts the shepherd.
> >
> > But even if that would be solved, since the runtime directory was nuked,
> > there is no shepherd socket around anymore, so the (still running)
> > shepherd from previous login session cannot be contacted by herd.
> 
> Hmm, when is /run/user/UID deleted?
> 
> > Of the top of my head I can think of two possible solutions:
> >
> > 1. Stop the shepherd on log out.  So as we have on-first-login, we would
> > have on-last-logout.  I have no idea how to implement that.  Maybe we
> > could use ~/.bash_logout?  Or some PAM thing?
> 
> Or some elogind thing, rather?
> 
> But then, how do we make it work on other distros?  Maybe on systemd
> distros shepherd receives SIGTERM or something, in which case it
> terminates properly.
> 
> > 2. Shepherd could shutdown gracefully when the control socket is deleted
> > from the file system.  It is arguable how useful running shepherd is
> > without the socket anyway.
> 
> I don’t think that’s workable: you’d need to poll/inotify for the
> existence of that socket, but even if it exists on the file system, you
> cannot tell whether it matches the socket you’re accepting on.
> 
> Ludo’.
> 
> 
> 

I wonder how many guix-daemon-process-relationship type problems would be simplified
if (radical vision) one let wayland's inner event-driven loop/protocol be the dispatcher
for guix processes instead of the current guix daemon switching between its collection of threads.
I.e., all the guix threads would be individual login or spawned user processes securely communicating
virtualizably (shared memory or networked rendezvous buffers etc) for offloading?

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]


I am not sure how this relates to this specific bug report, but

bokr@bokr.com writes:

> I wonder how many guix-daemon-process-relationship type problems would be simplified
> if (radical vision) one let wayland's inner event-driven loop/protocol
> be the dispatcher

not everyone uses wayland.

> for guix processes instead of the current guix daemon switching between its collection of threads.
> I.e., all the guix threads would be individual login or spawned user processes securely communicating
> virtualizably (shared memory or networked rendezvous buffers etc) for offloading?

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

bug#74912: Shepherd: Growing number of user shepherds when relogging

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 2722 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Tomas Volf <~@wolfsden.cz> skribis:
>
>> When you have another login session active when you log out and in
>> again, new shepherd is *not* spawned.  I am guessing here but probably
>> last log out causes XDG_RUNTIME_DIR to be removed (by elogind in my
>> case), so on log in there is no /run/user/$UID/on-first-login-executed,
>> so it runs again and starts the shepherd.
>>
>> But even if that would be solved, since the runtime directory was nuked,
>> there is no shepherd socket around anymore, so the (still running)
>> shepherd from previous login session cannot be contacted by herd.
>
> Hmm, when is /run/user/UID deleted?

I believe it is done by elogind (in my setup) when last user session
(for the given UID) logs out.  If I grepped right, it is done by
user_finalize function in logind-user.c.

It (AFAIUT) it should be performed when last session of the seat
terminates.  So if you log only into a single TTY, the XDG_RUNTIME_DIR
will be removed on every log out.

>
>> Of the top of my head I can think of two possible solutions:
>>
>> 1. Stop the shepherd on log out.  So as we have on-first-login, we would
>> have on-last-logout.  I have no idea how to implement that.  Maybe we
>> could use ~/.bash_logout?  Or some PAM thing?
>
> Or some elogind thing, rather?

I looked around the manual page, but did not found anything.  There is
KillUserProcesses, but that feels like fairly big hammer, and something
that should *not* be enabled by default.

We could patch elogind to add new RemoveRuntimeDirectory boolean flag to
allow keeping the XDG_RUNTIME_DIR even after last log out (I personally
would prefer that behavior anyway).  I am not sure what our policy
regarding patches here is.

>
> But then, how do we make it work on other distros?  Maybe on systemd
> distros shepherd receives SIGTERM or something, in which case it
> terminates properly.

No idea here.  ~/.bash_logout?

>
>> 2. Shepherd could shutdown gracefully when the control socket is deleted
>> from the file system.  It is arguable how useful running shepherd is
>> without the socket anyway.
>
> I don’t think that’s workable: you’d need to poll/inotify for the
> existence of that socket, but even if it exists on the file system, you
> cannot tell whether it matches the socket you’re accepting on.

For files I would suggest checking if both `stat:dev' and `stat:ino'
match in order to detect whether it is the same file.  Not sure if same
strategy can be used for unix sockets.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]