bug#57844: Shepherd fails to start in user session ~50% of the time

bug#57844: Shepherd fails to start in user session ~50% of the time

[Tom Willemse]

Hi Guix!

I've been using Guix on Archlinux for a little while now, and ever since
I've started using Guix Home on my laptop to start up user-level
services I've been having the issue that about 50% of the time when I
boot my laptop shepherd fails to start. 

My .xsession-errors says:

> shepherd: while opening socket '/run/user/1000/shepherd/socket': bind:
> Address already in use

and looking at my shepherd log:

> 2022-09-15 11:47:18 Service root has been started.
> 2022-09-15 11:47:18 Service root has been started.
> 2022-09-15 11:47:19 Starting services...
> 2022-09-15 11:47:19 Starting services...
> 2022-09-15 11:47:19 Exiting shepherd...
> 2022-09-15 11:47:19 Service dunst has been started.
> 2022-09-15 11:47:19 Service unclutter has been started.
> 2022-09-15 11:47:19 Service syncthing has been started.
> 2022-09-15 11:47:19 Service polybar has been started.
> 2022-09-15 11:47:19 Service cmst has been started.
> 2022-09-15 11:47:19 Service kdeconnect has been started.
> 2022-09-15 11:47:20 Service xbindkeys has been started.
> 2022-09-15 11:47:20 Service picom has been started.
> 2022-09-15 11:47:20 Service xmodmap has been started.
> 2022-09-15 11:47:20 Service redshift has been started.
> 2022-09-15 11:47:20 Exiting shepherd...
> 2022-09-15 11:47:20 Service syncthing has been stopped.
> 2022-09-15 11:47:20 Service xbindkeys has been stopped.
> 2022-09-15 11:47:20 Service redshift has been stopped.
> 2022-09-15 11:47:20 Service cmst has been stopped.
> 2022-09-15 11:47:20 Service kdeconnect has been stopped.
> 2022-09-15 11:47:20 Service polybar has been stopped.
> 2022-09-15 11:47:20 Service dunst has been stopped.
> 2022-09-15 11:47:20 Service picom has been stopped.
> 2022-09-15 11:47:20 Service unclutter has been stopped.
> 2022-09-15 11:47:20 Exiting.

It looks like it starts twice and then exits both, but I'm not sure why.
I'm guessing it's the ~/.guix-home/activate and
~/.guix-home/on-first-login that are trying to start it.

I'm not sure what other information I can provide you that will help, so
please let me know!


Cheers,

Tom

bug#57844: Shepherd fails to start in user session ~50% of the time

[Ludovic Courtès]

Hi,

Andrew, does the bug report below ring a bell?

  https://issues.guix.gnu.org/57844

(I haven’t hit that problem myself.)

Ludo’.

Tom Willemse <tom@ryuslash.org> skribis:

> Hi Guix!
>
> I've been using Guix on Archlinux for a little while now, and ever since
> I've started using Guix Home on my laptop to start up user-level
> services I've been having the issue that about 50% of the time when I
> boot my laptop shepherd fails to start. 
>
> My .xsession-errors says:
>
>> shepherd: while opening socket '/run/user/1000/shepherd/socket': bind:
>> Address already in use
>
> and looking at my shepherd log:
>
>> 2022-09-15 11:47:18 Service root has been started.
>> 2022-09-15 11:47:18 Service root has been started.
>> 2022-09-15 11:47:19 Starting services...
>> 2022-09-15 11:47:19 Starting services...
>> 2022-09-15 11:47:19 Exiting shepherd...
>> 2022-09-15 11:47:19 Service dunst has been started.
>> 2022-09-15 11:47:19 Service unclutter has been started.
>> 2022-09-15 11:47:19 Service syncthing has been started.
>> 2022-09-15 11:47:19 Service polybar has been started.
>> 2022-09-15 11:47:19 Service cmst has been started.
>> 2022-09-15 11:47:19 Service kdeconnect has been started.
>> 2022-09-15 11:47:20 Service xbindkeys has been started.
>> 2022-09-15 11:47:20 Service picom has been started.
>> 2022-09-15 11:47:20 Service xmodmap has been started.
>> 2022-09-15 11:47:20 Service redshift has been started.
>> 2022-09-15 11:47:20 Exiting shepherd...
>> 2022-09-15 11:47:20 Service syncthing has been stopped.
>> 2022-09-15 11:47:20 Service xbindkeys has been stopped.
>> 2022-09-15 11:47:20 Service redshift has been stopped.
>> 2022-09-15 11:47:20 Service cmst has been stopped.
>> 2022-09-15 11:47:20 Service kdeconnect has been stopped.
>> 2022-09-15 11:47:20 Service polybar has been stopped.
>> 2022-09-15 11:47:20 Service dunst has been stopped.
>> 2022-09-15 11:47:20 Service picom has been stopped.
>> 2022-09-15 11:47:20 Service unclutter has been stopped.
>> 2022-09-15 11:47:20 Exiting.
>
> It looks like it starts twice and then exits both, but I'm not sure why.
> I'm guessing it's the ~/.guix-home/activate and
> ~/.guix-home/on-first-login that are trying to start it.
>
> I'm not sure what other information I can provide you that will help, so
> please let me know!
>
>
> Cheers,
>
> Tom

bug#57844: Shepherd fails to start in user session ~50% of the time

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 4086 bytes --]

On 2022-10-19 18:21, Ludovic Courtès wrote:

> Hi,
>
> Andrew, does the bug report below ring a bell?
>

Yes, I don't remember if I created a thread on that (probably not) or
just discussed it in some chat, but when shepherd stops it doesn't clean
up its socket file, so you can't start shepherd again until manually
remove socket.

Checked it right now:
--8<---------------cut here---------------start------------->8---
herd stop root
shepherd # fails with Address already in use
--8<---------------cut here---------------end--------------->8---

I found it out, when was experimenting with the place, where I start
shepherd https://issues.guix.gnu.org/57692.  To inherit graphical
environment variables I start it by sway compositor, not login shell and
if in addition to sway session I login on another tty, elogind won't
remove XDG_RUNTIME_DIR => shepherd/socket is not removed => shepherd
fails to start after sway restart.

>   https://issues.guix.gnu.org/57844
>
> (I haven’t hit that problem myself.)
>
> Ludo’.
>
> Tom Willemse <tom@ryuslash.org> skribis:
>
>> Hi Guix!
>>
>> I've been using Guix on Archlinux for a little while now, and ever since
>> I've started using Guix Home on my laptop to start up user-level
>> services I've been having the issue that about 50% of the time when I
>> boot my laptop shepherd fails to start. 
>>
>> My .xsession-errors says:
>>
>>> shepherd: while opening socket '/run/user/1000/shepherd/socket': bind:
>>> Address already in use
>>
>> and looking at my shepherd log:
>>
>>> 2022-09-15 11:47:18 Service root has been started.
>>> 2022-09-15 11:47:18 Service root has been started.
>>> 2022-09-15 11:47:19 Starting services...
>>> 2022-09-15 11:47:19 Starting services...
>>> 2022-09-15 11:47:19 Exiting shepherd...
>>> 2022-09-15 11:47:19 Service dunst has been started.
>>> 2022-09-15 11:47:19 Service unclutter has been started.
>>> 2022-09-15 11:47:19 Service syncthing has been started.
>>> 2022-09-15 11:47:19 Service polybar has been started.
>>> 2022-09-15 11:47:19 Service cmst has been started.
>>> 2022-09-15 11:47:19 Service kdeconnect has been started.
>>> 2022-09-15 11:47:20 Service xbindkeys has been started.
>>> 2022-09-15 11:47:20 Service picom has been started.
>>> 2022-09-15 11:47:20 Service xmodmap has been started.
>>> 2022-09-15 11:47:20 Service redshift has been started.
>>> 2022-09-15 11:47:20 Exiting shepherd...
>>> 2022-09-15 11:47:20 Service syncthing has been stopped.
>>> 2022-09-15 11:47:20 Service xbindkeys has been stopped.
>>> 2022-09-15 11:47:20 Service redshift has been stopped.
>>> 2022-09-15 11:47:20 Service cmst has been stopped.
>>> 2022-09-15 11:47:20 Service kdeconnect has been stopped.
>>> 2022-09-15 11:47:20 Service polybar has been stopped.
>>> 2022-09-15 11:47:20 Service dunst has been stopped.
>>> 2022-09-15 11:47:20 Service picom has been stopped.
>>> 2022-09-15 11:47:20 Service unclutter has been stopped.
>>> 2022-09-15 11:47:20 Exiting.
>>
>> It looks like it starts twice and then exits both, but I'm not sure why.
>> I'm guessing it's the ~/.guix-home/activate and
>> ~/.guix-home/on-first-login that are trying to start it.

~/.guix-home/activate should be launched only by guix home reconfigure,
so it shouldn't be touched during startup of the session at all, also
they both have a condition, which must prevent the start of shepherd if
socket exists.

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services/shepherd.scm?h=883fb8f41b08a8455f16c736a83fb1ae8a3df0a1#n105
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services/shepherd.scm?h=883fb8f41b08a8455f16c736a83fb1ae8a3df0a1#n131

Tom, can you show your startup scripts, please (like xsession or
whatever you use for starting graphical environment)?  Sharing home
environment config can be useful as well.  Do you use some display/login
manager?

>>
>> I'm not sure what other information I can provide you that will help, so
>> please let me know!
>>


-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#57844: Shepherd fails to start in user session ~50% of the time

[Ludovic Courtès]

Hi,

Andrew Tropin <andrew@trop.in> skribis:

> Yes, I don't remember if I created a thread on that (probably not) or
> just discussed it in some chat, but when shepherd stops it doesn't clean
> up its socket file, so you can't start shepherd again until manually
> remove socket.

Right, but in the context of Guix Home, it should be started only once
anyway, right?

Thanks,
Ludo’.

bug#57844: Shepherd fails to start in user session ~50% of the time

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 489 bytes --]

On 2022-10-20 23:44, Ludovic Courtès wrote:

> Hi,
>
> Andrew Tropin <andrew@trop.in> skribis:
>
>> Yes, I don't remember if I created a thread on that (probably not) or
>> just discussed it in some chat, but when shepherd stops it doesn't clean
>> up its socket file, so you can't start shepherd again until manually
>> remove socket.
>
> Right, but in the context of Guix Home, it should be started only once
> anyway, right?

Right.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#57844: Shepherd fails to start in user session ~50% of the time

[Tom Willemse]

Hey!

Thanks for getting back to me on this! :)

Andrew Tropin <andrew@trop.in> writes:

> On 2022-10-19 18:21, Ludovic Courtès wrote:
>
>>> It looks like it starts twice and then exits both, but I'm not sure why.
>>> I'm guessing it's the ~/.guix-home/activate and
>>> ~/.guix-home/on-first-login that are trying to start it.
>
> ~/.guix-home/activate should be launched only by guix home reconfigure,
> so it shouldn't be touched during startup of the session at all, also
> they both have a condition, which must prevent the start of shepherd if
> socket exists.

Yeah I know that they both have a check for the pid file, I remember
finding your bug report and patch earlier and hoping that it would fix
my issues, although now I understand that this would never have helped.
I thought that they might still both try to start shepherd, since
neither check nor execution of shepherd is atomic, so I figured they
might end up both starting shepherd, one creating the pid and starting,
the other failing to create it and crashing, but I couldn't think of why
the first one would stop as well then. Anyway, I guess that is
completely wrong if they don't actually ever get run at the same time :)

> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services/shepherd.scm?h=883fb8f41b08a8455f16c736a83fb1ae8a3df0a1#n105
> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services/shepherd.scm?h=883fb8f41b08a8455f16c736a83fb1ae8a3df0a1#n131
>
> Tom, can you show your startup scripts, please (like xsession or
> whatever you use for starting graphical environment)?  Sharing home
> environment config can be useful as well.  Do you use some display/login
> manager?

My ~/.xinitrc is literally just:

    exec /gnu/store/rl6f55i52lldzmbgg6z0ywr41ni4kbjg-herbstluftwm-0.9.4/bin/herbstluftwm --autostart /gnu/store/lqif0y90y7ipvf9aakdjisygnffk4999-autostart

I use the lightdm display manager installed with the Archlinux package
manager.

My home configuration on my laptop should be based on this:

https://git.sr.ht/~ryuslash/dotfiles/tree/7771b3e1fd16a30bee09b11ecbaf2b1b69b3921b/item/oni/home/data/config.scm

It's spread across a few files, so if there's anything specific you want
to know please tell me, or if you need it in a different format I can
provide that too, I figured this would be the easiest way.

Thanks for your help!


Cheers,

Tom

bug#57844: (no subject)

[Zacchaeus Scheffer]

Hi all,

Not sure if it is relevant, but I have often had this problem, and it
has always been from an orphaned syncthing process.  I.e, the user login
session which has my guix home services running in it ends, but the
syncthing process is not terminated.  Then I start a new login, and it
tries to start a NEW syncthing service, and gives this same error.

You can check this (next time syncthing fails to start) with:
ps -e | grep syncthing

Hope this helps,
Zacchae

bug#57844:

[Zacchaeus Scheffer]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

Oh wow, should have read closer.  That's a shepherd socket, not a syncthing
socket.  (happened across this thread searching syncthing)

Please Disregard,
Zacchae

[-- Attachment #2: Type: text/html, Size: 255 bytes --]