bug#66297: guix-daemon not starting after boot

bug#66297: guix-daemon not starting after boot

[Sharlatan Hellseher]

Hi Guix!

After the recent pull and system reconfigure I've started experiencing
guix-daemon not starting up after the system boot.

~# guix describe
Generation 29   Sep 27 2023 21:59:12    (current)
  guix 0500af5
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 0500af5556307d0a4c14a23864e9e4bccd2643d7
  nonguix bb184bd
    repository URL: https://gitlab.com/nonguix/nonguix
    branch: master
    commit: bb184bd0a8f91beec3a00718759e96c7828853de

~# guix pull
guix pull: error: failed to connect to
`/var/guix/daemon-socket/socket': Connection refused
~# herd status guix-daemon
Status of guix-daemon:
  It is stopped.
  It is enabled.
  Provides (guix-daemon).
  Requires (user-processes).
  Will be respawned.

Logs are empty:
~# tail /var/log/guix-daemon.log | wc -l
0


-- 
… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

bug#66297: guix-daemon not starting after boot

[Ludovic Courtès]

Hi,

Sharlatan Hellseher <sharlatanus@gmail.com> skribis:

> After the recent pull and system reconfigure I've started experiencing
> guix-daemon not starting up after the system boot.

Anything in /var/log/messages?

What does ‘herd log’ say?

Thanks,
Ludo’.

bug#66297: guix-daemon not starting after boot

[Sharlatan Hellseher]

Hi,

I think I just get used to kick it manually and forgot to replay on
this issue =)

No Herd/Shepherd logs since Sept 30 in messages

 sudo grep -i "shepherd\|herd" /var/log/messages  | tail
grep: /var/log/messages: binary file matches
Sep 30 21:16:03 localhost shepherd[1]: Service console-font-tty3
running with value #t.
Sep 30 21:16:03 localhost shepherd[1]: Service console-font-tty4 started.
Sep 30 21:16:03 localhost vmunix: [    2.427621] shepherd[1]:
Sep 30 21:16:03 localhost vmunix: [    2.428262] shepherd[1]:
Sep 30 21:16:03 localhost vmunix: [    2.429207] shepherd[1]: Service
sysctl has been started.
Sep 30 21:16:03 localhost vmunix: [    2.429434] shepherd[1]: Service
sysctl started.
Sep 30 21:16:03 localhost vmunix: [    2.429666] shepherd[1]: Service
sysctl running with value #t.
Sep 30 21:16:03 localhost vmunix: [    2.430082] shepherd[1]: Starting
service virtual-terminal...
Sep 30 21:16:03 localhost vmunix: [    2.430416] shepherd[1]: Service
virtual-terminal started.
Sep 30 21:16:03 localhost vmunix: [    2.430649] shepherd[1]: Service
virtual-terminal running with value #t.

There this no track of guix-daemon was starting in herd log

herd log | grep daemon
10 Oct 2023 00:03:30    service upower-daemon is being started
10 Oct 2023 00:03:30    service upower-daemon is running
10 Oct 2023 00:03:30    service ssh-daemon is being started
10 Oct 2023 00:03:30    service ssh-daemon is running

Thanks,
Oleg

On Thu, 5 Oct 2023 at 14:39, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi,
>
> Sharlatan Hellseher <sharlatanus@gmail.com> skribis:
>
> > After the recent pull and system reconfigure I've started experiencing
> > guix-daemon not starting up after the system boot.
>
> Anything in /var/log/messages?
>
> What does ‘herd log’ say?
>
> Thanks,
> Ludo’.



-- 

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

bug#66297: guix-daemon not starting after boot

[Ludovic Courtès]

Hi Sharlatan,

Sharlatan Hellseher <sharlatanus@gmail.com> skribis:

> I think I just get used to kick it manually and forgot to replay on
> this issue =)

Ouch.  :-)

>  sudo grep -i "shepherd\|herd" /var/log/messages  | tail
> grep: /var/log/messages: binary file matches

Looks like grep didn’t actually output matches, did it?

Could you screen the file, starting from the latest boot, in search of
references to guix-daemon?

> There this no track of guix-daemon was starting in herd log
>
> herd log | grep daemon
> 10 Oct 2023 00:03:30    service upower-daemon is being started
> 10 Oct 2023 00:03:30    service upower-daemon is running
> 10 Oct 2023 00:03:30    service ssh-daemon is being started
> 10 Oct 2023 00:03:30    service ssh-daemon is running

What about ‘sudo herd status’?  Does guix-daemon show up there?

Also, could you check:

  guix system shepherd-graph /run/current-system/configuration.scm | \
    guix shell xdot -- xdot -

to make sure guix-daemon actually is in your config, to be 100% sure?

Thanks,
Ludo’.

bug#66297: guix-daemon not starting after boot

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]

Hi Ludo,

Thank you for your time and help.

I creaed a simple script, it may be helpfull for quick glance on guix healh in
general =)

I try to include all discussed steps I've tried to have a broad picture.

> ./guix-health.sh
> [1697156132.285]: [ PASS ] verification of required commands completed
> [1697156132.315]: [ PASS ] Guix socket file is present in /var/guix/daemon-socket/socket
> [1697156132.318]: [ WARN ] guix-daemon is stopped
> [1697156132.323]: [ WARN ] guix-daemon is not listed in current system config

Sharlatan Hellseher (1):
  etc: Add guix-health.sh

 etc/guix-health.sh | 145 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 145 insertions(+)
 create mode 100755 etc/guix-health.sh


base-commit: 5a3fb306535c2ec0a118f2f0bc8f6a162b85c7d6
-- 
2.41.0

On Thu, 12 Oct 2023 at 08:20, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi Sharlatan,
>
> Sharlatan Hellseher <sharlatanus@gmail.com> skribis:
>
> > I think I just get used to kick it manually and forgot to replay on
> > this issue =)
>
> Ouch.  :-)
>
> >  sudo grep -i "shepherd\|herd" /var/log/messages  | tail
> > grep: /var/log/messages: binary file matches
>
> Looks like grep didn’t actually output matches, did it?
>
> Could you screen the file, starting from the latest boot, in search of
> references to guix-daemon?
>
> > There this no track of guix-daemon was starting in herd log
> >
> > herd log | grep daemon
> > 10 Oct 2023 00:03:30    service upower-daemon is being started
> > 10 Oct 2023 00:03:30    service upower-daemon is running
> > 10 Oct 2023 00:03:30    service ssh-daemon is being started
> > 10 Oct 2023 00:03:30    service ssh-daemon is running
>
> What about ‘sudo herd status’?  Does guix-daemon show up there?
>
> Also, could you check:
>
>   guix system shepherd-graph /run/current-system/configuration.scm | \
>     guix shell xdot -- xdot -
>
> to make sure guix-daemon actually is in your config, to be 100% sure?
>
> Thanks,
> Ludo’.



-- 

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

[-- Attachment #2: 0001-etc-Add-guix-health.sh.patch --]
[-- Type: text/x-patch, Size: 4504 bytes --]

From cf698536c6393032274c6d8539f366acaa036e98 Mon Sep 17 00:00:00 2001
Message-ID: <cf698536c6393032274c6d8539f366acaa036e98.1697156963.git.sharlatanus@gmail.com>
In-Reply-To: <cover.1697156963.git.sharlatanus@gmail.com>
References: <cover.1697156963.git.sharlatanus@gmail.com>
From: Sharlatan Hellseher <sharlatanus@gmail.com>
Date: Fri, 13 Oct 2023 01:27:17 +0100
Subject: [PATCH 1/1] etc: Add guix-health.sh

* etc/guix-health.sh: New file.
---
 etc/guix-health.sh | 145 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 145 insertions(+)
 create mode 100755 etc/guix-health.sh

diff --git a/etc/guix-health.sh b/etc/guix-health.sh
new file mode 100755
index 0000000000..9078d52ce6
--- /dev/null
+++ b/etc/guix-health.sh
@@ -0,0 +1,145 @@
+#!/bin/sh
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2023 sharlatan <sharlatanus@gmail.com>
+
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+# We require Bash but for portability we'd rather not use /bin/bash or
+# /usr/bin/env in the shebang, hence this hack.
+
+## Commentary:
+#
+# General purpose check script which may be used as report during diagnostics of
+# issues with running system in general and Guix in particularly.
+
+## Code:
+
+if [ "x$BASH_VERSION" = "x" ]
+then
+    exec bash "$0" "$@"
+fi
+
+set -eo pipefail
+
+[ "$UID" -eq 0 ] || { echo "This script must be run as root."; exit 1; }
+
+REQUIRE=(
+    "herd"
+    "stat"
+    "grep"
+    "cat"
+)
+
+PAS=$'[ \033[32;1mPASS\033[0m ] '
+ERR=$'[ \033[31;1mFAIL\033[0m ] '
+WAR=$'[ \033[33;1mWARN\033[0m ] '
+INF="[ INFO ] "
+
+DEBUG=0
+
+# Map of all vital predictable files required to diagnose health condition of
+# the guix-daemon.
+declare -A GUIX_FILE
+GUIX_FILE=(
+    [socket]="/var/guix/daemon-socket/socket"
+    [configuration]="/run/current-system/configuration.scm"
+)
+
+# ------------------------------------------------------------------------------
+#+UTILITIES
+
+_err()
+{ # All errors go to stderr.
+    printf "[%s]: %s\n" "$(date +%s.%3N)" "$1"
+}
+
+_msg()
+{ # Default message to stdout.
+    printf "[%s]: %s\n" "$(date +%s.%3N)" "$1"
+}
+
+_debug()
+{
+    if [ "${DEBUG}" = '1' ]; then
+        printf "[%s]: %s\n" "$(date +%s.%3N)" "$1"
+    fi
+}
+
+die()
+{
+    _err "${ERR}$*"
+    exit 1
+}
+
+chck_require()
+{ # Check that every required command is available.
+    declare -a warn
+    local c
+
+    _debug "--- [ ${FUNCNAME[0]} ] ---"
+
+    for c in "$@"; do
+        command -v "$c" &>/dev/null || warn+=("$c")
+    done
+
+    [ "${#warn}" -ne 0 ] && die "Missing commands: ${warn[*]}."
+
+    _msg "${PAS}verification of required commands completed"
+}
+
+# ------------------------------------------------------------------------------
+#+CHECKS :: Collection of grouped check to validate Guix system health.
+
+chck_daemon()
+{ # Basic assumptions and check list to show what's going on with guix-daemon,
+  # some of the checks can't be performed if daemon is not running (e.g guix
+  # system shepherd-graph)
+
+    local guix_daemon_status
+
+    _debug "--- [ ${FUNCNAME[0]} ] ---"
+
+    guix_daemon_status="$(herd status | grep guix-daemon | cut -d' ' -f2)"
+
+    if [ "$(stat --format=%F ${GUIX_FILE[socket]})" == 'socket' ]
+    then
+        _msg "${PAS}Guix socket file is present in ${GUIX_FILE[socket]}"
+    fi
+
+    if [ "$guix_daemon_status" == "+" ]
+    then
+        _msg "${PAS}guix-daemon is running"
+    elif [ "$guix_daemon_status" == "-" ]
+    then
+        _msg "${WAR}guix-daemon is stopped"
+    fi
+
+    if grep guix-daemon "${GUIX_FILE[configuration]}" &>/dev/null
+    then
+        _msg "${PAS}guix-daemon is listed in current system config"
+    else
+        _msg "${WAR}guix-daemon is not listed in current system config"
+    fi
+}
+
+main()
+{
+    chck_require
+    chck_daemon
+    # XXX: Add more, event more!
+}
+
+main "$@"
-- 
2.41.0


[-- Attachment #3: guix-config.png --]
[-- Type: image/png, Size: 571393 bytes --]

bug#66297: guix-daemon not starting after boot

[Ludovic Courtès]

Hello,

Sharlatan Hellseher <sharlatanus@gmail.com> skribis:

> I creaed a simple script, it may be helpfull for quick glance on guix healh in
> general =)

Heh.  :-)  Could you answer my earlier questions:

>> > There this no track of guix-daemon was starting in herd log
>> >
>> > herd log | grep daemon
>> > 10 Oct 2023 00:03:30    service upower-daemon is being started
>> > 10 Oct 2023 00:03:30    service upower-daemon is running
>> > 10 Oct 2023 00:03:30    service ssh-daemon is being started
>> > 10 Oct 2023 00:03:30    service ssh-daemon is running
>>
>> What about ‘sudo herd status’?  Does guix-daemon show up there?
>>
>> Also, could you check:
>>
>>   guix system shepherd-graph /run/current-system/configuration.scm | \
>>     guix shell xdot -- xdot -
>>
>> to make sure guix-daemon actually is in your config, to be 100% sure?

TIA,
Ludo’.