bug#28772: guix system reconfigure after kernel panic user or group not created

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

Hello Guix,

During 'guix system reconfigure' I got a kernel panic.

Getting it quite often while 'guix system reconfigure' at Linux magnolia
4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
report.

After reboot I tried another attempt to make a 'guix system
reconfigure', but because of /etc/group.lock and /etc/passwd.lock
operating-system groups and users fields didn't applied.

I tried to get a creation of adbusers group and add my user to it, but
because of those files I cannot do it, until manually removed them.

So, do we need to check if *.lock files are exist and remove them at the
start of 'guix system reconfigure' OR abort reconfigure and creation of
Grub boot entry OR something else?

Thanks,
Oleg.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

Oleg Pykhalov <go.wigust@gmail.com> writes:

[...]

> Getting it quite often while 'guix system reconfigure' at Linux magnolia
> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
> report.

Kernel panic was caused by my miss configuring.  I used a Guile module
on reconfigure via GUILE_LOAD_PATH and (use-modles (my-module) in
config.scm.  I guess it's also required after reconfigure, according to
some times getting errors after reconfigure: cannot find my-module.  So
I need to add my-module globally on system.

[...]

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

Hello,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> During 'guix system reconfigure' I got a kernel panic.

Can you show the exact command and its output?

A user-land program is not supposed to be able to cause a kernel panic;
if it does, that’s a kernel bug.

But perhaps you got the kernel panic *after* rebooting in the
reconfigured system?  That could well be a GuixSD bug, indeed.

> After reboot I tried another attempt to make a 'guix system
> reconfigure', but because of /etc/group.lock and /etc/passwd.lock
> operating-system groups and users fields didn't applied.
>
> I tried to get a creation of adbusers group and add my user to it, but
> because of those files I cannot do it, until manually removed them.
>
> So, do we need to check if *.lock files are exist and remove them at the
> start of 'guix system reconfigure' OR abort reconfigure and creation of
> Grub boot entry OR something else?

I’m not sure I understand what happened.  Could you copy/paste commands
and outputs, or transcribe what happened?

TIA,
Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> Oleg Pykhalov <go.wigust@gmail.com> writes:
>
> [...]
>
>> Getting it quite often while 'guix system reconfigure' at Linux magnolia
>> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
>> report.
>
> Kernel panic was caused by my miss configuring.  I used a Guile module
> on reconfigure via GUILE_LOAD_PATH and (use-modles (my-module) in
> config.scm.  I guess it's also required after reconfigure, according to
> some times getting errors after reconfigure: cannot find my-module.  So
> I need to add my-module globally on system.

Without seeing the config, I cannot tell whether it’s a bug on our side
or not.

However, if you’re confident that the bug is in your code, that’s good.
;-)  Should we close this bug?

Thanks,
Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

Hello Ludovic,

apologies for not adding logs before.  It's hard to do when I do guix
commands from Xterm and not from Emacs.  Emacs *shell* or *compilation*
buffers will eat all memory if they get too much text.

Probably need to redirect STDOUT STDERR in file when Xterm do guix.

I heard Guix folks work on implementing tiny log output to console and
redirect everything else to a log file.  This will be my life saver.

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Oleg Pykhalov <go.wigust@gmail.com> skribis:
>
>> During 'guix system reconfigure' I got a kernel panic.
>
> Can you show the exact command and its output?

Sorry, as I said this is not a topic and I don't want to do it again and
I caution to make it on my current system.

I will setup a specific Guix VM for this, where I could make a 'system
reconfigure'.  Then I'll create a new bug report with full log.

Neverless I'll leave a how-to reproduce it below for at least for myself
TODO list.

The problem
===========

The bigger problem from my view are files like /etc/group.lock and
/etc/passwd.lock.  For example:

    sudo touch /etc/group.lock

/etc/config.scm

    (operating-system
      ;; …
      (groups (cons
         (user-group (name "test"))
         %base-groups)))

reconfigure log

    

[-- Attachment #2: $ guix system reconfigure $HOME/dotfiles/guix/system-magnolia.scm --]
[-- Type: text/plain, Size: 3572 bytes --]

$ guix system reconfigure $HOME/dotfiles/guix/system-magnolia.scm

substitute: updating list of substitutes from 'https://berlin.guixsd.org'... 100.0%
The following derivations will be built:
   /gnu/store/v9dp6193rpxrx1rqfdw59s5ss4wlrfdh-system.drv
   /gnu/store/carkycnf6zcarbmnk5745pgsx1nv478y-grub.cfg.drv
   /gnu/store/r5p953fx3dl18aav1ggwmiy2bqnv991s-activate-service.drv
   /gnu/store/pjjm6595562ysk40zjrznhmsfsid1k8r-activate.drv
   /gnu/store/l41adszqk24sb200dwm8sj6ky71ivwpi-boot.drv
/gnu/store/qqhzapsv5w8mrbz3s8hgy7w42r3dbyv9-system
/gnu/store/b4i4drp7lpxmgpcfkbvgmrig2hlszl3j-grub.cfg
/gnu/store/0b459jxdmyz5vf22avav9sm8ig03173k-grub-efi-2.02
/gnu/store/ijw065yljn1np4x0p5l1qkx9w4z9ikcl-bootloader-installer
activating system...
making '/gnu/store/qqhzapsv5w8mrbz3s8hgy7w42r3dbyv9-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/iyr9ji3idg3iphi3fskh2hqjlmg4h5w0-etc...
usermod: no changes
adding group 'test'...
groupadd: existing lock file /etc/group.lock without a PID
groupadd: cannot lock /etc/group; try again later.
usermod: no changes
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: no changes
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: existing lock file /etc/group.lock without a PID
usermod: cannot lock /etc/group; try again later.
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
usermod: no changes
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: [alert] could not open error log file: open() "/gnu/store/vyj2vkmdmlpxn3mnj71vz8zc8j30ahkf-nginx-1.12.1/logs/error.log" failed (2: No such file or directory)
nginx: the configuration file /gnu/store/xms1g2z62rcj2h9i9d6fbqyl65a4yycm-nginx.conf syntax is ok
nginx: configuration file /gnu/store/xms1g2z62rcj2h9i9d6fbqyl65a4yycm-nginx.conf test is successful
guix system: unloading service 'user-homes'...
shepherd: Removing service 'user-homes'...
shepherd: Done.
guix system: loading new services: user-homes...
shepherd: Evaluating user expression (register-services (primitive-load "/gnu/sto?")).
shepherd: Service user-homes could not be started.
Installing for x86_64-efi platform.
Installation finished. No error reported.

[-- Attachment #3: Type: text/plain, Size: 2480 bytes --]


The new system generation was produced without "test" group, so you
could reboot into it.  And it could lead to problems if we will have a
tiny output to console and big output to a log file, I guess.

> A user-land program is not supposed to be able to cause a kernel panic;
> if it does, that’s a kernel bug.

                       How to make a kernel panic

The problem will be No defined variable IPTABLES-SSH after 'guix system
reconfigure' and kernel crash after.

$HOME/src/iptables/iptables/ru.scm

    (define-module (iptables ru)
      ;; …
      )

    (define %iptables-ssh
       "-A INPUT -p tcp --dport 22 \
    -m state --state NEW -m recent --set --name SSH -j ACCEPT")

/etc/config.scm

    (use-modules ;; …
                 (iptables ru))

    (define start-firewall
      #~(let ((iptables
               (lambda (str)
                 (zero? (system (string-join `(,#$(file-append iptables
                                                               "/sbin/iptables")
                                               ,str) " "))))))
          (format #t "Install iptables rules.~%")
          (and
           ;; …
           (iptables %iptables-ssh))))

    (define firewall-service
      (simple-service 'firewall shepherd-root-service-type
                      (list
                       (shepherd-service
                        (provision '(firewall))
                        (requirement '())
                        (start #~(lambda _
                                   #$start-firewall))
                        (respawn? #f)
                        (stop #~(lambda _
                                  (zero?
                                   (system* #$(file-append iptables
                                                           "/sbin/iptables")
                                            "-F"))))))))

    (operating-system
      ;; …
        (services (cons* ;; …
                         firewall-service)))


Make a kernel panic

    sudo GUILE_LOAD_PATH=\"$HOME/src/iptables\
    :$GUILE_LOAD_PATH\" guix system reconfigure \
    $HOME/dotfiles/guix/system-magnolia.scm

    # Run above again and kernel will panic.

> But perhaps you got the kernel panic *after* rebooting in the
> reconfigured system?  That could well be a GuixSD bug, indeed.

No, it happens after second 'guix system reconfigure' with howto above.

[...]

Thanks,
Oleg.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> apologies for not adding logs before.  It's hard to do when I do guix
> commands from Xterm and not from Emacs.  Emacs *shell* or *compilation*
> buffers will eat all memory if they get too much text.

I sympathize…

> The problem
> ===========
>
> The bigger problem from my view are files like /etc/group.lock and
> /etc/passwd.lock.  For example:
>
>     sudo touch /etc/group.lock
>
> /etc/config.scm
>
>     (operating-system
>       ;; …
>       (groups (cons
>          (user-group (name "test"))
>          %base-groups)))
>
> reconfigure log

I think we can avoid the problem by forcefully removing these two lock
files at boot time:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 570 bytes --]

diff --git a/gnu/services.scm b/gnu/services.scm
index 329b7b151..2ef1d8530 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -368,6 +368,8 @@ boot."
                                                 #t))))
                     ;; Ignore I/O errors so the system can boot.
                     (fail-safe
+                     (delete-file "/etc/group.lock")
+                     (delete-file "/etc/passwd.lock")
                      (delete-file-recursively "/tmp")
                      (delete-file-recursively "/var/run")
                      (mkdir "/tmp")

[-- Attachment #3: Type: text/plain, Size: 1579 bytes --]


>                        How to make a kernel panic
>
> The problem will be No defined variable IPTABLES-SSH after 'guix system
> reconfigure' and kernel crash after.

[...]

> Make a kernel panic
>
>     sudo GUILE_LOAD_PATH=\"$HOME/src/iptables\
>     :$GUILE_LOAD_PATH\" guix system reconfigure \
>     $HOME/dotfiles/guix/system-magnolia.scm
>
>     # Run above again and kernel will panic.

I tried to reproduce it with my user’s shepherd, but that didn’t work:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ cat ,t.scm
(define s
  (make <service>
    #:provides '(nothing)
    #:start (lambda _ unbound)))

(register-services s)
(start s)
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
herd: exception caught while executing 'load' on service 'root':
ERROR: Unbound variable: foo
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
herd: exception caught while executing 'start' on service 'nothing':
ERROR: Unbound variable: unbound
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
Assertion (null? (lookup-services (canonical-name new))) failed.
herd: exception caught while executing 'load' on service 'root':
ERROR: Throw to key `assertion-failed' with args `()'.
ludo@ribbon ~/src/guix$ echo $?
1
ludo@ribbon ~/src/guix$ herd status

[...]

ludo@ribbon ~/src/guix$ echo $?
0
--8<---------------cut here---------------end--------------->8---

IOW, shepherd caught the exceptions and didn’t die.

What am I missing?

Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

Hello Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> I think we can avoid the problem by forcefully removing these two lock
> files at boot time:
>
> diff --git a/gnu/services.scm b/gnu/services.scm
> index 329b7b151..2ef1d8530 100644
> --- a/gnu/services.scm
> +++ b/gnu/services.scm
> @@ -368,6 +368,8 @@ boot."
>                                                  #t))))
>                      ;; Ignore I/O errors so the system can boot.
>                      (fail-safe
> +                     (delete-file "/etc/group.lock")
> +                     (delete-file "/etc/passwd.lock")
>                       (delete-file-recursively "/tmp")
>                       (delete-file-recursively "/var/run")
>                       (mkdir "/tmp")
>
>

There is also a '/etc/.pwd.lock'.  Info about this file
https://lists.debian.org/debian-user/2005/07/msg02949.html


I'm not sure if any files are exist.  Days past after reconfigure
failure.

$ sudo find /etc -name '*.lock' # Shows nothing.

[...]

> IOW, shepherd caught the exceptions and didn’t die.
>
> What am I missing?

I'll try to make a reproducible thing later.

Thanks,
Oleg.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> I think we can avoid the problem by forcefully removing these two lock
>> files at boot time:
>>
>> diff --git a/gnu/services.scm b/gnu/services.scm
>> index 329b7b151..2ef1d8530 100644
>> --- a/gnu/services.scm
>> +++ b/gnu/services.scm
>> @@ -368,6 +368,8 @@ boot."
>>                                                  #t))))
>>                      ;; Ignore I/O errors so the system can boot.
>>                      (fail-safe
>> +                     (delete-file "/etc/group.lock")
>> +                     (delete-file "/etc/passwd.lock")
>>                       (delete-file-recursively "/tmp")
>>                       (delete-file-recursively "/var/run")
>>                       (mkdir "/tmp")
>>
>>
>
> There is also a '/etc/.pwd.lock'.  Info about this file
> https://lists.debian.org/debian-user/2005/07/msg02949.html
>
>
> I'm not sure if any files are exist.  Days past after reconfigure
> failure.
>
> $ sudo find /etc -name '*.lock' # Shows nothing.

I’ve pushed it as aad8a143000600abec5c8ebfadec4c09f34f1b73.

> [...]
>
>> IOW, shepherd caught the exceptions and didn’t die.
>>
>> What am I missing?
>
> I'll try to make a reproducible thing later.

Awesome.

Thanks,
Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

Hi,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> I think we can avoid the problem by forcefully removing these two lock
>> files at boot time:
>>
>> diff --git a/gnu/services.scm b/gnu/services.scm
>> index 329b7b151..2ef1d8530 100644
>> --- a/gnu/services.scm
>> +++ b/gnu/services.scm
>> @@ -368,6 +368,8 @@ boot."
>>                                                  #t))))
>>                      ;; Ignore I/O errors so the system can boot.
>>                      (fail-safe
>> +                     (delete-file "/etc/group.lock")
>> +                     (delete-file "/etc/passwd.lock")
>>                       (delete-file-recursively "/tmp")
>>                       (delete-file-recursively "/var/run")
>>                       (mkdir "/tmp")
>>
>>
>
> There is also a '/etc/.pwd.lock'.  Info about this file
> https://lists.debian.org/debian-user/2005/07/msg02949.html
>
>
> I'm not sure if any files are exist.  Days past after reconfigure
> failure.
>
> $ sudo find /etc -name '*.lock' # Shows nothing.
>
> [...]
>
>> IOW, shepherd caught the exceptions and didn’t die.
>>
>> What am I missing?
>
> I'll try to make a reproducible thing later.

Did you eventually gather more info?

Thanks,
Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

Hello Ludovic,

Apologies for the late reply.

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Did you eventually gather more info?

Yes, I got an undefined %iptables-rst variable in reconfigure output
at first run.

But at second run substitutions didn't work.  GuixSD rebuilds the world.
I didn't wait for this.

I also didn't make guix pull.

/tmp/guixsd
├── bootstrap.sh
├── iptables
│   └── iptables.scm
├── panic.sh
└── vm-image.scm

1 directory, 4 files


[-- Attachment #2: bootstrap.sh --]
[-- Type: application/x-sh, Size: 873 bytes --]

[-- Attachment #3: panic.sh --]
[-- Type: application/x-sh, Size: 242 bytes --]

[-- Attachment #4: vm-image.scm --]
[-- Type: application/octet-stream, Size: 1657 bytes --]

(use-modules (gnu)
             (iptables))

(use-service-modules shepherd)
(use-package-modules linux)

(define start-firewall
  #~(let ((iptables
           (lambda (str)
             (zero? (system (string-join `(,#$(file-append iptables
                                                           "/sbin/iptables")
                                           ,str) " "))))))
      (format #t "Install iptables rules.~%")
      (and
       (iptables %iptables-rst))))

(define firewall-service
  (simple-service 'firewall shepherd-root-service-type
                  (list
                   (shepherd-service
                    (provision '(firewall))
                    (requirement '())
                    (start #~(lambda _
                               #$start-firewall))
                    (respawn? #f)
                    (stop #~(lambda _
                              (zero?
                               (system* #$(file-append iptables
                                                       "/sbin/iptables")
                                        "-F"))))))))

(operating-system
  (host-name "gnu")
  (timezone "Etc/UTC")
  (locale "en_US.utf8")
  (bootloader (grub-configuration (device "/dev/sda")
                                  (terminal-outputs '(console))))
  (file-systems (cons (file-system
                        (device "my-root")
                        (title 'label)
                        (mount-point "/")
                        (type "ext4"))
                      %base-file-systems))
  (users %base-user-accounts)
  (packages %base-packages)
  (services (cons firewall-service
                  %base-services)))

[-- Attachment #5: iptables.scm --]
[-- Type: application/octet-stream, Size: 378 bytes --]

(define-module (iptables)
  #:use-module (guix packages)
  #:use-module (guix gexp)
  #:use-module (guix monads)
  #:use-module (guix store)
  #:use-module (gnu packages)
  #:use-module (gnu packages linux)
  #:use-module (guix utils)
  #:use-module (srfi srfi-1)
  #:use-module (ice-9 match))

(define %iptables-rst
  "-A INPUT -p tcp --sport 443 --tcp-flags RST RST -j DROP")

bug#28772: guix system reconfigure after kernel panic user or group not created

[Oleg Pykhalov]

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Did you eventually gather more info?

In addition to previous message.

--- /home/natsu/r/bootstrap.sh	2017-10-22 04:48:42.992394510 +0300
+++ ./bootstrap.sh	2017-10-22 23:33:54.113495658 +0300
@@ -11,7 +11,7 @@
 SIG=guixsd-vm-image-$VERSION.$KERNEL.xz.sig
 
 pull_release () {
-    if [ ! -f $RELEASE ] || [ ! -f $RELEASE.xz ] || ! gpg --verify $SIG
+    if [ ! -f $RELEASE ] || [ ! -f $RELEASE.xz ]
     then
         gpg --keyserver pgp.mit.edu --recv-keys $GPG_KEY
         wget --output-document=$RELEASE.xz https://alpha.gnu.org/gnu/guix/$RELEASE.xz 
@@ -23,6 +23,7 @@
 
 if pull_release
 then qemu-system-x86_64 \
+     -enable-kvm \
      -daemonize \
      -m 1024 \
      -virtfs local,path=$PWD,security_model=none,mount_tag=TAG_pwd \

Also need to increase a size of qemu image.  I don't know how I
reconfigured successfully in previous message.

bug#28772: guix system reconfigure after kernel panic user or group not created

[Ludovic Courtès]

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> Did you eventually gather more info?
>
> Yes, I got an undefined %iptables-rst variable in reconfigure output
> at first run.
>
> But at second run substitutions didn't work.  GuixSD rebuilds the world.
> I didn't wait for this.

In my previous message I showed a way to (attempt to) reproduce the
problem by directly using ‘herd load’.

Could you try to reproduce the problem in this way?

You should even be able to check with a shepherd instance not running as
PID 1, which is more convenient.

Thanks in advance,
Ludo’.

bug#28772: guix system reconfigure after kernel panic user or group not created

[zimoun]

Hi,

On Tue, 10 Oct 2017 at 08:51, Oleg Pykhalov <go.wigust@gmail.com> wrote:

> During 'guix system reconfigure' I got a kernel panic.
>
> Getting it quite often while 'guix system reconfigure' at Linux magnolia
> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
> report.
>
> After reboot I tried another attempt to make a 'guix system
> reconfigure', but because of /etc/group.lock and /etc/passwd.lock
> operating-system groups and users fields didn't applied.
>
> I tried to get a creation of adbusers group and add my user to it, but
> because of those files I cannot do it, until manually removed them.
>
> So, do we need to check if *.lock files are exist and remove them at the
> start of 'guix system reconfigure' OR abort reconfigure and creation of
> Grub boot entry OR something else?


The last message for bug 28772 [1] is 22 Oct 2017 and it is marked
’moreinfo’ since 2017.  I propose to close or provide more details.

1: <http://issues.guix.gnu.org/issue/28772>


Cheers,
simon