cannot boot after installation on VPS (via rescue system)

cannot boot after installation on VPS (via rescue system)

[Giovanni Biscuolo]

[-- Attachment #1.1: Type: text/plain, Size: 467 bytes --]

Hello,

following the good guidelines from (info "(guix-cookbook) Running Guix
on a Kimsufi Server") and (info "(guix-cookbook) Running Guix on a
Linode Server") I'm developing a shell script to automate the "manual"
installation of Guix on bare metal and VPS, you can find it attached to
this email as bootstrap-guix.sh or at this git repo URL:
https://gitlab.com/softwareworkers/swws/-/blob/master/infrastructure/hosts/cornouiller/bootstrap-guix.sh?ref_type=heads


[-- Attachment #1.2: bootstrap-guix.sh --]
[-- Type: text/x-sh, Size: 12313 bytes --]

#!/bin/bash
# Copyright © 2023 Giovanni Biscuolo <g@xelera.eu>
#
# bootstrap-guix.sh is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 3 of the
# License, or (at your option) any later version.
#
# bootstrap-guix.sh is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# A copy of the GNU General Public License is available at
# <http://www.gnu.org/licenses/>.

# bootstrap-guix.sh is a very opinionated script to install Guix
# System on a host booted in "rescue" mode
#
# The system is installed on a single disk BTRFS filesystem

# Used variables MUST be initialized.
set -o nounset

# ---------------------------------------------------------------------
# Variables

# Disks
# TODO: transform this in array TARGET_DISKS[TARGET_NUMDISKS], for multi disk setups
export TARGET_NUMDISKS=1
export TARGET_DISK_PART_SUFFIX=""
export TARGET_DISK1="/dev/sda"
export TARGET_SWAP_SIZE="6GB"

# Hostname
export TARGET_HOSTNAME="cornouiller"

# User and pub key (only one admin user for basic installation)
export TARGET_USERNAME="g"
export TARGET_USERGECOS="Giovanni Biscuolo"
TARGET_USERKEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICqpr0unFxPo2PnQTmmO2dIUEECsCL3vVvjhk5Dx80Yb g@xelera.eu"

# --------------------------------------------------------------------
# Source os-release information
test -e /etc/os-release && os_release='/etc/os-release' || os_release='/usr/lib/os-release'
. "${os_release}"
echo "### INFO - Detected GNU/Linux distribution: ${PRETTY_NAME}."

# ---------------------------------------------------------------------
# Get package dependencies
export AUTO_INSTALLED=0

if [ $AUTO_INSTALLED -eq 0 ]; then
    # Install dependencies with Guix, if available
    if command -v guix &> /dev/null; then
	echo "### INFO - Installing dependencies via guix..."
	guix install util-linux parted dosfstools btrfs-progs gnupg
	export AUTO_INSTALLED=1
	echo "### END - Installing dependencies via guix."
    fi
fi

if [ $AUTO_INSTALLED -eq 0 ]; then
    # Install dependencies with apt, if available
    if command -v apt &> /dev/null; then
	echo "### INFO - Installing dependencies via apt..."
	apt install util-linux parted dosfstools btrfs-progs gnupg
	export AUTO_INSTALLED=1
	echo "### END - Installing dependencies via apt."
    fi
fi

# Give up installing and notify users
if [ $AUTO_INSTALLED -eq 0 ] ; then
    echo "### INFO - I'm not able to automatically install dependencies on ${PRETTY_NAME}."
    echo "Please check you have the following commands available: wipefs, parted, mkfs.fat, mkswap, mkfs.btrfs, btrfs, gpg."
    echo "### END - Checking dependencies."
fi

# Abort on any error, from now
set -e 

# ###########################
# DO NOT EDIT this variables
# unless for debugging

# (minimal) OS configuration file name
export OS_CONFIG_FILE="bootstrap-config.scm"

# Target OS mount point
export TARGET_MOUNTPOINT="/mnt/guix"

# ---------------------------------------------------------------------
# Prepare the target system filesystem
echo "### START - Downloading guix install script."
wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
chmod +x guix-install.sh
echo "### END - Downloading guix install script."  

# ---------------------------------------------------------------------
# Prepare the target system filesystem

# Wipe the disks
# TODO: use the array TARGET_DISKS[]
echo "### START - Wiping disks."
wipefs -af ${TARGET_DISK1}*
echo "### END - Wiping disks."

# Partition the disks
echo "### START - EFI system check."
if [ -e "/sys/firmware/efi/efivars" ]; then
    IS_EFI=true
    echo "System is EFI based. ($IS_EFI)"
else
    IS_EFI=false
    echo "System is BIOS grub based. ($IS_EFI)"
fi
echo "### END - EFI system check."

## Disk 1
echo "### START - partitioning ${TARGET_DISK1}."
parted ${TARGET_DISK1} --align=opt -s -m -- mklabel gpt

# partition p1 will be system boot
if $IS_EFI; then
    # EFI system partition
    BOOT_SIZE="501MiB"
    parted ${TARGET_DISK1} --align=opt -s -m -- \
	   mkpart primary fat32 1MiB $BOOT_SIZE \
	   name 1 EFI-system \
	   set 1 esp on
else
    # BIOS grub system partition
    BOOT_SIZE="5MiB"
    parted ${TARGET_DISK1} --align=opt -s -m -- \
	   mkpart grub 1MiB $BOOT_SIZE \
	   name 1 grub-1 \
	   set 1 bios_grub on
fi

# partition p2 will be swap
parted ${TARGET_DISK1} --align=opt -s -m -- \
       mkpart primary linux-swap $BOOT_SIZE ${TARGET_SWAP_SIZE} \
       name 2 swap-1

# partition p3 will be BTRFS device
parted ${TARGET_DISK1} --align=opt -s -m -- \
       mkpart primary btrfs ${TARGET_SWAP_SIZE} 100% \
       name 3 BTRFS

partprobe ${TARGET_DISK1}

sleep 5s  # Needed sometimes to allow new partition access
echo "### END - partitioning ${TARGET_DISK1}."

# Create FAT32 filesystem on p1 if it's EFI system
if $IS_EFI; then
    echo "### START - Making EFI filesystem"
    mkfs.fat -F32 ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}1
    fatlabel ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}1 "EFI-system"
    echo "### END - Making EFI filesystem"    
fi

# Make swap on p2 partitions and turn them on
echo "### START - Making swap."
mkswap ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}2
# swaplabel -L "swap-1" ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}2
swapon ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}2
echo "### END - Making swap."

# Create BTRFS filesystem
echo "### START - Making BTRFS flesystem and subvolumes."
mkfs.btrfs -f ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}3
# btrfs filesystem label ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}3 "root"

# Mount the target Guix System root
mkdir -p ${TARGET_MOUNTPOINT}
mount -o compress=zstd ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}3 ${TARGET_MOUNTPOINT}

# Create subvolumes for target system
# btrfs subvolume create ${TARGET_MOUNTPOINT}/var
# btrfs subvolume create ${TARGET_MOUNTPOINT}/home
# btrfs subvolume create ${TARGET_MOUNTPOINT}/srv
# btrfs subvolume create ${TARGET_MOUNTPOINT}/root
# btrfs subvolume create ${TARGET_MOUNTPOINT}/gnu
echo "### END - Making BTRFS flesystem and subvolumes."

# Mount the ESP on /boot/efi
if $IS_EFI; then
    echo "### START - Mounting EFI filesystem"
    mkdir -p ${TARGET_MOUNTPOINT}/boot/efi
    mount ${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}1 ${TARGET_MOUNTPOINT}/boot/efi
    echo "### END - Mounting EFI filesystem"    
fi

# ---------------------------------------------------------------------
# Prepare basic OS configuration
echo "### START - Creating basic Guix system config in ${OS_CONFIG_FILE}."
if $IS_EFI; then
    BOOTLOADER_SCM="
        (bootloader (bootloader-configuration
                     (bootloader grub-efi-bootloader)
                     (targets (list \"/boot/efi\"))
                     (keyboard-layout keyboard-layout)))"
	             
else
    BOOTLOADER_SCM="
        (bootloader (bootloader-configuration
                     (bootloader grub-bootloader)
	             (targets (list \"${TARGET_DISK1}\"))
	             (keyboard-layout keyboard-layout)))"
fi

EFI_FS_SCM=""
if $IS_EFI; then
    EFI_FS_SCM="
        (file-system
	  (mount-point "/boot/efi")
	  (device (file-system-label "EFI-system" 'fat))
	  (type "vfat"))"
fi

# Heredoc with scheme OS configuration
cat > ${OS_CONFIG_FILE} << EO_CONFIG
;; Very basic Guix System
(use-modules (gnu))
(use-service-modules admin networking ssh linux)

;; Definitions
(define (sysadmin name full-name)
  (user-account
   (name name)
   (comment full-name)
   (group "users")
   (supplementary-groups '("wheel" "kvm"))
   (home-directory (string-append "/home/" name))))

(define %accounts
  (list (sysadmin "${TARGET_USERNAME}" "${TARGET_USERGECOS}")))

;; operating-system
(operating-system
  (locale "en_US.utf8")
  (timezone "Europe/Rome")
  (keyboard-layout (keyboard-layout "it" "winkeys"))
  (host-name "${TARGET_HOSTNAME}")

  ${BOOTLOADER_SCM}

  (file-systems (append
		 (list (file-system
			(mount-point "/")
			(device "${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}3")
			(type "btrfs")
			(options "compress=zstd"))
                  ${EFI_FS_SCM})
		 %base-file-systems))
  
  (swap-devices
   (list (swap-space (target "${TARGET_DISK1}${TARGET_DISK_PART_SUFFIX}2"))))

  (issue
   ;; Default contents for /etc/issue.
   "\\
This a Guix system.  Welcome.\n")

  (users (append %accounts %base-user-accounts))

  (sudoers-file
   (plain-file "sudoers" "\\
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL\n"))

  ;; Globally-installed packages.
  (packages (append (list (specification->package "st")
			  (specification->package "nss-certs"))
		    %base-packages))

  (services
   (append %base-services
           (list
            (service dhcp-client-service-type)
            (service openssh-service-type
        	     (openssh-configuration
        	      (port-number 22)
                      (password-authentication? #f)
                      (permit-root-login 'prohibit-password)
                      (extra-content "ListenAddress 0.0.0.0")
        	      (authorized-keys
        	       \`(("${TARGET_USERNAME}" ,(plain-file "${TARGET_USERNAME}.pub" "${TARGET_USERKEY}"))
                          ("root" ,(plain-file "${TARGET_USERNAME}.pub" "${TARGET_USERKEY}"))))))))))

EO_CONFIG
echo "### END - Creating basic Guix system config."


# ---------------------------------------------------------------------
# Mount the /gnu store copy-on-write using ${TARGET_MOUNTPOINT}
echo "### START - Mounting cow-store"

if [ "${ID:-linux}" = "guix" ]; then
    herd start cow-store ${TARGET_MOUNTPOINT}
else
    # Make the store copy-on-write, using TARGET as the backing store.
    # This is useful when TARGET is on a hard disk, whereas the current
    # store is on a RAM disk.  Ported from mount-cow-store in
    # gnu/build/install.scm, used by "herd start cow-store".

    mkdir -p /gnu/store
    RWDIR=${TARGET_MOUNTPOINT}/tmp/guix-inst
    WORKDIR=${RWDIR}/../.overlayfs-workdir
    mkdir -p ${RWDIR}
    mkdir -p ${WORKDIR}
    chmod 775 ${RWDIR}
    mount -t overlay -o lowerdir=/gnu/store,upperdir=${RWDIR},workdir=${WORKDIR} overlay /gnu/store
    # systemctl daemon-reload # FIXME: is this needed?!?
fi
echo "### END - Mounting cow-store"

# Collect some partitioning and mount points info
mount > bootstrap-mount-points.txt
lsblk -f ${TARGET_DISK1} -J > bootstrap-lsblk-`basename ${TARGET_DISK1}`.json

# ---------------------------------------------------------------------
# Install GNU Guix if needed
if [ "${ID:-linux}" = "guix" ]; then
    echo "### INFO - No need to install the guix binary."
else
    # ---------------------------------------------------------------------
    # Install guix using binary installation
    echo "### START - Installing guix binary."
    ./guix-install.sh
    hash guix
    echo "### END - Installing guix binary."  
fi

guix describe > bootstrap-guix-version.txt

# ---------------------------------------------------------------------
# Install Guix on target filesystem
echo "### START - Installing Guix on ${TARGET_MOUNTPOINT}"
mkdir ${TARGET_MOUNTPOINT}/etc/
cp bootstrap-mount-points.txt ${TARGET_MOUNTPOINT}/etc/
cp bootstrap-lsblk-*.json ${TARGET_MOUNTPOINT}/etc/
cp bootstrap-guix-version.txt ${TARGET_MOUNTPOINT}/etc/
cp ${OS_CONFIG_FILE} ${TARGET_MOUNTPOINT}/etc/config.scm
guix system init ${TARGET_MOUNTPOINT}/etc/config.scm ${TARGET_MOUNTPOINT}
echo "### END - Installing Guix on ${TARGET_MOUNTPOINT}"

# FIXME: umount cow-store and delete tmp files

# ---------------------------------------------------------------------
# End game
echo ""
echo "### DONE! - Target system in ${TARGET_MOUNTPOINT} is ready..."
echo ""
echo "Please remember to copy ${OS_CONFIG_FILE} to a safe remote location"
echo ""
echo "...and reboot to start your new Guix System!  Bye."

[-- Attachment #1.3: Type: text/plain, Size: 1928 bytes --]


The main purpose of the script is to allow me to install Guix on
machines (bare metal or virtual) hosted by vendors who do not offer Guix
in their OS installation options, since almost all vendors provides a
rescue system that can be used to "manually" set up the environment,
partition the target disk and lastly "guix system init..."

I've extensively tested the script locally (on bare metal and QEMU VMs),
both booting the machines via grml [1] and Guix install image and it
works as expected; I've also used it to install Guix on some bare metal.

Now I'm trying to use it on two VPS from two different vendors, booted
in rescue mode, but after the installation (via bootstrap-guix.sh) when
I reboot the VPS I get the usual grub menu but the boot process suddenly
fails with this error (manually copied from web console, sorry for
possible typos):

--8<---------------cut here---------------start------------->8---

Scanning for Btrfs filesystems
ice-9/boot9.scm:1685:16: In procedure raise-exception:
In procedure mount: No such file or directory
GRUB loading...
Entering a new prompt.  Type ',bt' for a backtrace or ',q' to continue.
[...]
scheme@(guile-user)> ,bt
In gnu/build/linux-boot.scm:
    637:8  3 (_)
    435:8  2 (mount-root-filesystem "/dev/sda3" "btrfs" # _ #:flags ?)
In unknown file:
           1 (mount "/dev/sda3" "/root" "btrfs" 0 "compress=zstd")
In ice-9/boot9.scm:
  1685:16: 0 (raise-exception _ #:continuable? _)

--8<---------------cut here---------------end--------------->8---

In particular, I don't understand why the boot script is trying to mount
the root filesystem at "/root" and not at "/" as it should: am I missing
something?

I suspect that the issue depends on how the rescue system root is
mounted (overlay filesystem) but I'm not sure; I cannot find a solution.

I'm also attaching:

- bootstrap-config.scm: automatically generated by the script and used
by "guix system init..."


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.4: bootstrap-config.scm --]
[-- Type: text/x-script.guile, Size: 2169 bytes --]

;; Very basic Guix System
(use-modules (gnu))
(use-service-modules admin networking ssh linux)

;; Definitions
(define (sysadmin name full-name)
  (user-account
   (name name)
   (comment full-name)
   (group "users")
   (supplementary-groups '("wheel" "kvm"))
   (home-directory (string-append "/home/" name))))

(define %accounts
  (list (sysadmin "g" "Giovanni Biscuolo")))

;; operating-system
(operating-system
  (locale "en_US.utf8")
  (timezone "Europe/Rome")
  (keyboard-layout (keyboard-layout "it" "winkeys"))
  (host-name "cornouiller")

  
        (bootloader (bootloader-configuration
                     (bootloader grub-bootloader)
	             (targets (list "/dev/sda"))
	             (keyboard-layout keyboard-layout)))

  (file-systems (append
		 (list (file-system
			(mount-point "/")
			(device "/dev/sda3")
			(type "btrfs")
			(options "compress=zstd"))
                  )
		 %base-file-systems))
  
  (swap-devices
   (list (swap-space (target "/dev/sda2"))))

  (issue
   ;; Default contents for /etc/issue.
   "\
This a Guix system.  Welcome.\n")

  (users (append %accounts %base-user-accounts))

  (sudoers-file
   (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL\n"))

  ;; Globally-installed packages.
  (packages (append (list (specification->package "st")
			  (specification->package "nss-certs"))
		    %base-packages))

  (services
   (append %base-services
           (list
            (service dhcp-client-service-type)
            (service openssh-service-type
        	     (openssh-configuration
        	      (port-number 22)
                      (password-authentication? #f)
                      (permit-root-login 'prohibit-password)
                      (extra-content "ListenAddress 0.0.0.0")
        	      (authorized-keys
        	       `(("g" ,(plain-file "g.pub" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICqpr0unFxPo2PnQTmmO2dIUEECsCL3vVvjhk5Dx80Yb g@xelera.eu"))
                          ("root" ,(plain-file "g.pub" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICqpr0unFxPo2PnQTmmO2dIUEECsCL3vVvjhk5Dx80Yb g@xelera.eu"))))))))))


[-- Attachment #1.5: Type: text/plain, Size: 76 bytes --]


- bootstrap-mount-points.txt: automatically collected during installation


[-- Attachment #1.6: bootstrap-mount-points.txt --]
[-- Type: text/plain, Size: 2833 bytes --]

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=1956752k,nr_inodes=489188,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
[2a01:4ff:ff00::b007:1]:/nfs on /root/.oldroot/nfs type nfs (ro,noatime,vers=3,rsize=8192,wsize=8192,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noresvport,proto=tcp6,timeo=600,retrans=2,sec=sys,mountaddr=2a01:4ff:ff00::b007:1,mountvers=3,mountproto=tcp6,local_lock=all,addr=2a01:4ff:ff00::b007:1)
overlay on / type overlay (rw,relatime,lowerdir=/nfsroot,upperdir=/ramfs/root,workdir=/ramfs/work,uuid=on)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=786480k,nr_inodes=819200,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=393236k,nr_inodes=98309,mode=700)
/dev/sda3 on /mnt/guix type btrfs (rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
overlay on /gnu/store type overlay (rw,relatime,lowerdir=/gnu/store,upperdir=/mnt/guix/tmp/guix-inst,workdir=/mnt/guix/tmp/guix-inst/../.overlayfs-workdir,uuid=on)

[-- Attachment #1.7: Type: text/plain, Size: 310 bytes --]


You can find all the files also here:
https://gitlab.com/softwareworkers/swws/-/tree/master/infrastructure/hosts/cornouiller?ref_type=heads

Any idea please? :-O

Happy hacking! Gio'



[1] https://grml.org/, a rescue system based on Debian


-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[jbranso]

February 21, 2024 at 4:54 AM, "Giovanni Biscuolo" <g@xelera.eu> wrote:



> 
> Hello,
> 
> following the good guidelines from (info "(guix-cookbook) Running Guix
> on a Kimsufi Server") and (info "(guix-cookbook) Running Guix on a
> Linode Server") I'm developing a shell script to automate the "manual"
> installation of Guix on bare metal and VPS, you can find it attached to
> this email as bootstrap-guix.sh or at this git repo URL:
> https://gitlab.com/softwareworkers/swws/-/blob/master/infrastructure/hosts/cornouiller/bootstrap-guix.sh?ref_type=heads
> 
> The main purpose of the script is to allow me to install Guix on
> machines (bare metal or virtual) hosted by vendors who do not offer Guix
> in their OS installation options, since almost all vendors provides a
> rescue system that can be used to "manually" set up the environment,
> partition the target disk and lastly "guix system init..."
> 
> I've extensively tested the script locally (on bare metal and QEMU VMs),
> both booting the machines via grml [1] and Guix install image and it
> works as expected; I've also used it to install Guix on some bare metal.
> 
> Now I'm trying to use it on two VPS from two different vendors, booted
> in rescue mode, but after the installation (via bootstrap-guix.sh) when
> I reboot the VPS I get the usual grub menu but the boot process suddenly
> fails with this error (manually copied from web console, sorry for
> possible typos):
> 

I just logged into my linode server...your script defaults to a btrfs
filesystem right?  When I tried to add an additional disk in linode
just now, the only supported filesystem was ext4.  Does linode support
btrfs?

Joshua 

> --8<---------------cut here---------------start------------->8---
> 
> Scanning for Btrfs filesystems
> ice-9/boot9.scm:1685:16: In procedure raise-exception:
> In procedure mount: No such file or directory
> GRUB loading...
> Entering a new prompt. Type ',bt' for a backtrace or ',q' to continue.
> [...]
> scheme@(guile-user)> ,bt
> In gnu/build/linux-boot.scm:
>  637:8 3 (_)
>  435:8 2 (mount-root-filesystem "/dev/sda3" "btrfs" # _ #:flags ?)
> In unknown file:
>  1 (mount "/dev/sda3" "/root" "btrfs" 0 "compress=zstd")
> In ice-9/boot9.scm:
>  1685:16: 0 (raise-exception _ #:continuable? _)
> 
> --8<---------------cut here---------------end--------------->8---
> 
> In particular, I don't understand why the boot script is trying to mount
> the root filesystem at "/root" and not at "/" as it should: am I missing
> something?
> 
> I suspect that the issue depends on how the rescue system root is
> mounted (overlay filesystem) but I'm not sure; I cannot find a solution.
> 
> I'm also attaching:
> 
> - bootstrap-config.scm: automatically generated by the script and used
> by "guix system init..."
> 
> - bootstrap-mount-points.txt: automatically collected during installation
> 
> You can find all the files also here:
> https://gitlab.com/softwareworkers/swws/-/tree/master/infrastructure/hosts/cornouiller?ref_type=heads
> 
> Any idea please? :-O
> 
> Happy hacking! Gio'
> 
> [1] https://grml.org/, a rescue system based on Debian
> 
> -- 
> Giovanni Biscuolo
> 
> Xelera IT Infrastructures
>

Re: cannot boot after installation on VPS (via rescue system)

[Wojtek Kosior via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 5303 bytes --]

> --8<---------------cut here---------------start------------->8---
> 
> Scanning for Btrfs filesystems
> ice-9/boot9.scm:1685:16: In procedure raise-exception:
> In procedure mount: No such file or directory
> GRUB loading...
> Entering a new prompt.  Type ',bt' for a backtrace or ',q' to continue.
> [...]
> scheme@(guile-user)> ,bt
> In gnu/build/linux-boot.scm:
>     637:8  3 (_)
>     435:8  2 (mount-root-filesystem "/dev/sda3" "btrfs" # _ #:flags ?)
> In unknown file:
>            1 (mount "/dev/sda3" "/root" "btrfs" 0 "compress=zstd")
> In ice-9/boot9.scm:
>   1685:16: 0 (raise-exception _ #:continuable? _)
> 
> --8<---------------cut here---------------end--------------->8---

Maybe the device file is called different from /dev/sda3?  On one VPS of
mine (which also happens to have Guix installed via rescue mode) the
root is mounted from /dev/vda1.

> In particular, I don't understand why the boot script is trying to mount
> the root filesystem at "/root" and not at "/" as it should: am I missing
> something?

Linux-based systems typically start with initrd filesystem mounted at
/.  They then mount the real root at some subdirectory of / and use
either chroot or pivot-root system call to make the processes see it as
if it were mounted at / in the first place.

I'm not an expert in early boot process so please forgive me any
mistakes I might have made in this explanation :)

Best,
Wojtek


-- (sig_start)
website: https://koszko.org/koszko.html
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
follow me on Fediverse: https://friendica.me/profile/koszko/profile

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Wed, 21 Feb 2024 10:54:16 +0100 Giovanni Biscuolo <g@xelera.eu> wrote:

> From: Giovanni Biscuolo <g@xelera.eu>
> To: guix-devel@gnu.org
> Subject: cannot boot after installation on VPS (via rescue system)
> Date: Wed, 21 Feb 2024 10:54:16 +0100
> Sender: guix-devel-bounces+koszko=koszko.org@gnu.org
> Organization: Xelera.eu
> 
> Hello,
> 
> following the good guidelines from (info "(guix-cookbook) Running Guix
> on a Kimsufi Server") and (info "(guix-cookbook) Running Guix on a
> Linode Server") I'm developing a shell script to automate the "manual"
> installation of Guix on bare metal and VPS, you can find it attached to
> this email as bootstrap-guix.sh or at this git repo URL:
> https://gitlab.com/softwareworkers/swws/-/blob/master/infrastructure/hosts/cornouiller/bootstrap-guix.sh?ref_type=heads
> 
> 
> [bootstrap-guix.sh  text/x-sh (11954 bytes)] 
> 
> 
> The main purpose of the script is to allow me to install Guix on
> machines (bare metal or virtual) hosted by vendors who do not offer Guix
> in their OS installation options, since almost all vendors provides a
> rescue system that can be used to "manually" set up the environment,
> partition the target disk and lastly "guix system init..."
> 
> I've extensively tested the script locally (on bare metal and QEMU VMs),
> both booting the machines via grml [1] and Guix install image and it
> works as expected; I've also used it to install Guix on some bare metal.
> 
> Now I'm trying to use it on two VPS from two different vendors, booted
> in rescue mode, but after the installation (via bootstrap-guix.sh) when
> I reboot the VPS I get the usual grub menu but the boot process suddenly
> fails with this error (manually copied from web console, sorry for
> possible typos):
> 
> --8<---------------cut here---------------start------------->8---
> 
> Scanning for Btrfs filesystems
> ice-9/boot9.scm:1685:16: In procedure raise-exception:
> In procedure mount: No such file or directory
> GRUB loading...
> Entering a new prompt.  Type ',bt' for a backtrace or ',q' to continue.
> [...]
> scheme@(guile-user)> ,bt
> In gnu/build/linux-boot.scm:
>     637:8  3 (_)
>     435:8  2 (mount-root-filesystem "/dev/sda3" "btrfs" # _ #:flags ?)
> In unknown file:
>            1 (mount "/dev/sda3" "/root" "btrfs" 0 "compress=zstd")
> In ice-9/boot9.scm:
>   1685:16: 0 (raise-exception _ #:continuable? _)
> 
> --8<---------------cut here---------------end--------------->8---
> 
> In particular, I don't understand why the boot script is trying to mount
> the root filesystem at "/root" and not at "/" as it should: am I missing
> something?
> 
> I suspect that the issue depends on how the rescue system root is
> mounted (overlay filesystem) but I'm not sure; I cannot find a solution.
> 
> I'm also attaching:
> 
> - bootstrap-config.scm: automatically generated by the script and used
> by "guix system init..."
> 
> 
> [bootstrap-config.scm  text/x-script.guile (2097 bytes)] 
> 
> 
> - bootstrap-mount-points.txt: automatically collected during installation
> 
> 
> [bootstrap-mount-points.txt  text/plain (2833 bytes)] 
> 
> 
> You can find all the files also here:
> https://gitlab.com/softwareworkers/swws/-/tree/master/infrastructure/hosts/cornouiller?ref_type=heads
> 
> Any idea please? :-O
> 
> Happy hacking! Gio'
> 
> 
> 
> [1] https://grml.org/, a rescue system based on Debian
> 
> 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

Hi Joshua!

jbranso@dismail.de writes:

[...]

>> Now I'm trying to use it on two VPS from two different vendors, booted
>> in rescue mode, but after the installation (via bootstrap-guix.sh) when
>> I reboot the VPS I get the usual grub menu but the boot process suddenly
>> fails with this error (manually copied from web console, sorry for
>> possible typos):
>
> I just logged into my linode server...your script defaults to a btrfs
> filesystem right?

Right, and now that you pointed to that I realized that may be not a
good idea for a QEMU virtual machine [1]: I'll switch to ext4 to be sure

> When I tried to add an additional disk in linode just now, the only
> supported filesystem was ext4.  Does linode support btrfs?

I'm not on Linode, I'm working on OVH and Hetzner VPSs

Anyway I don't think that the boot issue is connected with BTRFS, but
I'll investigate!

Thank you! Gio'

[...]

[1] I don't know the host filesystem but if it's BTRFS, placing a COW
(qcow2) filesystem on top of another COW have a very bad performance,
see
https://www.qemu.org/docs/master/system/qemu-block-drivers.html#cmdoption-qcow2-arg-nocow


-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

Hi Wojtek!

Wojtek Kosior <koszko@koszko.org> writes:

>> --8<---------------cut here---------------start------------->8---
>> 
>> Scanning for Btrfs filesystems
>> ice-9/boot9.scm:1685:16: In procedure raise-exception:
>> In procedure mount: No such file or directory
>> GRUB loading...
>> Entering a new prompt.  Type ',bt' for a backtrace or ',q' to continue.
>> [...]
>> scheme@(guile-user)> ,bt
>> In gnu/build/linux-boot.scm:
>>     637:8  3 (_)
>>     435:8  2 (mount-root-filesystem "/dev/sda3" "btrfs" # _ #:flags ?)
>> In unknown file:
>>            1 (mount "/dev/sda3" "/root" "btrfs" 0 "compress=zstd")
>> In ice-9/boot9.scm:
>>   1685:16: 0 (raise-exception _ #:continuable? _)
>> 
>> --8<---------------cut here---------------end--------------->8---
>
> Maybe the device file is called different from /dev/sda3?

Maybe, but I also tried using UUID (the usual method I use) and label,
both failing... I'll investigate

> On one VPS of mine (which also happens to have Guix installed via
> rescue mode) the root is mounted from /dev/vda1.

Out of curiosity: what's the hoster, please?

>> In particular, I don't understand why the boot script is trying to mount
>> the root filesystem at "/root" and not at "/" as it should: am I missing
>> something?
>
> Linux-based systems typically start with initrd filesystem mounted at
> /.  They then mount the real root at some subdirectory of / and use
> either chroot or pivot-root system call to make the processes see it as
> if it were mounted at / in the first place.

Yes! Thank you for your explanation: I checked gnu/build/linux-boot.scm
and it's just as you pointed out; I simply overlooked the error was in
the "initrd phase"... and now I'm starting to barely understand what's
happening

> I'm not an expert in early boot process so please forgive me any
> mistakes I might have made in this explanation :)

No mistakes :-D

Thank you for your help! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

>>> Now I'm trying to use it on two VPS from two different vendors, booted
>>> in rescue mode, but after the installation (via bootstrap-guix.sh) when
>>> I reboot the VPS I get the usual grub menu but the boot process suddenly
>>> fails with this error (manually copied from web console, sorry for
>>> possible typos):
>>>
>>> [...]
> 
> I'm not on Linode, I'm working on OVH and Hetzner VPSs

I had to add "virtio_scsi" to initrd-modules to get Guix to boot on a
Hetzner VPS. Maybe that could be the problem here, too?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

Hi Saku!

Saku Laesvuori <saku@laesvuori.fi> writes:

>>>> Now I'm trying to use it on two VPS from two different vendors, booted
>>>> in rescue mode, but after the installation (via bootstrap-guix.sh) when
>>>> I reboot the VPS I get the usual grub menu but the boot process suddenly
>>>> fails with this error (manually copied from web console, sorry for
>>>> possible typos):
>>>>
>>>> [...]
>> 
>> I'm not on Linode, I'm working on OVH and Hetzner VPSs
>
> I had to add "virtio_scsi" to initrd-modules to get Guix to boot on a
> Hetzner VPS. Maybe that could be the problem here, too?

Yes: you got it!

I asked Hetzner support to have the Guix installer ISO as "custom ISO"
and was able to do a guided install... and it (re)booted fine.  The
first thing that got my attention was this line in the working
config.scm:

--8<---------------cut here---------------start------------->8---

(initrd-modules (append '("virtio_scsi") %base-initrd-modules))
 
--8<---------------cut here---------------end--------------->8---

...and it makes sense! [1]

I also missed that line in (info "(guix-cookbook) Running Guix on a
Linode Server") ;-(

I still have not tested that fix with my bootstrap-guix.sh installation
script, but I'm pretty sure it will work.

Thank you so much! Gio'


[1] I also need to enhance my qemu scripts needed for testing

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: cannot boot after installation on VPS (via rescue system)

[Wojtek Kosior via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

> > On one VPS of mine (which also happens to have Guix installed via
> > rescue mode) the root is mounted from /dev/vda1.  
> 
> Out of curiosity: what's the hoster, please?

https://rapiddc.pl/

It's a result of a long search for a decent Europe-based provider.  I
didn't choose any of the mainstream providers because of their ToS (even
well-reputed Gandi had somewhat customer-hostile ToS; the requirement
to purchase an insurance finally convinced me not to use them).  1984
was pretty fair but offered a DPA in horribly broken english (I still
use them for non-business needs).

The one I chose isn't perfect, either.  As you can see, they have
GDPR-incompliant Google spyware on their website (though it causes no
legal risk to their customers) and they rely on nonfree JS.  I couldn't
find a provider who does everything perfectly so I chose based on
priorities.  From my PoV it's a domestic provider which also makes
things easier for me (and possibly harder for others who'd like to
use this message as some kind of recommendation).

Good to see you solved your issue :)
Wojtek

-- (sig_start)
website: https://koszko.org/koszko.html
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
follow me on Fediverse: https://friendica.me/profile/koszko/profile

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Thu, 22 Feb 2024 08:25:57 +0100 Giovanni Biscuolo <g@xelera.eu> wrote:

> > On one VPS of mine (which also happens to have Guix installed via
> > rescue mode) the root is mounted from /dev/vda1.  
> 
> Out of curiosity: what's the hoster, please?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]