Re: btrfs and subvolumes for root, take 2

Re: btrfs and subvolumes for root, take 2

[Ludovic Courtès]

Hello Fredrik,

Fredrik Salomonsson <plattfot@gmail.com> skribis:

> guix archive --authorize < /gnu/store/cw55zvxzi3d9cjmhfvxsryz31jxb1y6k-guix-0.15.0-1.4876bc8/share/guix/berlin.guixsd.org.pub
> guix pull --commit=d9f8e84 --substitute-urls="http://berlin.guixsd.org http://mirror.hydra.gnu.org"
> guix system init /mnt/etc/config.scm /mnt --substitute-urls="http://berlin.guixsd.org http://mirror.hydra.gnu.org"

The Guile backtrace you sent shows that /etc/ssl already existed when
your system booted and was not a symlink.  This led the “activation
code” of GuixSD to fail:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/activation.scm#n320

The solution is to remove /etc/ssl (is it coming from another distro
previously installed on this device?).  You can boot a separate medium, 
mount the root partition, and “rm -rf /etc/ssl” from there.  Or you can,
at the boot REPL that you get after the backtrace, type something like:

  ,use (guix build utils)
  (delete-file-recursively "/etc/ssl")
  ,q

Note that you might have similar issues with /etc/pam.d, for instance,
if there’s such a stale directory.

HTH,
Ludo’.

Re: btrfs and subvolumes for root, take 2

[Fredrik Salomonsson]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]


Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> The Guile backtrace you sent shows that /etc/ssl already existed when
> your system booted and was not a symlink.  This led the “activation
> code” of GuixSD to fail:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/activation.scm#n320
>
> The solution is to remove /etc/ssl (is it coming from another distro
> previously installed on this device?).  You can boot a separate medium, 
> mount the root partition, and “rm -rf /etc/ssl” from there.  Or you can,
> at the boot REPL that you get after the backtrace, type something like:
>
>   ,use (guix build utils)
>   (delete-file-recursively "/etc/ssl")
>   ,q
>
> Note that you might have similar issues with /etc/pam.d, for instance,
> if there’s such a stale directory.

Thanks for the reply. That pointed me in the right direction. Although
the solution you suggested wasn't an option for me. As it turned out, it
was actually mounting my Arch Linux root (__current/arch-root). Which I
had set to be the default subvolume if no ~subvol=~ option is given when
mounting the disk.

Since my goal is to be able to dual boot, I don't want to nuke my arch
subvolume.

But that gave me an idea. What if I just change the default subvolume to
my GuixSD root (__current/guixsd-root). I did that and what do you know
it booted!

The question is though why did it mount my default subvolume? I
specified it to mount __current/guixsd-root for the root filesystem. As
well as setting that to be the subvolume to mount in the rootflags for
the kernel command line when booting. Is there a third step that I'm
missing?

In either case thank you so much for your help. Been trying to get this
going for over a year now. So it feels good to finally see the login
screen!

I attached my install log, which has all the steps I needed to do in
order to install GuixSD on a btrfs disk. In case someone is interested.

-- 
s/Fred[re]+i[ck]+/Fredrik/g

[-- Attachment #2: Install log for GuixSD --]
[-- Type: text, Size: 16171 bytes --]

#+author: Fredrik Salomonsson

* Install guix-0.15
** Setup network and ssh locally
   These things need to be first run locally on the machine you're
   installing GuixSD.
**** Setup wireless and get ip

     #+BEGIN_SRC sh
       wpa_supplicant -i INTERFACE -B -c <(wpa_passphrase SSID PASSPHRASE)
       dhclient -v interface
     #+END_SRC

**** Setup ssh
     Change the password to be able to connect to ssh
     #+BEGIN_SRC sh
       passwd
     #+END_SRC

     Then start the ssh service using shepherd
     #+BEGIN_SRC sh
       herd start ssh-daemon
     #+END_SRC

** Install Process

   NOTE: Change ~192.168.4.112~ to the ip you got from calling
   ~dhclient~ earlier.

**** Fix known_hosts
     Note: If this is the first time, you can skip this step.

     Remove guix from known_hosts, as its key changes every reboot.
     #+BEGIN_SRC bash :dir ~/
       sed -iE 's/^192.168.4.112 .*//' .ssh/known_hosts
     #+END_SRC

**** Mount disks

     This assumes that you have already setup an encrypted disk and
     setup the following layout for ~/dev/sda3~.

     Partion layout for /dev/sda3 under __current/
     | subvol      | Mountpoint   | Comment            | Shared   |
     |-------------+--------------+--------------------+----------|
     | arch-root   | /            | root for Arch      | no       |
     | guixsd-root | /            | root for GuixSD    | no       |
     | grub        | /boot/grub   | grub config        | yes      |
     | guix        | /var/guix    | guix stuff         | yes      |
     | gnu         | /gnu         | Store etc          | yes      |
     | home        | /home        | home partition     | yes      |

     Decrypt the disk
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       cryptsetup open --type luks /dev/sda2 swap && cryptsetup open --type luks /dev/sda3 root
     #+END_SRC

     Mount the different subvolumes
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       mount -o defaults,discard,compress=lzo,space_cache,autodefrag,subvol=__current/guixsd-root LABEL=root /mnt/
       mount -o defaults,discard,compress=lzo,space_cache,autodefrag,subvol=__current/grub LABEL=root /mnt/boot/grub/
       mount -o defaults,discard,compress=lzo,space_cache,autodefrag,subvol=__current/guix LABEL=root /mnt/var/guix
       mount -o defaults,discard,compress=lzo,space_cache,autodefrag,subvol=__current/gnu LABEL=root /mnt/gnu
     #+END_SRC

     Enable swap
     #+BEGIN_SRC bash :dir /ssh:root@192.168.4.112: :session
       mkswap /dev/mapper/swap
       swapon /dev/mapper/swap
     #+END_SRC
**** Config
     Copy the config files

     The guixsd configuration
     #+BEGIN_SRC scheme :session :tangle /ssh:root@192.168.4.112:/mnt/etc/config.scm :mkdirp yes
       ;; This is an operating system configuration template
       ;; for a "desktop" setup without full-blown desktop
       ;; environments.

       (use-modules (gnu)
                    (gnu packages)
                    (gnu system nss)
                    (gnu system locale)
                    (gnu services nfs)
                    (ice-9 rdelim)
                    (ice-9 format))
       (use-service-modules desktop networking ssh base xorg)
       (use-package-modules wm certs shells xdisorg)

       (define plattfot
         (user-account
          (name "plattfot")
          (group "users")
          ;; Define a G-Expr to find the path of the zsh binary:
          ;; https://gitlab.com/rain1/guix-wiki/wikis/FAQ#how-do-i-make-my-login-shell-zsh
          (shell #~(string-append #$zsh "/bin/zsh"))
          (supplementary-groups '("wheel" "netdev" "audio" "video"))
          (home-directory "/home/plattfot")))

       ;; (define keyboard-conf
       ;;   (call-with-input-file "/etc/config.d/00-keyboard.conf" read-string))

       ;; Specify a mapped device for the encrypted root partition.
       ;; The UUID is that returned by 'cryptsetup luksUUID'.
       (define mapped-root
         (mapped-device
          (source (uuid "ab43f8be-1a18-4999-836d-71dac382dfb5"))
          (target "root")
          (type luks-device-mapping)))

       (define mapped-swap
         (mapped-device
          (source (uuid "9f04f917-efd3-4036-b3f5-24705fee7ffa"))
          (target "swap")
          (type luks-device-mapping)))

       ;; Partion layout for /dev/sda3
       ;; under __current/
       ;; | subvol      | Mountpoint   | Comment            | Shared   |
       ;; |-------------+--------------+--------------------+----------|
       ;; | arch-root   | /            | root for Arch      | no       |
       ;; | guixsd-root | /            | root for GuixSD    | no       |
       ;; | grub        | /boot/grub   | grub config        | yes      |
       ;; | guix        | /var/guix    | guix stuff         | yes      |
       ;; | gnu         | /gnu         | Store etc          | yes      |
       ;; | home        | /home        | home partition     | yes      |

       (define btrfs-common-options
         '("defaults" "discard" "compress=lzo" "space_cache" "autodefrag"))

       (define (btrfs-mount-options subvol)
         "Return the btrfs mount options I use.
       Where SUBVOL is the subvolume to mount"
         (string-join `(,@btrfs-common-options ,(format #f "subvol=~a" subvol)) ","))

       (define fs-root
         (file-system
          (mount-point "/")
          (type "btrfs")
          (device (file-system-label "root"))
          (options (btrfs-mount-options "__current/guixsd-root"))
          (needed-for-boot? #t)
          (dependencies `(,mapped-root))))

       (define fs-grub
         (file-system
          (mount-point "/boot/grub")
          (type "btrfs")
          (device (file-system-label "root"))
          (options (btrfs-mount-options "__current/grub"))
          (needed-for-boot? #t)
          (dependencies `(,fs-root))))

       (define fs-gnu
         (file-system
          (mount-point "/gnu")
          (type "btrfs")
          (device (file-system-label "root"))
          (options (btrfs-mount-options "__current/gnu"))
          (needed-for-boot? #t)
          (dependencies `(,fs-root))))

       (define fs-guix
         (file-system
          (mount-point "/var/guix")
          (type "btrfs")
          (device (file-system-label "root"))
          (options (btrfs-mount-options "__current/guix"))
          (needed-for-boot? #t)
          (dependencies `(,fs-root))))

       (define fs-home
         (file-system
          (mount-point "/home")
          (type "btrfs")
          (device (file-system-label "root"))
          (options (btrfs-mount-options "__current/home"))
          (needed-for-boot? #t)
          (dependencies `(,fs-root))))

       (define fs-valhalla
         (file-system
          (device "fafner:/srv/nfs4/Valhalla")
          (mount-point "/media/Valhalla")
          (type "nfs4")
          (mount? #f)
          (check? #f)))

       (define menu-arch
         (menu-entry
          (label "Arch Linux")
          (linux "/boot/vmlinux")
          (linux-arguments
           '("luks.uuid=ab43f8be-1a18-4999-836d-71dac382dfb5"
             "luks.name=ab43f8be-1a18-4999-836d-71dac382dfb5=root"
             "luks.key=ab43f8be-1a18-4999-836d-71dac382dfb5=/boot/rootkey.bin"
             "luks.options=ab43f8be-1a18-4999-836d-71dac382dfb5=discard,luks"
             "luks.uuid=9f04f917-efd3-4036-b3f5-24705fee7ffa"
             "luks.name=9f04f917-efd3-4036-b3f5-24705fee7ffa=swap"
             "luks.key=9f04f917-efd3-4036-b3f5-24705fee7ffa=/boot/swapkey.bin"
             "luks.options=9f04f917-efd3-4036-b3f5-24705fee7ffa=swap,discard,luks"
             "root=LABEL=root"
             "resume=/dev/mapper/swap"
             "rootflags=compress=lzo,subvol=__current/arch-root"))
          (initrd "/boot/initramfs-linux.img")))

       (operating-system
        (host-name "loke")
        (timezone "Canada/Pacific")
        (locale "en_US.utf8")
        (locale-definitions
         (list
          (locale-definition (name "en_US.utf8") (source "en_US") (charset "UTF-8"))
          (locale-definition (name "sv_SE.utf8") (source "sv_SE") (charset "UTF-8"))))
        ;; Assuming /dev/sda is the target hard disk, and "root"
        ;; is the label of the target root file system.
        (bootloader
         (grub-configuration (target "/dev/sda")
                             ;; Need to mount __current/arch-root
                             ;; (menu-entries '(menu-arch))
                             ))
        ;; Kernel arguments
        (kernel-arguments '("rootflags=compress=lzo,subvol=__current/guixsd-root"))
        (mapped-devices (list mapped-root mapped-swap))
        (file-systems
         (cons*
          fs-home
          fs-grub
          fs-gnu
          fs-guix
          fs-root
          %base-file-systems))
        (swap-devices '("/dev/mapper/swap"))
        (users (cons plattfot %base-user-accounts))

        ;; Add a bunch of window managers; we can choose one at
        ;; the log-in screen with F1.
        (packages (cons* i3-wm i3status rofi ;window managers
                         zsh
                         nss-certs            ;for HTTPS access
                         %base-packages))

        ;; Use the "desktop" services, which include the X11
        ;; log-in service, networking with Wicd, and more.
        (services
         %desktop-services
         ;; (cons* ;; (service openssh-service-type
         ;;        ;;          (openssh-configuration
         ;;        ;;           (port-number 6060)
         ;;        ;;           (password-authentication? #f)))
         ;;        ;; (extra-special-file "/bin/env" (file-append coreutils "/bin/env"))
         ;;        %desktop-services
         ;;        ;; (modify-services %desktop-services
         ;;        ;;	   (slim-service-type
         ;;        ;;    config => (slim-configuration
         ;;        ;;       (inherit config)
         ;;        ;;     (startx (xorg-start-command
         ;;        ;;	#:configuration-file
         ;;        ;;(xorg-configuration-file
         ;;        ;;#:extra-config
         ;;        ;;(list keyboard-conf)))))))
         ;;        )
         )

        ;; Allow resolution of '.local' host names with mDNS.
        (name-service-switch %mdns-host-lookup-nss))

     #+END_SRC

     The xorg file for configuring the keyboard
     #+BEGIN_SRC conf :session :tangle /ssh:root@192.168.4.112:/mnt/etc/config.d/00-keyboard.conf :mkdirp yes
       # Map Ctrl to caps, toggle between us and swedish keyboard layout. Scroll lock led is on when using swedish layout.
       Section "InputClass"
               Identifier "system-keyboard"
               MatchIsKeyboard "on"
               Option "XkbLayout" "us,se"
               Option "XkbOptions" "ctrl:nocaps,grp:sclk_toggle,grp_led:scroll,:2"
       EndSection
     #+END_SRC

**** Install

     Start cow-store on /mnt
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       herd start cow-store /mnt/
     #+END_SRC

     Authorize berlin, much faster than hydra.
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       guix archive --authorize < /gnu/store/cw55zvxzi3d9cjmhfvxsryz31jxb1y6k-guix-0.15.0-1.4876bc8/share/guix/berlin.guixsd.org.pub
     #+END_SRC

***** Updated to a newer version (optional)

      Note: this might take some time, so running it in org-babel
      might not be the best.

      Find a commit with good coverage at
      http://berlin.guixsd.org/jobset/guix-master

      Update guix to that.
      #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
        guix pull --commit=d9f8e84 --substitute-urls="http://berlin.guixsd.org http://mirror.hydra.gnu.org"
      #+END_SRC

***** System init

      Note: this might take some time, so running it in org-babel
      might not be the best.

      Install
      #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session

        guix system init /mnt/etc/config.scm /mnt --substitute-urls="http://berlin.guixsd.org http://mirror.hydra.gnu.org"
      #+END_SRC

**** Hack grub

     Currently need to hack the grub file for it to boot correctly.

     First make it writeable
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       chmod +w /mnt/boot/grub/grub.cfg
     #+END_SRC

     #+RESULTS:

     Then change:
     #+BEGIN_SRC conf
       # Set 'root' to the partition that contains /gnu/store.
       search --label --set root

       if loadfont /store/<hash>-grub-2.02/share/grub/unicode.pf2; then
       setup_gfxterm
       fi
     #+END_SRC

     To

     #+BEGIN_SRC conf
       insmod part_gpt
       insmod cryptodisk
       insmod luks
       insmod gcry_rijndael
       insmod gcry_rijndael
       insmod gcry_sha256
       insmod btrfs
       cryptomount -u ab43f8be1a184999836d71dac382dfb5
       set root='cryptouuid/ab43f8be1a184999836d71dac382dfb5'

       search --no-floppy --fs-uuid --set=root --hint='cryptouuid/ab43f8be1a184999836d71dac382dfb5'  7cd60921-2b01-487d-8369-046a23a00de5

       font="/__current/gnu/store/<hash>-grub-2.02/share/grub/unicode.pf2"

       if loadfont $font; then
       setup_gfxterm
       fi
     #+END_SRC

     Where cryptmount -u hash is the same as specified in
     (mapped-device ...) for root in config.scm.

     And the last hash is the one of the mapped disk (/dev/mapper/root).

     Correct the path for background_image.

     Then change the menuentry of guix
     #+BEGIN_SRC conf
       search --label --set root

       linux /gnu/store/<hash>-linux-libre-4.17.3/bzImage --root=root --system=/gnu/store/<hash>-system --load=/gnu/store/<hash>-system/boot rootflags=compress=lzo,subvol=__current/guixsd-root
       initrd /gnu/store/<hash>-raw-initrd/initrd
     #+END_SRC

     To
     #+BEGIN_SRC conf
       set gfxpayload=keep
       insmod gzio
       insmod part_gpt
       insmod cryptodisk
       insmod luks
       insmod gcry_rijndael
       insmod gcry_rijndael
       insmod gcry_sha256
       insmod btrfs
       cryptomount -u ab43f8be1a184999836d71dac382dfb5
       set root='cryptouuid/ab43f8be1a184999836d71dac382dfb5'
       search --no-floppy --fs-uuid --set=root --hint='cryptouuid/ab43f8be1a184999836d71dac382dfb5'  7cd60921-2b01-487d-8369-046a23a00de5

       linux /__current/gnu/store/<hash>-linux-libre-4.17.3/bzImage --root=root --system=/gnu/store/<hash>-system --load=/gnu/store/<hash>-system/boot rootflags=compress=lzo,subvol=__current/guixsd-root
       initrd /__current/gnu/store/<hash>-raw-initrd/initrd
     #+END_SRC

     Note that <hash> just indicates that there's a hash there, not that
     all the hashes are the same. And that only update the path for the
     bxImage and initrd and not the one for --system= and --load.

     I.e add the modules to be able to decrypt (not sure if all are
     needed) then set the correct path for the stuff in the
     store. Keeping the hashes the same. I.e. you cannot just backup
     grub.cfg then copy it back, as then you would just get the old
     install.

     After edit make it read only again, and you're done.

     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       chmod -w /mnt/boot/grub/grub.cfg
     #+END_SRC

     #+RESULTS:
**** Hack btrfs

     Something doesn't seem to respect the mount options (rootflags)
     when booting linux. Instead of mounting ~__current/guixsd-root~
     as /, ~__current/arch-root~ gets mounted instead (Which is the
     default subvolume.). That throws a wrench in the whole boot
     process as GuixSD cannot setup the environment. You will see
     errors like, "cannot symlink /etc/ssl, file already exist"

     To work around that we need to change the default subvolume to
     ~__current/guixsd-root~.

     Fetch the Subvolume id for ~__current/guixsd-root~, assumes it's
     mounted at ~/mnt~.

     #+NAME: btrfs-subvolume-id
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       btrfs subvolume show /mnt | grep "Subvolume ID:" | grep -oE "[0-9]+"
     #+END_SRC

     Then set it as the default.
     #+BEGIN_SRC sh :dir /ssh:root@192.168.4.112: :session
       btrfs subvolume set-default <<btrfs-subvolume-id()>> /mnt
     #+END_SRC

Re: btrfs and subvolumes for root, take 2

[Ludovic Courtès]

Hi,

Fredrik Salomonsson <plattfot@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The Guile backtrace you sent shows that /etc/ssl already existed when
>> your system booted and was not a symlink.  This led the “activation
>> code” of GuixSD to fail:
>>
>>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/activation.scm#n320
>>
>> The solution is to remove /etc/ssl (is it coming from another distro
>> previously installed on this device?).  You can boot a separate medium, 
>> mount the root partition, and “rm -rf /etc/ssl” from there.  Or you can,
>> at the boot REPL that you get after the backtrace, type something like:
>>
>>   ,use (guix build utils)
>>   (delete-file-recursively "/etc/ssl")
>>   ,q
>>
>> Note that you might have similar issues with /etc/pam.d, for instance,
>> if there’s such a stale directory.
>
> Thanks for the reply. That pointed me in the right direction. Although
> the solution you suggested wasn't an option for me. As it turned out, it
> was actually mounting my Arch Linux root (__current/arch-root). Which I
> had set to be the default subvolume if no ~subvol=~ option is given when
> mounting the disk.

Clearly / or /etc cannot be shared between GuixSD and another distro;
each distro needs to have full control over these.  My suggestion would
be to share nothing but /home (and /gnu, /var/guix, and /etc/guix if you
want to able to use Guix on the other distro).

I can’t really advise more than this since the specifics are then a
matter of taste.  :-)

Thanks,
Ludo’.

Re: btrfs and subvolumes for root, take 2

[Fredrik Salomonsson]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Fredrik Salomonsson <plattfot@gmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> The Guile backtrace you sent shows that /etc/ssl already existed when
>>> your system booted and was not a symlink.  This led the “activation
>>> code” of GuixSD to fail:
>>>
>>>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/activation.scm#n320
>>>
>>> The solution is to remove /etc/ssl (is it coming from another distro
>>> previously installed on this device?).  You can boot a separate medium, 
>>> mount the root partition, and “rm -rf /etc/ssl” from there.  Or you can,
>>> at the boot REPL that you get after the backtrace, type something like:
>>>
>>>   ,use (guix build utils)
>>>   (delete-file-recursively "/etc/ssl")
>>>   ,q
>>>
>>> Note that you might have similar issues with /etc/pam.d, for instance,
>>> if there’s such a stale directory.
>>
>> Thanks for the reply. That pointed me in the right direction. Although
>> the solution you suggested wasn't an option for me. As it turned out, it
>> was actually mounting my Arch Linux root (__current/arch-root). Which I
>> had set to be the default subvolume if no ~subvol=~ option is given when
>> mounting the disk.
>
> Clearly / or /etc cannot be shared between GuixSD and another distro;
> each distro needs to have full control over these.  My suggestion would
> be to share nothing but /home (and /gnu, /var/guix, and /etc/guix if you
> want to able to use Guix on the other distro).
>
> I can’t really advise more than this since the specifics are then a
> matter of taste.  :-)

I think you misunderstood what I meant. I'm not sharing / or /etc
between GuixSD and Arch linux. Both have their own / (and /etc), in
their own subvolume (think partitions for ext4).

See this thread for a full explanation of my disk layout:
https://lists.gnu.org/archive/html/help-guix/2017-08/msg00011.html

But in short, only /home, /gnu, /var/guix and /boot/grub are shared
between the two. The issue I had was that for some reason when booting
GuixSD. It ignore the rootflags for mounting GuixSD's /, instead it
mounted the / for Arch, as that was the default. Which causes the
error. Clearly I do not want it to share /etc between the two, that
would be madness. :)

I worked around that by changing the default to GuixSD's /. So now it's
mounting the correct subvolumes. Sorry about the confusion.

-- 
s/Fred[re]+i[ck]+/Fredrik/g