[bug#40207] [PATCH 0/2] Add a desktop graphical installer test.

[bug#40207] [PATCH 0/2] Add a desktop graphical installer test.

[Mathieu Othacehe]

Hello,

Here are two patches that allow testing the graphical installation of a Guix
System with all desktop-environments included.

Thanks,

Mathieu

Mathieu Othacehe (2):
  installer: tests: Use a filter to select desktop-environments.
  tests: install: Add %test-gui-installed-desktop-os-encrypted.

 gnu/installer/tests.scm | 11 +++--
 gnu/tests/install.scm   | 90 +++++++++++++++++++++++++++++++++--------
 2 files changed, 81 insertions(+), 20 deletions(-)

-- 
2.25.1

[bug#40207] [PATCH 1/2] installer: tests: Use a filter to select desktop-environments.

[Mathieu Othacehe]

* gnu/installer/tests.scm (choose-services): Turn desktop-environments list
into a choose-desktop-environment procedure. This way, it is easier to select
all desktop-environments or none, in the same way as choose-network-service?
and choose-network-management-tool? arguments.
---
 gnu/installer/tests.scm | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/gnu/installer/tests.scm b/gnu/installer/tests.scm
index 6f5393e3ab..07f3121083 100644
--- a/gnu/installer/tests.scm
+++ b/gnu/installer/tests.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -211,7 +212,7 @@ ROOT-PASSWORD, and USERS."
 
 (define* (choose-services port
                           #:key
-                          (desktop-environments '("GNOME"))
+                          (choose-desktop-environment? (const #f))
                           (choose-network-service?
                            (lambda (service)
                              (or (string-contains service "SSH")
@@ -220,10 +221,14 @@ ROOT-PASSWORD, and USERS."
                            (lambda (service)
                              (string-contains service "DHCP"))))
   "Converse over PORT to choose networking services."
+  (define desktop-environments '())
+
   (converse port
     ((checkbox-list (title "Desktop environment") (text _)
-                    (items _))
-     desktop-environments)
+                    (items ,services))
+     (let ((services* (filter choose-desktop-environment? services)))
+       (set! desktop-environments services*)
+       services*))
     ((checkbox-list (title "Network service") (text _)
                     (items ,services))
      (filter choose-network-service? services))
-- 
2.25.1

[bug#40207] [PATCH 2/2] tests: install: Add %test-gui-installed-desktop-os-encrypted.

[Mathieu Othacehe]

* gnu/tests/install.scm (run-install): Make sure that the default target-size
is used if #f is passed,
(gui-test-program): add a desktop? argument, and pass it to choose-services,
(guided-installation-test): add desktop? and target-size arguments. If
desktop? is #t, make sure that all desktop-environments are available. Pass
target-size to run-install call and desktop? to gui-test-program call.
(%test-gui-installed-desktop-os-encrypted): New variable.
---
 gnu/tests/install.scm | 90 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 73 insertions(+), 17 deletions(-)

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 83988873c2..9a4b36d5e7 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -32,15 +32,23 @@
   #:use-module (gnu packages cryptsetup)
   #:use-module (gnu packages linux)
   #:use-module (gnu packages ocr)
+  #:use-module (gnu packages openbox)
   #:use-module (gnu packages package-management)
+  #:use-module (gnu packages ratpoison)
+  #:use-module (gnu packages suckless)
   #:use-module (gnu packages virtualization)
+  #:use-module (gnu packages wm)
+  #:use-module (gnu packages xorg)
+  #:use-module (gnu services desktop)
   #:use-module (gnu services networking)
+  #:use-module (gnu services xorg)
   #:use-module (guix store)
   #:use-module (guix monads)
   #:use-module (guix packages)
   #:use-module (guix grafts)
   #:use-module (guix gexp)
   #:use-module (guix utils)
+  #:use-module (srfi srfi-1)
   #:export (%test-installed-os
             %test-installed-extlinux-os
             %test-iso-image-installer
@@ -52,7 +60,8 @@
             %test-jfs-root-os
 
             %test-gui-installed-os
-            %test-gui-installed-os-encrypted))
+            %test-gui-installed-os-encrypted
+            %test-gui-installed-desktop-os-encrypted))
 
 ;;; Commentary:
 ;;;
@@ -203,13 +212,14 @@ reboot\n")
                                                 (gnu installer tests)
                                                 (guix combinators))))
                       (installation-disk-image-file-system-type "ext4")
-                      (target-size (* 2200 MiB)))
+                      (target-size #f))
   "Run SCRIPT (a shell script following the system installation procedure) in
 OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
 the installed system.  The packages specified in PACKAGES will be appended to
 packages defined in installation-os."
 
-  (mlet* %store-monad ((_      (set-grafting #f))
+  (mlet* %store-monad ((target-size -> (or target-size (* 2200 MiB)))
+                       (_      (set-grafting #f))
                        (system (current-system))
                        (target (operating-system-derivation target-os))
 
@@ -941,7 +951,10 @@ build (current-guix) and then store a couple of full system images.")
 
 (define %root-password "foo")
 
-(define* (gui-test-program marionette #:key (encrypted? #f))
+(define* (gui-test-program marionette
+                           #:key
+                           (desktop? #f)
+                           (encrypted? #f))
   #~(let ()
       (define (screenshot file)
         (marionette-control (string-append "screendump " file)
@@ -998,7 +1011,8 @@ build (current-guix) and then store a couple of full system images.")
       (screenshot "installer-services.ppm")
 
       (marionette-eval* '(choose-services installer-socket
-                                          #:desktop-environments '()
+                                          #:choose-desktop-environment?
+                                          (const #$desktop?)
                                           #:choose-network-service?
                                           (const #f))
                         #$marionette)
@@ -1038,7 +1052,11 @@ build (current-guix) and then store a couple of full system images.")
                         (gnu installer tests)
                         (guix combinators))))
 
-(define* (guided-installation-test name #:key encrypted?)
+(define* (guided-installation-test name
+                                   #:key
+                                   (desktop? #f)
+                                   encrypted?
+                                   (target-size #f))
   (define os
     (operating-system
       (inherit %minimal-os)
@@ -1055,26 +1073,56 @@ build (current-guix) and then store a couple of full system images.")
                             (supplementary-groups
                              '("wheel" "audio" "video"))))
                      %base-user-accounts))
+      (keyboard-layout (and desktop?
+                            (keyboard-layout "us" "altgr-intl")))
       ;; The installer does not create a swap device in guided mode with
       ;; encryption support.
       (swap-devices (if encrypted? '() '("/dev/vdb2")))
-      (services (cons (service dhcp-client-service-type)
-                      (operating-system-user-services %minimal-os)))))
+
+      ;; Make sure that all the packages and services that may be used by the
+      ;; graphical installer are available.
+      (packages (append
+                 (if desktop?
+                     (list openbox awesome i3-wm i3status
+                           dmenu st ratpoison xterm)
+                     '())
+                 %base-packages))
+      (services
+       (if desktop?
+           (append
+            (list (service gnome-desktop-service-type)
+                  (service xfce-desktop-service-type)
+                  (service mate-desktop-service-type)
+                  (service enlightenment-desktop-service-type)
+                  (set-xorg-configuration
+                   (xorg-configuration
+                    (keyboard-layout keyboard-layout)))
+                  (service marionette-service-type
+                           (marionette-configuration
+                            (imported-modules '((gnu services herd)
+                                                (guix build utils)
+                                                (guix combinators))))))
+            %desktop-services)
+           (cons (service dhcp-client-service-type)
+                 (operating-system-user-services %minimal-os))))))
 
   (system-test
    (name name)
    (description
     "Install an OS using the graphical installer and test it.")
    (value
-    (mlet* %store-monad ((image   (run-install os '(this is unused)
-                                               #:script #f
-                                               #:os installation-os-for-gui-tests
-                                               #:gui-test
-                                               (lambda (marionette)
-                                                 (gui-test-program
-                                                  marionette
-                                                  #:encrypted? encrypted?))))
-                         (command (qemu-command/writable-image image)))
+    (mlet* %store-monad
+        ((image   (run-install os '(this is unused)
+                               #:script #f
+                               #:os installation-os-for-gui-tests
+                               #:target-size target-size
+                               #:gui-test
+                               (lambda (marionette)
+                                 (gui-test-program
+                                  marionette
+                                  #:desktop? desktop?
+                                  #:encrypted? encrypted?))))
+         (command (qemu-command/writable-image image)))
       (run-basic-test os command name
                       #:initialization (and encrypted? enter-luks-passphrase)
                       #:root-password %root-password)))))
@@ -1087,4 +1135,12 @@ build (current-guix) and then store a couple of full system images.")
   (guided-installation-test "gui-installed-os-encrypted"
                             #:encrypted? #t))
 
+;; Building a desktop image is very time and space consuming. Install all
+;; desktop environments in a single test to reduce the overhead.
+(define %test-gui-installed-desktop-os-encrypted
+  (guided-installation-test "gui-installed-desktop-os-encrypted"
+                            #:desktop? #t
+                            #:encrypted? #t
+                            #:target-size (* 9000 MiB)))
+
 ;;; install.scm ends here
-- 
2.25.1

[bug#40207] [PATCH 2/2] tests: install: Add %test-gui-installed-desktop-os-encrypted.

[Ludovic Courtès]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> * gnu/tests/install.scm (run-install): Make sure that the default target-size
> is used if #f is passed,
> (gui-test-program): add a desktop? argument, and pass it to choose-services,
> (guided-installation-test): add desktop? and target-size arguments. If
> desktop? is #t, make sure that all desktop-environments are available. Pass
> target-size to run-install call and desktop? to gui-test-program call.
> (%test-gui-installed-desktop-os-encrypted): New variable.

[...]

> -                      (target-size (* 2200 MiB)))
> +                      (target-size #f))
>    "Run SCRIPT (a shell script following the system installation procedure) in
>  OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
>  the installed system.  The packages specified in PACKAGES will be appended to
>  packages defined in installation-os."
>  
> -  (mlet* %store-monad ((_      (set-grafting #f))
> +  (mlet* %store-monad ((target-size -> (or target-size (* 2200 MiB)))

[...]

> +(define* (guided-installation-test name
> +                                   #:key
> +                                   (desktop? #f)
> +                                   encrypted?
> +                                   (target-size #f))

What about giving #:target-size a default value here so that the hunk
above in unnecessary?

>    (define os
>      (operating-system
>        (inherit %minimal-os)
> @@ -1055,26 +1073,56 @@ build (current-guix) and then store a couple of full system images.")
>                              (supplementary-groups
>                               '("wheel" "audio" "video"))))
>                       %base-user-accounts))
> +      (keyboard-layout (and desktop?
> +                            (keyboard-layout "us" "altgr-intl")))
>        ;; The installer does not create a swap device in guided mode with
>        ;; encryption support.
>        (swap-devices (if encrypted? '() '("/dev/vdb2")))
> -      (services (cons (service dhcp-client-service-type)
> -                      (operating-system-user-services %minimal-os)))))
> +
> +      ;; Make sure that all the packages and services that may be used by the
> +      ;; graphical installer are available.
> +      (packages (append
> +                 (if desktop?
> +                     (list openbox awesome i3-wm i3status
> +                           dmenu st ratpoison xterm)
> +                     '())
> +                 %base-packages))
> +      (services
> +       (if desktop?
> +           (append
> +            (list (service gnome-desktop-service-type)
> +                  (service xfce-desktop-service-type)
> +                  (service mate-desktop-service-type)
> +                  (service enlightenment-desktop-service-type)
> +                  (set-xorg-configuration
> +                   (xorg-configuration
> +                    (keyboard-layout keyboard-layout)))
> +                  (service marionette-service-type
> +                           (marionette-configuration
> +                            (imported-modules '((gnu services herd)
> +                                                (guix build utils)
> +                                                (guix combinators))))))
> +            %desktop-services)
> +           (cons (service dhcp-client-service-type)
> +                 (operating-system-user-services %minimal-os))))))

What about making a separate OS definition for the desktop installation
case?  We’d move the ‘os’ variable to the top-level and add an ‘os’
parameter to ‘guided-installation-test’.

Having two separate OS definitions (one for desktop install, one for
“bare-bones” install) should be clearer than having a single definition
with conditionals.

> +;; Building a desktop image is very time and space consuming. Install all
> +;; desktop environments in a single test to reduce the overhead.
> +(define %test-gui-installed-desktop-os-encrypted
> +  (guided-installation-test "gui-installed-desktop-os-encrypted"
> +                            #:desktop? #t
> +                            #:encrypted? #t
> +                            #:target-size (* 9000 MiB)))

Wo0t!

Thanks a lot for working on it!  You can stop by on #guix if you feel
like it so we’re all less lonely during confinement.  :-)

Ludo’.

[bug#40207] [PATCH 2/2] tests: install: Add %test-gui-installed-desktop-os-encrypted.

[Mathieu Othacehe]

Hey,

> What about making a separate OS definition for the desktop installation
> case?  We’d move the ‘os’ variable to the top-level and add an ‘os’
> parameter to ‘guided-installation-test’.
>
> Having two separate OS definitions (one for desktop install, one for
> “bare-bones” install) should be clearer than having a single definition
> with conditionals.

I fixed all your remarks. But there's still an issue, the size of the
installation image is guessed badly in "run-install" for
"%test-gui-installed-desktop-os-encrypted"

--8<---------------cut here---------------start------------->8---
(image  (system-disk-image
         (operating-system-with-gc-roots
          os (list target))
         #:disk-image-size 'guess ;guess is too low.
         #:file-system-type
         installation-disk-image-file-system-type)))
--8<---------------cut here---------------end--------------->8---

applying the dirty snippet below fixes it, but you may have a better idea?

--8<---------------cut here---------------start------------->8---
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -233,7 +233,7 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
   "Return the estimated size of a partition that can store the store items
 given by GRAPHS, a list of file names produced by #:references-graphs."
   ;; Simply add a 25% overhead.
-  (round (* 1.25 (closure-size graphs))))
+  (round (* 1.30 (closure-size graphs))))

 (define* (initialize-partition-table device partitions
--8<---------------cut here---------------end--------------->8---

Thanks,

Mathieu

[bug#40207] [PATCH 2/2] tests: install: Add %test-gui-installed-desktop-os-encrypted.

[Ludovic Courtès]

Hi!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> What about making a separate OS definition for the desktop installation
>> case?  We’d move the ‘os’ variable to the top-level and add an ‘os’
>> parameter to ‘guided-installation-test’.
>>
>> Having two separate OS definitions (one for desktop install, one for
>> “bare-bones” install) should be clearer than having a single definition
>> with conditionals.
>
> I fixed all your remarks. But there's still an issue, the size of the
> installation image is guessed badly in "run-install" for
> "%test-gui-installed-desktop-os-encrypted"
>
> (image  (system-disk-image
>          (operating-system-with-gc-roots
>           os (list target))
>          #:disk-image-size 'guess ;guess is too low.
>          #:file-system-type
>          installation-disk-image-file-system-type)))
>
>
> applying the dirty snippet below fixes it, but you may have a better idea?
>
> --- a/gnu/build/vm.scm
> +++ b/gnu/build/vm.scm
> @@ -233,7 +233,7 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
>    "Return the estimated size of a partition that can store the store items
>  given by GRAPHS, a list of file names produced by #:references-graphs."
>    ;; Simply add a 25% overhead.
> -  (round (* 1.25 (closure-size graphs))))
> +  (round (* 1.30 (closure-size graphs))))

Weird, the 25% hack had always worked fine.

I don’t have a better idea though.  Perhaps it’s better to pass an
explicit size in ‘run-install’ than to modify the 25% hack above?

Thanks,
Ludo’.

[bug#40207] [PATCH 2/2] tests: install: Add %test-gui-installed-desktop-os-encrypted.

[Mathieu Othacehe]

Hey,

> I don’t have a better idea though.  Perhaps it’s better to pass an
> explicit size in ‘run-install’ than to modify the 25% hack above?

Seems fair, fixed and pushed!

Thanks,

Mathieu

[bug#40207] [PATCH 1/2] installer: tests: Use a filter to select desktop-environments.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> * gnu/installer/tests.scm (choose-services): Turn desktop-environments list
> into a choose-desktop-environment procedure. This way, it is easier to select
> all desktop-environments or none, in the same way as choose-network-service?
> and choose-network-management-tool? arguments.

[...]

> +     (let ((services* (filter choose-desktop-environment? services)))
> +       (set! desktop-environments services*)
> +       services*))

Maybe rename ‘services*’ to ‘desktops’?

‘set!’ hurts my eyes :-) but let’s get over it.  Eventually we can
change convert to run expressions in ‘%state-monad’ or something.

Ludo’.

[bug#40207] [PATCH 1/2] installer: tests: Use a filter to select desktop-environments.

[Mathieu Othacehe]

Hey!

> Maybe rename ‘services*’ to ‘desktops’?
>
> ‘set!’ hurts my eyes :-) but let’s get over it.  Eventually we can
> change convert to run expressions in ‘%state-monad’ or something.

Heh, also feel terribly guilty when using it :)

Thanks for reviewing,

Mathieu