[bug#28288] [PATCH] Test and fix the ISO Image Installer

[bug#28288] [PATCH] Test and fix the ISO Image Installer

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

Hey,

So at the GNU Hackers Meeting, I attempted to build the installer system
as an ISO image, and test installing Guix on a Bytemark VM with this. I
encountered problems, and therefore set about writing a system test to
see if this could be reproduced.

The failures did seem reproducible locally, and Ludo helped with some
code changes. I'll send the work in progress patches to this bug.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 1/2] tests: Add test for installing from an ISO Image.

[Christopher Baines]

* gnu/tests/install.scm (%test-iso-image-installer): New variable.
  (run-install): Add #:installation-disk-image-file-system-type as a keyword
  argument.
---
 gnu/tests/install.scm | 48 +++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 11 deletions(-)

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 866bf885c..7596347a3 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -37,6 +37,7 @@
   #:use-module (guix utils)
   #:export (%test-installed-os
             %test-installed-extlinux-os
+            %test-iso-image-installer
             %test-separate-store-os
             %test-separate-home-os
             %test-raid-root-os
@@ -196,6 +197,7 @@ reboot\n")
                              (kernel-arguments '("console=ttyS0")))
                            #:imported-modules '((gnu services herd)
                                                 (guix combinators))))
+                      (installation-disk-image-file-system-type "ext4")
                       (target-size (* 1200 MiB)))
   "Run SCRIPT (a shell script following the GuixSD installation procedure) in
 OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
@@ -213,7 +215,9 @@ packages defined in installation-os."
                        (image  (system-disk-image
                                 (operating-system-with-gc-roots
                                  os (list target))
-                                #:disk-image-size (* 1500 MiB))))
+                                #:disk-image-size (* 1500 MiB)
+                                #:file-system-type
+                                installation-disk-image-file-system-type)))
     (define install
       (with-imported-modules '((guix build utils)
                                (gnu build marionette))
@@ -229,16 +233,24 @@ packages defined in installation-os."
 
             (define marionette
               (make-marionette
-               (cons (which #$(qemu-command system))
-                     (cons* "-no-reboot" "-m" "800"
-                            "-drive"
-                            (string-append "file=" #$image
-                                           ",if=virtio,readonly")
-                            "-drive"
-                            (string-append "file=" #$output ",if=virtio")
-                            (if (file-exists? "/dev/kvm")
-                                '("-enable-kvm")
-                                '())))))
+               `(,(which #$(qemu-command system))
+                 "-no-reboot"
+                 "-m" "800"
+                 ,@(case installation-disk-image-file-system-type
+                     (("ext4")
+                      `("-drive"
+                        ,(string-append "file=" #$image
+                                        ",if=virtio,readonly")))
+                     (("iso9660") '("-cdrom" #$image))
+                     (else
+                      (error
+                       "unsupported installation-disk-image-file-system-type"
+                       installation-disk-image-file-system-type)))
+                 "-drive"
+                 ,(string-append "file=" #$output ",if=virtio")
+                 ,@(if (file-exists? "/dev/kvm")
+                       '("-enable-kvm")
+                       '()))))
 
             (pk 'uname (marionette-eval '(uname) marionette))
 
@@ -312,6 +324,20 @@ per %test-installed-os, this test is expensive in terms of CPU and storage.")
       (run-basic-test %minimal-extlinux-os command
                       "installed-extlinux-os")))))
 
+(define %test-iso-image-installer
+  (system-test
+   (name "iso-image-installer")
+   (description
+    "")
+   (value
+    (mlet* %store-monad ((image   (run-install
+                                   %minimal-os
+                                   %minimal-os-source
+                                   #:installation-disk-image-file-system-type
+                                   "iso9660"))
+                         (command (qemu-command/writable-image image)))
+      (run-basic-test %minimal-os command name)))))
+
 \f
 ;;;
 ;;; Separate /home.
-- 
2.14.1

[bug#28288] [PATCH 2/2] WIP

[Christopher Baines]

---
 gnu/build/vm.scm  | 19 +++++++++++++++++--
 gnu/system/vm.scm | 16 +++++++++++-----
 2 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 727494ad9..e2659cb5e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -366,12 +366,27 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (error "failed to create GRUB EFI image"))))
 
 (define* (make-iso9660-image grub config-file os-drv target
-                             #:key (volume-id "GuixSD_image") (volume-uuid #f))
+                             #:key (volume-id "GuixSD_image") (volume-uuid #f)
+                             register-closures? (closures '()))
   "Given a GRUB package, creates an iso image as TARGET, using CONFIG-FILE as
 GRUB configuration and OS-DRV as the stuff in it."
-  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue")))
+  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue"))
+        (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+
+    ;; XXX: 'guix-register' wants to palpate the things it registers, so
+    ;; bind-mount the store on the target.
+    (mkdir-p target-store)
+    (mount (%store-directory) target-store "" MS_BIND)
+
+    (when register-closures?
+      (display "registering closures...\n")
+      (for-each (lambda (closure)
+                  (register-closure target
+                                    (string-append "/xchg/" closure)))
+                closures))
+
     (unless (zero? (apply system*
                           `(,grub-mkrescue "-o" ,target
                             ,(string-append "boot/grub/grub.cfg=" config-file)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4494af003..b106dff0a 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -192,6 +192,7 @@ made available under the /xchg CIFS share."
                         os-drv
                         bootcfg-drv
                         bootloader
+                        register-closures?
                         (inputs '()))
   "Return a bootable, stand-alone iso9660 image.
 
@@ -207,8 +208,13 @@ INPUTS is a list of inputs (as for packages)."
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools xorriso)
                            (map canonical-package
-                                (list sed grep coreutils findutils gawk))))
+                                (list sed grep coreutils findutils gawk))
+                           (if register-closures? (list guix) '())))
 
+
+               (graphs     '#$(match inputs
+                                   (((names . _) ...)
+                                    names)))
                ;; This variable is unused but allows us to add INPUTS-TO-COPY
                ;; as inputs.
                (to-register
@@ -222,6 +228,8 @@ INPUTS is a list of inputs (as for packages)."
                                #$bootcfg-drv
                                #$os-drv
                                "/xchg/guixsd.iso"
+                               #:register-closures? #$register-closures?
+                               #:closures graphs
                                #:volume-id #$file-system-label
                                #:volume-uuid #$file-system-uuid)
            (reboot))))
@@ -391,6 +399,7 @@ to USB sticks meant to be read-only."
                          #:file-system-label root-label
                          #:file-system-uuid #f
                          #:os-drv os-drv
+                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
                                         (operating-system-bootloader os))
@@ -403,10 +412,7 @@ to USB sticks meant to be read-only."
                                     (operating-system-bootloader os))
                       #:disk-image-size disk-image-size
                       #:disk-image-format "raw"
-                      #:file-system-type (if (string=? "iso9660"
-                                                       file-system-type)
-                                             "ext4"
-                                             file-system-type)
+                      #:file-system-type file-system-type
                       #:file-system-label root-label
                       #:copy-inputs? #t
                       #:register-closures? #t
-- 
2.14.1

[bug#28288] [PATCH 2/2] WIP

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Currently, I get the following error when running this test.

registering closures...
error: initialising database schema: disk I/O error
ERROR: In procedure scm-error:
ERROR: failed to register store items "/xchg/system"


I also need the patch from
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273 otherwise I can't
even run the test.

Danny, I guess you have been looking at installing from an ISO image as
well, as I spotted you added it to the release target. Have you been
able to successfully use it yet?

When attempting to use it initially, I encountered a problem where a
glibc in the store was partially deleted. /var/guix/db/db.sqlite also
initially didn't exist, which the WIP patch here attempts to address. 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 2/2] WIP

[Danny Milosavljevic]

Hi Christopher,

On Wed, 30 Aug 2017 07:31:13 +0100
Christopher Baines <mail@cbaines.net> wrote:

> Currently, I get the following error when running this test.
> 
> registering closures...
> error: initialising database schema: disk I/O error
> ERROR: In procedure scm-error:
> ERROR: failed to register store items "/xchg/system"
> 
> 
> I also need the patch from
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273 otherwise I can't
> even run the test.
> 
> Danny, I guess you have been looking at installing from an ISO image as
> well, as I spotted you added it to the release target. Have you been
> able to successfully use it yet?

Does that mean with the wip-installer-2 branch?

Otherwise in master I always tested it using gnu/system/install.scm until it has booted up successfully, which it did.

I've asked about the installation part before but there was no response.

> When attempting to use it initially, I encountered a problem where a
> glibc in the store was partially deleted. /var/guix/db/db.sqlite also
> initially didn't exist, which the WIP patch here attempts to address. 

Hmm, do you have logs of the release creation process?  Maybe xorriso complained?

[bug#28288] [PATCH 2/2] WIP

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

On Wed, 30 Aug 2017 10:38:00 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> Hi Christopher,
> 
> On Wed, 30 Aug 2017 07:31:13 +0100
> Christopher Baines <mail@cbaines.net> wrote:
> 
> > Currently, I get the following error when running this test.
> > 
> > registering closures...
> > error: initialising database schema: disk I/O error
> > ERROR: In procedure scm-error:
> > ERROR: failed to register store items "/xchg/system"
> > 
> > 
> > I also need the patch from
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273 otherwise I
> > can't even run the test.
> > 
> > Danny, I guess you have been looking at installing from an ISO
> > image as well, as I spotted you added it to the release target.
> > Have you been able to successfully use it yet?  
> 
> Does that mean with the wip-installer-2 branch?

I've just been testing with the master branch, and with the changes I
sent to this bug. Are there any changes on the wip-installer-2 that are
not solely related to the installer (which if I understand correctly is
something more complicated than running guix system init).

> Otherwise in master I always tested it using gnu/system/install.scm
> until it has booted up successfully, which it did.
> 
> I've asked about the installation part before but there was no
> response.

Ok, that's good to know. I would guess it's pretty close now, great
work on moving the ISO stuff forward!

> > When attempting to use it initially, I encountered a problem where a
> > glibc in the store was partially deleted. /var/guix/db/db.sqlite
> > also initially didn't exist, which the WIP patch here attempts to
> > address.   
> 
> Hmm, do you have logs of the release creation process?  Maybe xorriso
> complained?

From talking to Ludo at the GHM, I think there is some extra steps
needed to make installing from the ISO possible, but I don't yet
understand this. I'll CC Ludo, as he might be able to explain better.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 2/2] WIP

[Danny Milosavljevic]

On Wed, 30 Aug 2017 23:56:47 +0100
Christopher Baines <mail@cbaines.net> wrote:

> I've just been testing with the master branch, and with the changes I
> sent to this bug. Are there any changes on the wip-installer-2 that are
> not solely related to the installer 

No

>(which if I understand correctly is something more complicated than running guix system init).

Yes

[bug#28288] [PATCH 2/2] WIP

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

On Thu, 31 Aug 2017 15:04:48 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> On Wed, 30 Aug 2017 23:56:47 +0100
> Christopher Baines <mail@cbaines.net> wrote:
> 
> > I've just been testing with the master branch, and with the changes
> > I sent to this bug. Are there any changes on the wip-installer-2
> > that are not solely related to the installer   
> 
> No
> 
> >(which if I understand correctly is something more complicated than
> >running guix system init).  
> 
> Yes

Great, ok, good to know I haven't missed anything.

I'll hopefully be able to make some time soonish to take a look at
getting the ISO image in to a state where running guix system init from
it works, unless someone beats me to it!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 2/2] WIP

[Danny Milosavljevic]

> +    ;; XXX: 'guix-register' wants to palpate the things it registers, so
> +    ;; bind-mount the store on the target.
> +    (mkdir-p target-store)
> +    (mount (%store-directory) target-store "" MS_BIND)

What does this do?

gnu/build/vm.scm make-iso9660-image already provides /gnu/store a few lines down:

    (unless (zero? (apply system*
                          `(,grub-mkrescue "-o" ,target
                            ,(string-append "boot/grub/grub.cfg=" config-file)
                            ,(string-append "gnu/store=" os-drv "/..")
                                            ^^^^

Maybe I'm missing something...

Otherwise LGTM!

[bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

[Christopher Baines]

This mimics the functionality in the root-partition-initializer used in
creating the QEMU image. This helps when trying to run guix system init from
the generated ISO image.

* gnu/build/vm.scm (make-iso9660-image): Add support for registering closures.
---
 gnu/build/vm.scm | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 727494ad9..606257d8c 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -366,12 +366,27 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (error "failed to create GRUB EFI image"))))
 
 (define* (make-iso9660-image grub config-file os-drv target
-                             #:key (volume-id "GuixSD_image") (volume-uuid #f))
+                             #:key (volume-id "GuixSD_image") (volume-uuid #f)
+                             register-closures? (closures '()))
   "Given a GRUB package, creates an iso image as TARGET, using CONFIG-FILE as
 GRUB configuration and OS-DRV as the stuff in it."
-  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue")))
+  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue"))
+        (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+
+    (mkdir-p target-store)
+    (mount (%store-directory) target-store "" MS_BIND)
+
+    (when register-closures?
+      (display "registering closures...\n")
+      (for-each (lambda (closure)
+                  (register-closure
+                   "/tmp/root"
+                   (string-append "/xchg/" closure)
+                   #:deduplicate? #f))
+                closures))
+
     (unless (zero? (apply system*
                           `(,grub-mkrescue "-o" ,target
                             ,(string-append "boot/grub/grub.cfg=" config-file)
-- 
2.14.1

[bug#28288] [PATCH 2/5] vm: Create /mnt in the generated ISO image in make-iso9660-image.

[Christopher Baines]

This is used in the installation process, as the mountpoint for the target
filesystem.

* gnu/build/vm.scm (make-iso9660-image): Create /mnt within the generated ISO
  image.
---
 gnu/build/vm.scm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 606257d8c..f6228b40b 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -374,6 +374,7 @@ GRUB configuration and OS-DRV as the stuff in it."
         (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+    (mkdir-p "/tmp/root/mnt")
 
     (mkdir-p target-store)
     (mount (%store-directory) target-store "" MS_BIND)
@@ -393,6 +394,10 @@ GRUB configuration and OS-DRV as the stuff in it."
                             ,(string-append "gnu/store=" os-drv "/..")
                             "var=/tmp/root/var"
                             "run=/tmp/root/run"
+                            ;; /mnt is used as part of the installation
+                            ;; process, as the mount point for the target
+                            ;; filesystem, so create it.
+                            "mnt=/tmp/root/mnt"
                             "--"
                             ;; Store two copies of the headers.
                             ;; The resulting ISO-9660 image has a DOS MBR and
-- 
2.14.1

[bug#28288] [PATCH 2/5] vm: Create /mnt in the generated ISO image in make-iso9660-image.

[Danny Milosavljevic]

LGTM!

[bug#28288] [PATCH 3/5] vm: Add support for registering closures to iso9660-image.

[Christopher Baines]

* gnu/system/vm.scm (iso9660-image): Add support for registering closures.
---
 gnu/system/vm.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4494af003..fc55935aa 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -192,6 +192,7 @@ made available under the /xchg CIFS share."
                         os-drv
                         bootcfg-drv
                         bootloader
+                        register-closures?
                         (inputs '()))
   "Return a bootable, stand-alone iso9660 image.
 
@@ -207,8 +208,13 @@ INPUTS is a list of inputs (as for packages)."
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools xorriso)
                            (map canonical-package
-                                (list sed grep coreutils findutils gawk))))
+                                (list sed grep coreutils findutils gawk))
+                           (if register-closures? (list guix) '())))
 
+
+               (graphs     '#$(match inputs
+                                   (((names . _) ...)
+                                    names)))
                ;; This variable is unused but allows us to add INPUTS-TO-COPY
                ;; as inputs.
                (to-register
@@ -222,6 +228,8 @@ INPUTS is a list of inputs (as for packages)."
                                #$bootcfg-drv
                                #$os-drv
                                "/xchg/guixsd.iso"
+                               #:register-closures? #$register-closures?
+                               #:closures graphs
                                #:volume-id #$file-system-label
                                #:volume-uuid #$file-system-uuid)
            (reboot))))
-- 
2.14.1

[bug#28288] [PATCH 3/5] vm: Add support for registering closures to iso9660-image.

[Danny Milosavljevic]

LGTM!

[bug#28288] [PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

[Christopher Baines]

* gnu/system/vm.scm (system-disk-image): Call iso9660-image with
  #:register-closures? as #t.
---
 gnu/system/vm.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index fc55935aa..9cd973385 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -399,6 +399,7 @@ to USB sticks meant to be read-only."
                          #:file-system-label root-label
                          #:file-system-uuid #f
                          #:os-drv os-drv
+                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
                                         (operating-system-bootloader os))
-- 
2.14.1

[bug#28288] [PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

[Danny Milosavljevic]

LGTM!

[bug#28288] [PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> LGTM!

+1!

Excellent.

Ludo'.

[bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

[Christopher Baines]

* gnu/tests/install.scm (%test-iso-image-installer): New variable.
  (run-install): Add #:installation-disk-image-file-system-type as a keyword
  argument.
---
 gnu/tests/install.scm | 110 +++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 99 insertions(+), 11 deletions(-)

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 866bf885c..93c8a89cd 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -37,6 +37,7 @@
   #:use-module (guix utils)
   #:export (%test-installed-os
             %test-installed-extlinux-os
+            %test-iso-image-installer
             %test-separate-store-os
             %test-separate-home-os
             %test-raid-root-os
@@ -196,6 +197,7 @@ reboot\n")
                              (kernel-arguments '("console=ttyS0")))
                            #:imported-modules '((gnu services herd)
                                                 (guix combinators))))
+                      (installation-disk-image-file-system-type "ext4")
                       (target-size (* 1200 MiB)))
   "Run SCRIPT (a shell script following the GuixSD installation procedure) in
 OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
@@ -213,7 +215,9 @@ packages defined in installation-os."
                        (image  (system-disk-image
                                 (operating-system-with-gc-roots
                                  os (list target))
-                                #:disk-image-size (* 1500 MiB))))
+                                #:disk-image-size (* 1500 MiB)
+                                #:file-system-type
+                                installation-disk-image-file-system-type)))
     (define install
       (with-imported-modules '((guix build utils)
                                (gnu build marionette))
@@ -229,16 +233,25 @@ packages defined in installation-os."
 
             (define marionette
               (make-marionette
-               (cons (which #$(qemu-command system))
-                     (cons* "-no-reboot" "-m" "800"
-                            "-drive"
-                            (string-append "file=" #$image
-                                           ",if=virtio,readonly")
-                            "-drive"
-                            (string-append "file=" #$output ",if=virtio")
-                            (if (file-exists? "/dev/kvm")
-                                '("-enable-kvm")
-                                '())))))
+               `(,(which #$(qemu-command system))
+                 "-no-reboot"
+                 "-m" "800"
+                 #$@(cond
+                     ((string=? "ext4" installation-disk-image-file-system-type)
+                      `("-drive"
+                        ,(file-append "file=" image
+                                      ",if=virtio,readonly")))
+                     ((string=? "iso9660" installation-disk-image-file-system-type)
+                      `("-cdrom" ,image))
+                     (else
+                      (error
+                       "unsupported installation-disk-image-file-system-type:"
+                       installation-disk-image-file-system-type)))
+                 "-drive"
+                 ,(string-append "file=" #$output ",if=virtio")
+                 ,@(if (file-exists? "/dev/kvm")
+                       '("-enable-kvm")
+                       '()))))
 
             (pk 'uname (marionette-eval '(uname) marionette))
 
@@ -313,6 +326,81 @@ per %test-installed-os, this test is expensive in terms of CPU and storage.")
                       "installed-extlinux-os")))))
 
 \f
+;;;
+;;; Installation through an ISO image.
+;;;
+
+(define-os-with-source (%minimal-os-on-vda %minimal-os-on-vda-source)
+  ;; The OS we want to install.
+  (use-modules (gnu) (gnu tests) (srfi srfi-1))
+
+  (operating-system
+    (host-name "liberigilo")
+    (timezone "Europe/Paris")
+    (locale "en_US.UTF-8")
+
+    (bootloader (grub-configuration (target "/dev/vda")))
+    (kernel-arguments '("console=ttyS0"))
+    (file-systems (cons (file-system
+                          (device "my-root")
+                          (title 'label)
+                          (mount-point "/")
+                          (type "ext4"))
+                        %base-file-systems))
+    (users (cons (user-account
+                  (name "alice")
+                  (comment "Bob's sister")
+                  (group "users")
+                  (supplementary-groups '("wheel" "audio" "video"))
+                  (home-directory "/home/alice"))
+                 %base-user-accounts))
+    (services (cons (service marionette-service-type
+                             (marionette-configuration
+                              (imported-modules '((gnu services herd)
+                                                  (guix combinators)))))
+                    %base-services))))
+
+(define %simple-installation-script-for-/dev/vda
+  ;; Shell script of a simple installation.
+  "\
+. /etc/profile
+set -e -x
+guix --version
+
+export GUIX_BUILD_OPTIONS=--no-grafts
+guix build isc-dhcp
+parted --script /dev/vda mklabel gpt \\
+  mkpart primary ext2 1M 3M \\
+  mkpart primary ext2 3M 1G \\
+  set 1 boot on \\
+  set 1 bios_grub on
+mkfs.ext4 -L my-root /dev/vda2
+mount /dev/vda2 /mnt
+df -h /mnt
+herd start cow-store /mnt
+mkdir /mnt/etc
+cp /etc/target-config.scm /mnt/etc/config.scm
+guix system init /mnt/etc/config.scm /mnt --no-substitutes
+sync
+reboot\n")
+
+(define %test-iso-image-installer
+  (system-test
+   (name "iso-image-installer")
+   (description
+    "")
+   (value
+    (mlet* %store-monad ((image   (run-install
+                                   %minimal-os-on-vda
+                                   %minimal-os-on-vda-source
+                                   #:script
+                                   %simple-installation-script-for-/dev/vda
+                                   #:installation-disk-image-file-system-type
+                                   "iso9660"))
+                         (command (qemu-command/writable-image image)))
+      (run-basic-test %minimal-os-on-vda command name)))))
+
+\f
 ;;;
 ;;; Separate /home.
 ;;;
-- 
2.14.1

[bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On Sun,  3 Sep 2017 11:50:41 +0100
Christopher Baines <mail@cbaines.net> wrote:

> * gnu/tests/install.scm (%test-iso-image-installer): New variable.
>   (run-install): Add #:installation-disk-image-file-system-type as a
> keyword argument.
> ---
>  gnu/tests/install.scm | 110
> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 99
> insertions(+), 11 deletions(-)

This test now passes! I've also successfully generated an ISO image
that I used to install GuixSD to a Bytemark VM [1].

Most of the changes are educated guesses, mostly from copying the code
relating to QEMU images, so it would good to get some review of these.

1: https://www.bytemark.co.uk/cloud-hosting/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Sun,  3 Sep 2017 11:50:41 +0100
> Christopher Baines <mail@cbaines.net> wrote:
>
>> * gnu/tests/install.scm (%test-iso-image-installer): New variable.
>>   (run-install): Add #:installation-disk-image-file-system-type as a
>> keyword argument.
>> ---
>>  gnu/tests/install.scm | 110
>> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 99
>> insertions(+), 11 deletions(-)
>
> This test now passes! I've also successfully generated an ISO image
> that I used to install GuixSD to a Bytemark VM [1].

Woohoo, really nice!  \o/

Thanks for persevering!

Ludo’.

bug#28288: [PATCH 5/5] tests: Add test for installing from an ISO Image.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

On Tue, 05 Sep 2017 15:18:22 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Christopher Baines <mail@cbaines.net> skribis:
> 
> > On Sun,  3 Sep 2017 11:50:41 +0100
> > Christopher Baines <mail@cbaines.net> wrote:
> >  
> >> * gnu/tests/install.scm (%test-iso-image-installer): New variable.
> >>   (run-install): Add #:installation-disk-image-file-system-type as
> >> a keyword argument.
> >> ---
> >>  gnu/tests/install.scm | 110
> >> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed,
> >> 99 insertions(+), 11 deletions(-)  
> >
> > This test now passes! I've also successfully generated an ISO image
> > that I used to install GuixSD to a Bytemark VM [1].  
> 
> Woohoo, really nice!  \o/
> 
> Thanks for persevering!

Danny, Ludo, thanks for reviewing :) I've now pushed these changes.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> This mimics the functionality in the root-partition-initializer used in
> creating the QEMU image. This helps when trying to run guix system init from
> the generated ISO image.
>
> * gnu/build/vm.scm (make-iso9660-image): Add support for registering closures.

[...]

> +    (when register-closures?
> +      (display "registering closures...\n")
> +      (for-each (lambda (closure)
> +                  (register-closure
> +                   "/tmp/root"
> +                   (string-append "/xchg/" closure)
> +                   #:deduplicate? #f))
> +                closures))

Was there any specific reason for #:deduplicate? #f here?

Otherwise LGTM!

Ludo’.

[bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Tue, 05 Sep 2017 15:17:32 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Hi!
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> > This mimics the functionality in the root-partition-initializer
> > used in creating the QEMU image. This helps when trying to run guix
> > system init from the generated ISO image.
> >
> > * gnu/build/vm.scm (make-iso9660-image): Add support for
> > registering closures.  
> 
> [...]
> 
> > +    (when register-closures?
> > +      (display "registering closures...\n")
> > +      (for-each (lambda (closure)
> > +                  (register-closure
> > +                   "/tmp/root"
> > +                   (string-append "/xchg/" closure)
> > +                   #:deduplicate? #f))
> > +                closures))  
> 
> Was there any specific reason for #:deduplicate? #f here?

Yep, the image creation process fails if this is not specified.

error: cannot link
`/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'
to
`/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':
Invalid cross-device link

There is probably another way around this though. I'm not sure what
impact skipping the deduplication has on the size of the image.

> Otherwise LGTM!
> 
> Ludo’.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Tue, 05 Sep 2017 15:17:32 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hi!
>> 
>> Christopher Baines <mail@cbaines.net> skribis:
>> 
>> > This mimics the functionality in the root-partition-initializer
>> > used in creating the QEMU image. This helps when trying to run guix
>> > system init from the generated ISO image.
>> >
>> > * gnu/build/vm.scm (make-iso9660-image): Add support for
>> > registering closures.  
>> 
>> [...]
>> 
>> > +    (when register-closures?
>> > +      (display "registering closures...\n")
>> > +      (for-each (lambda (closure)
>> > +                  (register-closure
>> > +                   "/tmp/root"
>> > +                   (string-append "/xchg/" closure)
>> > +                   #:deduplicate? #f))
>> > +                closures))  
>> 
>> Was there any specific reason for #:deduplicate? #f here?
>
> Yep, the image creation process fails if this is not specified.
>
> error: cannot link
> `/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'
> to
> `/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':
> Invalid cross-device link

Hmm, OK.  Looks like it’s trying to deduplicate between /tmp/root and
some other store.  (Or are we using a bind mount here?)

> There is probably another way around this though. I'm not sure what
> impact skipping the deduplication has on the size of the image.

It probably doesn’t matter much, so it’s not a blocker IMO.

Maybe leave an XXX comment mentioning the cross-device link issue.

Thanks!

Ludo’.

[bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]

On Wed, 06 Sep 2017 15:20:44 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Christopher Baines <mail@cbaines.net> skribis:
> 
> > On Tue, 05 Sep 2017 15:17:32 +0200
> > ludo@gnu.org (Ludovic Courtès) wrote:
> >  
> >> Hi!
> >> 
> >> Christopher Baines <mail@cbaines.net> skribis:
> >>   
> >> > This mimics the functionality in the root-partition-initializer
> >> > used in creating the QEMU image. This helps when trying to run
> >> > guix system init from the generated ISO image.
> >> >
> >> > * gnu/build/vm.scm (make-iso9660-image): Add support for
> >> > registering closures.    
> >> 
> >> [...]
> >>   
> >> > +    (when register-closures?
> >> > +      (display "registering closures...\n")
> >> > +      (for-each (lambda (closure)
> >> > +                  (register-closure
> >> > +                   "/tmp/root"
> >> > +                   (string-append "/xchg/" closure)
> >> > +                   #:deduplicate? #f))
> >> > +                closures))    
> >> 
> >> Was there any specific reason for #:deduplicate? #f here?  
> >
> > Yep, the image creation process fails if this is not specified.
> >
> > error: cannot link
> > `/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'
> > to
> > `/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':
> > Invalid cross-device link  
> 
> Hmm, OK.  Looks like it’s trying to deduplicate between /tmp/root and
> some other store.  (Or are we using a bind mount here?)

I think there is a bind mount in use here, so maybe that has something
to do with it.

> > There is probably another way around this though. I'm not sure what
> > impact skipping the deduplication has on the size of the image.  
> 
> It probably doesn’t matter much, so it’s not a blocker IMO.
> 
> Maybe leave an XXX comment mentioning the cross-device link issue.

Yep, I'll put in a comment :)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]