[bug#27661] [PATCH] build: Make ISO-9660 image bootable from USB flash drive.

[bug#27661] [PATCH] build: Make ISO-9660 image bootable from USB flash drive.

[Danny Milosavljevic]

* gnu/build/vm.scm (make-iso9660-image): Make image bootable from USB flash
drive.
---
 gnu/build/vm.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 2d41c5756..b686ef298 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -367,6 +367,8 @@ Grub configuration and OS-DRV as the stuff in it."
                             "var=/tmp/root/var"
                             "run=/tmp/root/run"
                             "--"
+                            ;; Store two copies of the headers
+                            "-boot_image" "any" "partition_offset=16"
                             "-volid" ,(string-upcase volume-id)
                             ,@(if volume-uuid
                                   `("-volume_date" "uuid"

[bug#27661] [PATCH] build: Make ISO-9660 image bootable from USB flash drive.

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/vm.scm (make-iso9660-image): Make image bootable from USB flash
> drive.
> ---
>  gnu/build/vm.scm | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
> index 2d41c5756..b686ef298 100644
> --- a/gnu/build/vm.scm
> +++ b/gnu/build/vm.scm
> @@ -367,6 +367,8 @@ Grub configuration and OS-DRV as the stuff in it."
>                              "var=/tmp/root/var"
>                              "run=/tmp/root/run"
>                              "--"
> +                            ;; Store two copies of the headers
> +                            "-boot_image" "any" "partition_offset=16"

I think you’ll have to expound in the comment because it doesn’t sound
trivial.  :-)

Thanks,
Ludo’.

[bug#27661] [PATCH v2] build: Make ISO-9660 image bootable from USB flash drive.

[Danny Milosavljevic]

* gnu/build/vm.scm (make-iso9660-image): Make image bootable from USB flash
drive.
---
 gnu/build/vm.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 2d41c5756..d8c53ef37 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -367,6 +367,19 @@ Grub configuration and OS-DRV as the stuff in it."
                             "var=/tmp/root/var"
                             "run=/tmp/root/run"
                             "--"
+                            ;; Store two copies of the headers.
+                            ;; The resulting ISO-9660 image has a DOS MBR and
+                            ;; one protective partition (with type 0xCD).
+                            ;; Because GuixSD only uses actual partitions
+                            ;; rather than what /proc/partitions returns, work
+                            ;; around it by storing the primary volume
+                            ;; descriptor twice, once where it should be and
+                            ;; once in the partition.
+                            ;; Allegedly, otherwise, many other GNU tools
+                            ;; (automounters etc) would also be confused by
+                            ;; the extra partition so it makes sense to
+                            ;; store two copies in any case.
+                            "-boot_image" "any" "partition_offset=16"
                             "-volid" ,(string-upcase volume-id)
                             ,@(if volume-uuid
                                   `("-volume_date" "uuid"

[bug#27661] [PATCH v2] build: Make ISO-9660 image bootable from USB flash drive.

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/vm.scm (make-iso9660-image): Make image bootable from USB flash
> drive.
> ---
>  gnu/build/vm.scm | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
> index 2d41c5756..d8c53ef37 100644
> --- a/gnu/build/vm.scm
> +++ b/gnu/build/vm.scm
> @@ -367,6 +367,19 @@ Grub configuration and OS-DRV as the stuff in it."
>                              "var=/tmp/root/var"
>                              "run=/tmp/root/run"
>                              "--"
> +                            ;; Store two copies of the headers.
> +                            ;; The resulting ISO-9660 image has a DOS MBR and
> +                            ;; one protective partition (with type 0xCD).
> +                            ;; Because GuixSD only uses actual partitions
> +                            ;; rather than what /proc/partitions returns, work
> +                            ;; around it by storing the primary volume
> +                            ;; descriptor twice, once where it should be and
> +                            ;; once in the partition.
> +                            ;; Allegedly, otherwise, many other GNU tools
> +                            ;; (automounters etc) would also be confused by
> +                            ;; the extra partition so it makes sense to
> +                            ;; store two copies in any case.
> +                            "-boot_image" "any" "partition_offset=16"

Now I understand.  :-)

Perfect, thank you!

Ludo’.

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Thanks!

Pushed.

If we want we can (in the future) provide only the iso9660 image on the homepage (for x86_64).

It will work from CD and USB flash drive - that should cover all the options.

(Usually, ARM systems have a more involved boot process and will not work, neither from SD card nor from CD-ROM - and booting from USB flash drive is out of the question).

Of course it would be nice if people could test the iso9660 image a bit (especially on UEFI systems of which I have none).

To generate one, for example do this:

guix system disk-image -t iso9660 foo.scm # will print the name of the directory that contains "guixsd.iso" in the end.

where foo.scm contains:

(use-modules (gnu))
(use-package-modules linux bootloaders)
(use-service-modules base networking ssh)

(operating-system
  (host-name "beagle-bone-black")
  (timezone "Europe/Zurich")
  (locale "en_US.UTF-8")

  (bootloader (grub-configuration (device "/dev/vda")))
  (kernel linux-libre)

  (file-systems
   (cons
    (file-system
      (mount-point "/")
      (type "iso9660")
      (title 'label)
      (device "gnu-disk-image"))
    %base-file-systems))

  (users
   (cons*
    (user-account
     (name "test")
     (group "users")
     (supplementary-groups '("wheel" "video"))
     (home-directory "/home/test")
     (password "sa5JEXtYx/rm6")) ; Password is pass.
    %base-user-accounts))

  (services
   (cons*
    (dropbear-service)
    (dhcp-client-service)
    %base-services)))

[bug#27661] ISO-9660 image working and ready

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

Danny Milosavljevic transcribed 1.4K bytes:
> Thanks!
> 
> Pushed.
> 
> If we want we can (in the future) provide only the iso9660 image on the homepage (for x86_64).
> 
> It will work from CD and USB flash drive - that should cover all the options.

Woo! Thanks for your work on this :)

I'll test this tomorrow evening.
Exciting news.

> (Usually, ARM systems have a more involved boot process and will not work, neither from SD card nor from CD-ROM - and booting from USB flash drive is out of the question).
> 
> Of course it would be nice if people could test the iso9660 image a bit (especially on UEFI systems of which I have none).
> 
> To generate one, for example do this:
> 
> guix system disk-image -t iso9660 foo.scm # will print the name of the directory that contains "guixsd.iso" in the end.
> 
> where foo.scm contains:
> 
> (use-modules (gnu))
> (use-package-modules linux bootloaders)
> (use-service-modules base networking ssh)
> 
> (operating-system
>   (host-name "beagle-bone-black")
>   (timezone "Europe/Zurich")
>   (locale "en_US.UTF-8")
> 
>   (bootloader (grub-configuration (device "/dev/vda")))
>   (kernel linux-libre)
> 
>   (file-systems
>    (cons
>     (file-system
>       (mount-point "/")
>       (type "iso9660")
>       (title 'label)
>       (device "gnu-disk-image"))
>     %base-file-systems))
> 
>   (users
>    (cons*
>     (user-account
>      (name "test")
>      (group "users")
>      (supplementary-groups '("wheel" "video"))
>      (home-directory "/home/test")
>      (password "sa5JEXtYx/rm6")) ; Password is pass.
>     %base-user-accounts))
> 
>   (services
>    (cons*
>     (dropbear-service)
>     (dhcp-client-service)
>     %base-services)))
> 
> 
> 
> 

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ISO-9660 image working and ready

[Ludovic Courtès]

Hey hey!

(This is a followup to <https://bugs.gnu.org/27661>.)

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> If we want we can (in the future) provide only the iso9660 image on the homepage (for x86_64).
>
> It will work from CD and USB flash drive - that should cover all the options.

Are you saying that the same image could be either dd’d to a USB key or
burnt on an actual CD?

Are there any downsides to using ISO9660 as the file system for the
media, like limitations on file names or file name lengths, restrictions
on the type of files, etc.?  (That doesn’t seem to be the case, but I
vaguely remember ISO9660 as having annoying limitations.)

> (Usually, ARM systems have a more involved boot process and will not work, neither from SD card nor from CD-ROM - and booting from USB flash drive is out of the question).
>
> Of course it would be nice if people could test the iso9660 image a bit (especially on UEFI systems of which I have none).
>
> To generate one, for example do this:
>
> guix system disk-image -t iso9660 foo.scm # will print the name of the directory that contains "guixsd.iso" in the end.

I’ve just built:

  ./pre-inst-env guix system disk-image -t iso9660 gnu/system/install.scm

on v0.13.0-1321-gc96ed0091 (current master), booted it with QEMU,
worked fine with ‘lsblk’ showing only /dev/{fd0,sr0}.   Woohoo!  \o/

The image has this 2KiB /boot.catalog file; is that expected?

Otherwise the file names look alright as if Joliet extensions were used,
but maybe they are?  “guix gc --verify=contents” in the image is happy
(and surprisingly fast).

Anyway awesome work, and a great milestone.  Many users who had been
complaining about the lack of ISOs will thank you!  :-)

Ludo’.

Re: ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Ludo,

On Wed, 12 Jul 2017 23:20:26 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> > It will work from CD and USB flash drive - that should cover all the options.  
> 
> Are you saying that the same image could be either dd’d to a USB key or
> burnt on an actual CD?

Yes.

> Are there any downsides to using ISO9660 as the file system for the
> media, like limitations on file names or file name lengths, restrictions
> on the type of files, etc.?  (That doesn’t seem to be the case, but I
> vaguely remember ISO9660 as having annoying limitations.)

It uses the Rock Ridge extension.  That means basenames are limited to 255 characters at most, allowed are all characters except NUL and "/".

>   ./pre-inst-env guix system disk-image -t iso9660 gnu/system/install.scm
> 
> on v0.13.0-1321-gc96ed0091 (current master), booted it with QEMU,
> worked fine with ‘lsblk’ showing only /dev/{fd0,sr0}.   Woohoo!  \o/

Now try qemu ...  -hda thesamefile.iso :)

> The image has this 2KiB /boot.catalog file; is that expected?

Yeah, that's the El Torito specification for the first-stage bootloader.  It contains what system architectures are supported and what kind of weird boot emulation the BIOS is supposed to use (look like a floppy drive, look like a hard drive, just be yourself etc).

> Otherwise the file names look alright as if Joliet extensions were used,
> but maybe they are?

Rock Ridge :)

UNIX permissions work, too.

>  “guix gc --verify=contents” in the image is happy
> (and surprisingly fast).

> Anyway awesome work, and a great milestone.  Many users who had been
> complaining about the lack of ISOs will thank you!  :-)

Thanks for your help :)

Re: ISO-9660 image working and ready

[Ricardo Wurmus]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> On Wed, 12 Jul 2017 23:20:26 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> > It will work from CD and USB flash drive - that should cover all the options.  
>> 
>> Are you saying that the same image could be either dd’d to a USB key or
>> burnt on an actual CD?
>
> Yes.

Wow, this is great!  Thanks a lot!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: ISO-9660 image working and ready

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Wed, 12 Jul 2017 23:20:26 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> > It will work from CD and USB flash drive - that should cover all the options.  
>> 
>> Are you saying that the same image could be either dd’d to a USB key or
>> burnt on an actual CD?
>
> Yes.

Awesome!

Would you like to update the ‘release’ Makefile.am target as well as
“System Installation” in guix.texi to reflect that?

>> Are there any downsides to using ISO9660 as the file system for the
>> media, like limitations on file names or file name lengths, restrictions
>> on the type of files, etc.?  (That doesn’t seem to be the case, but I
>> vaguely remember ISO9660 as having annoying limitations.)
>
> It uses the Rock Ridge extension.  That means basenames are limited to 255 characters at most, allowed are all characters except NUL and "/".

Cool.

>>   ./pre-inst-env guix system disk-image -t iso9660 gnu/system/install.scm
>> 
>> on v0.13.0-1321-gc96ed0091 (current master), booted it with QEMU,
>> worked fine with ‘lsblk’ showing only /dev/{fd0,sr0}.   Woohoo!  \o/
>
> Now try qemu ...  -hda thesamefile.iso :)

Incredible.  :-)

>> The image has this 2KiB /boot.catalog file; is that expected?
>
> Yeah, that's the El Torito specification for the first-stage bootloader.  It contains what system architectures are supported and what kind of weird boot emulation the BIOS is supposed to use (look like a floppy drive, look like a hard drive, just be yourself etc).

OK.

>> Otherwise the file names look alright as if Joliet extensions were used,
>> but maybe they are?
>
> Rock Ridge :)

Oh right, Rock Ridge.  Thanks for explaining!

Ludo’.

Re: ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Ludo,

On Mon, 17 Jul 2017 15:41:26 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Would you like to update the ‘release’ Makefile.am target as well as
> “System Installation” in guix.texi to reflect that?

Sure, but if possible, let's make sure that the image is tested in all cases first.  I hope someone tests the image on a real EFI computer again.  (It works in qemu - but with the complexity of the EFI specification and all I'd rather someone burned it to DVD and tried to boot from it)

Then something like the following?

diff --git a/Makefile.am b/Makefile.am
index 4d1512f8c..1d4364bce 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -632,14 +632,15 @@ release: dist
          image=`$(top_builddir)/pre-inst-env                                           \
            guix system disk-image                                                      \
             --system=$$system                                                          \
+            --file-system-type=iso9660                                                  \
            gnu/system/install.scm` ;                                                   \
          if [ ! -f "$$image" ] ; then                                                  \
            echo "failed to produced GuixSD installation image for $$system" >&2 ;      \
            exit 1 ;                                                                    \
          fi ;                                                                          \
-         xz < "$$image" > "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp" ;       \
+         xz < "$$image"/guixsd.iso > "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp" ;    \
          mv "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp"                       \
-            "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz" ;                         \
+            "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.iso.xz" ;                     \
        done
        for system in $(GUIXSD_VM_SYSTEMS) ; do                                         \
          image=`$(top_builddir)/pre-inst-env                                           \

Re: ISO-9660 image working and ready

[Adam Van Ymeren]

On July 17, 2017 1:54:49 PM EDT, Danny Milosavljevic <dannym@scratchpost.org> wrote:
>
>Sure, but if possible, let's make sure that the image is tested in all
>cases first.  I hope someone tests the image on a real EFI computer
>again.  (It works in qemu - but with the complexity of the EFI
>specification and all I'd rather someone burned it to DVD and tried to
>boot from it)

I will try to test it on a real EFI system tonight.

Re: ISO-9660 image working and ready

[Adam Van Ymeren]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Ludo,
>
> On Mon, 17 Jul 2017 15:41:26 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Would you like to update the ‘release’ Makefile.am target as well as
>> “System Installation” in guix.texi to reflect that?
>
> Sure, but if possible, let's make sure that the image is tested in all
> cases first.  I hope someone tests the image on a real EFI computer
> again.  (It works in qemu - but with the complexity of the EFI
> specification and all I'd rather someone burned it to DVD and tried to
> boot from it)

No luck booting the prodced guixsd.iso burned to a DVD on my UEFI
system.  It does boot fine if I disable UEFI boot.

That being said, it's possible that my system just doesn't supporting
booting from DVDs in UEFI mode.  I'm going to try to rule that out by
finding some other iso reported to work on UEFI systems.

Re: ISO-9660 image working and ready

[Ludovic Courtès]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> Would you like to update the ‘release’ Makefile.am target as well as
>> “System Installation” in guix.texi to reflect that?

[...]

> Then something like the following?
>
> diff --git a/Makefile.am b/Makefile.am
> index 4d1512f8c..1d4364bce 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -632,14 +632,15 @@ release: dist
>           image=`$(top_builddir)/pre-inst-env                                           \
>             guix system disk-image                                                      \
>              --system=$$system                                                          \
> +            --file-system-type=iso9660                                                  \
>             gnu/system/install.scm` ;                                                   \
>           if [ ! -f "$$image" ] ; then                                                  \
>             echo "failed to produced GuixSD installation image for $$system" >&2 ;      \
>             exit 1 ;                                                                    \
>           fi ;                                                                          \
> -         xz < "$$image" > "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp" ;       \
> +         xz < "$$image"/guixsd.iso > "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp" ;    \
>           mv "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz.tmp"                       \
> -            "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.xz" ;                         \
> +            "$(releasedir)/$(GUIXSD_IMAGE_BASE).$$system.iso.xz" ;                     \
>         done
>         for system in $(GUIXSD_VM_SYSTEMS) ; do                                         \
>           image=`$(top_builddir)/pre-inst-env                                           \

Yes, but what about removing the “/guixsd.iso” suffix first, as
discussed in another thread?

We also need to pay attention to “System Installation” in guix.texi.

Ludo’.

[bug#27661] ISO-9660 image working and ready

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Thanks!
>
> Pushed.
>
> If we want we can (in the future) provide only the iso9660 image on the homepage (for x86_64).
>
> It will work from CD and USB flash drive - that should cover all the options.
>
> (Usually, ARM systems have a more involved boot process and will not work, neither from SD card nor from CD-ROM - and booting from USB flash drive is out of the question).
>
> Of course it would be nice if people could test the iso9660 image a bit (especially on UEFI systems of which I have none).

It does not work on UEFI. I tested in qemu like this:

qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin \
  -m 256M -enable-kvm -boot c -cdrom /gnu/store/...-disk-image/guixsd.iso

There doesn't seem to be an EFI system partition? How did you dodge my
nasty partition hack :)

Also, the derivation creates two files alongside the ISO, are those
actually required or can we make it it return the ISO directly?

Impressive work nonetheless! Many users have asked for ISO support for
use with e.g. hosting providers. And it's really cool to be able to
create a CDROM/USB, BIOS/UEFI* multi-hybrid at a whim!

* Soon™

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Marius,

On Thu, 13 Jul 2017 20:53:50 +0200
Marius Bakke <mbakke@fastmail.com> wrote:

> It does not work on UEFI. I tested in qemu like this:
> 
> qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin \
>   -m 256M -enable-kvm -boot c -cdrom /gnu/store/...-disk-image/guixsd.iso
> 
> There doesn't seem to be an EFI system partition? How did you dodge my
> nasty partition hack :)

It's not using initialize-harddisk but rather grub-mkrescue and xorriso in order to create the filesystem.

That's because ISO-9660 filesystems are read-only and have to be created *including* the entire content in one go.

> Also, the derivation creates two files alongside the ISO, are those
> actually required 

No.

>or can we make it it return the ISO directly?

I don't know how...

> Impressive work nonetheless! Many users have asked for ISO support for
> use with e.g. hosting providers. And it's really cool to be able to
> create a CDROM/USB, BIOS/UEFI* multi-hybrid at a whim!
> 
> * Soon™

Bug #27689 fixes it a bit if the bootloader used is grub-efi.

However, that image then doesn't work with normal (non-EFI) BIOS and neither does it work with "qemu -hda".

Unfortunately I can't debug the "qemu -hda" problem properly because it seems that when UEFI is in use, "grub linux16" doesn't work any more which means I can't increase the number of lines on screen anymore (also, there's a huge penguin in the way) which means I can't see the entire error message that the guest prints.

How can I see the entire Linux kernel crash message in qemu (UEFI)?

[bug#27661] ISO-9660 image working and ready

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> Also, the derivation creates two files alongside the ISO, are those
>> actually required 
>
> No.
>
>>or can we make it it return the ISO directly?
>
> I don't know how...

I also vote for returning the ISO directly, for consistency with what
‘guix system disk-image’ does for the other formats.  WDYT, Danny?  :-)

Ludo’.

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Ludo,

> I also vote for returning the ISO directly, for consistency with what
> ‘guix system disk-image’ does for the other formats.  WDYT, Danny?  :-)

Sure, I just don't know how to do that.  The other two files are "system" and "bootcfg" and contain some kind of list of gnu store paths and single-digit numbers.  Anyone can go ahead and remove those - and make it somehow pick the guixsd.iso ... *shrugs*

iso9660-image uses expression->derivation-in-linux-vm to run make-iso9660-image to create /xchg/guixsd.iso - not sure how the other two files came into being.

I could make iso9660-image delete the two other files after the fact, I guess.  But that wouldn't make system-disk-image use just one regular file as the output - without the directory.

load-in-linux-vm has a flag "make-disk-image?".  When not set (and in this case, it's really not set), it does

    (mkdir output)
    (copy-recursively "xchg" output)

.

On the other hand, when set, it creates a huge qemu disk image we don't need.

We could special-case "if there's a guixsd.iso in there, take it instead" there, but doesn't seem so nice, does it?

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

I tried something like the following.  But now it throws away the output entirely.  That's not an improvement :)

But I think this is overcomplicating things - and the easiest way with only aesthetic downsides would be to punt and do nothing...

Or we could replace make-disk-image? and make-output-directory? by two parameters "initializer" and "serializer" which would be procedures that could do whatever they wanted.  I think that would be overly generic, though :)

Also, what amount of rebuilding would any change cause?  The procedures modified are already used quite often...

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index d8c53ef37..2be060902 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -76,6 +76,7 @@
                            (qemu (qemu-command)) (memory-size 512)
                            linux initrd
                            make-disk-image?
+                           (make-output-directory? #f)
                            (disk-image-size (* 100 (expt 2 20)))
                            (disk-image-format "qcow2")
                            (references-graphs '()))
@@ -87,7 +88,9 @@ DISK-IMAGE-SIZE bytes resulting from the execution of BUILDER, which may
 access it via /dev/hda.
 
 REFERENCES-GRAPHS can specify a list of reference-graph files as produced by
-the #:references-graphs parameter of 'derivation'."
+the #:references-graphs parameter of 'derivation'.
+
+If CREATE_OUTPUT-DIRECTORY?, creates a directory OUTPUT."
   (when make-disk-image?
     (format #t "creating ~a image of ~,2f MiB...~%"
             disk-image-format (/ disk-image-size (expt 2 20)))
@@ -137,8 +140,10 @@ the #:references-graphs parameter of 'derivation'."
 
   ;; When MAKE-DISK-IMAGE? is true, the image is in OUTPUT already.
   (unless make-disk-image?
-    (mkdir output)
-    (copy-recursively "xchg" output)))
+    (if make-output-directory?
+        (begin
+          (mkdir output)
+          (copy-recursively "xchg" output)))))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 66a2448ce..69ebf2377 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -106,6 +106,7 @@
                                               (%guile-for-build))
 
                                              (make-disk-image? #f)
+                                             (make-output-directory? #f)
                                              (references-graphs #f)
                                              (memory-size 256)
                                              (disk-image-format "qcow2")
@@ -121,6 +122,9 @@ DISK-IMAGE-FORMAT (e.g., 'qcow2' or 'raw'), of DISK-IMAGE-SIZE bytes and
 return it.  When DISK-IMAGE-SIZE is 'guess, estimate the image size based
 based on the size of the closure of REFERENCES-GRAPHS.
 
+When MAKE-OUTPUT-DIRECTORY? is true, then create a directory OUTPUT
+and copy the contents of /xchg there.
+
 When REFERENCES-GRAPHS is true, it must be a list of file name/store path
 pairs, as for `derivation'.  The files containing the reference graphs are
 made available under the /xchg CIFS share."
@@ -164,6 +168,7 @@ made available under the /xchg CIFS share."
                                 #:linux linux #:initrd initrd
                                 #:memory-size #$memory-size
                                 #:make-disk-image? #$make-disk-image?
+                                #:make-output-directory? #$make-output-directory?
                                 #:disk-image-format #$disk-image-format
                                 #:disk-image-size size
                                 #:references-graphs graphs)))))

[bug#27661] ISO-9660 image working and ready

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> I also vote for returning the ISO directly, for consistency with what
>> ‘guix system disk-image’ does for the other formats.  WDYT, Danny?  :-)
>
> Sure, I just don't know how to do that.  The other two files are
> "system" and "bootcfg" and contain some kind of list of gnu store
> paths and single-digit numbers.

They are “references graphs” files, describing the closure of the given
store items.

I gave it a stab and ended up with the attached patch, which Works For
Me™ and is not unreasonably ugly to my eye.  :-)

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix system disk-image -t iso9660 gnu/system/install.scm 
/gnu/store/5pk71pmjp2yshkh0xdfgsnmaxfvdsms1-image.iso
ludo@ribbon ~/src/guix/+master$ file /gnu/store/5pk71pmjp2yshkh0xdfgsnmaxfvdsms1-image.iso
/gnu/store/5pk71pmjp2yshkh0xdfgsnmaxfvdsms1-image.iso: DOS/MBR boot sector; GRand Unified Bootloader, stage1 version 0x79, boot drive 0xbb, stage2 address 0x8e70, 1st sector stage2 0xb8db31c3, stage2 segment 0x201 ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'GUIXSD_IMAGE' (bootable)
--8<---------------cut here---------------end--------------->8---

WDYT?

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 5248 bytes --]

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 086f38ade..305df5933 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -76,11 +76,13 @@
                            (qemu (qemu-command)) (memory-size 512)
                            linux initrd
                            make-disk-image?
+                           single-file-output?
                            (disk-image-size (* 100 (expt 2 20)))
                            (disk-image-format "qcow2")
                            (references-graphs '()))
   "Run BUILDER, a Scheme file, into a VM running LINUX with INITRD, and copy
-the result to OUTPUT.
+the result to OUTPUT.  Unless SINGLE-FILE-OUTPUT? is true, the result is
+copied recursively to OUTPUT.
 
 When MAKE-DISK-IMAGE? is true, OUTPUT will contain a VM image of
 DISK-IMAGE-SIZE bytes resulting from the execution of BUILDER, which may
@@ -137,8 +139,14 @@ the #:references-graphs parameter of 'derivation'."
 
   ;; When MAKE-DISK-IMAGE? is true, the image is in OUTPUT already.
   (unless make-disk-image?
-    (mkdir output)
-    (copy-recursively "xchg" output)))
+    (if single-file-output?
+        (let ((graph? (lambda (name stat)
+                        (member (basename name) references-graphs))))
+          (copy-file (first (find-files "xchg" (negate graph?)))
+                     output))
+        (begin
+          (mkdir output)
+          (copy-recursively "xchg" output)))))
 
 \f
 ;;;
@@ -356,7 +364,7 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
 (define* (make-iso9660-image grub config-file os-drv target
                              #:key (volume-id "GuixSD_image") (volume-uuid #f))
   "Given a GRUB package, creates an iso image as TARGET, using CONFIG-FILE as
-Grub configuration and OS-DRV as the stuff in it."
+GRUB configuration and OS-DRV as the stuff in it."
   (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue")))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index ec3fb031a..45006b2ba 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -108,16 +108,18 @@
                                              (guile-for-build
                                               (%guile-for-build))
 
+                                             (single-file-output? #f)
                                              (make-disk-image? #f)
                                              (references-graphs #f)
                                              (memory-size 256)
                                              (disk-image-format "qcow2")
                                              (disk-image-size 'guess))
   "Evaluate EXP in a QEMU virtual machine running LINUX with INITRD (a
-derivation).  In the virtual machine, EXP has access to all its inputs from the
-store; it should put its output files in the `/xchg' directory, which is
-copied to the derivation's output when the VM terminates.  The virtual machine
-runs with MEMORY-SIZE MiB of memory.
+derivation).  In the virtual machine, EXP has access to all its inputs from
+the store; it should put its output files in the `/xchg' directory, which is
+copied to the derivation's output when the VM terminates, recursively, unless
+SINGLE-FILE-OUTPUT? is true.  The virtual machine runs with MEMORY-SIZE MiB of
+memory.
 
 When MAKE-DISK-IMAGE? is true, then create a QEMU disk image of type
 DISK-IMAGE-FORMAT (e.g., 'qcow2' or 'raw'), of DISK-IMAGE-SIZE bytes and
@@ -167,6 +169,7 @@ made available under the /xchg CIFS share."
                                 #:linux linux #:initrd initrd
                                 #:memory-size #$memory-size
                                 #:make-disk-image? #$make-disk-image?
+                                #:single-file-output? #$single-file-output?
                                 #:disk-image-format #$disk-image-format
                                 #:disk-image-size size
                                 #:references-graphs graphs)))))
@@ -222,6 +225,7 @@ INPUTS is a list of inputs (as for packages)."
            (reboot))))
    #:system system
    #:make-disk-image? #f
+   #:single-file-output? #t
    #:references-graphs inputs))
 
 (define* (qemu-image #:key
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 65dd92e8b..0fcb6a9b0 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -579,8 +579,12 @@ PATTERN, a string.  When PATTERN is #f, display all the system generations."
                                                 (* 70 (expt 2 20)))
                                             #:mappings mappings))
     ((disk-image)
-     (system-disk-image os #:disk-image-size image-size
-                           #:file-system-type file-system-type))))
+     (system-disk-image os
+                        #:name (match file-system-type
+                                 ("iso9660" "image.iso")
+                                 (_         "disk-image"))
+                        #:disk-image-size image-size
+                        #:file-system-type file-system-type))))
 
 (define (maybe-suggest-running-guix-pull)
   "Suggest running 'guix pull' if this has never been done before."

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Ludo,

On Tue, 18 Jul 2017 15:08:49 +0200 ludo@gnu.org (Ludovic Courtès) wrote:
>--- a/gnu/build/vm.scm
>+the result to OUTPUT.  Unless SINGLE-FILE-OUTPUT? is true, the result is
>+copied recursively to OUTPUT.

What happens when SINGLE-FILE-OUTPUT? is true? :)

I'd say something like,

If SINGLE-FILE-OUTPUT? is true, copy a single file from /xchg to OUTPUT.
Otherwise, copy the contents of /xchg to a new directory OUTPUT.

>+    (if single-file-output?
>+        (let ((graph? (lambda (name stat)
>+                        (member (basename name) references-graphs))))
>+          (copy-file (first (find-files "xchg" (negate graph?)))

Maybe raise error when there's more than one possibility?

>+derivation).  In the virtual machine, EXP has access to all its inputs from
>+the store; it should put its output files in the `/xchg' directory, which is
>+copied to the derivation's output when the VM terminates, recursively, unless
>+SINGLE-FILE-OUTPUT? is true. 

What if it's true?

>+++ b/guix/scripts/system.scm
[...]
>+     (system-disk-image os
>+                        #:name (match file-system-type
>+                                 ("iso9660" "image.iso")
>+                                 (_         "disk-image"))

If we called it "disk-image" in the iso9660 case as well we'd not have this special case.  Do we want to?

Overall LGTM!

[bug#27661] ISO-9660 image working and ready

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Tue, 18 Jul 2017 15:08:49 +0200 ludo@gnu.org (Ludovic Courtès) wrote:
>>--- a/gnu/build/vm.scm
>>+the result to OUTPUT.  Unless SINGLE-FILE-OUTPUT? is true, the result is
>>+copied recursively to OUTPUT.
>
> What happens when SINGLE-FILE-OUTPUT? is true? :)
>
> I'd say something like,
>
> If SINGLE-FILE-OUTPUT? is true, copy a single file from /xchg to OUTPUT.
> Otherwise, copy the contents of /xchg to a new directory OUTPUT.
>
>>+    (if single-file-output?
>>+        (let ((graph? (lambda (name stat)
>>+                        (member (basename name) references-graphs))))
>>+          (copy-file (first (find-files "xchg" (negate graph?)))
>
> Maybe raise error when there's more than one possibility?
>
>>+derivation).  In the virtual machine, EXP has access to all its inputs from
>>+the store; it should put its output files in the `/xchg' directory, which is
>>+copied to the derivation's output when the VM terminates, recursively, unless
>>+SINGLE-FILE-OUTPUT? is true. 
>
> What if it's true?

All good points, indeed.  Pushed with these changes as
8d033e3e1607d5722ef7288208551d0331c8a853.

>>+++ b/guix/scripts/system.scm
> [...]
>>+     (system-disk-image os
>>+                        #:name (match file-system-type
>>+                                 ("iso9660" "image.iso")
>>+                                 (_         "disk-image"))
>
> If we called it "disk-image" in the iso9660 case as well we'd not have this special case.  Do we want to?

I find it marginally nicer to have an explicit “.iso” suffix.  I pushed
this part as 5058bf56843baf3c0d82fbf0addbb30f00572428.

Thank you!

Ludo’.

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Debugging:

$ qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin -m 1G -enable-kvm -hda Zuefi -serial stdio

Then add "console=ttyS0" to Linux cmdline in grub menu edit.

I get:

[    0.923605] scsi 0:0:0:0: Direct-Access     ATA      QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
[    0.926558] sd 0:0:0:0: [sda] 2215580 512-byte logical blocks: (1.13 GB/1.06 GiB)
[    0.928683] sd 0:0:0:0: [sda] Write Protect is off
[    0.930155] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    0.931705] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.934772] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
[    0.949039] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[    0.952674] cdrom: Uniform CD-ROM driver Revision: 3.20
[    0.955449]  sda: sda1 sda2 sda3 sda4
[    0.956995] sd 0:0:0:0: [sda] Attached SCSI disk
[    0.959031] sr 1:0:0:0: Attached scsi generic sg1 type 5
GC Warning: Couldn't read /proc/stat
Welcome, this is GNU's early boot Guile.
Use '--repl' for an initrd REPL.
loading '/gnu/store/48d1w55iy2vpcm5ass5lsp8almghszv8-system/boot'...
[    1.235733] attempt to access beyond end of device
[    1.237066] sda1: rw=0, want=949216, limit=76376
[    1.238565] attempt to access beyond end of device
[    1.240115] sda1: rw=0, want=949216, limit=76376
[    1.242095] attempt to access beyond end of device
[    1.243545] sda1: rw=0, want=949216, limit=76376
ERROR: In procedure primitive-load:
ERROR: In procedure fport_fill_input: Input/output error

Partitions are:

Disk Zuefi: 1.1 GiB, 1134376960 bytes, 2215580 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DB26D0AA-65F4-4CE1-87F8-85CAD51B5C94

Device       Start     End Sectors  Size Type
Zuefi1          64   76439   76376 37.3M Microsoft basic data <----- here is a suspect
Zuefi2       76440   82199    5760  2.8M EFI System
Zuefi3       82200 2214931 2132732    1G Apple HFS/HFS+
Zuefi4     2214932 2215531     600  300K Microsoft basic data

Backtrace:

Impossible.  It did not actually enter the prompt.

[bug#27661] ISO-9660 image working and ready

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> Debugging:
>
> $ qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin -m 1G -enable-kvm -hda Zuefi -serial stdio
>
> Then add "console=ttyS0" to Linux cmdline in grub menu edit.
>
> I get:
>
> [    0.923605] scsi 0:0:0:0: Direct-Access     ATA      QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
> [    0.926558] sd 0:0:0:0: [sda] 2215580 512-byte logical blocks: (1.13 GB/1.06 GiB)
> [    0.928683] sd 0:0:0:0: [sda] Write Protect is off
> [    0.930155] sd 0:0:0:0: Attached scsi generic sg0 type 0
> [    0.931705] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [    0.934772] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
> [    0.949039] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
> [    0.952674] cdrom: Uniform CD-ROM driver Revision: 3.20
> [    0.955449]  sda: sda1 sda2 sda3 sda4
> [    0.956995] sd 0:0:0:0: [sda] Attached SCSI disk
> [    0.959031] sr 1:0:0:0: Attached scsi generic sg1 type 5
> GC Warning: Couldn't read /proc/stat
> Welcome, this is GNU's early boot Guile.
> Use '--repl' for an initrd REPL.
> loading '/gnu/store/48d1w55iy2vpcm5ass5lsp8almghszv8-system/boot'...
> [    1.235733] attempt to access beyond end of device
> [    1.237066] sda1: rw=0, want=949216, limit=76376
> [    1.238565] attempt to access beyond end of device
> [    1.240115] sda1: rw=0, want=949216, limit=76376
> [    1.242095] attempt to access beyond end of device
> [    1.243545] sda1: rw=0, want=949216, limit=76376
> ERROR: In procedure primitive-load:
> ERROR: In procedure fport_fill_input: Input/output error

It looks like the partition detection code in (gnu build file-systems)
tries to read beyond the end of disk or something.  Could you try adding
a few ‘pk’ or ‘format’ calls in there so see what’s going on?

Alternately, you could run the same QEMU command line that the
derivation spawns, so you would get a REPL (and backtrace).  For that
you can “ps aux | grep qemu” while the derivation is building, and
copy/paste the QEMU command from there.

HTH,
Ludo’.

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Hi Ludo,

On Mon, 17 Jul 2017 10:25:08 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> It looks like the partition detection code in (gnu build file-systems)
> tries to read beyond the end of disk or something.  Could you try adding
> a few ‘pk’ or ‘format’ calls in there so see what’s going on?

Yeah, it tries to read beyond the end of the partition.  That's because grub-mkrescue creates a tiny fake partition which happens to start where the ISO-9660 filesystem starts and GuixSD mounts it and tries to boot from it (because it has a valid volume id).  As it is going on booting, the system boot wanders off the end of the partition and Linux tells it that it can't do that.

I've posted bug# 27705 which is the minimal set of patches to get it to work.  I just read the root filesystem directly from the whole disk there.

By now, I've also tested the resulting image on the bare metal on a real DVD on a non-EFI, non-Libreboot system.  Works fine :)

Marius, could you also test it on EFI ?

As for fixing the original partitioning problem, I didn't manage to find the place yet.  As far as I can see, grub-mkrescue uses xorriso which uses mkisofs to actually generate the ISO image.  There's a "protective" DOS MBR generated somewhere, but where.  It seems that Xorriso_coordinate_system_area and make_isohybrid_mbr are at least related somehow.  Also, it's questionable whether it's possible to fix it without creating overlapped partitions, see below.

The resulting partition table is:

Disk J: 1.1 GiB, 1137074176 bytes, 2220848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F86B53DD-3E7B-4B5E-A4C9-3A1A0AA56D22

Device   Start     End Sectors  Size Type
J1          64   76871   76808 37.5M Microsoft basic data <--- GuixSD tried to use this one as root
J2       76872   82631    5760  2.8M EFI System <-- mounts and reads fine as FAT FS
J3       82632 2220199 2137568    1G Apple HFS/HFS+ <--- can't find valid HFS on there
J4     2220200 2220799     600  300K Microsoft basic data <-- can't mount it

P.S. grub-mkrescue (with EFI) has a surprising number of supported architectures.  It might be that a grub-mkrescue-generated image could also boot on MIPS, Sparc, PowerPC, ARM64.

> Alternately, you could run the same QEMU command line that the
> derivation spawns, so you would get a REPL (and backtrace).  For that
> you can “ps aux | grep qemu” while the derivation is building, and
> copy/paste the QEMU command from there.

Thanks!  Very useful.

[bug#27661] ISO-9660 image working and ready

[Danny Milosavljevic]

Hi,

Adam Van Ymeren wrote on Mon, 17 Jul 2017 22:34:34 -0400:
>No luck booting the prodced guixsd.iso burned to a DVD on my UEFI system.  It does boot fine if I disable UEFI boot.

>That being said, it's possible that my system just doesn't supporting booting from DVDs in UEFI mode.  I'm going to try to rule that out by finding some other iso reported to work on UEFI systems.

Hmm, was that with patches from bug# 27695 (grub-hybrid-bootloader) or with patches from bug# 27705 (grub-efi patched to include some grub-pc files) ?