GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements]

GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements]

[ng0]

Hi, a short update on the multiple providers I'm talking with at the moment.
This just in after the holidays from 1984 ehf, iceland.

All in all not problematic, and they'd even add GuixSD to the list of available isos
once there is an iso.

So, what's the status on the ISO, who's working on it?

----- Forwarded message from Siggi via RT <1984@rtracker.1984.is> -----

> Hi,
> 
> The IP is completely static and needs to be set, agetty is not a requirement to use the out of band access.  We run kvm - qemu, and you do not need non-free software for the guest nor the host to run it successfully.  If you have an iso I would happily add it to the list of isos available to use.  If or when you are ready I can also dd an image to your VPS setup with us, if you like.
> 
> Best
> 
> Siggi
> 
> 
> On Thu Apr 13 16:32:33 2017, contact.ng0@cryptolab.net wrote:
> > Hi,
> > 
> > at Guix/GuixSD (https://gnu.org/s/guix) we are currently looking into
> > all the special requirements of virtual server platforms, testing them
> > out to make the little tweaks that must be applied unnecessary later on,
> > improving GuixSD that way.
> > 
> > This month I won't have the money to spawn a server at 1984hosting, but I'm
> > interested in details in advance (and potentially I want to host a part
> > of the infrastructure for my collective (https://pragmatique.xyz) there).
> > 
> > The usual cases for digitalocean (and linode) looks like this:
> > https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00139.html
> > and for IN-Berlin (a non-profit in Germany) I'm still waiting for some
> > GRUB changes to be merged into our sources (to communicate with the
> > out-of-band console server (agetty redirection)).
> > Furthermore there are cases where you can simply be lucky and
> > "assimilate" the mimimal host system into GuixSD:
> > https://n0is.noblogs.org/post/2017/04/11/installing-guixsd-the-borg-way/
> > 
> > GuixSD is currently just distributed as an raw image, but we're working
> > on making it accessible as an .iso with dual efi+legacy bios support.
> > 
> > Potential issues I'm interested in are: is agetty an requirement for
> > the "out of band" console,
> > do you run an dhcpd or is the IP completely
> > static and details need to be specified (we have services and
> > configuration tools for both cases),
> > do the virtual servers require any special drivers for hardware (which
> > would typically be found in 'non-free' apt source of debian for example),
> > and anything you can point out which would help to clear up the
> > situation in advance.
> > 
> > 
> > Thanks!
> 
> 
> -- 
> - * - * - * - * - * -
> Ráðuneyti ástarinnar/Ministry of love
> 1984 ehf.
> www.1984.is
> 1984@1984.is
> Sími/phone: 546 1984
> 

----- End forwarded message -----

-- 
PGP and more: https://people.pragmatique.xyz/ng0/

Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements]

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 3940 bytes --]

ng0 <contact.ng0@cryptolab.net> writes:

> Hi, a short update on the multiple providers I'm talking with at the moment.
> This just in after the holidays from 1984 ehf, iceland.
>
> All in all not problematic, and they'd even add GuixSD to the list of available isos
> once there is an iso.
>
> So, what's the status on the ISO, who's working on it?

I've been looking into this off and on over the last few weeks, but I
haven't made any breakthroughs.  The closest I got was this:

1) Create a disk image for testing:

  ./pre-inst-env guix system --root=/tmp/disk-image-gc-root --fallback disk-image ~/guix/gnu/system/install.scm
  cp $the_resulting_path /tmp/disk-image
  
2) Try to boot it (with an attached hard disk), and watch it fail:

  qemu-img create -f qcow2 /tmp/test 10G
  sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/disk-image" "/tmp/test"

3) Mount it as loopback device:

  sudo losetup -P /dev/loop0 /tmp/disk-image
  sudo mkdir /mnt/disk-image-partition-1
  sudo mount /dev/loop0p1 /mnt/disk-image-partition-1
  
4) Make a bootable CD-ROM image of it (see (grub) Invoking grub-mkrescue):

  sudo grub-mkrescue -o /tmp/test-img.iso /mnt/disk-image-partition-1

5) Try to boot (partial success):

  sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/disk-image" "/tmp/test"

There appear to be (at least) two problem that prevent this naive
solution from working, which might point us in the right direction:

First, the GRUB menu is trying to find a file system with label
"gnu-disk-image" (via "search --label --set gnu-disk-image"), which
won't work because there is no file system with that label in the
resulting image.  Possible fix: the manual for grub-mkrescue says "The
root device will be set up appropriately on entering your 'grub.cfg'
configuration file", so perhaps we can simply omit our --search.  FYI,
the boot process continues successfully past this point precisely
because GRUB has already set the root; the fact that our search command
failed generates an error message but does not change the fact that it
succeeds in booting to the initrd.
  
Second, the init process from the initrd (I think that's what it's
called?) is trying to look for a file system with label
"gnu-disk-image", which it never finds.  It just sits there waiting to
find it, and it never shows up, so it freaks out.  Possible solution:
modify the behavior of our initrd's init process.  I'm not sure how to
customize the init process here, but there must be a way.  We'll
probably also need the kernel module that enables reading of iso9660
file systems, if it wasn't present already.

If you don't like grub-mkrescue, you can "roll your own" ISO generation
program, like Nix does by customizing the xorriso invocation [1]...  But
honestly, it looks pretty complicated [2].  So if we can let
grub-mkrescue do that work for us, that would be swell.

I've attached my raw notes that I took while trying to figure this out.
Hopefully somebody will find it useful.  Like most things that have to
do with computers, it's surprisingly complicated.  I'm very impressed by
what the NixOS folks were able to accomplish!

[1]
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/iso-image.nix
https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.nix
https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh

[2] The xorriso-fs manual page (info xorriso-fs) explains most of the
options that Nix uses pretty well.  The xorriso manual (info xorriso) is
a bit more dense.  To be honest, I had a hard time figuring out what we
would have to do while reading xorriso's manual; xorriso-fs's manual is
more approachable.

-- 
Chris

[-- Attachment #1.2: guix-iso.txt --]
[-- Type: text/plain, Size: 9585 bytes --]

Goal: generate a bootable CD image.  Ideally, augment disk-image to be hybrid!

Already know that Nix creates one like this:

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/iso-image.nix
https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.nix
https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh

Plan: try to do what Nix has done, but in Guix!

Phase 1: Figure out how to verify that it works.

Do this by getting an image of Nix and setting it up in qemu.  We
should be able to do exactly the same sort of thing to verify that
Guix is working...once we've implemented all of this!

Get Nix ISO: https://nixos.org/nixos/download.html

Start up qemu:
Create a qemu disk (to hold the installed system):

$ qemu-img create -f qcow2 /tmp/test 10G
Formatting 'test', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

This works for testing it:

$ sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name nixos -cdrom "/tmp/nixos-minimal-16.09.1836.067e66a-x86_64-linux.iso" "/tmp/test"

Confirm that this does not work with our current image:

$ qemu-img create -f qcow2 /tmp/test 10G
$ sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/disk-image" "/tmp/test"

OK, it fails.  Great!  Now we have a way to test for success!

Phase 2: Manually invoke xorriso to create the image.

Now that we know how to verify success, let's see if we can manually
create the image.  We'll invoke xorriso, following Nix's example.

We need the following stuff:

* A bootloader image.  What to use?  

* All the files that will reside in the OS's file system.  Clearly,
  this will include all the /gnu/store files, the /var/guix stuff, the
  bootloader configuration files, the system activation scripts - all
  of it.

* The right options for use with xorriso.  What to use? Nix provides
  an example; we'll use whatever we need to to get it working
  minimally.

First, let's start by trying to figure out how to invoke xorriso
correctly.  Conveniently, we already have all the files we need
sitting in the disk image.  It isn't an ISO, but it does contain
exactly the files we want, so let's worry first about how to invoke
xorriso after correctly generating the files we need.  We can worry
about the best way to correctly generate those files later.

To get at the files contained in the disk image, let's make it into a
loopback device:

$ sudo losetup -P /dev/loop0 /tmp/disk-image

$ sudo blkid /dev/loop0*
/dev/loop0: PTUUID="00831a15" PTTYPE="dos"
/dev/loop0p1: LABEL="root" UUID="fe1943fc-4cb8-44c6-a8a5-343a87bed681" TYPE="ext4" PARTUUID="00831a15-01"

$ ls /dev/loop0*
/dev/loop0  /dev/loop0p1

$ sudo mkdir /mnt/disk-image-partition-1

$ sudo mount /dev/loop0p1 /mnt/disk-image-partition-1

$ ls /mnt/disk-image-partition-1/
bin/  boot/  etc/  gnu/  home/  lost+found/  mnt/  root/  run/  tmp/  var/

Make a list of all the files:

$ sudo find /mnt/disk-image-partition-1/ > /tmp/all_files

$ head /tmp/all_files 
/mnt/disk-image-partition-1/
/mnt/disk-image-partition-1/tmp
/mnt/disk-image-partition-1/bin
/mnt/disk-image-partition-1/gnu
/mnt/disk-image-partition-1/gnu/store
/mnt/disk-image-partition-1/gnu/store/ykdzlcdyjjfhivids91d1xs36hmzrrp6-gmp-6.1.1
/mnt/disk-image-partition-1/gnu/store/ykdzlcdyjjfhivids91d1xs36hmzrrp6-gmp-6.1.1/include
/mnt/disk-image-partition-1/gnu/store/ykdzlcdyjjfhivids91d1xs36hmzrrp6-gmp-6.1.1/include/gmpxx.h
/mnt/disk-image-partition-1/gnu/store/ykdzlcdyjjfhivids91d1xs36hmzrrp6-gmp-6.1.1/include/gmp.h
/mnt/disk-image-partition-1/gnu/store/ykdzlcdyjjfhivids91d1xs36hmzrrp6-gmp-6.1.1/lib


The Xorriso manual seems to say I need to run this:

     El Torito only for GRUB: -boot_image "grub" "patch"

But it also says:

El Torito boot images have to be added to the ISO image by normal
     means (image loading, -map, -add, ...)

So I guess we have to add the files "first".

     *system_area=*disk_path copies at most 32768 bytes from the given
     disk file to the very start of the ISO image.  This System Area is
     reserved for system dependent boot software, e.g.  an MBR which can
     be used to boot from USB stick or hard disk.

OK, sure.  We'll try that.

     *grub2_mbr=*disk_path works like "any" system_area= with additional
     patching for modern GRUB MBRs.  The content start address of the
     first boot image is converted to a count of 512 byte blocks, and an
     offset of 4 is added.  The result is written as 64 bit
     little-endian number to byte address 0x1b0.
     This feature can be revoked either by grub2_mbr= with empty disk
     path, or by submitting a disk_path via system_area=.

What?  OK, sure, maybe we'll try that...  This is getting absurd.

Turns out, xororisofs manual is the one that contains the options that
Nix is using.

from xorrisofs manual (sec. "Bootable")
xorriso composes the boot catalog according to the boot image files
given and structured by options -b, -e, -el-torito-alt-boot, and
--efi-boot.

'xorrisofs' supports the example options out of the ISOLINUX wiki, the
options used in GRUB script grub-mkrescue, and the example in the
FreeBSD AvgLiveCD wiki.

OK, that's neat, but what are those options and what do they do?

need for sure:

-eltorito-boot
-eltorito-catalog
-no-emul-boot
-boot-load-size 4
-boot-info-table


Don't worry about EFI.  Don't worry about making it a hybrid ISO.
Just get El Torito boot working with PC-BIOS first.

Where do the boot image and catalog files come from?  GRUB package,
probably.

Hold the phone. According to GRUB's manual, it has a program that
makes bootable CD ROMs.  It's called grub-mkrescue.  Let's try
grub-mkrescue to see if we can save ourselves some pain.

$ sudo grub-mkrescue -o /tmp/test-img.iso /mnt/disk-image-partition-1GNU xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev 'stdio:/tmp/test-img.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 66.9g free
Added to ISO image: directory '/'='/tmp/grub.Qwzvrl'
xorriso : UPDATE : 318 files added in 1 seconds
xorriso : UPDATE : 15300 files added in 1 seconds
xorriso : UPDATE : 32000 files added in 2 seconds
xorriso : UPDATE : 37500 files added in 3 seconds
xorriso : UPDATE : 46400 files added in 4 seconds
xorriso : UPDATE : 48700 files added in 5 seconds
xorriso : UPDATE : 50300 files added in 6 seconds
xorriso : UPDATE : 51600 files added in 7 seconds
xorriso : UPDATE : 52600 files added in 8 seconds
xorriso : UPDATE : 53600 files added in 9 seconds
xorriso : UPDATE : 54500 files added in 10 seconds
xorriso : UPDATE : 55400 files added in 11 seconds
xorriso : UPDATE : 56300 files added in 13 seconds
xorriso : UPDATE : 57100 files added in 14 seconds
xorriso : UPDATE : 57900 files added in 15 seconds
xorriso : UPDATE : 58700 files added in 16 seconds
xorriso : UPDATE : 59400 files added in 17 seconds
xorriso : UPDATE : 60100 files added in 18 seconds
xorriso : UPDATE : 60800 files added in 19 seconds
xorriso : UPDATE : 61400 files added in 20 seconds
xorriso : UPDATE : 62000 files added in 21 seconds
xorriso : UPDATE : 62600 files added in 22 seconds
xorriso : UPDATE : 63200 files added in 23 seconds
xorriso : UPDATE : 63800 files added in 24 seconds
Added to ISO image: directory '/'='/mnt/disk-image-partition-1'
xorriso : UPDATE : 66811 files added in 26 seconds
xorriso : NOTE : Copying to System Area: 512 bytes from file '/gnu/store/2hxz9cpipsbf2hkiz5aq70k73wjj0fw1-grub-2.02rc1/lib/grub/i386-pc/boot_hybrid.img'
xorriso : UPDATE :  0.92% done
xorriso : UPDATE :  11.00% done
xorriso : UPDATE :  16.79% done, estimate finish Mon Apr 17 20:52:18 2017
xorriso : UPDATE :  23.60% done, estimate finish Mon Apr 17 20:52:19 2017
xorriso : UPDATE :  28.89% done, estimate finish Mon Apr 17 20:52:20 2017
xorriso : UPDATE :  34.79% done, estimate finish Mon Apr 17 20:52:20 2017
xorriso : UPDATE :  41.42% done, estimate finish Mon Apr 17 20:52:20 2017
xorriso : UPDATE :  49.49% done, estimate finish Mon Apr 17 20:52:20 2017
xorriso : UPDATE :  58.22% done, estimate finish Mon Apr 17 20:52:19 2017
xorriso : UPDATE :  69.48% done, estimate finish Mon Apr 17 20:52:19 2017
xorriso : UPDATE :  83.75% done, estimate finish Mon Apr 17 20:52:17 2017
xorriso : UPDATE :  97.00% done
ISO image produced: 361772 sectors
Written to medium : 361772 sectors at LBA 0
Writing to 'stdio:/tmp/test-img.iso' completed successfully.


let's try it out!

  sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/test-img.iso" "/tmp/test"

didn't work, can't find gnu-disk-image.  but GRUB got to the menu, and
then it tried to boot.

two problems:

* GRUB menu is trying to find a fs with label "gnu-disk-image", which
  won't work.  we could instead just tell it to use (cd,msdos1)
  directly through a custom entry, maybe.  or do a file search.
  
* init process is trying to look for a fs with label "gnu-disk-image",
  which it never finds.  We could instead just arrange for it to just
  load up the right partition.



Phase 3: Use Guix to build the ISO.

haven't gotten that far yet! :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements]

[Ludovic Courtès]

Heya,

Chris Marusich <cmmarusich@gmail.com> skribis:

> I've been looking into this off and on over the last few weeks, but I
> haven't made any breakthroughs.  The closest I got was this:
>
> 1) Create a disk image for testing:
>
>   ./pre-inst-env guix system --root=/tmp/disk-image-gc-root --fallback disk-image ~/guix/gnu/system/install.scm
>   cp $the_resulting_path /tmp/disk-image
>   
> 2) Try to boot it (with an attached hard disk), and watch it fail:
>
>   qemu-img create -f qcow2 /tmp/test 10G
>   sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/disk-image" "/tmp/test"
>
> 3) Mount it as loopback device:
>
>   sudo losetup -P /dev/loop0 /tmp/disk-image
>   sudo mkdir /mnt/disk-image-partition-1
>   sudo mount /dev/loop0p1 /mnt/disk-image-partition-1
>   
> 4) Make a bootable CD-ROM image of it (see (grub) Invoking grub-mkrescue):
>
>   sudo grub-mkrescue -o /tmp/test-img.iso /mnt/disk-image-partition-1
>
> 5) Try to boot (partial success):
>
>   sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom "/tmp/disk-image" "/tmp/test"
>
> There appear to be (at least) two problem that prevent this naive
> solution from working, which might point us in the right direction:
>
> First, the GRUB menu is trying to find a file system with label
> "gnu-disk-image" (via "search --label --set gnu-disk-image"), which
> won't work because there is no file system with that label in the
> resulting image.

So it seems that the crux of the problem is that ISO9660 lacks file
system labels, which breaks the whole thing, right?

> Possible fix: the manual for grub-mkrescue says "The
> root device will be set up appropriately on entering your 'grub.cfg'
> configuration file", so perhaps we can simply omit our --search.  FYI,
> the boot process continues successfully past this point precisely
> because GRUB has already set the root; the fact that our search command
> failed generates an error message but does not change the fact that it
> succeeds in booting to the initrd.

Good.

> Second, the init process from the initrd (I think that's what it's
> called?) is trying to look for a file system with label
> "gnu-disk-image", which it never finds.  It just sits there waiting to
> find it, and it never shows up, so it freaks out.  Possible solution:
> modify the behavior of our initrd's init process.  I'm not sure how to
> customize the init process here, but there must be a way.  We'll
> probably also need the kernel module that enables reading of iso9660
> file systems, if it wasn't present already.

So we could try detecting the root partition by a mechanism other than
partition labels/UUIDs, but I don’t know which mechanism.  Ideas?  How
do people address this?

> If you don't like grub-mkrescue, you can "roll your own" ISO generation
> program, like Nix does by customizing the xorriso invocation [1]...  But
> honestly, it looks pretty complicated [2].  So if we can let
> grub-mkrescue do that work for us, that would be swell.

Indeed, though of course grub-mkrescue invokes xorriso behind the
scenes.

Once we’ve figured out the partition designation issue above, I guess we
could integrate that and have, say,

  guix system iso-image …

to produce an ISO image.  Sounds doable!

Thanks for the explanations,
Ludo’.

Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements]

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3184 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

>> There appear to be (at least) two problem that prevent this naive
>> solution from working, which might point us in the right direction:
>>
>> First, the GRUB menu is trying to find a file system with label
>> "gnu-disk-image" (via "search --label --set gnu-disk-image"), which
>> won't work because there is no file system with that label in the
>> resulting image.
>
> So it seems that the crux of the problem is that ISO9660 lacks file
> system labels, which breaks the whole thing, right?

Yes, probably.  It seems we can set a "volume id", but not a label like
in EXT file systems.  For example, see here:

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/iso-image.nix

>> Second, the init process from the initrd (I think that's what it's
>> called?) is trying to look for a file system with label
>> "gnu-disk-image", which it never finds.  It just sits there waiting to
>> find it, and it never shows up, so it freaks out.  Possible solution:
>> modify the behavior of our initrd's init process.  I'm not sure how to
>> customize the init process here, but there must be a way.  We'll
>> probably also need the kernel module that enables reading of iso9660
>> file systems, if it wasn't present already.
>
> So we could try detecting the root partition by a mechanism other than
> partition labels/UUIDs, but I don’t know which mechanism.  Ideas?  How
> do people address this?

Volume ID seems like the right thing to try first.  What do we need to
change to support this, I wonder?  I'm not familiar with the initrd
stuff but would be willing to poke around in there and experiment.  I
just don't really know where to begin.

>> If you don't like grub-mkrescue, you can "roll your own" ISO generation
>> program, like Nix does by customizing the xorriso invocation [1]...  But
>> honestly, it looks pretty complicated [2].  So if we can let
>> grub-mkrescue do that work for us, that would be swell.
>
> Indeed, though of course grub-mkrescue invokes xorriso behind the
> scenes.
>
> Once we’ve figured out the partition designation issue above, I guess we
> could integrate that and have, say,
>
>   guix system iso-image …
>
> to produce an ISO image.  Sounds doable!

Yeah, that'd be cool.  It looks like with the right incantations (see
Nix's scripts), you can create a "hybrid" ISO image that boots correctly
in PC-BIOS, UEFI, from CD/DVD or USB stick.  But any ISO image that will
boot from a CD is a good start.

By the way, do you think it's possible to build an ISO image from a
manifest of files?  I ask because it seems like that's what Nix is
doing; however, the code involved in 'guix system disk-image' seems to
actually boot the target system before the final disk image is built.  I
was playing around by mounting the disk image as a loopback device only
because that provided me with a convenient way to access all the files
that need to be in a bootable system; it seems to me like it would be
great if we could build an ISO image without first building an entire
disk image.  That seems redundant.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

GuixSD bootable ISO-9669 image (was: Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements])

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4745 bytes --]

Chris Marusich <cmmarusich@gmail.com> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>>> There appear to be (at least) two problem that prevent this naive
>>> solution from working, which might point us in the right direction:
>>>
>>> First, the GRUB menu is trying to find a file system with label
>>> "gnu-disk-image" (via "search --label --set gnu-disk-image"), which
>>> won't work because there is no file system with that label in the
>>> resulting image.
>>
>> So it seems that the crux of the problem is that ISO9660 lacks file
>> system labels, which breaks the whole thing, right?
>
> Yes, probably.  It seems we can set a "volume id", but not a label like
> in EXT file systems.  For example, see here:
>
> https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/iso-image.nix

I discovered that grub-mkrescue also lets you pass through options to
xorriso.  So, you can set the volume ID as follows:

--8<---------------cut here---------------start------------->8---
$ sudo grub-mkrescue -V gnu-disk-image -v -o /tmp/test-img.iso /mnt/disk-image-partition-1
...
grub-mkrescue: info: enabling BIOS support ....
grub-mkrescue: info: grub-mkimage --directory '/gnu/store/2hxz9cpipsbf2hkiz5aq70k73wjj0fw1-grub-2.02rc1/lib/grub/i386-pc' --prefix '/boot/grub' --output '/tmp/grub.zilpbK/boot/grub/i386-pc/eltorito.img' --format 'i386-pc-eltorito' --compression 'auto'  --config '/tmp/grub.N4OuA5' 'biosdisk' 'iso9660' 
...
grub-mkrescue: info: executing xorriso -as mkisofs -graft-points --modification-date=2017042404413800 -b boot/grub/i386-pc/eltorito.img -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info --grub2-mbr /gnu/store/2hxz9cpipsbf2hkiz5aq70k73wjj0fw1-grub-2.02rc1/lib/grub/i386-pc/boot_hybrid.img --protective-msdos-label -o /tmp/test-img.iso -r /tmp/grub.zilpbK --sort-weight 0 / --sort-weight 1 /boot -V gnu-disk-image /mnt/disk-image-partition-1.
...
xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
...
Writing to 'stdio:/tmp/test-img.iso' completed successfully.
--8<---------------cut here---------------end--------------->8---

Looks like setting the volume ID worked (despite the warning above):

--8<---------------cut here---------------start------------->8---
$ sudo blkid /tmp/test-img.iso
/tmp/test-img.iso: UUID="2017-04-24-04-41-38-00" LABEL="gnu-disk-image" TYPE="iso9660" PTTYPE="dos"
--8<---------------cut here---------------end--------------->8---

>>> Second, the init process from the initrd (I think that's what it's
>>> called?) is trying to look for a file system with label
>>> "gnu-disk-image", which it never finds.  It just sits there waiting to
>>> find it, and it never shows up, so it freaks out.  Possible solution:
>>> modify the behavior of our initrd's init process.  I'm not sure how to
>>> customize the init process here, but there must be a way.  We'll
>>> probably also need the kernel module that enables reading of iso9660
>>> file systems, if it wasn't present already.
>>
>> So we could try detecting the root partition by a mechanism other than
>> partition labels/UUIDs, but I don’t know which mechanism.  Ideas?  How
>> do people address this?
>
> Volume ID seems like the right thing to try first.  What do we need to
> change to support this, I wonder?  I'm not familiar with the initrd
> stuff but would be willing to poke around in there and experiment.  I
> just don't really know where to begin.

Unfortunately, even after setting the volume ID, booting as follows does
not work.  The init process still doesn't find the partition with the
label "gnu-disk-image":

--8<---------------cut here---------------start------------->8---
$ sudo qemu-system-x86_64 -machine type=pc-i440fx-2.5,accel=kvm -boot order=dc,menu=on -m size=4G -k en-us -name guixsd -cdrom /tmp/test-img.iso /tmp/test
--8<---------------cut here---------------end--------------->8---

Judging by the contents of gnu/build/file-systems.scm, we don't have any
support for ISO-9669 file systems.  Perhaps if we add support here, the
machinery to find the partition labeled "gnu-disk-image" will work.
What do you think?

> By the way, do you think it's possible to build an ISO image from a
> manifest of files?

I'm still curious to hear your answer regarding this.  It isn't clear to
me whether we need to generate a disk image first, just to create the
ISO-9669 image.  It seems redundant.  I'm just playing around with the
disk image because it's an easy way to get at all the files.  If we can
put all the files in a single place without creating a disk image, then
we can probably invoke xorriso or grub-mkrescue directly on them.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Hello!

Chris Marusich <cmmarusich@gmail.com> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>>> There appear to be (at least) two problem that prevent this naive
>>>> solution from working, which might point us in the right direction:
>>>>
>>>> First, the GRUB menu is trying to find a file system with label
>>>> "gnu-disk-image" (via "search --label --set gnu-disk-image"), which
>>>> won't work because there is no file system with that label in the
>>>> resulting image.
>>>
>>> So it seems that the crux of the problem is that ISO9660 lacks file
>>> system labels, which breaks the whole thing, right?
>>
>> Yes, probably.  It seems we can set a "volume id", but not a label like
>> in EXT file systems.  For example, see here:
>>
>> https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/iso-image.nix
>
> I discovered that grub-mkrescue also lets you pass through options to
> xorriso.  So, you can set the volume ID as follows:
>
> $ sudo grub-mkrescue -V gnu-disk-image -v -o /tmp/test-img.iso /mnt/disk-image-partition-1
> ...
> grub-mkrescue: info: enabling BIOS support ....
> grub-mkrescue: info: grub-mkimage --directory '/gnu/store/2hxz9cpipsbf2hkiz5aq70k73wjj0fw1-grub-2.02rc1/lib/grub/i386-pc' --prefix '/boot/grub' --output '/tmp/grub.zilpbK/boot/grub/i386-pc/eltorito.img' --format 'i386-pc-eltorito' --compression 'auto'  --config '/tmp/grub.N4OuA5' 'biosdisk' 'iso9660' 
> ...
> grub-mkrescue: info: executing xorriso -as mkisofs -graft-points --modification-date=2017042404413800 -b boot/grub/i386-pc/eltorito.img -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info --grub2-mbr /gnu/store/2hxz9cpipsbf2hkiz5aq70k73wjj0fw1-grub-2.02rc1/lib/grub/i386-pc/boot_hybrid.img --protective-msdos-label -o /tmp/test-img.iso -r /tmp/grub.zilpbK --sort-weight 0 / --sort-weight 1 /boot -V gnu-disk-image /mnt/disk-image-partition-1.
> ...
> xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
> ...
> Writing to 'stdio:/tmp/test-img.iso' completed successfully.
>
>
> Looks like setting the volume ID worked (despite the warning above):
>
> $ sudo blkid /tmp/test-img.iso
> /tmp/test-img.iso: UUID="2017-04-24-04-41-38-00" LABEL="gnu-disk-image" TYPE="iso9660" PTTYPE="dos"

Excellent.  So we would first need to add support for reading ISO9660
labels in (gnu build file-systems), which is what we use instead of
libblkid.  With that in place, we should be almost done.

Ludo’.

Re: GuixSD bootable ISO-9669 image (was: Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements])

[Danny Milosavljevic]

>... -V gnu-disk-image /mnt/disk-image-partition-1.
> xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules

That's because all characters in an ISO 9660 volume identification need to be one of "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_".

If you use more complicated labels you additionally need to support an extension like Joliet (which provides its own seperate directory tree in addition to the one of ISO 9660!).  It's easier to use uppercase letters only and not have a second tree.

See <http://wiki.osdev.org/ISO_9660>.

That page also describes how the disklabel looks:
- The first 32 KiB contain some non-ISO9660 stuff - it might be misdetected as something non-ISO9660 if you take it seriously.  So skip that.
- A sector has 2 KiB.
- The volume descriptors start at sector 16.
- There can be multiple volume descriptors.
- Each volume descriptor is one sector long.
- A volume descriptor can be either a boot record, a primary volume descriptor, a supplementary volume descriptor (for Joliet), a volume partition descriptor, or a volume descriptor set terminator.
- The list of volume descriptors have to be terminated by a volume descriptor set terminator.

- The primary volume descriptor contains the volume identifier at offset 40, length 32 Byte.

See also <http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf>.

Re: GuixSD bootable ISO-9669 image (was: Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements])

[Danny Milosavljevic]

Something like this (totally untested):

diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
index 0cb84b8aa..be512d59c 100644
--- a/gnu/build/file-systems.scm
+++ b/gnu/build/file-systems.scm
@@ -230,6 +230,55 @@ Trailing spaces are trimmed."
 
 ^L
 ;;;
+;;; ISO9660 file systems.
+;;;
+
+;; <http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf>.
+
+;(define-syntax %iso9660-endianness
+;  ;; Endianness of iso9660 file systems that we use.
+;  ;; Actually, iso9660 has redundant data (i.e. data for both endiannesses).
+;  (identifier-syntax (endianness little)))
+
+(define (iso9660-superblock? sblock)
+  "Return #t when SBLOCK is a iso9660 superblock."
+  (bytevector=? (sub-bytevector sblock 1 6)
+                ;; Note: "\x01" is the volume descriptor format version
+                (string->utf8 "CD001\x01")))
+
+(define (read-iso9660-primary-volume-descriptor device offset)
+  "Find and read the first primary volume descriptor, starting at OFFSET.
+   Return #f if not found."
+  (let* ((sblock (read-superblock device offset 2048 iso9660-superblock?))
+         (type-code (array-ref sblock 0))) ; FIXME if sblock
+    (match type-code
+      (255 #f) ; Volume Descriptor Set Terminator.
+      (1 sblock) ; Primary Volume Descriptor
+      (_ (read-iso9660-primary-volume-descriptor device (+ offset 2048))))))
+
+(define (read-iso9660-superblock device)
+  "Return the raw contents of DEVICE's iso9660 superblock as a bytevector, or
+#f if DEVICE does not contain a iso9660 file system."
+  ;; Start reading at sector 16.
+  (read-iso9660-primary-volume-descriptor device (* 2048 16)))
+
+(define (iso9660-superblock-uuid sblock)
+  "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
+  ;; Note: The field is the volume creation time.
+  ;; FIXME Use only certain parts (See grub).
+  ;; FIXME treat "all 0" as invalid.
+  (sub-bytevector sblock 813 17))
+
+;; FIXME make result human-readable (See grub).
+;(define (iso9660-uuid->string uuid)
+
+(define (iso9660-superblock-volume-name sblock)
+  "Return the volume name of SBLOCK as a string.  The volume name is an ASCII
+string.  Trailing spaces are trimmed."
+  (string-trim-right (latin1->string (sub-bytevector sblock 40 32) (lambda (c) #f)) #\space))
+
+^L
+;;;
 ;;; LUKS encrypted devices.
 ;;;
 
@@ -340,7 +389,9 @@ partition field reader that returned a value."
     (_ #f)))
 
 (define %partition-label-readers
-  (list (partition-field-reader read-ext2-superblock
+  (list (partition-field-reader read-iso9660-superblock
+                                iso9660-superblock-volume-name)
+        (partition-field-reader read-ext2-superblock
                                 ext2-superblock-volume-name)
         (partition-field-reader read-btrfs-superblock
                                 btrfs-superblock-volume-name)
@@ -348,7 +399,9 @@ partition field reader that returned a value."
                                 fat32-superblock-volume-name)))
 
 (define %partition-uuid-readers
-  (list (partition-field-reader read-ext2-superblock
+  (list (partition-field-reader read-iso9660-superblock
+                                iso9660-superblock-uuid)
+        (partition-field-reader read-ext2-superblock
                                 ext2-superblock-uuid)
         (partition-field-reader read-btrfs-superblock
                                 btrfs-superblock-uuid)

Re: GuixSD bootable ISO-9669 image (was: Re: GuixSD on servers [Fwd: [rtracker.1984.is #131647] A question about VServer system specific requirements])

[Danny Milosavljevic]

Small fix below

On Thu, 27 Apr 2017 22:00:09 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> Something like this (totally untested):
> 
> diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
> index 0cb84b8aa..be512d59c 100644
> --- a/gnu/build/file-systems.scm
> +++ b/gnu/build/file-systems.scm
> @@ -230,6 +230,55 @@ Trailing spaces are trimmed."
>  
>  ^L
>  ;;;
> +;;; ISO9660 file systems.
> +;;;
> +
> +;; <http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf>.
> +
> +;(define-syntax %iso9660-endianness
> +;  ;; Endianness of iso9660 file systems that we use.
> +;  ;; Actually, iso9660 has redundant data (i.e. data for both endiannesses).
> +;  (identifier-syntax (endianness little)))
> +
> +(define (iso9660-superblock? sblock)
> +  "Return #t when SBLOCK is a iso9660 superblock."
> +  (bytevector=? (sub-bytevector sblock 1 6)
> +                ;; Note: "\x01" is the volume descriptor format version
> +                (string->utf8 "CD001\x01")))
> +
> +(define (read-iso9660-primary-volume-descriptor device offset)
> +  "Find and read the first primary volume descriptor, starting at OFFSET.
> +   Return #f if not found."
> +  (let* ((sblock (read-superblock device offset 2048 iso9660-superblock?))
> +         (type-code (array-ref sblock 0))) ; FIXME if sblock

+         (type-code (if sblock (array-ref sblock 0) 255)))

> +    (match type-code
> +      (255 #f) ; Volume Descriptor Set Terminator.
> +      (1 sblock) ; Primary Volume Descriptor
> +      (_ (read-iso9660-primary-volume-descriptor device (+ offset 2048))))))
> +
> +(define (read-iso9660-superblock device)
> +  "Return the raw contents of DEVICE's iso9660 superblock as a bytevector, or
> +#f if DEVICE does not contain a iso9660 file system."
> +  ;; Start reading at sector 16.
> +  (read-iso9660-primary-volume-descriptor device (* 2048 16)))
> +
> +(define (iso9660-superblock-uuid sblock)
> +  "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
> +  ;; Note: The field is the volume creation time.
> +  ;; FIXME Use only certain parts (See grub).
> +  ;; FIXME treat "all 0" as invalid.
> +  (sub-bytevector sblock 813 17))
> +
> +;; FIXME make result human-readable (See grub).
> +;(define (iso9660-uuid->string uuid)
> +
> +(define (iso9660-superblock-volume-name sblock)
> +  "Return the volume name of SBLOCK as a string.  The volume name is an ASCII
> +string.  Trailing spaces are trimmed."
> +  (string-trim-right (latin1->string (sub-bytevector sblock 40 32) (lambda (c) #f)) #\space))
> +
> +^L
> +;;;
>  ;;; LUKS encrypted devices.
>  ;;;
>  
> @@ -340,7 +389,9 @@ partition field reader that returned a value."
>      (_ #f)))
>  
>  (define %partition-label-readers
> -  (list (partition-field-reader read-ext2-superblock
> +  (list (partition-field-reader read-iso9660-superblock
> +                                iso9660-superblock-volume-name)
> +        (partition-field-reader read-ext2-superblock
>                                  ext2-superblock-volume-name)
>          (partition-field-reader read-btrfs-superblock
>                                  btrfs-superblock-volume-name)
> @@ -348,7 +399,9 @@ partition field reader that returned a value."
>                                  fat32-superblock-volume-name)))
>  
>  (define %partition-uuid-readers
> -  (list (partition-field-reader read-ext2-superblock
> +  (list (partition-field-reader read-iso9660-superblock
> +                                iso9660-superblock-uuid)
> +        (partition-field-reader read-ext2-superblock
>                                  ext2-superblock-uuid)
>          (partition-field-reader read-btrfs-superblock
>                                  btrfs-superblock-uuid)

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

Hi!

I just tested your patch, Danny (last version attached): I took the
image at
<http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-8.7.1-amd64-netinst.iso>
and then did:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,m (gnu build file-systems)
scheme@(gnu build file-systems)> (define sb (read-iso9660-superblock "/tmp/debian-8.7.1-amd64-netinst.iso.part"))
scheme@(gnu build file-systems)> (iso9660-superblock-volume-name sb)
$2 = "Debian 8.7.1 amd64 1"
scheme@(gnu build file-systems)> (iso9660-superblock-uuid sb)
$3 = #vu8(50 48 49 55 48 49 49 54 49 49 48 49 48 49 48 48 0)
scheme@(gnu build file-systems)> (bytevector-length $3)
$4 = 17
scheme@(gnu build file-systems)> (uuid->string $3)
$5 = "32303137-3031-3136-3131-303130313030"
--8<---------------cut here---------------end--------------->8---

Seems to work!

Could you clarify the two “See grub” in here:

--8<---------------cut here---------------start------------->8---
(define (iso9660-superblock-uuid sblock)
  "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
  ;; Note: The field is the volume creation time.
  ;; FIXME Use only certain parts (See grub).
  ;; FIXME treat "all 0" as invalid.
  (sub-bytevector sblock 813 17))

;; FIXME make result human-readable (See grub).
;(define (iso9660-uuid->string uuid)
--8<---------------cut here---------------end--------------->8---

Should ‘iso9660-uuid->string’ be different from ‘uuid->string’?

Anyway, I think we should polish and commit real soon.  :-)  Perhaps we
can add a note about endianness and assume little endian for now.

Thank you Danny!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3381 bytes --]

diff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scm
index 0cb84b8aa..4c826830d 100644
--- a/gnu/build/file-systems.scm
+++ b/gnu/build/file-systems.scm
@@ -230,6 +230,55 @@ Trailing spaces are trimmed."
 
 \f
 ;;;
+;;; ISO9660 file systems.
+;;;
+
+;; <http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf>.
+
+;(define-syntax %iso9660-endianness
+;  ;; Endianness of iso9660 file systems that we use.
+;  ;; Actually, iso9660 has redundant data (i.e. data for both endiannesses).
+;  (identifier-syntax (endianness little)))
+
+(define (iso9660-superblock? sblock)
+  "Return #t when SBLOCK is a iso9660 superblock."
+  (bytevector=? (sub-bytevector sblock 1 6)
+                ;; Note: "\x01" is the volume descriptor format version
+                (string->utf8 "CD001\x01")))
+
+(define (read-iso9660-primary-volume-descriptor device offset)
+  "Find and read the first primary volume descriptor, starting at OFFSET.
+   Return #f if not found."
+  (let* ((sblock    (read-superblock device offset 2048 iso9660-superblock?))
+         (type-code (if sblock (array-ref sblock 0) 255)))
+    (match type-code
+      (255 #f) ; Volume Descriptor Set Terminator.
+      (1 sblock) ; Primary Volume Descriptor
+      (_ (read-iso9660-primary-volume-descriptor device (+ offset 2048))))))
+
+(define (read-iso9660-superblock device)
+  "Return the raw contents of DEVICE's iso9660 superblock as a bytevector, or
+#f if DEVICE does not contain a iso9660 file system."
+  ;; Start reading at sector 16.
+  (read-iso9660-primary-volume-descriptor device (* 2048 16)))
+
+(define (iso9660-superblock-uuid sblock)
+  "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
+  ;; Note: The field is the volume creation time.
+  ;; FIXME Use only certain parts (See grub).
+  ;; FIXME treat "all 0" as invalid.
+  (sub-bytevector sblock 813 17))
+
+;; FIXME make result human-readable (See grub).
+;(define (iso9660-uuid->string uuid)
+
+(define (iso9660-superblock-volume-name sblock)
+  "Return the volume name of SBLOCK as a string.  The volume name is an ASCII
+string.  Trailing spaces are trimmed."
+  (string-trim-right (latin1->string (sub-bytevector sblock 40 32) (lambda (c) #f)) #\space))
+
+\f
+;;;
 ;;; LUKS encrypted devices.
 ;;;
 
@@ -340,7 +389,9 @@ partition field reader that returned a value."
     (_ #f)))
 
 (define %partition-label-readers
-  (list (partition-field-reader read-ext2-superblock
+  (list (partition-field-reader read-iso9660-superblock
+                                iso9660-superblock-volume-name)
+        (partition-field-reader read-ext2-superblock
                                 ext2-superblock-volume-name)
         (partition-field-reader read-btrfs-superblock
                                 btrfs-superblock-volume-name)
@@ -348,7 +399,9 @@ partition field reader that returned a value."
                                 fat32-superblock-volume-name)))
 
 (define %partition-uuid-readers
-  (list (partition-field-reader read-ext2-superblock
+  (list (partition-field-reader read-iso9660-superblock
+                                iso9660-superblock-uuid)
+        (partition-field-reader read-ext2-superblock
                                 ext2-superblock-uuid)
         (partition-field-reader read-btrfs-superblock
                                 btrfs-superblock-uuid)

Re: GuixSD bootable ISO-9669 image

[ng0]

Ludovic Courtès transcribed 5.2K bytes:
> Hi!
> 
> I just tested your patch, Danny (last version attached): I took the
> image at
> <http://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-8.7.1-amd64-netinst.iso>
> and then did:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> ,m (gnu build file-systems)
> scheme@(gnu build file-systems)> (define sb (read-iso9660-superblock "/tmp/debian-8.7.1-amd64-netinst.iso.part"))
> scheme@(gnu build file-systems)> (iso9660-superblock-volume-name sb)
> $2 = "Debian 8.7.1 amd64 1"
> scheme@(gnu build file-systems)> (iso9660-superblock-uuid sb)
> $3 = #vu8(50 48 49 55 48 49 49 54 49 49 48 49 48 49 48 48 0)
> scheme@(gnu build file-systems)> (bytevector-length $3)
> $4 = 17
> scheme@(gnu build file-systems)> (uuid->string $3)
> $5 = "32303137-3031-3136-3131-303130313030"
> --8<---------------cut here---------------end--------------->8---
> 
> Seems to work!
> 
> Could you clarify the two “See grub” in here:
> 
> --8<---------------cut here---------------start------------->8---
> (define (iso9660-superblock-uuid sblock)
>   "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
>   ;; Note: The field is the volume creation time.
>   ;; FIXME Use only certain parts (See grub).
>   ;; FIXME treat "all 0" as invalid.
>   (sub-bytevector sblock 813 17))
> 
> ;; FIXME make result human-readable (See grub).
> ;(define (iso9660-uuid->string uuid)
> --8<---------------cut here---------------end--------------->8---
> 
> Should ‘iso9660-uuid->string’ be different from ‘uuid->string’?
> 
> Anyway, I think we should polish and commit real soon.  :-)  Perhaps we
> can add a note about endianness and assume little endian for now.

Excuse me if I didn't pay much attention to the thread, I just flipped
through all of it. Isn't that (little endian) the most common assumption
most operation system have? I know that Gentoo defaults to little endian.

https://packages.gentoo.org/useflags/search?q=big-endian

And we don't support SuperH or similar Big Endian hardware. Or am I
thinking about different reasons why the endianness here matters?

> Thank you Danny!
> 
> Ludo’.
> 



-- 
https://pragmatique.xyz
PGP: https://people.pragmatique.xyz/ng0/

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

Hi Ludo,

On Tue, 02 May 2017 14:37:04 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Seems to work!

Nice :)

> Could you clarify the two “See grub” in here:
> 
> --8<---------------cut here---------------start------------->8---
> (define (iso9660-superblock-uuid sblock)
>   "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
>   ;; Note: The field is the volume creation time.
>   ;; FIXME Use only certain parts (See grub).
>   ;; FIXME treat "all 0" as invalid.
>   (sub-bytevector sblock 813 17))

Yeah.  If you check out grub, /dev/disk/by-uuid or blkid, you can see that for ISO9660 filesystems, it prints the "uuids" differently (because they are not uuids, they are timestamps...).

> Should ‘iso9660-uuid->string’ be different from ‘uuid->string’?

From a purity standpoint I'd say no.  From a compatibility standpoint, yes.

> Anyway, I think we should polish and commit real soon.  :-)  Perhaps we
> can add a note about endianness and assume little endian for now.

Turns out that for the parts that we need we don't care about the endianness either way.  So I dropped the endianness now.

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

> (define (iso9660-superblock-uuid sblock)
>   "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
>   ;; Note: The field is the volume creation time.
>   ;; FIXME Use only certain parts (See grub).

Grub drops the timezone.  So do blkid and /dev/disk/by-uuid .

>   ;; FIXME treat "all 0" as invalid.

The creation timestamp is actually optional in ISO-9660 (in case it's not given it's supposed to be 0) so it could be that the uuids on the system are not all that unique.  I think this case should be caught in the generic uuid matcher too.

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Heya,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Tue, 02 May 2017 14:37:04 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Seems to work!
>
> Nice :)
>
>> Could you clarify the two “See grub” in here:
>> 
>> --8<---------------cut here---------------start------------->8---
>> (define (iso9660-superblock-uuid sblock)
>>   "Return the Volume ID of a iso9660 superblock SBLOCK as a 4-byte bytevector."
>>   ;; Note: The field is the volume creation time.
>>   ;; FIXME Use only certain parts (See grub).
>>   ;; FIXME treat "all 0" as invalid.
>>   (sub-bytevector sblock 813 17))
>
> Yeah.  If you check out grub, /dev/disk/by-uuid or blkid, you can see that for ISO9660 filesystems, it prints the "uuids" differently (because they are not uuids, they are timestamps...).

OK.  Are these “UUIDs” 160-bit long like the “real” ones?  If they are,
I’d suggest ignoring the problem for now.

>> Should ‘iso9660-uuid->string’ be different from ‘uuid->string’?
>
> From a purity standpoint I'd say no.  From a compatibility standpoint, yes.

Makes sense.

>> Anyway, I think we should polish and commit real soon.  :-)  Perhaps we
>> can add a note about endianness and assume little endian for now.
>
> Turns out that for the parts that we need we don't care about the endianness either way.  So I dropped the endianness now.

Cool.

Thanks,
Ludo’.

Re: GuixSD bootable ISO-9669 image

[Mark H Weaver]

ng0 <contact.ng0@cryptolab.net> writes:

> Ludovic Courtès transcribed 5.2K bytes:
>> Anyway, I think we should polish and commit real soon.  :-)  Perhaps we
>> can add a note about endianness and assume little endian for now.
>
> Excuse me if I didn't pay much attention to the thread, I just flipped
> through all of it. Isn't that (little endian) the most common assumption
> most operation system have? I know that Gentoo defaults to little endian.
>
> https://packages.gentoo.org/useflags/search?q=big-endian
>
> And we don't support SuperH or similar Big Endian hardware. Or am I
> thinking about different reasons why the endianness here matters?

We should keep the door open to adding support for big-endian systems in
the future.  That job will be made more difficult if we incorporate code
that assumes little-endian.  Let's please strive for portability.

      Mark

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

Hi Ludo,

On Tue, 02 May 2017 23:11:05 +0200
ludo@gnu.org (Ludovic Courtès) wrote:
> OK.  Are these “UUIDs” 160-bit long like the “real” ones?  If they are,
> I’d suggest ignoring the problem for now.

No, unfortunately not.  There's also a similar problem in the FAT support in the same module.

I think Linux doesn't actually mean real DCE uuids when they say "uuid", so I don't think pretending that they are real uuids is going to be useful (because at some point control passes to /dev/disk/by-uuid, parted or grub - which interpret uuids like they want - which isn't necessarily as real uuids).

So I think the best course of action is to drop the real DCE uuids entirely and make uuids be free-form strings - like these other programs already decided...

What do you think?

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> Hi Ludo,
>
> On Tue, 02 May 2017 23:11:05 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>> OK.  Are these “UUIDs” 160-bit long like the “real” ones?  If they are,
>> I’d suggest ignoring the problem for now.
>
> No, unfortunately not.  There's also a similar problem in the FAT support in the same module.
>
> I think Linux doesn't actually mean real DCE uuids when they say "uuid", so I don't think pretending that they are real uuids is going to be useful (because at some point control passes to /dev/disk/by-uuid, parted or grub - which interpret uuids like they want - which isn't necessarily as real uuids).
>
> So I think the best course of action is to drop the real DCE uuids entirely and make uuids be free-form strings - like these other programs already decided...
>
> What do you think?

I’d prefer to add a special ‘iso-9660-uuid’ form similar to ‘uuid’ in
(gnu system file-systems).

That way we could detect that we get a valid UUID at macro-expansion
time or system-instantiation time, rather than end up with an error at
boot time.

WDYT?

Ludo’.

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

Hi Ludo,

>I’d prefer to add a special ‘iso-9660-uuid’ form similar to ‘uuid’ in (gnu system file-systems).
> That way we could detect that we get a valid UUID at macro-expansion
> time or system-instantiation time, rather than end up with an error at
> boot time.

Hmm, I've never seen that macro before.  Should it be used in config.scm ?

Right now config.scm could have:

  (file-systems (cons* (file-system
                        (device "1234-5678-....")
                        (title 'uuid)
                        (mount-point "/")
                        (type "ext4")
                        (needed-for-boot? #t))

Which kind of UUID is it then?  Should string->uuid try to be clever about it?  Or a new function string->uuid-like ?

find-partition-by-uuid right now uses read-partition-uuid and then uses bytevector=?.

So all these forms should continue to be bytevectors.  Then it would work fine without lots of changes.

So something like this?

- Add "string->uuid-like" and use it in canonical-title (instead of using "string->uuid").
- string->uuid-like would have a case analysis by some weird markers, then parse the uuid using the right parser into a bytevector with length typical-for-this-filesystem-but-not-unique.
- Modify uuid->string to have a case analysis by the bytevector length, then print the uuid using the right printer into a string typical-for-this-filesystem.  This will fail to detect ISO9660 correctly because these are the same length as real uuids (Linux uses 16 bytes of the 17-byte iso9660-uuid)
- Leave the uuid macro as-is since it should already do the right thing then?
- Add a new macro fat32-uuid, and a new macro iso9660-uuid.

String lengths:
- FAT32 UUID-string: 3 Byte to 9 Byte (variable-length numerals.  Is that actually how Linux writes them?  I misplaced my USB stick... where is it? *mumbles*)
- ISO9660 UUID-string: 16 Byte (fixed-length numerals).  That's the same length as the ISO9660 bytevector since it's actually the same content
- Real UUID-string: 36 Byte

Bytevector lengths (right now):
- FAT32 UUID: 4 Byte
- ISO9660 UUID: 16 Byte (oops...)
- Real UUID: 16 Byte

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>>I’d prefer to add a special ‘iso-9660-uuid’ form similar to ‘uuid’ in (gnu system file-systems).
>> That way we could detect that we get a valid UUID at macro-expansion
>> time or system-instantiation time, rather than end up with an error at
>> boot time.
>
> Hmm, I've never seen that macro before.  Should it be used in config.scm ?

Yes, see <https://gnu.org/software/guix/manual/html_node/File-Systems.html>.

> Right now config.scm could have:
>
>   (file-systems (cons* (file-system
>                         (device "1234-5678-....")
                                  ^^
This is wrong, should be (uuid "1234-…").

> Which kind of UUID is it then?  Should string->uuid try to be clever about it?  Or a new function string->uuid-like ?

‘uuid’ and ‘string->uuid’ are for DCE UUIDs, see (gnu build
file-systems).  I think it should remain this way: ‘string->uuid’
shouldn’t try to guess what kind of UUID it’s given; doing that could
lead to hard-to-debug problems.

> find-partition-by-uuid right now uses read-partition-uuid and then uses bytevector=?.

Right, file system UUIDs are represented as bytevectors internally in
(gnu build file-systems).

> So all these forms should continue to be bytevectors.  Then it would work fine without lots of changes.
>
> So something like this?
>
> - Add "string->uuid-like" and use it in canonical-title (instead of using "string->uuid").
> - string->uuid-like would have a case analysis by some weird markers, then parse the uuid using the right parser into a bytevector with length typical-for-this-filesystem-but-not-unique.
> - Modify uuid->string to have a case analysis by the bytevector length, then print the uuid using the right printer into a string typical-for-this-filesystem.  This will fail to detect ISO9660 correctly because these are the same length as real uuids (Linux uses 16 bytes of the 17-byte iso9660-uuid)
> - Leave the uuid macro as-is since it should already do the right thing then?
> - Add a new macro fat32-uuid, and a new macro iso9660-uuid.

I don’t like the idea of a polymorphical ‘string->uuid-like’.  I would
prefer to have:

  fat32-uuid->string (already exists)
  string->fat32-uuid
  fat32-uuid macro

  iso9660-uuid->string
  string->iso9660-uuid
  iso9660-uuid macro

Then users should pick the right one, usually the macro, and get an
error when they pass an invalid UUID.

WDYT?

Thanks,
Ludo’.

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

Hi Ludo,

On Fri, 12 May 2017 17:33:21 +0200
ludo@gnu.org (Ludovic Courtès) wrote:
> Then users should pick the right one, usually the macro, and get an
> error when they pass an invalid UUID.

Note that the directory /dev/disk/by-uuid doesn't contain any marker about what kind of uuid it is. So it would make things complicated for the user.

I wouldn't expect anyone that isn't a programmer to know that fat32 uuids are different from btrfs uuids.

If we have different uuid macros, we should have *-uuid macros for *every* possible filesystem type, whether the uuids differ or not - at least then there's a pattern for the user to learn.  Not like 3 macros and 14 filesystem types so you'd have to have a mapping function in your head.

But really, I myself use uuids only for file-system entries to be bus independent so I can plug known disks on whatever slot I want and they'll be mounted at the same place regardless.  I guess another use case is for the Guix LiveCD to find itself - for the same reason.  Not sure what the complication of bytevectors, multiple macros etcetc buys us.  The user is probably copy&pasting the uuid from /dev/disk/by-uuid anyway.  It's not like you type in uuids (ever).

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Fri, 12 May 2017 17:33:21 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>> Then users should pick the right one, usually the macro, and get an
>> error when they pass an invalid UUID.
>
> Note that the directory /dev/disk/by-uuid doesn't contain any marker about what kind of uuid it is. So it would make things complicated for the user.
>
> I wouldn't expect anyone that isn't a programmer to know that fat32 uuids are different from btrfs uuids.
>
> If we have different uuid macros, we should have *-uuid macros for *every* possible filesystem type, whether the uuids differ or not - at least then there's a pattern for the user to learn.  Not like 3 macros and 14 filesystem types so you'd have to have a mapping function in your head.
>
> But really, I myself use uuids only for file-system entries to be bus independent so I can plug known disks on whatever slot I want and they'll be mounted at the same place regardless.  I guess another use case is for the Guix LiveCD to find itself - for the same reason.  Not sure what the complication of bytevectors, multiple macros etcetc buys us.  The user is probably copy&pasting the uuid from /dev/disk/by-uuid anyway.  It's not like you type in uuids (ever).

The user interface (the ‘uuid’ macro for instance) and the internal
representation (bytevectors) are two different things.

The internal representation is fine, we don’t want to represent
something that’s literally a “byte vector” as a Unicode string, and have
parsing errors that could happen anywhere.

As for the user interface, I get your point.  We certainly don’t want to
make the user’s life unnecessarily difficult.

Honestly, I was under the assumptions that DCE UUIDs was all that
mattered: it works for ext2 and btrfs, right?  I would think that people
would rarely add FAT32 or ISO9660 UUIDs to the OS config.  That’s why I
though that having a separate macro for these two “special UUID types”
is okay.

What the macro buys us is expansion-time or system-instantiation-time
checks.  When someone uses /dev/disk/by-uuid/something-obviously-wrong
on another GNU/Linux system, they don’t get an error until they try to
boot the thing.  Here we have an opportunity to do that check before the
system is instantiated.

WDYT?

Ludo’.

Re: GuixSD bootable ISO-9669 image

[Danny Milosavljevic]

On Tue, 16 May 2017 10:31:36 +0200
ludo@gnu.org (Ludovic Courtès) wrote:
> Honestly, I was under the assumptions that DCE UUIDs was all that
> mattered: it works for ext2 and btrfs, right?  I would think that people
> would rarely add FAT32 or ISO9660 UUIDs to the OS config.  

With UEFI I thought that everyone has at least one FAT32 UUID in the OS config?

What's more, just by checking /dev/disk/by-uuid the user wouldn't know what kind of filesystem it is.  But he had to specify the file system type in the config anyway - so it would be no worse than before.  Okay then.

What do you all think of the following?

(define %iso9660-uuid-rx
  ;;                   Y                m                d                H                M                S                ss
  (make-regexp "^([[:digit:]]{4})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})$"))

(define (string->iso9660-uuid str)
  "Parse STR as a ISO9660 UUID (which is really a timestamp - see /dev/disk/by-uuid).
Return its contents as a 16-byte bytevector.  Return #f if STR is not a valid
ISO9660 UUID representation."
  (and=> (regexp-exec %iso9660-uuid-rx str)
         (lambda (match)
           (letrec-syntax ((match-numeral
                            (syntax-rules ()
                              ((_ index)
                               (match:substring match index)))))
             (let ((year (match-numeral 1))
                   (month (match-numeral 2))
                   (day (match-numeral 3))
                   (hour (match-numeral 4))
                   (minute (match-numeral 5))
                   (second (match-numeral 6))
                   (hundredths (match-numeral 7)))
               (string->utf8 (string-append year
                                            month
                                            day
                                            hour
                                            minute
                                            second
                                            hundredths)))))))

And also:

(define string->btrfs-uuid string->uid)
(define string->ext2-uuid string->uid)
(define string->ext3-uuid string->uid)
...

Re: GuixSD bootable ISO-9669 image

[Ludovic Courtès]

Howdy,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Tue, 16 May 2017 10:31:36 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>> Honestly, I was under the assumptions that DCE UUIDs was all that
>> mattered: it works for ext2 and btrfs, right?  I would think that people
>> would rarely add FAT32 or ISO9660 UUIDs to the OS config.  
>
> With UEFI I thought that everyone has at least one FAT32 UUID in the OS config?

I actually use the /dev device name but you’re right, we should be using
UUIDs in this case.

> What's more, just by checking /dev/disk/by-uuid the user wouldn't know what kind of filesystem it is.  But he had to specify the file system type in the config anyway - so it would be no worse than before.  Okay then.
>
> What do you all think of the following?
>
> (define %iso9660-uuid-rx
>   ;;                   Y                m                d                H                M                S                ss
>   (make-regexp "^([[:digit:]]{4})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})-([[:digit:]]{2})$"))
>
> (define (string->iso9660-uuid str)
>   "Parse STR as a ISO9660 UUID (which is really a timestamp - see /dev/disk/by-uuid).
> Return its contents as a 16-byte bytevector.  Return #f if STR is not a valid
> ISO9660 UUID representation."
>   (and=> (regexp-exec %iso9660-uuid-rx str)
>          (lambda (match)
>            (letrec-syntax ((match-numeral
>                             (syntax-rules ()
>                               ((_ index)
>                                (match:substring match index)))))
>              (let ((year (match-numeral 1))
>                    (month (match-numeral 2))
>                    (day (match-numeral 3))
>                    (hour (match-numeral 4))
>                    (minute (match-numeral 5))
>                    (second (match-numeral 6))
>                    (hundredths (match-numeral 7)))
>                (string->utf8 (string-append year
>                                             month
>                                             day
>                                             hour
>                                             minute
>                                             second
>                                             hundredths)))))))

LGTM!  Or even:

--8<---------------cut here---------------start------------->8---
(letrec-syntax ((match-numerals
                 (syntax-rules ()
                   ((_ index (name rest ...) body)
                    (let ((name (match:substring match index)))
                      (match-numerals (+ 1 index) (rest ...) body)))
                   ((_ index () body)
                    body))))
  (match-numerals 0 (year month day hour)
                  (string-append year month day)))
--8<---------------cut here---------------end--------------->8---

:-)

> And also:
>
> (define string->btrfs-uuid string->uid)
> (define string->ext2-uuid string->uid)
> (define string->ext3-uuid string->uid)
> ...

Yes, sounds reasonable.

Thanks!

Ludo’.