Re: bug#33639: ISO installer image is broken on i686

Re: bug#33639: ISO installer image is broken on i686

[Gábor Boskovits]

Hello people,

Thomas Schmitt <scdbackup@gmx.net> ezt írta (időpont: 2019. ápr. 15., H, 19:54):
>
> Hi,
>
> Florian Pelz wrote:
> > Well this is strange.  I got fine ISO images each time (fine with no
> > complaints from xorriso or fdisk and bootable in QEMU without errors),
> > but after dd’ing them to different USB flash drives each time I get
> > kernel output when inserting the flash drive:
> > [   10.025223] GPT:Primary header thinks Alt. header is not at the end of
> > the disk.
>
> The alternative/backup header is a property of GPT which makes it
> rather unsuitable for disk images. xorriso puts it correctly into the
> last 512-byte block of the image. But when copied to a storage device,
> it should move up to the last block of the device.
> Even worse, the main GPT header at 512-byte LBA 1 needs to learn the
> new address.
>

Yes, this is a really painful point.

Could we create a simple tool to write the disk images to a disk
correcting this problem?

Does not look too hard?

I am also forwarding this to guix devel. I removed the xorriso bug
list, as I feel this does not belong there.

Best regards,
g_bor

ISO installer image: GPT versus MBR partitions (was bug#33639)

[Thomas Schmitt]

Hi,

Gábor Boskovits wrote:
> I removed the xorriso bug list, as I feel this does not belong there.

As its admin i say that everything xorriso related belongs there. :))

But maybe we should remove  33639@debbugs.gnu.org  from the Cc ?
I herby give this a try.


i wrote:
> > The alternative/backup header is a property of GPT which makes it
> > rather unsuitable for disk images.

Gábor Boskovits wrote:
> Could we create a simple tool to write the disk images to a disk
> correcting this problem?

There are GPT partition editors which offer to repair the backup block
problem. gdisk has in its expert's menu 'x':
  e       relocate backup data structures to the end of the disk

The job is indeed not very difficult. Backup header block and backup
table entries array need to be moved up. Then a few block addresses
in both header blocks need to be updated. Finally new CRCs need to
be computed and put into the headers.

More headache is in the wish to have a fool-proof tool for copying
the image file to USB stick.
Compared with the risks of burning a DVD, the risks of writing a disk
image are quite unbearable. Covering all pitfalls might be a larger
adventure. (Ubuntu has mkusb: https://help.ubuntu.com/community/mkusb)

A problem with GPT relocation is that checksums of the image cannot be
verified on USB stick any more.
An MBR partitioned image would not have to be altered. /sbin/isosize could
tell the number of bytes which should be processed when comparing with
published checksums.

Further an MBR partitioned image would have no non-mountable partitions.

  $ /sbin/gdisk guixsd-install-0.15.0.i686-linux.iso
  ...
  Command (? for help): p
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     1              64           34107   16.6 MiB    0700  Gap0
     2           34108           39867   2.8 MiB     EF00  EFI boot partition
     3           39868         1815727   867.1 MiB   AF00  HFSPLUS
     4         1815728         1816327   300.0 KiB   0700  Gap1

Gap0 and Gap1 are not mountable. One can get access to the ISO's files
only via the base device. The HFS+ partition publishes the same files,
but you need to find a HFS+ driver which works with block size 2048.
(Even more difficult could be to find any machine that boots via
 APM and HFS+ and can make use of the binaries in ISO and HFS+.)

Repeating my example of yesterday of a grub-mkrescue ISO with MBR
partition table (and no APM, which could be added if really of interest):

  $ /sbin/fdisk -l output.iso
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28103   28040 13.7M 83 Linux
  output.iso2      28104 33863    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  33864

Of course, ISO and partition 1 would be much larger in a Guix ISO.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions (was bug#33639)

[pelzflorian (Florian Pelz)]

I would prefer an image that can be written just with dd, because then
it would also be easy to use another tool like gnome-multi-writer or
to use a not UNIX-like OS to create the install USB and give
GNU Guix System a try.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I would prefer an image that can be written just with dd,

A broken backup GPT is not supposed to stop EFI from finding its partition
and running its architecture specific start-up program.

But it becomes annoying when another partition shall be created for
a read-write filesystem to carry the user's extra data.

(With Debian and Fedora ISOs it's even worse. They carry a MBR partition
 table and GPT header blocks, which are tehrfore invalid. One has to zap
 them before letting fdisk create a new MBR partition.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 16, 2019 at 07:57:20PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I would prefer an image that can be written just with dd,
> 
> A broken backup GPT is not supposed to stop EFI from finding its partition
> and running its architecture specific start-up program.
> 

On the 2011 Macbook I am using, the boot process gets stuck before I
can select the drive to boot from, as long as a USB flash drive with
this GPT error is in a USB port.  The few other computers I tried have
no problem with booting from the flash drive.

This means I cannot install the Guix distribution on this Macbook
using the Guix ISO image.

(This is the default UEFI boot process that came with the Macbook.
I have never tried setting up Libreboot or anything similar.)

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
> On the 2011 Macbook I am using, the boot process gets stuck before I
> can select the drive to boot from, as long as a USB flash drive with
> this GPT error is in a USB port.  The few other computers I tried have
> no problem with booting from the flash drive.
> 

Actually it still gets stuck after I used fdisk to repair the drive’s
GPT.  I will check again with a Debian ISO tomorrow.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > A broken backup GPT is not supposed to stop EFI from finding its partition

Florian Pelz wrote:
> On the 2011 Macbook I am using, the boot process gets stuck before I
> can select the drive to boot from, as long as a USB flash drive with
> this GPT error is in a USB port.

Did you already test whether it works after you treated the stick
with gdisk expert option 'x' 'e' ?

If it really does work, then i am glad to have written "supposed".

It would then be interesting to see whether this picky machine boots
from a MBR partition table.
Is
  guixsd-install-0.15.0.i686-linux.iso
suitable for that machine (modulo GPT backup position) ?
If so, i will use my copy of this ISO to develop a xorriso run on
the mounted ISO which produces a new ISO with MBR-only layout.


(next mail from Florian):
> Actually it still gets stuck after I used fdisk to repair the drive’s
> GPT.  I will check again with a Debian ISO tomorrow.

Did the GPT repair really work ?
I.e. do other partition editors or kernels have no complaints about the
stick any more ?


Another try would be to deface the Apple Partition Map. It's a Mac.
So APM and HFS+ might have some impact. But other than expected by
grub-mkrescue.

With APM the first bytes of the ISO are
  45  52  08  00  eb  02  ff  ff
(which is the APM magic number and also x86 code without effect.)

Without APM they are other intentionally effectless x86 code:
  eb  63  90  90  90  90  90  90

If you write the latter 8 bytes over the start of your USB stick, then
APM should not be recognizable any more.


> I will check again with a Debian ISO tomorrow.

Debian installation ISOs have APM but no HFS+. I.e. their APM is quite
useless. These are: "netinst" ISOs, and the ISO sets for CD, DVD, BD, BDDL.

Debian 9 Live ISOs have no APM (and no HFS+).
Version 8 has no EFI equipment but only for legacy BIOS.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

The Parabola ISO parabola-systemd-cli-i686-netinstall-2018.06.02.iso
does not inhibit the boot process but also contains no EFI partition
so I cannot boot from it.

Debian 9.8.0 works, see the end of this mail.







On Tue, Apr 16, 2019 at 11:15:56PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > A broken backup GPT is not supposed to stop EFI from finding its partition
> 
> Florian Pelz wrote:
> > On the 2011 Macbook I am using, the boot process gets stuck before I
> > can select the drive to boot from, as long as a USB flash drive with
> > this GPT error is in a USB port.
> 
> Did you already test whether it works after you treated the stick
> with gdisk expert option 'x' 'e' ?
> 

Nope, it does not.




> If it really does work, then i am glad to have written "supposed".
> 
> It would then be interesting to see whether this picky machine boots
> from a MBR partition table.
> Is
>   guixsd-install-0.15.0.i686-linux.iso
> suitable for that machine (modulo GPT backup position) ?

Without changing the ISO’s partition tables, it does not work.

What should I change?







> If so, i will use my copy of this ISO to develop a xorriso run on
> the mounted ISO which produces a new ISO with MBR-only layout.
> 
> 
> (next mail from Florian):
> > Actually it still gets stuck after I used fdisk to repair the drive’s
> > GPT.  I will check again with a Debian ISO tomorrow.
> 
> Did the GPT repair really work ?
> I.e. do other partition editors or kernels have no complaints about the
> stick any more ?
> 
> 

I have overwritten the fdisk-corrected one, but the gdisk-corrected
flash drive is considered correct by fdisk.



> Another try would be to deface the Apple Partition Map. It's a Mac.
> So APM and HFS+ might have some impact. But other than expected by
> grub-mkrescue.
> 
> With APM the first bytes of the ISO are
>   45  52  08  00  eb  02  ff  ff
> (which is the APM magic number and also x86 code without effect.)
> 
> Without APM they are other intentionally effectless x86 code:
>   eb  63  90  90  90  90  90  90
> 
> If you write the latter 8 bytes over the start of your USB stick, then
> APM should not be recognizable any more.
> 
>

I inserted the gdisk-corrected USB flash drive and did:

$ ghex bytes  # put in eb  63  90  90  90  90  90  90
$ sudo dd if=bytes of=/dev/sdc
Passwort: 
0+1 Datensätze ein
0+1 Datensätze aus
8 bytes copied, 0,030945 s, 0,3 kB/s

It still does not work.




> > I will check again with a Debian ISO tomorrow.
> 
> Debian installation ISOs have APM but no HFS+. I.e. their APM is quite
> useless. These are: "netinst" ISOs, and the ISO sets for CD, DVD, BD, BDDL.
> 
> Debian 9 Live ISOs have no APM (and no HFS+).
> Version 8 has no EFI equipment but only for legacy BIOS.
> 
> 
> Have a nice day :)
> 
> Thomas
> 

The Debian ISO works fine and is bootable from its EFI partition.

gdisk prints:

$ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: present

Found valid MBR and GPT. Which do you want to use?
 1 - MBR
 2 - GPT
 3 - Create blank GPT

Your answer: 2
Using GPT and creating fresh protective MBR.
Warning! Main partition table overlaps the first partition by 64 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help): p
Disk Downloads/debian-live-9.8.0-amd64-xfce.iso: 3811392 sectors, 1.8 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): C05B92F2-4B16-4592-BA53-A2025E420B60
Partition table holds up to 248 entries
Main partition table begins at sector 2 and ends at sector 63
First usable sector is 64, last usable sector is 3811328
Partitions will be aligned on 8-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   2            1432            2263   416.0 KiB   0700  ISOHybrid1

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> $ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso
> ...
> Found valid MBR and GPT. Which do you want to use?
> 1 - MBR
> 2 - GPT
> ...
> Your answer: 2
> Using GPT and creating fresh protective MBR.
> Warning! Main partition table overlaps the first partition by 64 blocks!
> You will need to delete this partition or resize it in another utility.
> ...
> Number  Start (sector)    End (sector)  Size       Code  Name
>    2            1432            2263   416.0 KiB   0700  ISOHybrid1

This operation has killed the MBR partition table with the entry for
the EFI partition.
Due to the odd layout introduced for Fedora ISOs in SYSLINUX program
"isohybrid", there is a GPT partition entry which marks the EFI FAT image.
But it does not have the EFI type GUID. So EFI won't recognize it.

(This layout has been adopted by Debian and others who use ISOLINUX
 for BIOS and GRUB for EFI. It needs MBR partition type 0x00 for the
 ISO partition. The GPT ISO partition is madly starting at block 0.
 Partition editors are entitled to hate it.
 But well, the user decides what to do. I can only give advise.)

There remains the boot path of legacy BIOS via the MBR x86 code.


> The Debian ISO works fine and is bootable from its EFI partition.

Did you try booting before or after the gdisk run ?


> > Is guixsd-install-0.15.0.i686-linux.iso
> > suitable for that machine (modulo GPT backup position) ?

> Without changing the ISO’s partition tables, it does not work.
> What should I change?

I will develop and test a xorriso repack run for the mounted ISO
which produces MBR partitioning without GPT and APM.
Maybe this evening.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 12:23:56PM +0200, Thomas Schmitt wrote:
> This operation has killed the MBR partition table with the entry for
> the EFI partition.
> […]
> Did you try booting before or after the gdisk run ?
> 
>

Before.  I also did not write out.  Note that I also called gdisk on
the iso image and not on the USB disk.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

here are instructions to repack a Guix ISO with EFI support (via GPT)
to an ISO with MBR-only partition table.

Tested with guixsd-install-0.15.0.i686-linux.iso, because my copy
of 0.16.0 is one of the victims of bug#33639.

It becomes 130 MiB larger than the original, probably because there
were hardlinks in the original disk tree, which share their content in
the original ISO but which the Linux isofs driver cannot expose as
hardlink siblings when the ISO gets mounted.
E.g.
  /gnu/store/0br6mgbnc8b0myvvgpihhry9lfb67pdv-gmp-6.1.2/share/info/gmp.info-1.gz
  /gnu/store/2078rjc6rlj5yran6sx9kd5k4281j0vm-gmp-6.1.2/share/info/gmp.info-1.gz  /gnu/store/v78qymi7sxywv8l6kk5iydvvpdpncrsv-gmp-6.1.2/share/info/gmp.info-1.gz
have all the same LBA in the original ISO. But Linux shows them with
three different inode numbers, which it derives from the byte addresses
of the files' directory records. Shwoops ... 190 KiB more.
A few hundred such duplications would cause 130 MiB of growth.

About 17 MiB of the size difference are due to the second directory
tree which is needed to make partition 1 mountable despite not
beginning at block address 0.
Guix has a generously sized directory tree in its ISO. More than 30,000
data files. But i think a nice partition table is worth even 17 MiB.

=========================================================================

# Some paths which one may choose differently
BASE_ISO=guixsd-install-0.15.0.i686-linux.iso
MOUNT_POINT=/mnt/iso
EXTRACTED_MBR_IMG=extracted_mbr.img
REPACKED_ISO=guix-0.15.0.i686-mbr-only.iso

# (I myself prefer to become real superuser for mounting. But sudo is modern.)
PRIV_CMD=sudo

# Now for the work:

# Extract the MBR x86 code part from the original ISO
dd if="$BASE_ISO" bs=1 count=446 of="$EXTRACTED_MBR_IMG"

# The mounted original ISO will donate all its files to the repacked one
$PRIV_CMD mount -o loop "$BASE_ISO" "$MOUNT_POINT"

# The follow xorriso run builds the repacked ISO by native xorriso commands.
# Other than the mkisofs emulation it would not truncate an existing
# ISO image but rather complain about an unsuitabe multi-session setup.
# So remove an older image, if it exists:
test -e "$REPACKED_ISO" && rm "$REPACKED_ISO"

# Map the ISO data files into the new ISO, but delete /efi.img, which is the
# FAT filesystem image of the EFI partition.
# Install the extracted MBR with GRUB specific adjustments and start of
# partition 1 at byte 32768.
# Append the EFI partition image after the ISO filesystem (that is why the file
# is not needed inside the filesystem and then makes a much nicer partition).
# Set up the first El Torito catalog entry for PC-BIOS.
# Set up the second  El Torito catalog entry for EFI, using the appended
# partition.
xorriso \
   -outdev "$REPACKED_ISO" \
   -volid 'GUIXSD_IMAGE' \
   \
   -map "$MOUNT_POINT" "/" \
   -rm /efi.img -- \
   \
   -boot_image grub grub2_mbr="$EXTRACTED_MBR_IMG" \
   -boot_image any mbr_force_bootable=on \
   -boot_image any partition_table=on \
   -boot_image any iso_mbr_part_type=0x83 \
   -boot_image any partition_offset=16 \
   \
   -append_partition 2 0xef "$MOUNT_POINT"/efi.img \
   \
   -boot_image any cat_path='/boot.catalog' \
   \
   -boot_image grub bin_path='/boot/grub/i386-pc/eltorito.img' \
   -boot_image grub grub2_boot_info=on \
   -boot_image any emul_type=no_emulation \
   -boot_image any load_size=2048 \
   -boot_image any boot_info_table=on \
   \
   -boot_image any next \
   \
   -boot_image any efi_path='--interval:appended_partition_2:all::' \
   -boot_image any emul_type=no_emulation

=========================================================================

The result may be inspected by

  xorriso \
     -indev "$REPACKED_ISO" \
     -report_system_area plain \
     -report_el_torito plain

which should say

  ...
  Volume id    : 'GUIXSD_IMAGE'
  System area options: 0x00004a00
  System area summary: MBR grub2-mbr cyl-align-off
  ISO image size/512 : 2071064
  Partition offset   : 16
  MBR heads per cyl  : 64
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x83           64      2065240
  MBR partition      :   2   0x00  0xef      2065304         5760
  El Torito catalog  : 17333  1
  El Torito cat path : /boot.catalog
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
  El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       17334
  El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      516326
  El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
  El Torito img opts :   1  boot-info-table grub2-boot-info
  El Torito img blks :   2  1440

=========================================================================

For eyes which are not accustomed to xorriso's native command set, here is
the same with mkisofs emulation options (without trying to exclude
/mnt/iso/efi.img from the filesystem):

xorriso -as mkisofs \
   -o "$REPACKED_ISO" \
   -V 'GUIXSD_IMAGE' \
   \
   --grub2-mbr "$EXTRACTED_MBR_IMG" \
   --mbr-force-bootable \
   -iso_mbr_part_type 0x83 \
   -partition_offset 16 \
   \
   -append_partition 2 0xef "$MOUNT_POINT"/efi.img \
   \
   -c '/boot.catalog' \
   \
   -b '/boot/grub/i386-pc/eltorito.img' \
   --grub2-boot-info \
   -boot-info-table \
   -no-emul-boot \
   -boot-load-size 4 \
   \
   -eltorito-alt-boot \
   \
   -e '--interval:appended_partition_2:all::' \
   -no-emul-boot \
   \
   "$MOUNT_POINT"

Most users, including grub-mkrescue, prefer the mkisofs emulation because
they can recycle old knowledge from El-Torito-BIOS-only times.
The appeal of the native command set mostly shows up when the ISO content
is not shaped in a tree on hard disk but collected and arranged from
various files and trees.

Main difference is that sequence does strictly matter with the native set.
E.g. i cannot set boot attributes to files which are not mapped in yet.
The mkisofs emulation sets up a production job and only then assesses
the input file objects. (Imagine a shell which takes a dozen commands
from you and re-arranges them before executing them. Ewww ...)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 05:51:41PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> here are instructions to repack a Guix ISO with EFI support (via GPT)
> to an ISO with MBR-only partition table.
> 

Thank you!  However, even though Linux is happy about the repacked
ISO, my Macbook still does not like it.  I wonder what makes Debian so
likable.  I will investigate tomorrow.

(The ISO image file size is reproducable, the content is not, of
course, as before.)




> The result may be inspected by
> 
>   xorriso \
>      -indev "$REPACKED_ISO" \
>      -report_system_area plain \
>      -report_el_torito plain
> 
> which should say
> 
>   ...
>   Volume id    : 'GUIXSD_IMAGE'
>   System area options: 0x00004a00
>   System area summary: MBR grub2-mbr cyl-align-off
>   ISO image size/512 : 2071064
>   Partition offset   : 16
>   MBR heads per cyl  : 64
>   MBR secs per head  : 32
>   MBR partition table:   N Status  Type        Start       Blocks
>   MBR partition      :   1   0x80  0x83           64      2065240
>   MBR partition      :   2   0x00  0xef      2065304         5760
>   El Torito catalog  : 17333  1
>   El Torito cat path : /boot.catalog
>   El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
>   El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       17334
>   El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      516326
>   El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
>   El Torito img opts :   1  boot-info-table grub2-boot-info
>   El Torito img blks :   2  1440
> 

Looks good, I believe:

florian@florianmacbook ~$   xorriso \
>      -indev "$REPACKED_ISO" \
>      -report_system_area plain \
>      -report_el_torito plain
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : ISO image bears MBR with  -boot_image any partition_offset=16
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE :   32980 nodes read in 1 seconds
xorriso : UPDATE :   56154 nodes read in 2 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 742333 * 2048 , 5760 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'g7.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record  : El Torito , MBR grub2-mbr cyl-align-off
Media summary: 1 session, 743725 data blocks, 1453m data, 58.4g free
Volume id    : 'GUIXSD_IMAGE'
System area options: 0x00004a00
System area summary: MBR grub2-mbr cyl-align-off
ISO image size/512 : 2975092
Partition offset   : 16
MBR heads per cyl  : 91
MBR secs per head  : 32
MBR partition table:   N Status  Type        Start       Blocks
MBR partition      :   1   0x80  0x83           64      2969268
MBR partition      :   2   0x00  0xef      2969332         5760
El Torito catalog  : 21669  1
El Torito cat path : /boot.catalog
El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       21670
El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      742333
El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
El Torito img opts :   1  boot-info-table grub2-boot-info
El Torito img blks :   2  1440







> For eyes which are not accustomed to xorriso's native command set, here is
> the same with mkisofs emulation options (without trying to exclude
> /mnt/iso/efi.img from the filesystem):
> 

This yields a differently sized ISO (perhaps because of not excluding
/mnt/iso/efi.img) which too is now liked by Linux but still not by my
Macbook:

$ ls -l g7.iso g8.iso
-rw-r--r-- 1 florian users 1523580928 17. Apr 21:26 g7.iso
-rw-r--r-- 1 florian users 1525852160 17. Apr 22:03 g8.iso



$ xorriso      -indev "$REPACKED_ISO"      -report_system_area plain      -report_el_torito plain
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : ISO image bears MBR with  -boot_image any partition_offset=16
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE :   33890 nodes read in 1 seconds
xorriso : UPDATE :   56155 nodes read in 2 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 743455 * 2048 , 5760 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'g8.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record  : El Torito , MBR grub2-mbr cyl-align-off
Media summary: 1 session, 745045 data blocks, 1455m data, 57.0g free
Volume id    : 'GUIXSD_IMAGE'
System area options: 0x00004a00
System area summary: MBR grub2-mbr cyl-align-off
ISO image size/512 : 2980180
Partition offset   : 16
MBR heads per cyl  : 0
MBR secs per head  : 0
MBR partition table:   N Status  Type        Start       Blocks
MBR partition      :   1   0x80  0x83           64      2973756
MBR partition      :   2   0x00  0xef      2973820         5760
El Torito catalog  : 21351  1
El Torito cat path : /boot.catalog
El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       21352
El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      743455
El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
El Torito img opts :   1  boot-info-table grub2-boot-info
El Torito img blks :   2  1440

Again, thank you!  (Even though I would prefer if it worked on
Macbooks.  Many people I know own one of those.  I do not know if
newer Macbooks are as picky as mine.)

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> my Macbook still does not like it.

The content of the EFI partition of guixsd-install-0.15.0.i686-linux.iso
does not look like it is ready for 64-bit x86:

  $ mount guixsd-install-0.15.0.i686-linux.iso /mnt/iso
  $ mount /mnt/iso/efi.img /mnt/fat
  $ find /mnt/fat -exec ls -ld '{}' ';'
  drwxr-xr-x 3 root root 7680 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 1024 Jul  6  2018 /mnt/fat/efi
  drwxr-xr-x 2 root root 1024 Jul  6  2018 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 96256 Jul  6  2018 /mnt/fat/efi/boot/boot.efi
  -rwxr-xr-x 1 root root 96256 Jul  6  2018 /mnt/fat/efi/boot/bootia32.efi

According to UEFI specs, bootia32.efi is the start program for 32-bit x86
systems.
The name for 64-bit x86 systems would be bootx64.efi like in the EFI
partition of debian-9.3.0-amd64-netinst.iso :

  drwxr-xr-x 3 root root 16384 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 2048 Dec  9  2017 /mnt/fat/efi
  drwxr-xr-x 2 root root 2048 Dec  9  2017 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 390656 Dec  9  2017 /mnt/fat/efi/boot/bootx64.efi

(Afaik, the EFI partition image is created by grub-mkimage. But i may
 be wrong. Better check source of debian-cd or ask Steve McIntyre.)

Was GRUB configured to produce 64-bit x86 code ?


I only know how to do it with Debian. There i have to install the
packages "grub-pc" for BIOS, "grub-efi-ia32-bin" for 32 bit x86 EFI,
"grub-efi-amd64-bin" for 64 bit x86 EFI.

The EFI partition then has:

  drwxr-xr-x 3 root root 7680 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 1024 Feb  6  2016 /mnt/fat/efi
  drwxr-xr-x 2 root root 1024 Feb  6  2016 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 132608 Feb  6  2016 /mnt/fat/efi/boot/bootx64.efi
  -rwxr-xr-x 1 root root 95744 Feb  6  2016 /mnt/fat/efi/boot/bootia32.efi
  -rwxr-xr-x 1 root root 95744 Feb  6  2016 /mnt/fat/efi/boot/boot.efi
  -rwxr-xr-x 1 root root 1759 Dec 10  2017 /mnt/fat/NvVars

("grub-pc" caused bootable MBR x85 code.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

meanwhile i learned that "i686" means 32 bit.

So i downloaded
  guixsd-install-0.16.0.x86_64-linux.iso
and see indeed the boot program for 64 bit x86 in the EFI partition:

  -rwxr-xr-x 1 root root 132608 Dec  5 16:59 /mnt/fat/efi/boot/bootx64.efi

From Florian's reports i cannot tell which ISO he used as base for
repacking. Maybe one should distinguish the ISOs by their Volume Id. Like:
  GUIXSD_IMAGE_I686
  GUIXSD_IMAGE_X86_64


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 09:00:23AM +0200, Thomas Schmitt wrote:
> From Florian's reports i cannot tell which ISO he used as base for
> repacking. Maybe one should distinguish the ISOs by their Volume Id. Like:
>   GUIXSD_IMAGE_I686
>   GUIXSD_IMAGE_X86_64

The current git master for x86_64.  I can try another one this weekend.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > From Florian's reports i cannot tell which ISO he used

Florian Pelz wrote:
> The current git master for x86_64.

I begin to remember discussions about 64-bit Macs having 32-bit EFI
firmware.
Googling brought me to
  https://everymac.com/mac-answers/snow-leopard-mac-os-x-faq/mac-os-x-snow-leopard-64-bit-macs-64-bit-efi-boot-in-64-bit-mode.html
Note the combinations of "Processor 64" and "EFI 32".
(I think the "Boot" column is about MacOS, not the firmware.)


> ... > On the 2011 Macbook I am using

The rumors rather talk of the years up to 2010. But well ...


Yesterday:
> The Debian ISO works fine
> $ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso

Let's see what's in there ...

  drwxr-xr-x 3 root root 16384 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 2048 Feb 11 20:17 /mnt/fat/efi
  drwxr-xr-x 2 root root 2048 Feb 11 20:17 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 390656 Feb 11 20:17 /mnt/fat/efi/boot/bootx64.efi

No 32-bit start program. Grrr.


> I can try another one this weekend.

I am nearly out of ideas if not by surprise a test with Guix "i686" ISO
works.

The last resort would be to try the weird "isohybrid --uefi" layout of
questionable MBR partition table and invalid GPT.
(This might answer the riddle why Matthew Garrett kept the GPT data
 after he obviously decided for MBR partitioning.)

But before we do that, we should make absolutely sure that EFI does
not start the bootx64.efi of the Guix ISO. Only then, the partitioning
is to blame. After bootx64.efi was started, GRUB is in charge.
So its modules in both ISOs might make the difference.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 10:13:33AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > ... > On the 2011 Macbook I am using
> 
> The rumors rather talk of the years up to 2010. But well ...
> 
> 

Next week I can ask others who own newer Macbooks to try out the ISO.
If this only affects old Macbooks, fixing it might not be that
important.



> I am nearly out of ideas if not by surprise a test with Guix "i686" ISO
> works.
> 

I just tried guix-0.15.0.i686-mbr-only.iso and boot gets stuck too.
Remarkably, when using the original
guixsd-install-0.15.0.i686-linux.iso on the same x86_64 2011 Macbook,
I can at least still boot from other partitions, the Guix USB drive
merely does not show up (for Debian, it shows up as an EFI partition
in the boot menu).





> The last resort would be to try the weird "isohybrid --uefi" layout of
> questionable MBR partition table and invalid GPT.
> (This might answer the riddle why Matthew Garrett kept the GPT data
>  after he obviously decided for MBR partitioning.)
> 
> But before we do that, we should make absolutely sure that EFI does
> not start the bootx64.efi of the Guix ISO. Only then, the partitioning
> is to blame. After bootx64.efi was started, GRUB is in charge.
> So its modules in both ISOs might make the difference.
> 
> 

I find a efi/boot/bootx64.efi PE executable inside efi.img on the Guix
ISO.  On Debian the long name is EFI/boot/bootx64.iso in capital
letters; I do not know if this can make a difference.  How would I
check if it gets run?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I find a efi/boot/bootx64.efi PE executable inside efi.img on the Guix
> ISO. On Debian the long name is EFI/boot/bootx64.iso in capital letters

(I assume s/bootx64.iso/bootx64.efi/)

It's FAT filesystem. Upper case or lower case does not matter.
Most probably it is a matter of mount options whether you see capital
or small letters.
The image files seem to contain the names in capital letters:

  # mount guixsd-install-0.16.0.x86_64-linux.iso /mnt/iso

  $ strings /mnt/iso/efi.img | head -9
  MTOO4021
  Hf3NO NAME    FAT12
  EFI
  .
  ..
  BOOT
  .
  ..
  BOOTX64 EFI

  # umount /mnt/iso
  # mount debian-live-9.8.0-amd64-xfce.iso /mnt/iso

  $ strings /mnt/iso/boot/grub/efi.img | head -17
  mkfs.fat
  dNO NAME    FAT12
  This is not a bootable disk.  Please insert a bootable floppy and
  press any key to try again ...
  EFI
  KNKN
  .
  KNKN
  ..
  KNKN
  BOOT
  KNKN
  .
  KNKN
  ..
  KNKN
  BOOTX64 EFI


> How would I check if it gets run?

Interesting question. If it says "GRUB" or "Guix", then it runs. But up
to then it may start and fail silently.


> the Guix USB drive
> merely does not show up (for Debian, it shows up as an EFI partition
> in the boot menu).

So probably it is not run.

I'll develop a xorriso repack run for the layout of Debian ISOs, as
strange as it is.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

this is the xorriso run for producing a layout like "isohybrid --uefi"
with GRUB equipment for BIOS instead of ISOLINUX.

I prepended command -stdio_sync "off" in order to make use of my generous
amount of RAM for write buffering. The default syncs to disk every 16 MiB.
Good with low RAM or fast SSD.

xorriso treats its output as an optical drive.
Without sync, the speed reports on image files are rather funny:
  xorriso : UPDATE : Writing:   262432s  43.6%  fifo 88%  buf  50%  1169.9xD

===========================================================================

# Layout of isohybrid --uefi. Invented by Matthew J. Garrett for Fedora.
# The EFI partition is a file inside the ISO. The BIOS equipment is ISOLINUX
# instead of GRUB. But the EFI equipment is GRUB.
# Used by Fedora, Debian, Ubuntu, Mint, Arch, and more.
# (SuSE is different by using a pure MBR partition table.)

# Additionally to a Guix ISO, we need one of those isohybrid ISOs.
# The result will not be usable for booting via BIOS, because only the first
# stages of ISOLINUX are extracted. The rest would not fit Guix anyways.
ISOHYBRID_ISO=debian-live-9.8.0-amd64-xfce.iso
ISOLINUX_BIN_ISO=/isolinux/isolinux.bin
EXTRACTED_ISOLINUX_BIN=isolinux.bin
EXTRACTED_MBR_IMG=isohybrid_mbr.img

# Guix ISO related names
BASE_ISO=guixsd-install-0.16.0.x86_64-linux.iso
MOUNT_POINT=/mnt/iso
REPACKED_ISO=guix-0.16.0.x86_64-mjg.iso

PRIV_CMD=sudo

# Extract the ISOLINUX isohybrid MBR x86 code part from the Debian ISO
dd if="$ISOHYBRID_ISO" bs=1 count=446 of="$EXTRACTED_MBR_IMG"

# Extract the ISOLINUX boot image for El Torito
xorriso \
  -report_about warning \
  -osirrox on \
  -indev "$ISOHYBRID_ISO" \
  -report_about note \
  -extract "$ISOLINUX_BIN_ISO" "$EXTRACTED_ISOLINUX_BIN"

# The mounted original ISO will donate all its files to the repacked one
$PRIV_CMD mount -o loop "$BASE_ISO" "$MOUNT_POINT"

# Pack it all up
xorriso \
   -stdio_sync off \
   -outdev "$REPACKED_ISO" \
   -volid 'GUIXSD_IMAGE' \
   \
   -map "$MOUNT_POINT" "/" \
   -map "$EXTRACTED_ISOLINUX_BIN" /isolinux/isolinux.bin \
   \
   -boot_image isolinux system_area="$EXTRACTED_MBR_IMG" \
   -boot_image any mbr_force_bootable=on \
   -boot_image any partition_offset=0 \
   \
   -boot_image any cat_path='/isolinux/boot.cat' \
   -boot_image isolinux bin_path='/isolinux/isolinux.bin'  \
   -boot_image any emul_type=no_emulation \
   -boot_image any load_size=2048 \
   -boot_image any boot_info_table=on \
   \
   -boot_image any next \
   \
   -boot_image any efi_path=/efi.img \
   -boot_image any emul_type=no_emulation \
   -boot_image isolinux partition_entry=gpt_basdat

#Clean up disk
rm "$EXTRACTED_MBR_IMG" "$EXTRACTED_ISOLINUX_BIN"

===========================================================================

Expected inspection result:

  $ xorriso -indev "$REPACKED_ISO" -report_system_area plain
  ...
  Volume id    : 'GUIXSD_IMAGE'
  System area options: 0x00000102
  System area summary: MBR isohybrid cyl-align-on GPT
  ISO image size/512 : 2405888
  Partition offset   : 0
  MBR heads per cyl  : 74
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x00            0      2405888
  MBR partition      :   2   0x00  0xef        35132         2880
  MBR partition path :   2  /efi.img
  GPT                :   N  Info
  GPT disk GUID      :      451fe71ed0ff5d48a1c144a0316dd901
  GPT entry array    :      2  248  overlapping
  GPT lba range      :      64  2405824  2405887
  GPT partition name :   1  490053004f00480079006200720069006400
  GPT partname local :   1  ISOHybrid
  GPT partition GUID :   1  451fe71ed0ff5d48a1c044a0316dd901
  GPT type GUID      :   1  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   1  0x1000000000000001
  GPT start and size :   1  0  2405824
  GPT partition name :   2  490053004f004800790062007200690064003100
  GPT partname local :   2  ISOHybrid1
  GPT partition GUID :   2  451fe71ed0ff5d48a1c344a0316dd901
  GPT type GUID      :   2  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   2  0x1000000000000001
  GPT start and size :   2  35132  2880
  GPT partition path :   2  /efi.img

For comparison, the output with debian-live-9.8.0-amd64-xfce.iso :

  Volume id    : 'd-live 9.8.0 xf amd64'
  System area options: 0x00000102
  System area summary: MBR isohybrid cyl-align-on GPT
  ISO image size/512 : 3811392
  Partition offset   : 0
  MBR heads per cyl  : 117
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x00            0      3811392
  MBR partition      :   2   0x00  0xef         1432          832
  MBR partition path :   2  /boot/grub/efi.img
  GPT                :   N  Info
  GPT disk GUID      :      f2925bc0164b9245ba53a2025e420b60
  GPT entry array    :      2  248  overlapping
  GPT lba range      :      64  3811328  3811391
  GPT partition name :   1  490053004f00480079006200720069006400
  GPT partname local :   1  ISOHybrid
  GPT partition GUID :   1  f2925bc0164b9245ba52a2025e420b60
  GPT type GUID      :   1  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   1  0x1000000000000001
  GPT start and size :   1  0  3811328
  GPT partition name :   2  490053004f004800790062007200690064003100
  GPT partname local :   2  ISOHybrid1
  GPT partition GUID :   2  f2925bc0164b9245ba51a2025e420b60
  GPT type GUID      :   2  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   2  0x1000000000000001
  GPT start and size :   2  1432  832
  GPT partition path :   2  /boot/grub/efi.img


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 11:28:13PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> this is the xorriso run for producing a layout like "isohybrid --uefi"
> with GRUB equipment for BIOS instead of ISOLINUX.
> 

No, it still gets stuck. :(


> Expected inspection result:
> 

The GPT partition GUID is different each time I repeat the process.
Otherwise I get the same result.



> For comparison, the output with debian-live-9.8.0-amd64-xfce.iso :
> 

The debian output is fully identical.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 03:50:59PM +0200, Thomas Schmitt wrote:
> > the Guix USB drive
> > merely does not show up (for Debian, it shows up as an EFI partition
> > in the boot menu).
> 
> So probably it is not run.
> 

Boot gets stuck on x86_64 Guix images only, but the one i686 I tried
did not show up in the boot menu.  I believe bootx64.efi is run for
x86_64 and is the culprit.

I could try burning a DVD and attempt to boot from it.

It would also be interesting to find out how to change the
bootx64.efi’s code and make it display debug information in some way.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > this is the xorriso run for producing a layout like "isohybrid --uefi"

Florian Pelz wrote:
> No, it still gets stuck. :(

Then it must be something inside the EFI partition which makes the
Macbook firmware dismiss the partition for booting.


> The GPT partition GUID is different each time I repeat the process.

This can be avoided by setting environment variable SOURCE_DATE_EPOCH
to the same value in each production run.
(I just tested that this works as expected.)


Well, i am out of ideas how the Guix ISO layout or content differs from
the stuff in Debian ISOs, so that an EFI implementation dismisses the
former and accepts the latter.

It really does not look like an issue with partitioning.
I don't think that it goes deeper into the boot equipment than the
files in the EFI partition.

Maybe GRUB experts have ideas what to examine there. After all it is
grub-mkrescue which fails to produce a recognizable partition, whereas
Debian's effort with (most probably) grub-mkimage can do the trick.
grub-mkrescue uses grub-mkimage, too, afaik. So it probably is about
some magic ingredient to the production of the EFI start programs or
the FAT image file.

We could ask at debian-cd mailing list whether the problem rings a bell
from a past problem fix.
(I am subscribed there and ready to assist with technical details about
 our findings.)


Next mail from Florian:
> Boot gets stuck on x86_64 Guix images only, but the one i686 I tried
> did not show up in the boot menu.  I believe bootx64.efi is run for
> x86_64 and is the culprit.

This fits into my findings if EFI looks into the EFI partitions
and dismisses those without its CPU specific flavor of start program.

Can you get EFI to confess that the x86_64 USB stick indeed has a
recognizable EFI partition ?


> I could try burning a DVD and attempt to boot from it.

On DVD the El Torito boot equipment should lead EFI to the same partition
image as the partition table should do on USB stick.
But a try cannot harm.


> It would also be interesting to find out how to change the
> bootx64.efi’s code and make it display debug information in some way.

The highest control layer is
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c
The name "BOOTX64.EFI" appears in
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-install.c
Somewhere inbetween must be its source. :o)

------------------------------------------------------------------------

For now i say thanks for three found bugs in my own software, during
this discussion:

- Environment variable SOURCE_DATE_EPOCH did not affect synthetic files.
  Thanks to Florian and Ludovic for pointing out the non-reproducibility
  issue.
  Fixed by libisoburn 5b62c55
  Workaround for Guix and currently released versions of xorriso:
  Give the ISO root directory fixed atime, mtime, and ctime by xorriso
  command -alter_date. (Would work with future xorriso versions, too.)

- SIGSEGV happened if "-boot_image grub grub2_mbr=" was used and no
  El Torito boot image was defined.
  This one i found by a bad backslash while testing my xorriso run
  proposals to Florian.
  Fixed by libisofs 4b21386
  No workaround needed because no usable result can emerge anyways.

- I was unable yesterday to get "isohybrid --uefi" layout when using GRUB
  equipment for BIOS. I thought it was possible. My grub-mkrescue argument
  filter script falsely promises to do it.
  Still unresolved.
  Workaround was to use body parts of a real isohybrid ISO.

------------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 11:01:51AM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > this is the xorriso run for producing a layout like "isohybrid --uefi"
> 
> Florian Pelz wrote:
> > No, it still gets stuck. :(
> 
> Then it must be something inside the EFI partition which makes the
> Macbook firmware dismiss the partition for booting.
> 
> 

For x86_64 iso images, the Macbook does *not dismiss*.  It gets stuck
in the lightgray startup screen forever without showing a menu with
the drives to boot from, except for the Debian ISO image, for which a
menu is shown with an »EFI partition« that runs a bootloader when
chosen.






> Can you get EFI to confess that the x86_64 USB stick indeed has a
> recognizable EFI partition ?
> 
> 

I do not currently know how.  Maybe by changing the bootx64.efi on its
EFI partition?



> > I could try burning a DVD and attempt to boot from it.
> 
> On DVD the El Torito boot equipment should lead EFI to the same partition
> image as the partition table should do on USB stick.
> But a try cannot harm.
> 
>

Will try now.




> For now i say thanks for three found bugs in my own software, during
> this discussion:
> 

I am happy something good came of it.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> For x86_64 iso images, the Macbook does *not dismiss*.  It gets stuck
> in the lightgray startup screen forever without showing a menu with
> the drives to boot from, except for the Debian ISO image, for which a
> menu is shown with an »EFI partition« that runs a bootloader when
> chosen.

So the problem might be during the inspection of the EFI partition whether
it is worth to be offered for booting.

Comparing the output of program "strings" from both bootx64.efi files,
i see in the Debian file text about Secure Boot, which are not in the
Guix file:

  SecureBoot
  SetupMode
  ...
  Secure Boot forbids loading module from %s
  ...
  grub_efi_secure_boot

Older Debian netinst maybe ? ... digging in my collection ...

The youngest without "Secure" in its bootx64.efi is
  debian-7.7.0-amd64-netinst.iso

You could check whether this hangs like Guix.

Debian's own ISO archive of 7.7.0 only offers CD-1 as jigdo.
Probably with the same EFI equipment.
But let's google ...

  http://ftp.riken.go.jp/Linux/debian/debian-cdimage/archive/7.7.0/amd64/iso-cd/debian-7.7.0-amd64-netinst.iso

It matches my old copy. 232,783,872 bytes.
SHA512:
  5cb6e4fea55fbb5173f90c3a545b843c6c193e29c3aa32b3306c9bbdfb1ad6a6a36ae8be50e91af9d03d5f21c472bd05d04d3508172e0b519e76714333c7c74b


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 12:58:58PM +0200, Thomas Schmitt wrote:
> The youngest without "Secure" in its bootx64.efi is
>   debian-7.7.0-amd64-netinst.iso
> 
> You could check whether this hangs like Guix.
> 

No, it does not hang, it is bootable.


> SHA512:
>   5cb6e4fea55fbb5173f90c3a545b843c6c193e29c3aa32b3306c9bbdfb1ad6a6a36ae8be50e91af9d03d5f21c472bd05d04d3508172e0b519e76714333c7c74b
> 

Yes, for me the sha512sum is the same.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 11:01:51AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > I could try burning a DVD and attempt to boot from it.
> 
> On DVD the El Torito boot equipment should lead EFI to the same partition
> image as the partition table should do on USB stick.
> But a try cannot harm.
> 
>

The DVD works and is bootable. :)  It displays twice as “GRUB 2.02”
and “Windows”.  I presume one of them is booted in legacy BIOS mode
and one is booted in UEFI mode.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

What is this --hfs-bless option of mkisofs?  This is not related by
any chance?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> The DVD works and is bootable. :)

Hrmpf. I hate it when success spoils a theory.


I wrote:
> > The youngest without "Secure" in its bootx64.efi is
> >   debian-7.7.0-amd64-netinst.iso

> No, it does not hang, it is bootable.

It is about time for black candles and the toad mummy resurrection ritual.


Summary:

Something brings the firmware of this Macbook to a halt, probably when
it assesses the offers of bootable partitions.
It cannot entirely be about the partition table, because it even happens
when we achieve the table type and partition entries of Debian ISOs.
But it cannot entirely be about the content of the EFI partition, because
the same EFI partition is recognized and boots if it is accessed via the
El Torito boot catalog from DVD (*).

(*) Assumed that the Macbook EFI halfways obeys UEFI specs about which
    table to interpret depending on the medium type.

---------------------------------------------------------------------

Next mail:
> What is this --hfs-bless option of mkisofs?  This is not related by
> any chance?

Rather not. (Unless the toad becomes mother and the candles turn white.)

"Blessing" in HFS or HFS+ is a way to mark a file or directory for a
special role in the filesystem. For the firmware and maybe for MacOS.

In guixsd-install-0.16.0.x86_64-linux.iso (which should be re-generated
for the download site) there is one blessed file:

  $ xorriso -hfsplus on -indev guixsd-install-0.16.0.x86_64-linux.iso \
            -find / -has_hfs_bless any -exec get_hfs_bless --
  ...
  intel_bootfile   '/System/Library/CoreServices/boot.efi'

Other possible blessings are "ppc_bootdir", "show_folder", "os9_folder",
and "osx_folder".

But in Debian Live there is no HFS+ and thus no opportunity for blessing.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive and
mounted it.  I am trying to make the boot not get stuck.

I replaced efi/boot/bootx64.efi by Debian’s bootx64.efi.  Still stuck.

I deleted efi/boot/bootx64.efi.  Still stuck.

I changed the EFI partition’s GPT type code with gdisk from EF00 to
0700.  Still stuck.

I deleted the EFI partition.  Now I can boot (but not from the USB
drive, of course).

I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
Boot gets stuck again.

I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
partition.  Now I can boot again (not from the Guix USB drive, of
course).

I wonder, what kind of FAT is the EFI partition using?  Could it be
the wrong kind of FAT?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I wonder, what kind of FAT is the EFI partition using?  Could it be
> the wrong kind of FAT?

I got a mail from a bystander, Ady Ady whom i know from SYSLINUX/ISOLINUX
mailing list. He proposes the experiment to exchange the whole EFI partition
of the Guix ISO by the one of Debian Live 9.

This will not be able to really boot Guix, of course. But if the Mac's
boot manager shows it and is willing to start the bootx64.efi program,
then the FAT filesystem's meta data become prime suspect.


The EFI partition of Debian Live 9 is smaller than the one of Guix 0.16.0.
So after these steps :

> I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
> Boot gets stuck again.
>
> I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
> partition.

mount the Debian Live 9 ISO (e.g. at /mnt/iso) and do

  dd if=/mnt/iso/boot/grub/efi.img of=/dev/sdc2


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i now look at the FAT filesystem images of Debian Live 9 and Guix 0.16.0:

Guix:

- There is an MBR partition table in the image file:
    Device            Boot Start   End Sectors  Size Id Type
    /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12

- Program "file" says:

  DOS/MBR boot sector, code offset 0x3c+2,
  OEM-ID "MTOO4021",
  root entries 224, sectors 2880 (volumes <=32 MB) ,
  sectors/FAT 9, sectors/track 18,
  serial number 0x336648a7, unlabeled,
  FAT (12 bit), followed by FAT

Debian:

- No partition table in image.

- Program "file" says:
  DOS/MBR boot sector, code offset 0x3c+2,
  OEM-ID "mkfs.fat",
  sectors/cluster 4, root entries 512, sectors 832 (volumes <=32 MB) ,
  Media descriptor 0xf8,
  sectors/FAT 1, sectors/track 32, heads 64,
  serial number 0x64c6c435, unlabeled, FAT (12 bit)

---------------------------------------------------------------------

My favorite suspect would be the partition table in Guix /efi.img.

To kill table entry 1 after having put the Guix EFI image back into
partition 2 of the USB stick:

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

---------------------------------------------------------------------

If this does not help, then create a new empty FAT filesystem image,
mount it, and copy the tree from the mounted Guix /efi.img into it.
The new filesystem should have between 200 KiB and 1.4 MiB.
(The Guix image of 1.4 MiB still has 90 % of its capacity free.)

Then put the new filesystem into /dev/sdc2 and see what happens.

---------------------------------------------------------------------

If this does not help, then copy again the Debian efi.img into partition 2,
mount it, and replace the file bootx64.efi by the one of the mounted
Guix efi.img.

---------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

... and if the erasure of the partition table entry helps, would it then
also help if that entry's partition does not start at LBA 0 ?

Change bytes 446 (dec) to 461 (dec) from (in hex):

  80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00

to

  80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

"02" = Sector 2 = LBA 1.
"01 00 00 00" = LBA 1.
"3f 0b 00 00" = bloc count 2879, rather than 2880.


Reasoning:

It would make a nice endless loop if EFI dives into any partition
and looks for sub partition tables. The one in GRUB's efi.img refers
to itself.
DVD might be immune because EFI does not do this diving when it finds
the partition by the El Torito boot catalog.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> To kill table entry 1 after having put the Guix EFI image back into
> partition 2 of the USB stick:
>
>   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

Of course i meant  of=/dev/sdc2

The 16 bytes written to /dev/sdc1 would deface the GPT's "protective" MBR
and thus make the GPT invalid.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

It works, see end of this e-mail.

On Sat, Apr 20, 2019 at 12:50:49PM +0200, Thomas Schmitt wrote:
> So after these steps :
> 
> > I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
> > Boot gets stuck again.
> >
> > I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
> > partition.
> 
> mount the Debian Live 9 ISO (e.g. at /mnt/iso) and do
> 
>   dd if=/mnt/iso/boot/grub/efi.img of=/dev/sdc2
> 
> 

It still does not get stuck. :)



On Sat, Apr 20, 2019 at 01:16:14PM +0200, Thomas Schmitt wrote:
> My favorite suspect would be the partition table in Guix /efi.img.
> 
> To kill table entry 1 after having put the Guix EFI image back into
> partition 2 of the USB stick:
> 
>    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1
>

I did

sudo dd if=Downloads/guixsd-install-0.16.0.x86_64-linux.iso of=/dev/sdc
sudo dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

Boot gets stuck.

On Sat, Apr 20, 2019 at 04:23:49PM +0200, Thomas Schmitt wrote:
> Of course i meant  of=/dev/sdc2
> 

Oops, too late.


> If this does not help, then create a new empty FAT filesystem image,
> mount it, and copy the tree from the mounted Guix /efi.img into it.
> The new filesystem should have between 200 KiB and 1.4 MiB.
> (The Guix image of 1.4 MiB still has 90 % of its capacity free.)
> 

I do not recreate the USB drive.  I do:

qemu-img create -f raw fatfs.img 1M
mkfs.fat fatfs.img # from package dosfstools like the EFI system
                   # partition on Arch wiki
sudo mount Downloads/guixsd-install-0.16.0.x86_64-linux.iso /mnt/iso
sudo mount fatfs.img /mnt/img
sudo mount /mnt/iso/efi.img /mnt/efiimg
sudo cp -r /mnt/efiimg/efi /mnt/img/
sudo umount /mnt/img

> Then put the new filesystem into /dev/sdc2 and see what happens.
>

sudo dd if=fatfs.img of=/dev/sdc2
sync

Hooray!  It boots!  Thank you so much for your patience.

I remember back when I used Parabola GNU/Linux-libre, I had to make
sure I used mkfs.fat instead of mkfs.msdos or else the EFI partition
did not work, I think.

But…  Guix’ gnu/build/vm.scm already uses mkfs.fat.  Strange…

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sat, Apr 20, 2019 at 04:54:24PM +0200, pelzflorian (Florian Pelz) wrote:
> sudo dd if=fatfs.img of=/dev/sdc2
> sync
> 

I can copy the same fatfs.img from Guix System 0.16.0 onto a current
Guix git master installer iso and it works too.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sat, Apr 20, 2019 at 04:54:24PM +0200, pelzflorian (Florian Pelz) wrote:
> But…  Guix’ gnu/build/vm.scm already uses mkfs.fat.  Strange…
> 

It uses mkfs.fat for creating FAT partitions, but maybe this
particular EFI partition is created by grub-mkrescue?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> mkfs.fat fatfs.img
> sudo mount fatfs.img /mnt/img
> sudo cp -r /mnt/efiimg/efi /mnt/img/
> Hooray!  It boots!

So it is indeed the filesystem hull, which is to blame.

We still need to find out whether the partition entry is the culprit.
So please do the experiment with the Guix EFI partition image in
partition 2 of the USB stick and

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

The partition entry in the Guix EFI partition image has all the
properties of the thing we are looking for:
- Not in the FAT filesystem, but also not outside of it.
- Plausible to be differently approached from USB stick and DVD.
- Self referring enough for an endless inspection loop in EFI.


> It uses mkfs.fat for creating FAT partitions, but maybe this
> particular EFI partition is created by grub-mkrescue?

Yes.
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812
has:
      rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
            efiimgfat, "::", NULL });

This run produces an image file with partition 1, starting at LBA 0.
A remark below
  https://sources.debian.org/src/mtools/4.0.23-1/mformat.c/#L1373
says:
      /* install fake partition table pointing to itself */

(But why does grub-mkrescue.c create a 2.8 MB floppy where Guix has a
 1.4 MB floppy image ? Does Guix use a patched version of grub-mkrescue ?)

---------------------------------------------------------------------------
Under the assumption that it is really the partition entry, i snooped ahead:

mformat options -2 and -k can suppress the production of the partition
entry. Option -2 is for some exotic floppy format from the 1990s.

Option -k seems better, but if i use it, program "file" becomes sparing
with words:

  $ file /u/test/mformat.fat
  /u/test/mformat.fat: DOS floppy 1440k
  $

It can be mounted by Linux and offers free space, though.

More similar to the current FAT image hull is the result of

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

I actually did with a copy of partition 2:
   dd if=/dev/zero bs=1 count=16 seek=446 conv=notrunc of=test.img
and the report of "file" did not change. fdisk reports no partitions
in test.img any more.


Assuming that we cannot easily convince GRUB to use mformat -k or to
switch to some other FAT creator, i currently ponder the opportunity
to let my man-in-the-middle script do this dd patching of the EFI image
before it gets copied into the ISO.

-------------------------------------------------------------------------
Second correction of my statement about zapping the partition entry:

i wrote:
> The 16 bytes written to /dev/sdc1 would deface the GPT's "protective" MBR
> and thus make the GPT invalid.

Wrong again. Because it is a grub-mkrescue ISO, it hit byte 446 ff of
the ISO Primary Volume Descriptor. Funnily this is the place where
xorriso placed its hallmark:
   XORRISO-1.5.0 2018...
So the misplaced dd was effectively without effect.

-------------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Sat, Apr 20, 2019 at 06:32:07PM +0200, Thomas Schmitt wrote:
> We still need to find out whether the partition entry is the culprit.
> So please do the experiment with the Guix EFI partition image in
> partition 2 of the USB stick and
> 
>    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2
> 

Yes, this fixes the issue.


I tried current Git guix master with ludo's reproducibility patches.
Of course with the reproducibility patches the Macbook still gets
stuck before booting.

> Yes.
>   http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812
> has:
>       rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
>             efiimgfat, "::", NULL });
> 

I added "-k" to it (see attached ugly Guix patch).  This fixes the
issue for me.  (I do not know if this has other unintended side
effects and I can only try on other Macbooks next week.)

Regards,
Florian

[-- Attachment #2: use-gnu-packages-patches-grub-make-mkrescue-use-k-op.patch --]
[-- Type: text/plain, Size: 2073 bytes --]

From 0d266ebe33d03816d200e285d172e76ff641dca6 Mon Sep 17 00:00:00 2001
From: Florian Pelz <pelzflorian@pelzflorian.de>
Date: Sun, 21 Apr 2019 08:34:17 +0200
Subject: [PATCH] use
 gnu/packages/patches/grub-make-mkrescue-use-k-option.patch

---
 gnu/packages/bootloaders.scm                          |  3 ++-
 .../patches/grub-make-mkrescue-use-k-option.patch     | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/grub-make-mkrescue-use-k-option.patch

diff --git a/gnu/packages/bootloaders.scm b/gnu/packages/bootloaders.scm
index b4eabaea48..013b112592 100644
--- a/gnu/packages/bootloaders.scm
+++ b/gnu/packages/bootloaders.scm
@@ -91,7 +91,8 @@
                "03vvdfhdmf16121v7xs8is2krwnv15wpkhkf16a4yf8nsfc3f2w1"))
              (patches (search-patches "grub-check-error-efibootmgr.patch"
                                       "grub-binutils-compat.patch"
-                                      "grub-efi-fat-serial-number.patch"))))
+                                      "grub-efi-fat-serial-number.patch"
+                                      "grub-make-mkrescue-use-k-option.patch"))))
     (build-system gnu-build-system)
     (arguments
      `(#:phases (modify-phases %standard-phases
diff --git a/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch b/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch
new file mode 100644
index 0000000000..fbb4c5f596
--- /dev/null
+++ b/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch
@@ -0,0 +1,11 @@
+--- grub-2.02/util/grub-mkrescue.c	2019-04-21 08:17:18.880000000 +0200
++++ grub-2.02/util/grub-mkrescue.c	2019-04-21 08:17:52.588000000 +0200
+@@ -791,7 +791,7 @@
+ 
+       const char *fat_serial_number = getenv ("GRUB_FAT_SERIAL_NUMBER");
+       const char *mformat_args[] =
+-	{ "mformat", "-C", "-f", "2880", "-L", "16",
++	{ "mformat", "-k", "-C", "-f", "2880", "-L", "16",
+ 	  fat_serial_number != NULL ? "-N" : "-C",
+ 	  fat_serial_number != NULL ? fat_serial_number : "-C",
+ 	  "-i", efiimgfat, "::", NULL };
-- 
2.21.0

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > with the Guix EFI partition image in
> > partition 2 of the USB stick and
> >    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Florian Pelz wrote:
> Yes, this fixes the issue.

I really had a good feeling with that theory. :))

-----------------------------------------------------------------------

Ok, now for the last experiment. Just for curiosity, as i believe it
is not a good solution (and mformat -k isn't either):
Does start LBA 1 of partition 1 prevent the endless loop, too ?

So please write to byte 446 (dec) to 461 (dec) of the FAT image:

  80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

In the FAT image treated by the "dd" above, these bytes are all 00.
In the original FAT image they are:

  80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00
          ==                      ==              ==  ==

If this works, then we can blame a bug in the Macbook's EFI which is
too stupid to avoid endless recursion when exploring a potential
tree of partition tables.
(If it does not work, then we only have a heuristic indication that
 a partition table entry in the EFI partition is bad for the Macbook.)


Nevertheless, start LBA 1 is less desirable than no partition table
entry at all, because LBA 1 causes a partition that is not mountable.
So an option for mformat would be of interest, which writes an MBR
(other than with -k) but does not create a partition entry.

Alternatively, the Guix ISO production could apply above zeroizing dd
run to the EFI image by help of a pseudo-xorriso wrapper under
grub-mkrescue. This wrapper would learn the EFI image file address from
the xorriso options, patch that file, and then run xorriso with the
arguments given by grub-mkrescue.

(I will give my grub-mkrescue-sed.sh wrapper such a feature as addon-on
 to its other manipulation offers.)

----------------------------------------------------------------------

Reading UEFI 2.4, 12.3.1 "System Partition" i do not find statements
about partition tables inside that partition, but only:

  "A System Partition supports backward compatibility with legacy systems
   by reserving the first block (sector) of the partition for compatibility
   code. On legacy systems, the first block (sector) of a partition is
   loaded into memory and execution is transferred to this code.
   EFI firmware does not execute the code in the MBR. The EFI firmware
   contains knowledge about the partition structure of various devices,
   and can understand legacy MBR, GPT, and “El Torito.”

So an MBR-like start block of the partition is explicitely encouraged.
"understand"ing partitioning offers a wide range of interpretation.
So the behavior of the Macbook is not totally against specs.
It is just not overly stable and helpful.

"I am not crazy. Don't call me crazy. I...I'm just not user friendly."
Roberto, Futurama.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 11:35:49AM +0200, Thomas Schmitt wrote:
> Ok, now for the last experiment. Just for curiosity, as i believe it
> is not a good solution (and mformat -k isn't either):
> Does start LBA 1 of partition 1 prevent the endless loop, too ?
> 
> So please write to byte 446 (dec) to 461 (dec) of the FAT image:
> 
>   80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> 
> In the FAT image treated by the "dd" above, these bytes are all 00.
> In the original FAT image they are:
> 
>   80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00
>           ==                      ==              ==  ==
> 

Why is 0b underlined?

OK, so I just wrote the Guix git master with ludo’s reproducibility
patches to a USB drive (boot gets stuck again) and then did:

sudo dd if=/dev/sdc2 of=mysdc2.img

When I open it with ghex, I find at position 446:

    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

Strange.  I write the guixsd-install-0.16.0.x86_64-linux.iso to
/dev/sdc (boot gets stuck) and repeat

sudo dd if=/dev/sdc2 of=mysdc2.img

I find:

    80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00

This matches what you wrote.  I change it to

    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

I dd the changed mysdc2.img back to /dev/sdc2.

Now it boots. :)  Strangely, I now have only one entry “GRUB 2.02” in
the boot selection, but “EFI Boot” (or what it was called) is gone.



> Nevertheless, start LBA 1 is less desirable than no partition table
> entry at all, because LBA 1 causes a partition that is not mountable.
> So an option for mformat would be of interest, which writes an MBR
> (other than with -k) but does not create a partition entry.
> 

This raises the question what should be done in Guix.  Is it a good
idea to add -k to mformat in grub-mkrescue for the upcoming Guix 1.0
release (even though you don’t like it)?  Could it break install on
other machines?  Or should mformat be patched instead?  Could any of
this be upstreamed?  What about your MBR repacking?  I’m sure Ludo has
an opinion on these things too.  I just doing what I’m told, but I
don’t quite understand what I’m doing here.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i plan to ask at gru-devel about mformat, its MBR, and the partition entry,
as soon as the last experiment is made (LBA 1 rather than LBA 0 as start
of the inner partition).

For that discussion i could also need the exact model name of the Macbook.
If its EFI has a name, then this too.


Florian Pelz wrote:
> +--- grub-2.02/util/grub-mkrescue.c     2019-04-21 08:17:18.880000000 +0200
> ++++ grub-2.02/util/grub-mkrescue.c     2019-04-21 08:17:52.588000000 +0200
> [...]
> +-      { "mformat", "-C", "-f", "2880", "-L", "16",
> ++      { "mformat", "-k", "-C", "-f", "2880", "-L", "16",

The difference between the contents of the first image block with and
without option -k is significant, at least if we ask program "file".

If a special version of gub-mkrescue is acceptable, then i would rather
propose to apply the dd treatment to   efiimgfat  after mformat created
it without option -k.

We could ask at grub-devel for advise how to write 16 bytes to that file
without needing another shell command. (It's C language, after all.)

The binary differences between both blocks 0, except partition entry, are:

/efi.img from Guix 0.16.0
#########################
Image from mformat -k

00000000 :    eb  3c  90  4d  54  4f  4f  34  30  32  31  00  02  01  01  00
             ### ### ### ### ### ### ### ### ### ### ###
00000000 :    00  00  00  00  00  00  00  00  00  00  00  00  02  01  01  00

00000020 :    00  00  00  00  00  00  29  a7  48  66  33  4e  4f  20  4e  41
                                         ### ### ### ###
00000020 :    00  00  00  00  00  00  29  42  87  4b  7d  4e  4f  20  4e  41

00000030 :    4d  45  20  20  20  20  46  41  54  31  32  20  20  20  fa  31
                                                                     ### ###
00000030 :    4d  45  20  20  20  20  46  41  54  31  32  20  20  20  00  00

00000040 :    c0  8e  d8  8e  c0  fc  b9  00  01  be  00  7c  bf  00  80  f3
             ### ### ### ### ### ### ###     ### ###     ### ###     ### ###
00000040 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

00000050 :    a5  ea  56  00  00  08  b8  01  02  bb  00  7c  ba  80  00  b9
             ### ### ###         ### ### ### ### ###     ### ### ###     ###
00000050 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

00000060 :    01  00  cd  13  72  05  ea  00  7c  00  00  cd  19  00  00  00
             ###     ### ### ### ### ###     ###         ### ###
00000060 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

...

000001f0 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  55  aa
                                                                     ### ###
000001f0 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 01:16:14PM +0200, Thomas Schmitt wrote:
> For that discussion i could also need the exact model name of the Macbook.
> If its EFI has a name, then this too.
> 

I booted my ancient version of macOS Yosemite and clicked on the apple
symbol in the top left corner and About this Mac.

MacBook Pro (13-inch, Mid 2010)
[...]
Serial Number  WQ04501UATM

When I click on System Report, I see:

Model Name:            MacBook Pro
Model Identifier:      MacBookPro7,1
Boot ROM Version:      MBP71.003F.B00
SMC Version (system):  1.62f7

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > ... 3f  0b
> > ... 40  0b
> > ... ==  ==

Florian Pelz wrote:
> Why is 0b underlined?

Too much enthusiasm on my side.


> OK, so I just wrote the Guix git master with ludo’s reproducibility
> patches to a USB drive (boot gets stuck again) and then did:
> sudo dd if=/dev/sdc2 of=mysdc2.img
> When I open it with ghex, I find at position 446:
>    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

Pinching eyes ...

Bootable flag is set.
Start C/H/S is 4/0/0. (x/y/1 is usual)
Partition type is 1.
End C/H/S is 1/36/15.
Start LBA is 0x80000000 = 2,147,483,648.
Block count is 0x16 = 22.

So what ever did you do to this mysdc2.img ?


> I change it to
>    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> I dd the changed mysdc2.img back to /dev/sdc2.
>
> Now it boots. :)

Start C/H/S is 0/0/2.
Partition type is 1.
End C/H/S ...  i'm getting too old for decoding MBR C/H/S bit fields.
Start LBA is 1.
Block count is 0x0b3f = 2879.

So Start LBA 0 is indeed the trigger of the problem.
As soon as the partition entry does not point to its hosting block any
more, the (now very probable) loop in EFI cannot happen.


> Strangely, I now have only one entry “GRUB 2.02” in
> the boot selection, but “EFI Boot” (or what it was called) is gone.

What do you see if the partition entry is zeroized entirely ?

(Elsewise i refer to the Futurama quote in my previous mail.)


> Is it a good
> idea to add -k to mformat in grub-mkrescue for the upcoming Guix 1.0
> release (even though you don’t like it)?

We need to ask at grub-devel. I have begun to compose a mail.
You will be Cc-ed.
Does guix-devel want to be Cc-ed ?

New mail:
> MacBook Pro (13-inch, Mid 2010)
> Model Name:            MacBook Pro
> Model Identifier:      MacBookPro7,1
> Boot ROM Version:      MBP71.003F.B00
> SMC Version (system):  1.62f7

I will put this into my mail to grub-devel.

> Serial Number  ...

... but this only if asked for.


Back to old mail:
> Or should mformat be patched instead? Could any of
> this be upstreamed?

That's why i asked Ludovic about our chances with mtools upstream.
I would propose an option to write the usual pseudo-MBR but without that
partition entry.
(Well, maybe somebody there can even remember why a partition entry
 is made by default.)


> What about your MBR repacking?

I will create an option for zeroizing bytes 446 to 461 of the EFI image.

When the dust has settled here, i will ask Ludovic for preparing the
ISO production code for a run with libisoburn's wrapper script
frontend/grub-mkrescue-sed.sh for MBR-only.

At that occasion, the EFI image repairer could be tested, too.

The script has a mode "original" which does not change the xorriso
options submitted by grub-mkrescue. So it produces original GPT.
(It's original purpose is spying on grub-mkrescue's xorriso options.)


> I just doing what I’m told, but I
> don’t quite understand what I’m doing here.

We had a wild ride over boot sectors and partition tables.
I collect my knowledge in libisofs file doc/boot_sectors.txt .

At days when our web certificates are valid, it is available at
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/doc/boot_sectors.txt

Hah. Bug #4. No doc/boot_sectors.txt in libisofs tarball.
But it is in GNU xorriso's tarball:
  https://fossies.org/linux/misc/xorriso-1.5.1.tar.gz/xorriso-1.5.1/doc/boot_sectors.txt

For completeness, a reserve address for my grub-mkrescue-xorriso wrapper:
  https://sources.debian.org/src/libisoburn/1.5.0-1/frontend/grub-mkrescue-sed.sh


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 02:27:10PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > ... 3f  0b
> > > ... 40  0b
> > > ... ==  ==
> 
> Florian Pelz wrote:
> > Why is 0b underlined?
> 
> Too much enthusiasm on my side.
> 
> 
> > OK, so I just wrote the Guix git master with ludo’s reproducibility
> > patches to a USB drive (boot gets stuck again) and then did:
> > sudo dd if=/dev/sdc2 of=mysdc2.img
> > When I open it with ghex, I find at position 446:
> >    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00
> 
> Pinching eyes ...
> 
> Bootable flag is set.
> Start C/H/S is 4/0/0. (x/y/1 is usual)
> Partition type is 1.
> End C/H/S is 1/36/15.
> Start LBA is 0x80000000 = 2,147,483,648.
> Block count is 0x16 = 22.
> 
> So what ever did you do to this mysdc2.img ?
> 
> 

Ooops I am sorry, it is not

80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

but

80  00  01  00  04  01  24  4f  00  00  00  00  80  16  00  00  00



> We need to ask at grub-devel. I have begun to compose a mail.
> You will be Cc-ed.
> Does guix-devel want to be Cc-ed ?
> 

Thank you for sending a mail at
https://lists.gnu.org/archive/html/grub-devel/2019-04/msg00099.html

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> it is not
> 80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00
> but
> 80  00  01  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

This makes more sense as an MBR partition table entry:
Start C/H/S is 0/0/1.
Start LBA is 0.
Block count is 0x1680 = 5760 = 2.8 MB floppy.
The last 00 belongs to the next partition entry slot.

Overall it looks like the output of mformat -f 2880 without -k.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ady Ady gave me a hint about why the EFI partition in Guix 0.16.0 ISO
has 1.4 MB of size, despite grub-mkrescue asks mformat for 2.8 MB.

Obviously Guix uses mtools 4.0.21 (image OEM-ID "MTOO4021") which is one
version before this bug fix:
  "Fixed -f flag for mformat (size is KBytes, rather than sectors)"
  http://lists.gnu.org/archive/html/info-gnu/2018-12/msg00000.html

(Actually this size is quite some waste. Even 1.4 MB is generously
 oversized for the three directories and one or two boot*.efi.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

A friend’s “early 2014” MacBookAir6,2 does not have issues with the
old iso image.  It seems likely that only old Macbooks are affected.

On Sun, Apr 21, 2019 at 02:27:10PM +0200, Thomas Schmitt wrote:
> > I change it to
> >    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> > I dd the changed mysdc2.img back to /dev/sdc2.
> >
> > Now it boots. :)
> > […]
> > Strangely, I now have only one entry “GRUB 2.02” in
> > the boot selection, but “EFI Boot” (or what it was called) is gone.
> 
> What do you see if the partition entry is zeroized entirely ?
>

I see both “GRUB 2.02” and “EFI Boot” when I zero it (I have not tried
on Guix 0.16.0 this time though).

So the options are:

· Update mtools and then somehow patch mformat to zero out this
  region.

· Patch grub-mkrescue instead to make it use mformat -k.  Vladimir
  Serbinenko on the grub-devel list said this removes only information
  that is “Probably” not useful anyway, see
  https://lists.gnu.org/archive/html/grub-devel/2019-04/msg00100.html

· Ignore the issue.  The only affected machines we know are old Macbooks.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian pelaz wrote:
> So the options are:
> · Update mtools and then somehow patch mformat to zero out this
>  region.

I will hopefully tonight post a proposal for

- Leave grub-mkrescue and mformat as they are. Rather do the partition
  entry removal in the course of Guix ISO production.

It will be windfall profit of my detailed proposal for creating ISOs
with MBR partition table and Linux mountable ISO partition.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
>> On the 2011 Macbook I am using, the boot process gets stuck before I
>> can select the drive to boot from, as long as a USB flash drive with
>> this GPT error is in a USB port.  The few other computers I tried have
>> no problem with booting from the flash drive.
>> 
>
> Actually it still gets stuck after I used fdisk to repair the drive’s
> GPT.  I will check again with a Debian ISO tomorrow.

Does it boot the 0.16.0 installation image?  It was built in exactly the
same way, with the misplaced GPT backup.

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 02:24:12PM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> 
> > On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
> >> On the 2011 Macbook I am using, the boot process gets stuck before I
> >> can select the drive to boot from, as long as a USB flash drive with
> >> this GPT error is in a USB port.  The few other computers I tried have
> >> no problem with booting from the flash drive.
> >> 
> >
> > Actually it still gets stuck after I used fdisk to repair the drive’s
> > GPT.  I will check again with a Debian ISO tomorrow.
> 
> Does it boot the 0.16.0 installation image?  It was built in exactly the
> same way, with the misplaced GPT backup.
> 

No, it does not boot.  I believe when I installed this Guix System, I
used `guix system disk-image gnu/system/install.scm` without
specifying --file-system-type=iso9660.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

after the adventure of diagnosing Macbook EFI and mformat, i want to
return to the original topic of this thread:
I deem it worthwhile to offer the opportunity to create the Guix ISO
with MBR partition table rather than with GPT.

If you just want to see my technical proposal, then hop to the next
section titled "How to do it".

The advantages of my proposal are:

- No GPT backup which needs to be relocated after the ISO was put
  onto USB stick.

- Mountable ISO 9660 partition. (The GPT of a pure grub-mkrescue ISO
  offers a HFS+ partition for mounting. But afaik, Linux hates the
  block size 2048.)

A disadvantage is that for best overall properties, this layout should
have the ISO partition start at 512-byte block 64. This needs a second
superblock and directory tree. About 17 MiB with Guix 0.16.0.

If the size increase matters, then i propose to sacrifice the HFS+
filesystem tree as compensation.
It is for Macs even older than Florian's.

Without that non-zero partition start, the ISO superblock (PVD) of the
overall image cannot claim the appended EFI partition 2 as part of its
filesystem. That's because the superblock is mountable as partition 1
and a partition filesystem should not be larger than its partition.

But with partition start at LBA 64, the overall superblock is not
mountable via a partition and thus free to claim as much space as it
wants.
So programs like /sbin/isosize can tell the image size even when the
ISO has already been copied to USB stick.

Partition editors, too, will love a partition start after the partition
table rather than at block 0.

----------------------------------------------------------------------

How to do it:

We can achieve the MBR partition table based layout by a script which
is part of libisoburn. It acts as pseudo-xorriso between grub-mkrescue
and real xorriso.
Yesterday i improved it by a partition table killer for the EFI image:
  MKRESCUE_SED_IN_EFI_NO_PT=yes.

For experiments it is best to download it from upstream.
Currently the https certificate is not valid. So for now unsafe:

  wget --no-check-certificate \
       https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

Size of today's version is 11077 bytes.
SHA512 is
  23cbc25aac8e03de82bd6e76b00a6ad249629c19822728616d6144f075dbb78f69811648559846ba33319c38baa21515e4a3a0f7bb3ca86f2427e887592101cf
(Ask if you get a different checksum. Maybe i'll make more changes.)

------------------------------------------------------------------------

In a shell script it would be used like this:

  export MKRESCUE_SED_MODE=mbr_hfs
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ... \
                ... with or without a -- switch to native command mode ... \
                -- \
                -boot_image any partition_offset=16 \
                -boot_image any iso_mbr_part_type=0x83

The result (with nearly no payload) looks like

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  34456

The HFS+ superblock and directory tree will be marked by a partition
in an Apple Partition Map. From xorriso -report_system_area :

  APM                :   N  Info
  APM block size     :      2048
  APM gap fillers    :      1
  APM partition name :   1  Gap0
  APM partition type :   1  ISO9660_data
  APM start and size :   1  16  150
  APM partition name :   2  HFSPLUS_Hybrid
  APM partition type :   2  Apple_HFS
  APM start and size :   2  166  7008

------------------------------------------------------------------------

If HFS+ shall be omitted for size reduction or other reasons, use
a different mode of grub-mkrescue-sed.sh :

  export MKRESCUE_SED_MODE="mbr_only"

rather than MKRESCUE_SED_MODE="mbr_hfs".
The result will be smaller by the size of the additional HFS+ tree.

  $ expr $(/sbin/isosize output.iso) / 512
  33864

(The true cost of HFS+ and second ISO directory tree will show up only
 with much more files in the trees.)

------------------------------------------------------------------------

The original GPT layout with additionally zeroed inner EFI partition
table can be achieved by:

  export MKRESCUE_SED_MODE=original
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ...

(Partition offset and MBR partition type are of no use with "original".)

The result is

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: gpt
  ...
  Device      Start   End Sectors  Size Type
  output.iso1    64   339     276  138K Microsoft basic data
  output.iso2   340  6099    5760  2.8M EFI System
  output.iso3  6100 34131   28032 13.7M Apple HFS/HFS+
  output.iso4 34132 34731     600  300K Microsoft basic data

  $ expr $(/sbin/isosize output.iso) / 512
  34780

------------------------------------------------------------------------

Another interesting layout is "gpt_appended". It moves the EFI partition
out of the ISO filesystem, as does "mbr_only". But it marks the partitions
in GPT. The ISO partition is mountable. Two directory trees are created
automatically.

  export MKRESCUE_SED_MODE=gpt_appended
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ...

Result:

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: gpt
  ...
  Device      Start   End Sectors  Size Type
  output.iso1    64 28103   28040 13.7M Microsoft basic data
  output.iso2 28104 33863    5760  2.8M EFI System

  $ expr $(/sbin/isosize output.iso) / 512
  33928

(With the next version of xorriso it will be possible to choose
   -boot_image any iso_mbr_part_type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
 for "Linux filesystem data".)

------------------------------------------------------------------------

As can be seen with MKRESCUE_SED_IN_EFI_NO_PT, such a pseudo-xorriso
program between grub-mkrescue and xorriso gives the caller control
over the stage between GRUB file preparation and ISO production run.

For example, it would be possible to make a smaller FAT image, copy
the files from grub-mkrescue's EFI images over to the small one,
and then use that as EFI partition.

It would of course be possible to create such a program in Guile.
Be invited to pick from grub-mkrescue-sed.sh what you need.

--------------------------------------------------------------------
Somewhat off topic:

The script also has a killer for the MBR signature in the EFI image
file. It gets activated together with partition table erasure by
  export MKRESCUE_SED_IN_EFI_NO_PT=extra
It is meant for those users who are curious whether block 0 of an EFI
partition really must look like an MBR to be accepted by EFI firmware.

It would be interesting to learn about EFIs which fail without MBR
signature.

The specs say that there may be "compatibility code" in a EFI System
Partition. They talk of "MBR" and understanding partitions in the EFI
partition. I understand it as "may be there, but does not have to".

Minimal test outside of Guix ISO production:

- Install GRUB and configure it for EFI for 32 or 64 bit as your firmware
  needs. (Both is combinable, as Debian GRUB packages demonstrate.)

- Create a minimal bootable ISO. It will not offer more than a GRUB
  command prompt after it was successfully booted:

    mkdir minimal
    touch minimal/empty-file.txt

    export MKRESCUE_SED_MODE=original
    export MKRESCUE_SED_IN_EFI_NO_PT=extra

    grub-mkrescue -o output.iso minimal \
                  --xorriso=./grub-mkrescue-sed.sh

- Put the resulting output.iso on USB stick and offer it to your firmware
  at boot time.
  Is it not found by the EFI boot manager ?
  If you let it boot, does the machine not show a GRUB prompt in the end ?

- Negative results with other mode settings in MKRESCUE_SED_MODE would be
  interesting, too.

--------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

This sounds nice and clean.  Thank you!

On Tue, Apr 23, 2019 at 08:14:48PM +0200, Thomas Schmitt wrote:
> A disadvantage is that for best overall properties, this layout should
> have the ISO partition start at 512-byte block 64. This needs a second
> superblock and directory tree. About 17 MiB with Guix 0.16.0.
> 

The ISO files are offered on alpha.gnu.org only with xz compression.
I suppose most of the 17 MiB would disappear with compression.


> So programs like /sbin/isosize can tell the image size even when the
> ISO has already been copied to USB stick.
> 

Sounds like this could be used to get a checksum or maybe GPG
verification of a USB drive that supposedly contains the iso.



> Somewhat off topic:
> 
> The script also has a killer for the MBR signature in the EFI image
> file. It gets activated together with partition table erasure by
>   export MKRESCUE_SED_IN_EFI_NO_PT=extra
> It is meant for those users who are curious whether block 0 of an EFI
> partition really must look like an MBR to be accepted by EFI firmware.
> 
> It would be interesting to learn about EFIs which fail without MBR
> signature.
> 
> The specs say that there may be "compatibility code" in a EFI System
> Partition. They talk of "MBR" and understanding partitions in the EFI
> partition. I understand it as "may be there, but does not have to".
> 
> Minimal test outside of Guix ISO production:
> 
> - Install GRUB and configure it for EFI for 32 or 64 bit as your firmware
>   needs. (Both is combinable, as Debian GRUB packages demonstrate.)
> 
> - Create a minimal bootable ISO. It will not offer more than a GRUB
>   command prompt after it was successfully booted:
> 
>     mkdir minimal
>     touch minimal/empty-file.txt
> 
>     export MKRESCUE_SED_MODE=original
>     export MKRESCUE_SED_IN_EFI_NO_PT=extra
> 
>     grub-mkrescue -o output.iso minimal \
>                   --xorriso=./grub-mkrescue-sed.sh
>

This prints

florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal --xorriso=./grub-mkrescue-sed.sh
grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'. Some features are disabled. Please use xorriso 1.2.9 or later..

and yields no output.iso for me.  Note that xorriso actually has
version 1.5.0.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal
> --xorriso=./grub-mkrescue-sed.sh
> grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'.
> Some features are disabled. Please use xorriso 1.2.9 or later..
>
> and yields no output.iso for me.  Note that xorriso actually has
> version 1.5.0.

Does the script offer x-permission ?
(Shame on me for not thinking of this in my download instructions.)

If so, what do you get from

  ./grub-mkrescue-sed.sh -as mkisofs -help 2>&1 | fgrep grub2-boot-info

If it says

  --grub2-boot-info           Patch boot image at byte 2548

then i wonder why grub-mkrescue is not satisfied in
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n608


I wrote:
> > So programs like /sbin/isosize can tell the image size even when the
> > ISO has already been copied to USB stick.

> Sounds like this could be used to get a checksum or maybe GPG
> verification of a USB drive that supposedly contains the iso.

That's what this property is advised for in Debian's CD FAQ.
One can read trailing garbage not only from USB sticks but also from
most DVD types.
(The layout with nested partitions obsoletes the cleanliness
 considerations about partition start at LBA 0.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 23, 2019 at 10:18:10PM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal
> > --xorriso=./grub-mkrescue-sed.sh
> > grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'.
> > Some features are disabled. Please use xorriso 1.2.9 or later..
> >
> > and yields no output.iso for me.  Note that xorriso actually has
> > version 1.5.0.
> 
> Does the script offer x-permission ?
> (Shame on me for not thinking of this in my download instructions.)
> 

This was the error.  It works after `chmod +x grub-mkrescue-sed.sh`.


On Tue, Apr 23, 2019 at 08:14:48PM +0200, Thomas Schmitt wrote:
> It would be interesting to learn about EFIs which fail without MBR
> signature.
> […]
> - Put the resulting output.iso on USB stick and offer it to your firmware
>   at boot time.
>   Is it not found by the EFI boot manager ?
>   If you let it boot, does the machine not show a GRUB prompt in the end ?
> 

It shows up twice in the boot selection screen as “GRUB 2.02” and “EFI
Boot” just like the Guix USB drive before.  It is bootable and shows a
GRUB prompt on my Macbook.



> - Negative results with other mode settings in MKRESCUE_SED_MODE would be
>   interesting, too.
> 

I can try tomorrow.

By the way, someone I know has a most peculiar machine called Lenovo
Ideapad 100S which does not boot Guix, but apparently is very picky in
general (cf. <https://forums.linuxmint.com/viewtopic.php?t=216476>).
Its CPU is claimed to support 64-bit, but its boot firmware rejects
most 64-bit isos.  I wonder if we could get that machine to boot Guix
(at least it currently cannot boot
guixsd-install-0.15.0.i686-linux.iso), but that may well be a waste of
time.  It might just not be feasible to please every EFI system.



> I wrote:
> > > So programs like /sbin/isosize can tell the image size even when the
> > > ISO has already been copied to USB stick.
> 
> > Sounds like this could be used to get a checksum or maybe GPG
> > verification of a USB drive that supposedly contains the iso.
> 
> That's what this property is advised for in Debian's CD FAQ.
> One can read trailing garbage not only from USB sticks but also from
> most DVD types.
> (The layout with nested partitions obsoletes the cleanliness
>  considerations about partition start at LBA 0.)
>

So… GPT is wrong for some optical media too?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > > > So programs like /sbin/isosize can tell the image size

> > (The [Debian] layout with nested partitions obsoletes the cleanliness
> >  considerations about partition start at LBA 0.)

Florian Pelz wrote
> So… GPT is wrong for some optical media too?

Not in this aspect. GPT partitions must not start at block 0.
So the overall ISO superblock is not mountable as partition and thus
free to claim the full image size as filesystem size.

The problem of restriction to partition size arises with MBR and
start block 0 of partition 1, when the nested partition mess of
"isohybrid -u" is untangled.


> By the way, someone I know has a most peculiar machine called Lenovo
> Ideapad 100S which does not boot Guix, but apparently is very picky in
> general (cf. <https://forums.linuxmint.com/viewtopic.php?t=216476>).
> Its CPU is claimed to support 64-bit, but its boot firmware rejects
> most 64-bit isos.

If Guix does not show up in the boot manager, try a plain grub-mkrescue
image that was made when GRUB was configured for both EFI variations:
32 bit x86 and 64 x86.
My experiments are made when it is configured for those two plus PC-BIOS.
The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.


> It might just not be feasible to please every EFI system.

Let's see how far we get. This research not only helps Guix but also
GRUB. (And it has a tendency to reveal my bugs in xorriso.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 24, 2019 at 08:56:57AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote
> > So… GPT is wrong for some optical media too?
> 
> Not in this aspect. GPT partitions must not start at block 0.
> So the overall ISO superblock is not mountable as partition and thus
> free to claim the full image size as filesystem size.
>

Sorry, I meant to quote this aspect:

> > One can read trailing garbage not only from USB sticks but also from
> > most DVD types.
> > […]
> >
> 
> So… GPT is wrong for some optical media too?
> 


> > By the way, someone I know has a most peculiar machine called Lenovo
> > Ideapad 100S which does not boot Guix, […]
> 
> If Guix does not show up in the boot manager, try a plain grub-mkrescue
> image that was made when GRUB was configured for both EFI variations:
> 32 bit x86 and 64 x86.
> My experiments are made when it is configured for those two plus PC-BIOS.
> The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.
> 

Will do.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Sorry, I meant to quote this aspect:

I wrote:
> > > One can read trailing garbage not only from USB sticks but also from
> > > most DVD types.

Florian Pelz wrote:
> > So… GPT is wrong for some optical media too?

I wrote:
> Not in this aspect. GPT partitions must not start at block 0.
> So the overall ISO superblock is not mountable as partition and thus
> free to claim the full image size as filesystem size.

I already understood you correctly ... i think. :))

GPT partitioned ISO are not hampered from telling the full image
size as size of the ISO 9660 filesystem.

  $ /sbin/gdisk -l guixsd-install-0.15.0.i686-linux.iso
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     1              64           34107   16.6 MiB    0700  Gap0
     2           34108           39867   2.8 MiB     EF00  EFI boot partition
     3           39868         1815727   867.1 MiB   AF00  HFSPLUS
     4         1815728         1816327   300.0 KiB   0700  Gap1

  $ expr $(/sbin/isosize guixsd-install-0.15.0.i686-linux.iso) / 512
  1816376

The isosize is a bit larger than the end of partition 4, because after
this partition comes the GPT backup.
The isosize matches the image file size:

  $ expr $(ls -l guixsd-install-0.15.0.i686-linux.iso | awk '{print $5}') / 512
  1816376


But an MBR partition which begins by block 0 and is mountable as ISO 9660
filesystem keeps its ISO superblock from telling a size larger than the
partition. So it cannot claim an appended partition 2 as part of its range.

  $ export MKRESCUE_SED_MODE=mbr_hfs
  $ grub-mkrescue --xorriso=./grub-mkrescue-sed.sh -o output.iso minimal \
                  --iso_mbr_part_type 0x83
  ...

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *        0 28371   28372 13.9M 83 Linux
  output.iso2      28372 34131    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  28372

This discrepancy of image file size and isosize is undesirable when the
image file itself cannot say its size any more, because it is on USB
stick or a DVD which delivers more bytes than were written as image.

So i propose partition offset 16 to get the partition superblock away
from being mountable by the base device.

  $ export MKRESCUE_SED_MODE=mbr_hfs
  $ grub-mkrescue --xorriso=./grub-mkrescue-sed.sh -o output.iso minimal \
                  -partition_offset 16 --iso_mbr_part_type 0x83

This yields

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

The ISO filesystem of the overall image then claims the image file size

  $ expr $(/sbin/isosize output.iso) / 512
  34456

With the partition's ISO filesystem we only get the partition size:

  $ dd if=output.iso bs=512 skip=64 of=partition1.iso

  $ expr $(/sbin/isosize partition1.iso) / 512
  28632

This is how it should be.


The ISOs of Fedora, Debian, Ubuntu, and others are MBR partitioned and
most of them have no partition offset 16. But despite that fact they can
claim the full image size as ISO 9660 filesystem size.

This is possible because partition 1 covers the whole image and partition 2
is located inside partition 1. Strictly illegal in UEFI specs, unless
the outer partition has MBR partition type 0x00 and thus does not exist
for EFI and some partition editors. (I.e. this is a really dirty hack.)

  $ /sbin/fdisk -l debian-live-9.8.0-amd64-xfce.iso
  ...
  Disklabel type: dos
  ...
  Device                            Boot Start     End Sectors  Size Id Type
  debian-live-9.8.0-amd64-xfce.iso1 *        0 3811391 3811392  1.8G  0 Empty
  debian-live-9.8.0-amd64-xfce.iso2       1432    2263     832  416K ef EFI (FAT-1

  $ expr $(/sbin/isosize debian-live-9.8.0-amd64-xfce.iso) / 512
  3811392

Old /sbin/gdisk only reports partition 2:

  $ /sbin/gdisk -l debian-live-9.8.0-amd64-xfce.iso
  ...
  Found valid MBR and GPT. Which do you want to use?
  ...
  Your answer: 1
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     2            1432            2263   416.0 KiB   EF00  EFI System



Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Hi Thomas,

thanks for the investigative work you are doing.

If someone is testing this with Guix, make sure to actually try to boot Guix
with it (until the root is mounted).  Back in the day I changed the root
discovery of Guix to make it also consider using the entire disk (as opposed to
a partition on it) as a viable root device (see guix master,
commit 9833bcfc08ef009b9e8b4398baa481ef65c80ad7).
That could maybe make it choke if there are multiple partitions/disks with
equal labels (or whatever field it's searching for) on it (though probably not--
as long as its immaterial which of the partitions/disks with the equal field
values is mounted).

FWIW, I'm all for having a MBR-only Guix installer disk as you suggest.
But I'd rather not do the hack the others are doing.  It's hard enough to
debug this stuff when everyone is playing nice.

How should we mount the root when the right way (with two non-overlapping
partitions) boots from DVD ?

(I'm surprised that it needs so much manual intervention to make grub-mkrescue
do the right thing.  One of the reasons I chose grub-mkrescue was that I
assumed that its output would boot fine on many different system types--
it being a rescue disk creator and all.  I guess there are tradeoffs to be
made...)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Danny Milosavljevic wrte:
> If someone is testing this with Guix, make sure to actually try to boot Guix
> with it (until the root is mounted). Back in the day I changed the root
> discovery of Guix to make it also consider using the entire disk (as opposed
> to a partition on it) as a viable root device (see guix master,
> commit 9833bcfc08ef009b9e8b4398baa481ef65c80ad7).
> That could maybe make it choke if there are multiple partitions/disks with
> equal labels (or whatever field it's searching for) on it (though probably
> not--
> as long as its immaterial which of the partitions/disks with the equal field
> values is mounted).

Volume Id and other superblock properties of the two possible ISO 9660 mount
candidates are supposed to be identical. So on USB stick there is indeed
an ambiguity, which superblock to choose (/dev/sdc versus /dev/sdc1).

But both candidates offer identical directory trees with identical file
properties and content. So it should make no difference which one gets
mounted.
The difference is in the block addresses by which superblock and directory
tree refer to file objects. They have to differ by 16. (Block size 2048.)

Unaware interpreters of hard disk partitioning will hardly consider
the base device for mounting but rather resort to the partitions.
Aware readers, like software in the ISO's initrd, will normally ignore
the partition table and act like on DVD.
But as said, in the end it does not matter which ISO gets mounted.


> FWIW, I'm all for having a MBR-only Guix installer disk as you suggest.

It should at least be tested.
Best with an option of the build process by which the user can choose
between "original", "mbr_hfs" + partition offset 16, "mbr_only" + offset,
and "gpt_appended".

Default should be "original" for now, but with the goal to switch to
"mbr_hfs" or "mbr_only" when enough courageous testers have reported
success.


> But I'd rather not do the hack the others are doing.

I am nagging against the Fedora-inspired layout since years. :))


> How should we mount the root when the right way (with two non-overlapping
> partitions) boots from DVD ?

On DVD there is no public offer of the partition's filesystem.
At least the Linux kernel is not supposed to look for partition tables
on optical media.
(Actually ioctl BLKRRPART is so CD-phobic that it does not even refresh
 the size perception of the medium but simply bails out.
 Else it would be a nice gesture after DVD burning to do
   hdparm -z /dev/sr0
 growisofs emulates this by putting out the tray and pulling it in again,
 if there is a tray motor. Watch your fingers.)


> I'm surprised that it needs so much manual intervention to make
> grub-mkrescue do the right thing.

It does the right thing for the main purpose of creating a widely bootable
ISO image. (The fact that Florian's Macbook takes offense from the EFI
partition image is independent of the partition layout.)

The complaints of Linux at partition assessment time are ugly but harmless.
The lack of a Linux-mountable ISO partition among the four GPT partitions
is ugly, too. But the user can at any time mount the base device of the
USB stick.

grub-mkrescue's partition layout becomes really suboptimal when the user
wants to use the remaining capacity of the USB stick for one or more
read-write data partitions.
Partition editors righteously complain about the misplaced GPT backup,
which cannot be avoided because the image does not get produced for a
storage device of a particular size.

Similarly suboptimal for partition editors is the Fedora-inspired layout
as produced by "isohybrid -u" or "xorrisofs ... -isohybrid-gpt-basdat".
Other than grub-mkrescue's GPT, this layout is barely legal.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

So what’s the plan?

Wait for another responce from the grub-devel mailing list?

Package grub-mkrescue-sed.sh and use
--xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?

Add an option to “guix system disk-image” to select which
grub-mkrescue-sed.sh environment variables to enable?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> So what’s the plan?
> Wait for another responce from the grub-devel mailing list?

They need time to to make up their mind. Maybe other incidents are needed to
push towards some change in grub-mkrescue.
We have a halfways positive response from the old owner and a somewhat
uninformed response from the new manager, indicating between the lines
that he is too busy with other things currently.


> Package grub-mkrescue-sed.sh and use
> --xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?

It is part of the libisoburn release tarball, from where Guix gets its
xorriso, afaik. It is also part of the GNU xorriso tarball, which builds
a static compilation of libburn, libisofs, libisoburn, and libjte.

Problem is that for the EFI partition fix, you need the current development
version. The partition definitions should be ok with the released version
1.5.0.
I plan a new release this year, but not right now.

How comfortable is the Guix patching system ? :))

  "New control variable for grub-mkrescue-sed.sh: MKRESCUE_SED_IN_EFI_NO_PT"
  https://dev.lovelyhq.com/libburnia/libisoburn/commit/3a2a3ba737a06162c22ace0ae09d33ba97aa2673

  "grub-mkrescue-sed.sh: MKRESCUE_SED_IN_EFI_NO_PT="extra"
   to erase MBR signature"
  https://dev.lovelyhq.com/libburnia/libisoburn/commit/1eb51f44dadb8b6c5f87533ca357186cdc1ac625

(The holidays are over. Our certificate works again.)


Looking at a commit from 2 years ago (i.e. in released version)
  "Made -iso_mbr_part_type 0x00 default of grub-mkrescue-sed.sh
   mode "mbr_only""
i remember that there was an EFI reported on grub-devel which strictly
refused on anything but a single MBR partition for EFI. So the camouflage
trick of partition type 0x0 was necessary for the ISO partition.

This is quite contrary to the expectations of partition editors, though.
I deem this EFI's behavior a much clearer bug than the Macbook EFI's.
So i stay with my proposal of native xorriso command
  -boot_image any iso_mbr_part_type=0x83
or mkisofs emulation option
  -iso_mbr_part_type 0x83

(To my knowledge, the Guix xorriso run switches from mkisofs emulation
 to native commands by "--". So -boot_image would be the one to use
 if it gets appended to the other arguments.)


> Add an option to “guix system disk-image” to select which
> grub-mkrescue-sed.sh environment variables to enable?

This would be great.
(I was brought up with HP BASIC and never was able to solve the Lisp
 puzzles in german magazine Bild der Wissenschaft of the 1980s.
 So i cannot help much with translating the usage gestures or even
 the script itself from shell to Guile.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4507 bytes --]

> How comfortable is the Guix patching system ? :))

Very comfortable.  But we'd like not to diverge from upstream too much.
We are all in this together, so I'd like upstream to eventually converge on
a solution.  I'm fine with carrying a patch if it's coordinated by people with
knowledge about the matter at hand.  For multiple projects, we already carry
patches while they are being actively reviewed upstream in order to fix bugs.

If the problem is not too bad, I'd like the change to take the normal path
instead of us interfering with upstream.  But here, you *are* upstream so
it's really up to you whether we should patch xorriso or not.

As for people testing possible avenues by patches in Guix, that's easy to do.

> This is quite contrary to the expectations of partition editors, though.
> I deem this EFI's behavior a much clearer bug than the Macbook EFI's.
> So i stay with my proposal of native xorriso command
>   -boot_image any iso_mbr_part_type=0x83
> or mkisofs emulation option
>   -iso_mbr_part_type 0x83
> 
> (To my knowledge, the Guix xorriso run switches from mkisofs emulation
>  to native commands by "--". So -boot_image would be the one to use
>  if it gets appended to the other arguments.)
> 
> 
> > Add an option to “guix system disk-image” to select which
> > grub-mkrescue-sed.sh environment variables to enable?  
> 
> This would be great.

Since grub-mkrescue-sed.sh is part of xorriso, it's easy to get it
to where it needs to be (gnu/build/vm.scm) from where xorriso is
refererred to (gnu/system/vm.scm).  (Just pass the package as argument)
I can do that part.

The patches would then go into the xorriso package definition we carry
(gnu/packages/cdrom.scm) and modify grub-mkrescue-sed.sh in its output
(the patching part would require no Guile knowledge, they are just
".patch" files we distribute and automatically apply).

I'm not sure about amending "guix system disk-image" by options that
aren't really end user options, but I guess it can't be helped.

Basically they would just be something that we'd tell users which
have problems booting the Guix iso to use.  Even that is a lot to ask
from normal users--building a new installer ISO for the sake of
installing Guix.

> (I was brought up with HP BASIC and never was able to solve the Lisp
>  puzzles in german magazine Bild der Wissenschaft of the 1980s.
>  So i cannot help much with translating the usage gestures or even
>  the script itself from shell to Guile.)

Oh, I see.  It's fine if it's in a shell script.

We aren't opposed to shell scripts--sometimes (in initrd etc) we already have
Guile there and don't want to bloat the image by shells and shell utils.

But in this case, we would be patching the iso generator which requires
a lot of the system anyway, so who cares if we pull in the shell and 20
small utils?

Also, we try very hard to keep packages modular.

But if no care is taken, shell scripts take stuff from the global environment
and thus are very difficult to predict and isolate for modularity.

Think of what happens when someone replaces the "test" binary
in $PATH, or "sed".  It will unintentionally potentially completely
change the output depending on what the user has installed (it's definitely
not what the user intended to happen).

As a first countermeasure, we build inside isolated build containers, which
makes the problem less bad.  But then grub-mkrescue will invoke
grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
as the sed to use?

(For that matter, think of what happens when another xorriso is somewhere
in $PATH and we use --xorriso=.../grub-mkrescue-sed.sh).  (I see that
you already take care to use a "closer" xorriso--but these kind of problems
pop up in shell scripts for almost every term used in there.  It's kinda
difficult to get them to behave)

What's more, shell scripts don't fail on error.  For example in
grub-mkrescue-sed.sh there's no "-e" in the shebang, which means that if
some command in there has an error, the shell will happily continue anyway.
I'm not blaming you, that's just the culture with shell scripts.

Even with "-e", if a command in a pipeline fails, it will sometimes still
not fail the shell script.  For that, you need "set -o pipefail" which is
a bash feature and not necessarily available on other shells (dash etc).

Guile, on the other hand, will just fail the script on error (usually).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Danny Milosavljevic wrote:
> you *are* upstream so
> it's really up to you whether we should patch xorriso or not.

Then do if you can make use of those two git commits.
They are the only ones to that file since release 1.5.0.


> I'm not sure about amending "guix system disk-image" by options that
> aren't really end user options, but I guess it can't be helped.

How about

  --file-system-type=iso9660[_$variation]

with $variation being "gpt_hfs" (= "original"), "mbr_hfs", "mbr_only",
and "gpt_appended", defaulting to "gpt_hfs" ?

This translates quite straightforward to the proposed variations of
grub-mkrescue-sed.sh in
  http://lists.gnu.org/archive/html/guix-devel/2019-04/msg00420.html

The partition offset 16 and MBR partition type 0x83 would be ordered
only with "mbr_*" variations.

The setting
  export MKRESCUE_SED_IN_EFI_NO_PT=yes
should be used unconditionally. It gets applied to the EFI partition
block 0, only if it is marked with an MBR signature 55 aa. Then we may
safely assume that bytes 446 to 509 are MBR partition table, which is
of no use and maybe harmfull. Only then they get zeroed.

The experimental setting
  export MKRESCUE_SED_IN_EFI_NO_PT=extra
is not yet sufficiently motivatable for production ISOs. It exists for
getting an impression whether mformat option -k would be safe in
grub-mkrescue. (I.e. it is research for general GRUB enlightenment.)


> Basically they would just be something that we'd tell users which
> have problems booting the Guix iso to use.

If the Guix download servers can afford two more ISOs, how about offering
"gpt_hfs" and "mbr_hfs" ISOs with the explanation that both should work
for booting and installing, but that "mbr_hfs" is more convenient when
the remaining capacity of a Guix ISO USB stick shall be used as partition
for a read-write filesystem.
(Moving the backup GPT is an expert option in fdisk. But without moving
 it there is no room for a new partition.)

On the long run and if no negative results arise, i propose to switch the
default to "mbr_hfs".


> But if no care is taken, shell scripts take stuff from the global
> environment and thus are very difficult to predict and isolate for
> modularity.

I tried hard to keep it conservative and am open for proposals to
make it more portable to older shells.


> But then grub-mkrescue will invoke
> grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
> as the sed to use?

grub-mkrescue runs the mformat program and the mcopy program before
it runs xorriso.


> (For that matter, think of what happens when another xorriso is somewhere
> in $PATH and we use --xorriso=.../grub-mkrescue-sed.sh).

Should not matter much, as long as this xorriso is young enough.
grub-mkrescue will ask by running effectively
  ./grub-mkrescue-sed.sh -as mkisofs -help 2>&1
and inspecting the options list of the real xorriso involved.

The only difference will be two message lines (one empty)
  frontend/grub-mkrescue-sed.sh manipulating xorriso arguments

but then it will say
  xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

  Usage: xorriso -as mkisofs [options] file...
  ...

If you can tell the absolute path to a good xorriso, then set

  export MKRESCUE_SED_XORRISO=...path.to.real.xorriso.binary...


> For example in grub-mkrescue-sed.sh there's no "-e" in the shebang

Is it conservative enough to add it ?


> Guile, on the other hand, will just fail the script on error (usually).

That's why i brought a Guile re-write into consideration.
Firstly it would be the language of choice and secondly it would make
Guix independent of my ideas how it should be done. As said, the
script-in-the-middle gives substantial control over the stage of
grub-mkrescue when the prepared files are still not packed up in the ISO.

(Or maybe it's time to re-write the script in C ... as was done with
 grub-mkrescue when it became too complicated.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4253 bytes --]

Hi Thomas,

On Thu, 25 Apr 2019 18:22:11 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Then do if you can make use of those two git commits.
> They are the only ones to that file since release 1.5.0.

Okay, I've picked them up.

> > I'm not sure about amending "guix system disk-image" by options that
> > aren't really end user options, but I guess it can't be helped.  
> 
> How about
> 
>   --file-system-type=iso9660[_$variation]
> 
> with $variation being "gpt_hfs" (= "original"), "mbr_hfs", "mbr_only",
> and "gpt_appended", defaulting to "gpt_hfs" ?

I'm not sure yet.  We have a lot of special-casing for iso9660
already.  If anything, at that point, we could pass an arbitrary list
of options or something (an "environment" if you will.  Hah).

> This translates quite straightforward to the proposed variations of
> grub-mkrescue-sed.sh in
>   http://lists.gnu.org/archive/html/guix-devel/2019-04/msg00420.html
> 
> The partition offset 16 and MBR partition type 0x83 would be ordered
> only with "mbr_*" variations.
> 
> The setting
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
> should be used unconditionally. It gets applied to the EFI partition
> block 0, only if it is marked with an MBR signature 55 aa. Then we may
> safely assume that bytes 446 to 509 are MBR partition table, which is
> of no use and maybe harmfull. Only then they get zeroed.

Okay, that makes sense.

> If the Guix download servers can afford two more ISOs, how about offering
> "gpt_hfs" and "mbr_hfs" ISOs with the explanation that both should work
> for booting and installing, but that "mbr_hfs" is more convenient when
> the remaining capacity of a Guix ISO USB stick shall be used as partition
> for a read-write filesystem.
> (Moving the backup GPT is an expert option in fdisk. But without moving
>  it there is no room for a new partition.)

I don't know these things.  But someone on the list will know.

> On the long run and if no negative results arise, i propose to switch the
> default to "mbr_hfs".

Yeah, I agree.

> I tried hard to keep it conservative and am open for proposals to
> make it more portable to older shells.

The problem is that there were horrible security problems in bash and that
made a lot of distributions switch to a more minimal shell ("dash") for
scripts.  I agree that that was a good move, but error handling is even
worse there.

> > But then grub-mkrescue will invoke
> > grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
> > as the sed to use?  
> 
> grub-mkrescue runs the mformat program and the mcopy program before
> it runs xorriso.

Yeah, we patch grub-mkrescue to use an absolute path for mformat and mcopy
in order to make them predictable.

> > For example in grub-mkrescue-sed.sh there's no "-e" in the shebang  
> 
> Is it conservative enough to add it ?

It will exit the shell when one of the toplevel commands has an
exit status != 0.
So as long as your script doesn't have toplevel commands return error status
in normal operation it should be fine.

Toplevel:

$ grep foo bar

Not toplevel:

$ if grep foo bar

(that won't fail the script even with "-e")

Questionable:

$ foo | bar

If foo fails, should the script fail?  Spoiler: It won't fail the script even with "-e".
If bar fails, should the script fail?  Spoiler: It will fail the script with "-e".

But:

$ set -e
$ set -o pipefail
$ foo | bar

Now if foo or bar fails, the script will fail.

> > Guile, on the other hand, will just fail the script on error (usually).  
> 
> That's why i brought a Guile re-write into consideration.
> Firstly it would be the language of choice and secondly it would make
> Guix independent of my ideas how it should be done. 

We're not trying to become iso bootloader experts ourselves :)

>As said, the
> script-in-the-middle gives substantial control over the stage of
> grub-mkrescue when the prepared files are still not packed up in the ISO.
> 
> (Or maybe it's time to re-write the script in C ... as was done with
>  grub-mkrescue when it became too complicated.)

Yeah, we'll see.  For now, let's try the shell script and make it more
paranoid.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> >   --file-system-type=iso9660[_$variation]

Danny Milosavljevic wrote:
> I'm not sure yet.  We have a lot of special-casing for iso9660
> already.  If anything, at that point, we could pass an arbitrary list
> of options or something (an "environment" if you will.  Hah).

How ever the options are brought to the grub-mkrescue run and the script,
i strongly urge not to expose all possible combinations of MKRESCUE_SED*
variables to the ISO production process.
Four consistent variations will be enough and avoid riddling failures.


> The problem is that there were horrible security problems in bash and that
> made a lot of distributions switch to a more minimal shell ("dash") for
> scripts.

It is supposed to run on dash, ksh, and BSD sh versions.


> we patch grub-mkrescue to use an absolute path for mformat and mcopy
> in order to make them predictable.

How deep shall this go ? dd, awk, od, head, find, test, mv, rm, ...


> > > For example in grub-mkrescue-sed.sh there's no "-e" in the shebang

> > Is it conservative enough to add it ?

> It will exit the shell when one of the toplevel commands has an
> exit status != 0.

But do all reasonable shells support -e ?
(I must dig in man ksh down to builtin command -set to get confirmation.)


> > That's why i brought a Guile re-write into consideration.
> > Firstly it would be the language of choice and secondly it would make
> > Guix independent of my ideas how it should be done.

> We're not trying to become iso bootloader experts ourselves :)

Currently Guix ISO production is not enough boss over what gets into the
ISO image. Neither with EFI partition properties nor with partition table
type.


> For now, let's try the shell script and make it more paranoid.

Please report all changes (and be ready to motivate them).


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

> How deep shall this go ? dd, awk, od, head, find, test, mv, rm, ...

Ideally everything.  That's why we don't use shell scripts that much:  They
are annoying to make behave.  Compare with C which will link a library and
with option "-Wl,-rpath" (which we use everywhere) will embed the absolute
path to the library in the executable and NEVER use another one than the
one it was compiled with.

Usually, we just wrap the entire shell script in a wrapper that sets
up PATH first.  As long as there are no naming conflicts in the
dependencies, it should be fine (since PATH just lists directories, not
regular files, it could still be bad).

I'm thinking of doing this PATH wrapper here, too.  It's easily automated on
the Guix side.

The end goal (for the most part reached) is to have a package be a
module that doesn't affect the rest of the system except when it is
supposed to.  Other distributions are spectacularily bad at this--to
say nothing about Windows where it's normal that installing one
application can break a totally unrelated application that had already
been there.

> But do all reasonable shells support -e ?
> (I must dig in man ksh down to builtin command -set to get confirmation.)

I don't know.

> > For now, let's try the shell script and make it more paranoid.  
> 
> Please report all changes (and be ready to motivate them).

Okay!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> So what’s the plan?
>
> Wait for another responce from the grub-devel mailing list?
>
> Package grub-mkrescue-sed.sh and use
> --xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?
>
> Add an option to “guix system disk-image” to select which
> grub-mkrescue-sed.sh environment variables to enable?

I lost track of the discussion.  You’re looking at a MacBook-specific
issue, right?

(The underlying question being: can this wait after 1.0 is out?  :-)
If it’s MacBook-specific, I would answer “yes.”)

Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 25, 2019 at 06:34:16PM +0200, Ludovic Courtès wrote:
> I lost track of the discussion.  You’re looking at a MacBook-specific
> issue, right?
> 
> (The underlying question being: can this wait after 1.0 is out?  :-)
> If it’s MacBook-specific, I would answer “yes.”)
> 

As I understand it, Thomas’ work is about
- making a correct ISO image and
- making it work on more machines.

I would have hoped the 1.0 ISO could already be made correct.
Older ISOs work on cooperative motherboards as well though, it appears
only old Macs are affected.  A more recent 2014 Macbook Air boots fine
though.

As for the latter, here are my findings when booting the USB
installer:


|                     | Nvidia PC with        | AMD GPU PC with                      |
|                     | Asus P8H67-V board    | Asus P8H67-?                         |
|---------------------+-----------------------+--------------------------------------|
| x86_64              | boots with UEFI       | boots but graphical installer        |
| unpatched master    | and legacy            | only works with UEFI                 |
|---------------------+-----------------------+--------------------------------------|
| x86_64 with Danny's | boots with UEFI       | boots but graphical installer        |
| patches (wip6)      | and legacy            | only works with UEFI                 |
|---------------------+-----------------------+--------------------------------------|
| i686                | boots graphical       | boots in legacy mode w/o             |
| unpatched master    | installer in legacy   | graphical installer, boots from      |
|                     | mode, boots from      | harddisk instead when selecting UEFI |
|                     | harddisk instead when |                                      |
|                     | selecting UEFI        |                                      |
|---------------------+-----------------------+--------------------------------------|
| i686 with Danny's   | boots graphical       | boots in legacy mode w/o             |
| patches (wip6)      | installer in legacy   | graphical installer, boots from      |
|                     | mode, boots from      | harddisk instead when selecting UEFI |
|                     | harddisk instead when |                                      |
|                     | selecting UEFI        |                                      |
|---------------------+-----------------------+--------------------------------------|

|                     | 2010 Macbook       | Lenovo Ideapad 100S |
|---------------------+--------------------+---------------------|
| x86_64              | boot gets stuck    | does not appear     |
| unpatched master    | before listing     | in the boot         |
|                     | devices to boot    | selection           |
|                     | from               |                     |
|---------------------+--------------------+---------------------|
| x86_64 with Danny's | boots fine with a  | does not appear     |
| patches (wip6)      | selection of “EFI  | in the boot         |
|                     | Boot” and “GRUB    | selection           |
|                     | 2.02”; both appear |                     |
|                     | to boot in UEFI    |                     |
|                     | mode though        |                     |
|---------------------+--------------------+---------------------|
| i686                | boot gets stuck    | boots fine (is      |
| unpatched master    | before listing     | only shown as EFI   |
|                     | devices to boot    | device)             |
|                     | from               |                     |
|---------------------+--------------------+---------------------|
| i686 with Danny's   | only “GRUB 2.02”   | boots fine (is      |
| patches (wip6)      | can be selected,   | only shown as EFI   |
|                     | but boots from     | device)             |
|                     | harddisk instead   |                     |
|---------------------+--------------------+---------------------|

I wonder why guixsd-install-0.15.0.i686-linux.iso did not boot on the
Lenovo Ideapad 100S before.  Windows claims the Lenovo is a 32-bit
machine only btw.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Thu, Apr 25, 2019 at 06:34:16PM +0200, Ludovic Courtès wrote:
>> I lost track of the discussion.  You’re looking at a MacBook-specific
>> issue, right?
>> 
>> (The underlying question being: can this wait after 1.0 is out?  :-)
>> If it’s MacBook-specific, I would answer “yes.”)
>> 
>
> As I understand it, Thomas’ work is about
> - making a correct ISO image and
> - making it work on more machines.
>
> I would have hoped the 1.0 ISO could already be made correct.
> Older ISOs work on cooperative motherboards as well though, it appears
> only old Macs are affected.  A more recent 2014 Macbook Air boots fine
> though.

So the ISO only “fails” (somehow?) on old Macs, is that correct?

If it’s “just” about old Macs, that’s not a 1.0 blocker for me (though
it’s something we should fix.)

If the issue might be broader, we need more data to estimate the scope
of the problem (your table mentioned UEFI vs. “legacy” boot and Lenovo
laptops, which I’m not sure how to interpret.)

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 26, 2019 at 04:41:27PM +0200, Ludovic Courtès wrote:
> So the ISO only “fails” (somehow?) on old Macs, is that correct?
> 
> If it’s “just” about old Macs, that’s not a 1.0 blocker for me (though
> it’s something we should fix.)
>

Old Macs are the only machines we know of that otherwise don’t work.
Lenovo Ideapad 100S had problems with Guix 0.15.0, but it seems
already fixed and unrelated.  There may be other affected machines.
As Thomas Schmitt wrote in his last e-mail, there are other advantages
when users want or need to modify an install USB drive.

If I understand correctly, Thomas Schmitt and Danny Milosavljevic
would rather work on a Guile implementation instead of submitting the
current patch to use a shell script from xorriso for
the 1.0 release (?).

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> > I lost track of the discussion.  You’re looking at a MacBook-specific
> > issue, right?

Florian Pelz wrote:
> As I understand it, Thomas’ work is about
> - making a correct ISO image and

The grub-mkrescue ISOs are correct but also sometimes inconvenient on
USB stick. So rather:
- Making ISO images which are more digestible for partition interpreters
  when presented unchanged on real storage devices. Especially for
  partition editors by which the user wants to add a read-write data
  partition to the USB stick.

> - making it work on more machines.

In this case it's about the Macbook EFI and mformat's partition table
entry. Exotic but also typical for firmware quirks.

----------------------------------------------------------------------

In general my proposal is about getting control over the GRUB-produced
files before xorriso packs them up, and over the arguments of the xorriso
run.

Danny pointed out that Guix ISO production software does not aim for
becomming expert for iso bootloaders.
But the Macbook EFI behavior and the Linux partition table complaints
do not demand deep knowledge of bootloaders. The remedies just need the
opportunity to make small adjustments to the work of grub-mkrescue.

Where such remedies come from, is a different question.
Count on frustration-tolerant users and curious bystanders.

The shell script implementation is somewhat questionable. We could use
it for exploring the best variation and then implement that variation
in a less tolerant programming language with less outer influences.

I asked Danny, whether the download servers could offer two partition
table variations of the ISOs: GPT and MBR (both with APM/HFS+).
Goal would be to gain experience with MBR and to finally replace GPT
by that layout.
He expressed the hope that other members of Guix could answer this.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> I asked Danny, whether the download servers could offer two partition
> table variations of the ISOs: GPT and MBR (both with APM/HFS+).
> Goal would be to gain experience with MBR and to finally replace GPT
> by that layout.
> He expressed the hope that other members of Guix could answer this.

I don’t feel like distributing two variants of the ISO per architecture.
In addition to the fact that it would take up more space on the GNU
servers, I suspect few users would know which one to choose (I
wouldn’t.)  It’s also something other distros don’t do: you get only one
ISO per architecture.

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> I don’t feel like distributing two variants of the ISO per architecture.

Then i guess the most early opportunity for my proposal to get
into production is after the upcomming Guix release.


> It’s also something other distros don’t do: you get only one
> ISO per architecture.

Not really. :))
Debian offers a wide range of sizes from mini.iso (32 MB) to BDDL (50 GB)
and two flavors for x86: Installer and Live (with several desktop flavors).
I have to acknowledge, though, that their boot equipment at most differs by
lagging behind debian-cd evolution.

If it's about what other distros do: They do MBR partition table.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Ludovic Courtès wrote:
>> I don’t feel like distributing two variants of the ISO per architecture.
>
> Then i guess the most early opportunity for my proposal to get
> into production is after the upcomming Guix release.

Sounds good to me.  Thanks again for helping out!

Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 24, 2019 at 11:13:23AM +0200, pelzflorian (Florian Pelz) wrote:
> On Wed, Apr 24, 2019 at 08:56:57AM +0200, Thomas Schmitt wrote:
> > Florian Pelz wrote
> > > By the way, someone I know has a most peculiar machine called Lenovo
> > > Ideapad 100S which does not boot Guix, […]
> > 
> > If Guix does not show up in the boot manager, try a plain grub-mkrescue
> > image that was made when GRUB was configured for both EFI variations:
> > 32 bit x86 and 64 x86.
> > My experiments are made when it is configured for those two plus PC-BIOS.
> > The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.
> > 
> 
> Will do.
> 

The Lenovo Ideapad 100S’s owner tried a USB drive prepared with a
purely 32-bit GRUB as per

On Thu, Apr 25, 2019 at 10:18:08AM +0200, Thomas Schmitt wrote:
>   wget https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
>   chmod u+x grub-mkrescue-sed.sh
> 
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
> 
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh  \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \

-o mbrhfs.iso

and it boots fine.

I do not know and did not try yet how to configure GRUB to produce
both 32-bit and 64-bit EFI.  (Would putting both 32-bit and 64-bit
bootefi files in EFI/Boot/ be a good idea for Guix ISOs in general, or
could this lead to problems with 64-bit machines or lead to confusing
boot selection screens?)

A Debian 9.8.0 32-bit also boots fine and a DVD with an untreated Guix
git master could not be booted but very likely due to an error with my
external DVD drive (it often does not work) and not due to an error
with the DVD.

I will try a current git master Guix iso built for 32-bit next.  If it
does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
will try the same with

dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I do not know and did not try yet how to configure GRUB to produce
> both 32-bit and 64-bit EFI.

As Debian user i am in the comfortable position to install the
three binary packages for BIOS, x86 EFI 32 bit, and x86 EFI 64 bit.
Then grub-mkrescue knows what to do.

Some digging in the source of grub-mkrescue and util/grub-install-common.c
reveils that it is about the existence of $cpu-$firmware directories
with lots of .mod files in them:

  $ ls /usr/lib/grub
  grub-mkconfig_lib  i386-efi  i386-pc  x86_64-efi

The binary Debian packages nearly entirely consist of one of these
directories, each:
  https://packages.debian.org/sid/amd64/grub-efi-amd64-bin/filelist
  https://packages.debian.org/sid/amd64/grub-efi-ia32-bin/filelist
  https://packages.debian.org/sid/amd64/grub-pc-bin/filelist

But i don't know the tool which is to be used for creating those
directories and for filling them with .mod files.

Maybe you find some GRUB tool usage in the Guix ISO production which
uses parameters like "x86_64" "efi".
Or the name "grub-mkconfig_lib" gives a hint ? man grub-mkconfig and
info chapter are quite sparse.


> (Would putting both 32-bit and 64-bit
> bootefi files in EFI/Boot/ be a good idea for Guix ISOs in general, or
> could this lead to problems with 64-bit machines or lead to confusing
> boot selection screens?)

It should be perfectly ok to combine several EFI start programs in
one EFI partition. The firmware is supposed to pick what it deems
suitable.
UEFI 2.4 3.4.1.1 "Removable Media Boot Behavior":

  "[...] the system firmware will attempt to boot from a removable media
   FilePathList[0] by adding a default file name in the form
     \EFI\BOOT\BOOT{machine type short-name}.EFI.
   Each file only contains one UEFI image type, and a system may
   support booting from one or more images types."

You could even combine "IA32", "X64", "ARM", and "AA64" programs if you
manage to keep all CPU specific files apart in the course of the further
operating system run.


> The Lenovo Ideapad 100S’s owner tried a USB drive prepared with a
> purely 32-bit GRUB [...]
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh  \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \

> and it boots fine.

So that's not the mad EFI which hates all MBR partitions except one 0xef.


> a DVD with an untreated Guix
> git master could not be booted but very likely due to an error with my
> external DVD drive (it often does not work)

We could start a separate endeavor in private for diagnosing that.
(My other sport besides ISO 9660 is burning flat round things.)


> I will try a current git master Guix iso built for 32-bit next.  If it
> does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
> will try the same with
> dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Was the machine similarly stuck as the Macbook ?
That would probably be the partition problem, indeed.

But failure when GRUB or a Linux kernel have taken over would be a
different problem. The minimal grub-mkrescue ISO would then possibly
work because it prompts the user before the problem can occur.
Or maybe because it has a .mod file that is needed for that machine ?


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 25, 2019 at 05:09:05PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I do not know and did not try yet how to configure GRUB to produce
> > both 32-bit and 64-bit EFI.
> […]
> Some digging in the source of grub-mkrescue and util/grub-install-common.c
> reveils that it is about the existence of $cpu-$firmware directories
> with lots of .mod files in them:
> 

This is interesting, but I am not familiar enough to try to quickly
change the guix package and I am not going to invest time in that.
The reason I was interested is that the CPU supposedly can run 64-bit
code, it is apparently just the UEFI that refuses 64-bit.  But one can
install 32-bit and then reconfigure Guix later on to use 64-bit, so it
is not important.



> > a DVD with an untreated Guix
> > git master could not be booted but very likely due to an error with my
> > external DVD drive (it often does not work)
> 
> We could start a separate endeavor in private for diagnosing that.
> (My other sport besides ISO 9660 is burning flat round things.)
> 
>

:D Burning is not the problem. ;)  This slim laptop has no DVD drive
itself and my external USB DVD drive works only sometimes and might
have required even more patience.  Booting got stuck waiting for the
DVD drive which continued to make noises, as it often does on other
machines when it has a bad day.  This was not a problem with the DVD
or the image, but with that DVD drive of mine.

I will test Danny’s patches on my Macbook and on the Ideapad.


> > I will try a current git master Guix iso built for 32-bit next.  If it
> > does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
> > will try the same with
> > dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2
> 
> Was the machine similarly stuck as the Macbook ?
> That would probably be the partition problem, indeed.
> 

With the USB flash drive booting was not stuck, it just did not appear
in the boot device selection.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i worte:
> > (My other sport besides ISO 9660 is burning flat round things.)

Florian Pelz wrote:
> :D Burning is not the problem. ;)

We could let xorriso put some read load on it and watch out for error
messages.


> With the USB flash drive booting was not stuck, it just did not appear
> in the boot device selection.

Then it is undecided yet. Either it dislikes the EFI partition's content
or it avoided the endless loop by discarding the iwhole partition.


Have a nice day :)

Thomas