bug#33639: ISO installer image is broken on i686

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hello,

The ISO installer image as produced on commit
4a0b87f0ec5b6c2dcf82b372dd20ca7ea6acdd9c by

  guix system disk-image --file-system-type=iso9660 \
    -s i686-linux gnu/system/install.scm

contains unreadable file(s), at least /var/guix/db/db.sqlite.

The build at <https://hydra.gnu.org/build/3151513> (2018-11-12,
64461ba20a07a0cf3197de3e97cb44e0f66cebdc) seems is the only occurrence
of the problem I could find on the build farms: while running the
installation off the ISO image, it fails like this:

--8<---------------cut here---------------start------------->8---
+ guix --version
guix (GNU Guix) 0.15.0-6.f9a8fce
Copyright (C) 2018 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ export GUIX_BUILD_OPTIONS=--no-grafts
+ GUIX_BUILD_OPTIONS=--no-grafts
+ guix build isc-dhcp
[   95.076694] attempt to access beyond end of device
[   95.080672] sr0: rw=524288, want=2118580, limit=2115840
[   95.082317] attempt to access beyond end of device
[   95.083730] sr0: rw=0, want=2118332, limit=2115840
[   95.097050] attempt to access beyond end of device
[   95.098175] sr0: rw=0, want=2118332, limit=2115840
guix build: error: build failed: cannot open Nix database `/var/guix/db/db.sqlite'
--8<---------------cut here---------------end--------------->8---

Indeed, if you spawn the image and run “cat /var/guix/db/db.sqlite”, it
fails with EIO and “attempt to access beyond end of device.”  I suspect
the bugs Mark reported at <https://issues.guix.info/issue/33362> and
<https://issues.guix.info/issue/33555> are related.

My guess is that the bug has always existed on ‘core-updates’ since
<https://berlin.guixsd.org/build/662745> (‘master’, 2018-11-30, i.e.,
just before ‘core-updates’ was merged) shows a successful installation.

I tried running the ISO image in qemu-system-{x86_64,i386}, with and
without KVM, and the I/O errors are always there, including with a
pre-core-updates QEMU.

I tried reverting xorriso to 1.4.8 to no avail (which is not surprising
since xorriso was upgraded on 2018-09-18 and the successful installation
above which 2018-11-30.)

At this point I can only suspect a toolchain issue, probably binutils or
libc since gcc didn’t change.

Thoughts?

This is holding the 0.16.0 release and I’m unavailable to do it next
week and with little time over the next few days.  Thus I’m considering
exceptionally releasing without the i686 GuixSD install image; thoughts?
The rest is all fine and ready to ship.

Thanks,
Ludo’.

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> The ISO installer image as produced on commit
> 4a0b87f0ec5b6c2dcf82b372dd20ca7ea6acdd9c by
>
>   guix system disk-image --file-system-type=iso9660 \
>     -s i686-linux gnu/system/install.scm
>
> contains unreadable file(s), at least /var/guix/db/db.sqlite.

I can reproduce the I/O error by mounting the image:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ sudo losetup /dev/loop0 /gnu/store/1yanxg3cz5wi6vhpvhipxvmjwm201fbm-image.iso
ludo@ribbon ~/src/guix$ sudo mount -t iso9660 /dev/loop /mnt/disk/
mount: /mnt/disk: WARNING: device write-protected, mounted read-only.
ludo@ribbon ~/src/guix$ cat < /mnt/disk/var/guix/db/db.sqlite > /dev/null
cat: -: Eraro de en-eligo
ludo@ribbon ~/src/guix$ dmesg |tail
[   41.186408] shepherd[1]: Service guix-daemon has been started.
[   45.725418] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   45.933911] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   49.496112] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   49.496165] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
[  203.358136] ISO 9660 Extensions: RRIP_1991A
[  215.199352] attempt to access beyond end of device
[  215.199357] loop0: rw=524288, want=1903876, limit=1899264
[  215.199362] attempt to access beyond end of device
[  215.199363] loop0: rw=0, want=1903532, limit=1899264
--8<---------------cut here---------------end--------------->8---

So the problems lies with the VM that creates the image.

Ludo’.

bug#33639: ISO installer image is broken on i686

[swedebugia]

On 2018-12-06 01:02, Ludovic Courtès wrote:
snip

> Indeed, if you spawn the image and run “cat /var/guix/db/db.sqlite”, it
> fails with EIO and “attempt to access beyond end of device.”  I suspect
> the bugs Mark reported at <https://issues.guix.info/issue/33362> and
> <https://issues.guix.info/issue/33555> are related.
> 
> My guess is that the bug has always existed on ‘core-updates’ since
> <https://berlin.guixsd.org/build/662745> (‘master’, 2018-11-30, i.e.,
> just before ‘core-updates’ was merged) shows a successful installation.
> 
> I tried running the ISO image in qemu-system-{x86_64,i386}, with and
> without KVM, and the I/O errors are always there, including with a
> pre-core-updates QEMU.
> 
> I tried reverting xorriso to 1.4.8 to no avail (which is not surprising
> since xorriso was upgraded on 2018-09-18 and the successful installation
> above which 2018-11-30.)
> 
> At this point I can only suspect a toolchain issue, probably binutils or
> libc since gcc didn’t change.
> 
> Thoughts?
> 
> This is holding the 0.16.0 release and I’m unavailable to do it next
> week and with little time over the next few days.  Thus I’m considering
> exceptionally releasing without the i686 GuixSD install image; thoughts?

Ok, I see.

Has anybody tested that guix pull from 0.15 -> 0.16 works on an install
ISO? (I don't know if we want/agreed to support this at all but 1 bug
suggests problems related to https: )

I say go for release and note it on the download page and provide
0.15-i686 image for now.

I'm using i686 GuixSD on my devlaptop.

-- 
Cheers 
Swedebugia

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Dear Xorriso hackers,

While building an ISO for i686, running Xorriso 1.5.0 built for i686
(actually ‘grub-mkrescue’, but that’s just a wrapper around Xorriso) in
qemu-system-i386, we end up with an ISO image containing files that lead
to I/O errors (“attempt to access beyond end of device”):

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ sudo losetup /dev/loop0 /gnu/store/1yanxg3cz5wi6vhpvhipxvmjwm201fbm-image.iso
ludo@ribbon ~/src/guix$ sudo mount -t iso9660 /dev/loop /mnt/disk/
mount: /mnt/disk: WARNING: device write-protected, mounted read-only.
ludo@ribbon ~/src/guix$ cat < /mnt/disk/var/guix/db/db.sqlite > /dev/null
cat: -: Eraro de en-eligo
ludo@ribbon ~/src/guix$ dmesg |tail
[   41.186408] shepherd[1]: Service guix-daemon has been started.
[   45.725418] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   45.933911] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   49.496112] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   49.496165] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
[  203.358136] ISO 9660 Extensions: RRIP_1991A
[  215.199352] attempt to access beyond end of device
[  215.199357] loop0: rw=524288, want=1903876, limit=1899264
[  215.199362] attempt to access beyond end of device
[  215.199363] loop0: rw=0, want=1903532, limit=1899264
--8<---------------cut here---------------end--------------->8---

The output of Xorriso and the kernel when it builds the image looks
good.

(More info at <https://issues.guix.info/issue/33639>.)

Using the exact same build process for x86_64 leads to valid ISO images.

Does that ring a bell or would you have advice to further debug it?

Thanks,
Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> [ 215.199357] loop0: rw=524288, want=1903876, limit=1899264

This looks much like a truncated ISO image. (For what reason ever.)

There are at least 4612 blocks = ~ 9 MiB missing.
In the original message of https://issues.guix.info/issue/33639 the
the minimum missing size is about 5 MiB.

Please consider local reasons for truncated ISO images.

In the following i will concentrate on a potential program bug.


> [...] running Xorriso 1.5.0 built for i686 [...] I/O errors [...]
> Using the exact same build process for x86_64 leads to valid ISO images.

Well, this would explain why 1.5.0 passed a regression test on my 64 bit
system with repacking about 200 ISOs, mounting them, and comparing them
with the monted original ISOs.
I currently lack of opportunities to build 32 bit xorriso.

Is there such a damaged ISO available for download ?

How much effort would it be to create a Guix installation for building
xorriso, running your ISO production, and possibly running xorriso under
gdb ?
(Something for a run like

  qemu-system-i386 \
     -enable-kvm \
     -nographic \
     -m 512 \
     -net nic \
     -net user,hostfwd=tcp::5555-:22 \
     -hda guix_on_qemu.img

with the opportunity to login from the host machine via SSH.
)

What do you get from this xorriso inspection run on a damaged ISO ?
(I tested it with the ISO from https://www.gnu.org/software/guix/download/):

  xorriso -indev guixsd-install-0.15.0.i686-linux.iso \
          -find / -sort_lba -exec report_lba -- \
          >/tmp/xorriso_indev_find.txt 2>&1

In a preliminary test with
  guixsd-install-0.15.0.i686-linux.iso
i get in /tmp/xorriso_indev_find.txt :

  ...
  Media summary: 1 session, 454094 data blocks,  887m data,  384g free
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,     8527 ,     1440 ,  2949120 , '/efi.img'
  ... many other files ...
  File data lba:  0 ,   453781 ,      122 ,   249856 , '/var/guix/db/db.sqlite'

The ISO image file size is 929984512 bytes = 454094 blocks.
The image by its inner size counter also claims 454094 blocks.
The data file with the highest storage address ends before block
453781 + 122 = 453903.
That's 191 blocks before the image end. Padding and GPT backup follow.
(The data block size is 2048 bytes.)

So this image looks ok. Let's read all its files:

  # mount guixsd-install-0.15.0.i686-linux.iso /mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only
  $ tar cf - /mnt/iso | wc
  tar: Removing leading `/' from member names
  7116387 35887498 1042391040
  $

No i/o error.


Unrelated observation:
xorriso command -pvd_info reports that the ISO was made with xorriso-1.4.8
with
  Creation Time: 1970010119010649
This means "1 Jan 1970 19:01:06". Something seems to be wrong with the
system clock of the producer machine.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi Thomas,

Thanks for the quick and insightful reply!

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>> [ 215.199357] loop0: rw=524288, want=1903876, limit=1899264
>
> This looks much like a truncated ISO image. (For what reason ever.)
>
> There are at least 4612 blocks = ~ 9 MiB missing.
> In the original message of https://issues.guix.info/issue/33639 the
> the minimum missing size is about 5 MiB.

OK.

> Please consider local reasons for truncated ISO images.

I’ve thought about this but that seem highly unlikely at this point.

> Is there such a damaged ISO available for download ?

No.

> How much effort would it be to create a Guix installation for building
> xorriso, running your ISO production, and possibly running xorriso under
> gdb ?
> (Something for a run like
>
>   qemu-system-i386 \
>      -enable-kvm \
>      -nographic \
>      -m 512 \
>      -net nic \
>      -net user,hostfwd=tcp::5555-:22 \
>      -hda guix_on_qemu.img

You could install Guix on top of your distro following the instructions
at
<https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html>.
Then you would need to run “guix pull” to get a current Guix (0.15.0
itself didn’t have this bug.)  And finally, run:

  guix system disk-image --file-system-type=iso9660 \
    -s i686-linux \
    ~/.config/guix/current/share/guile/site/2.2/gnu/system/install.scm

(This command works on an x86_64 machine.)

The result will be an ISO that’s corrupt.

> What do you get from this xorriso inspection run on a damaged ISO ?
> (I tested it with the ISO from https://www.gnu.org/software/guix/download/):
>
>   xorriso -indev guixsd-install-0.15.0.i686-linux.iso \
>           -find / -sort_lba -exec report_lba -- \
>           >/tmp/xorriso_indev_find.txt 2>&1

I get:

--8<---------------cut here---------------start------------->8---
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

libisoburn: WARNING : ISO image size 475636s larger than readable size 473456s
xorriso : NOTE : Loading ISO image tree from LBA 0
libburn : SORRY : Read start address 475635s larger than number of readable blocks 473456
xorriso : UPDATE :   46803 nodes read in 1 seconds
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev '/gnu/store/v13bryy1mrgrs694drsrknryf204q30j-image.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record  : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT APM
Media summary: 1 session, 473456 data blocks,  925m data, 45.6g free
Volume id    : 'GUIXSD_IMAGE'
xorriso : NOTE : Tolerated problem event of severity 'SORRY'
Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
File data lba:  0 ,     8612 ,      720 ,  1474560 , '/efi.img'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/1zzgag2ca7xzklss2j6phh4580cgkbl2-flac-1.3.2/share/doc/flac-1.3.2/FLAC.tag'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/55m1dng1zw7fq7ni73nm2v7b84wghpka-libx11-1.6.6/share/X11/locale/am_ET.UTF-8/XI18N_OBJS'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/55m1dng1zw7fq7ni73nm2v7b84wghpka-libx11-1.6.6/share/X11/locale/cs_CZ.UTF-8/XI18N_OBJS'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/55m1dng1zw7fq7ni73nm2v7b84wghpka-libx11-1.6.6/share/X11/locale/el_GR.UTF-8/XI18N_OBJS'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/55m1dng1zw7fq7ni73nm2v7b84wghpka-libx11-1.6.6/share/X11/locale/fi_FI.UTF-8/XI18N_OBJS'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/746645dl4fmz9h12x247nyznalswqyzp-groff-minimal-1.22.3/share/groff/1.22.3/tmac/mm/locale'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/746645dl4fmz9h12x247nyznalswqyzp-groff-minimal-1.22.3/share/groff/1.22.3/tmac/mm/se_locale'
File data lba:  0 ,    25032 ,        0 ,        0 , '/gnu/store/a1vpwa7wkxbxw18sz70rmp3cdfnf3jdj-libvorbis-1.3.6/share/doc/libvorbis-1.3.6/doxygen-build.stamp'
File data lba:  0 ,    25032 ,        0 ,        0 , '/mach_kernel'
File data lba:  0 ,    25034 ,     1173 ,  2400500 , '/boot/grub/fonts/unicode.pf2'
File data lba:  0 ,    26207 ,        1 ,     1520 , '/boot/grub/grub.cfg'
File data lba:  0 ,    26207 ,        1 ,     1520 , '/gnu/store/3zq39lvf12a87zcfrg87xgkllgfsyw3b-grub.cfg'
File data lba:  0 ,    26208 ,        5 ,     9928 , '/boot/grub/i386-efi/acpi.mod'

[…]

File data lba:  0 ,   475300 ,        1 ,     1651 , '/gnu/store/zrg4c2d0lvyw8z9xgh0darzglbxrm6b7-iptables-1.6.2/share/man/man8/iptables-restore.8.gz'
File data lba:  0 ,   475301 ,        1 ,     1137 , '/gnu/store/zrg4c2d0lvyw8z9xgh0darzglbxrm6b7-iptables-1.6.2/share/man/man8/iptables-save.8.gz'
File data lba:  0 ,   475302 ,        4 ,     7837 , '/gnu/store/zrg4c2d0lvyw8z9xgh0darzglbxrm6b7-iptables-1.6.2/share/man/man8/iptables.8.gz'
File data lba:  0 ,   475306 ,       47 ,    96256 , '/System/Library/CoreServices/boot.efi'
File data lba:  0 ,   475353 ,        1 ,      236 , '/System/Library/CoreServices/SystemVersion.plist'
File data lba:  0 ,   475354 ,        1 ,     1399 , '/System/Library/CoreServices/.disk_label'
File data lba:  0 ,   475355 ,        1 ,       10 , '/System/Library/CoreServices/.disk_label.contentDetails'
File data lba:  0 ,   475356 ,       88 ,   180224 , '/var/guix/db/db.sqlite'
xorriso : NOTE : -return_with SORRY 32 triggered by problem severity SORRY
--8<---------------cut here---------------end--------------->8---

Something’s fishy, and Xorriso is sorry.  :-)

Let me know if I can provide more info.

In the meantime I’ll see if I can build the image from x86_64 instead.

> Unrelated observation:
> xorriso command -pvd_info reports that the ISO was made with xorriso-1.4.8
> with
>   Creation Time: 1970010119010649
> This means "1 Jan 1970 19:01:06". Something seems to be wrong with the
> system clock of the producer machine.

For reproducibility purposes we set timestamps and related things to the
Epoch.  This pseudo-UUID/timestamps is actually derived from the config
of the operating system in the image.  It’s expected.  :-)

Thank you!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi again,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>> [ 215.199357] loop0: rw=524288, want=1903876, limit=1899264
>
> This looks much like a truncated ISO image. (For what reason ever.)
>
> There are at least 4612 blocks = ~ 9 MiB missing.
> In the original message of https://issues.guix.info/issue/33639 the
> the minimum missing size is about 5 MiB.

Based on this and on a suggestion Ricardo made on IRC, I passed
“-padding 10m” and that solved the problem.  \o/

I suppose you’ll have a scientific explanation, but I’m happy this
simple hacks works (and indeed, the documentation of “-padding” suggests
that this kind of problem is not uncommon.)

Thanks to both of you!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

i see that probably the kernel log talks of blocks of 512 bytes.
So the minimum missing size shrinks to 2.3 and 1.4 MiB, respectively.


I wrote:
> > Please consider local reasons for truncated ISO images.

Ludovic Courtès wrote:
> I’ve thought about this but that seem highly unlikely at this point.

It still looks like writing of the ISO image aborted prematurely.
Do you have the xorriso messages from the grub-mkrescue run ?

(If there are none, add the following three arguments to the grub-mkrescue 
 run:
   -- -- -report_about update
 The second "--" shall work around an intermediate version of grub-mkrescue
 which ate the first "--" instead of forwarding it to xorriso.
)


Reasoning:

> libisoburn: WARNING : ISO image size 475636s larger than readable size 473456s
> File data lba:  0 ,   475356 ,       88 ,   180224 , '/var/guix/db/db.sqlite'

When the ISO is assessed by libisoburn, its nominal block count is
192 blocks higher than the end of the last file. Insofar ok. But the
ISO image file is smaller than that.

After the warning, libisoburn corrects the displayed size to the readable
size. So the number in this subsequent message is rather insignificant:
> Media summary: 1 session, 473456 data blocks,  925m data, 45.6g free
(Only good that you also showed above warning message.)


The nominal count is recorded in the Primary Volume Descriptor, the
equivalent of a superblock. (Byte offset in the ISO file is 32768+80,
first as 4 byte little-endian, then again as 4 byte big-endian.)

The readable size is based on the byte size of the ISO file.

At ISO production time, the nominal block count is determined by libisofs
in a first dry-run. In the subsequent real production run, libisofs sticks
to the determined file sizes of the first run, even if some file changed
size inbetween. It would truncate or pad the copied file bytes to the
planned size. Directory data are written as assessed in the first run.

So from normal operation of libisofs it is guaranteed that the written
amount of data is the same as the nominal amount.

-----------------------------------------------------------------------

Possible glitches would be that libisofs skips to write some scheduled
data blocks, or that libburn drops blocks which were submitted by libisofs.
Both scenarios do not give me an idea how the difference between 32 bit
and 64 bit systems could be involved.

The theory of intermediately missing data blocks could be verified or
defuted by checking the content of the last file which sits in the
readable area. If it bears the expected content, then no blocks were
skipped or dropped inbetween.

So please look in the file listing for the last file which begins before
block 473456 and does not step over that limit by adding its "Blocks"
count (exact hit on the limit is ok).
If the filesystem refuses to obtain it, then use
  dd bs=2048 skip=$Startlba count=$Blocks
to cut it out from the ISO and then truncate it to the reported "Filesize".

In any case compare its content with the original.

If the contents match, then we have a flat premature end of file.
In this case there should be error messages from xorriso or its libraries.
(In case of GNU xorriso, the libraries are fixely compiled in from source.)


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> Based on this and on a suggestion Ricardo made on IRC, I passed
> “-padding 10m” and that solved the problem.  \o/

Ouchers. Do all files bear their expected content ?
Especially the last one: /var/guix/db/db.sqlite

If so, then something truncates the output stream of libisofs via libburn.
The only component that comes to my mind is the fifo between them.
The default fifo size is 4 MiB. Quite suspicious.

Try to reduce its size to the minimum by adding these grub-mkrescue
arguments:

  -- -- -fs 64k -padding 64k

If the fifo is to blame, then a padding of 64k should suffice to protect
the valuable blocks from a premature end.


--------------------------------------------------------------------
A bit off topic:

> the documentation of “-padding” suggests
> that this kind of problem is not uncommon.

It's normal purpose is to work around a traditional Linux kernel bug:

CDs written with write type Track-At-Once bear two unreadable blocks at
the end. Most CD drives report these blocks as part of the data range.
When Linux shall read a single block for isofs, it reads a larger chunk.
The chunk is not large enough to reach over the nominal end of the data
range, but it can reach the unreadable end blocks by mistake.
In this case Linux does not only miss the end blocks but also valid
payload blocks which are part of the filesystem. This yields I/O error.

The developer of cdrecord and the kernel people mistake this problem
for a "fuzziness" of a CD end by at most 2 seconds of audio play time.
This is wrong from reading the specs and from making experiments.
However, cdrecord introduced the tradition to add 150 blocks of padding
which would 2 seconds of sound.
As long as the read chunk of Linux is smaller than that, the padding
protects the operating system from touching the lead-out blocks of the
TAO track.

This cannot happen on hard disk or any optical media type other than CD.
If you write the CD by Session-At-Once it cannot happen. If you have one
of the rare CD drives which do not count the lead-out blocks to the
readable size of the CD, it cannot happen. (Currently 1 of my 7 drives
tells the truth.)

But who am i to stand against all others ?
So xorriso, too, adds 300 KiB of padding by default.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hello!

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>> Based on this and on a suggestion Ricardo made on IRC, I passed
>> “-padding 10m” and that solved the problem.  \o/
>
> Ouchers. Do all files bear their expected content ?
> Especially the last one: /var/guix/db/db.sqlite

It looks good, and there are no I/O errors left (I mounted it and run
“tar” over it.)

Note that the image is now available here:

  https://alpha.gnu.org/gnu/guix/guixsd-install-0.16.0.i686-linux.iso.xz

(I haven’t tried smaller padding.)

> If so, then something truncates the output stream of libisofs via libburn.
> The only component that comes to my mind is the fifo between them.
> The default fifo size is 4 MiB. Quite suspicious.
>
> Try to reduce its size to the minimum by adding these grub-mkrescue
> arguments:
>
>   -- -- -fs 64k -padding 64k
>
> If the fifo is to blame, then a padding of 64k should suffice to protect
> the valuable blocks from a premature end.

OK, I’ll try to test this, but note that I’ll be largely unavailable for
a week.

>> the documentation of “-padding” suggests
>> that this kind of problem is not uncommon.
>
> It's normal purpose is to work around a traditional Linux kernel bug:
>
> CDs written with write type Track-At-Once bear two unreadable blocks at
> the end. Most CD drives report these blocks as part of the data range.
> When Linux shall read a single block for isofs, it reads a larger chunk.
> The chunk is not large enough to reach over the nominal end of the data
> range, but it can reach the unreadable end blocks by mistake.
> In this case Linux does not only miss the end blocks but also valid
> payload blocks which are part of the filesystem. This yields I/O error.
>
> The developer of cdrecord and the kernel people mistake this problem
> for a "fuzziness" of a CD end by at most 2 seconds of audio play time.
> This is wrong from reading the specs and from making experiments.
> However, cdrecord introduced the tradition to add 150 blocks of padding
> which would 2 seconds of sound.
> As long as the read chunk of Linux is smaller than that, the padding
> protects the operating system from touching the lead-out blocks of the
> TAO track.
>
> This cannot happen on hard disk or any optical media type other than CD.
> If you write the CD by Session-At-Once it cannot happen. If you have one
> of the rare CD drives which do not count the lead-out blocks to the
> readable size of the CD, it cannot happen. (Currently 1 of my 7 drives
> tells the truth.)
>
> But who am i to stand against all others ?
> So xorriso, too, adds 300 KiB of padding by default.

I see, thanks for explaining!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> https://alpha.gnu.org/gnu/guix/guixsd-install-0.16.0.i686-linux.iso.xz
> (I haven’t tried smaller padding.)

I downloaded it and get on "xorriso -indev":
  libisoburn: WARNING : ISO image size 481129s larger than readable size 479184s

So the lack of 2k blocks is 1945 = nearly 4 MiB.
This is suspiciously near to the default fifo size.

The content of cleartext files near the payload end looks plausible:
  /System/Library/CoreServices/.disk_label
  /System/Library/CoreServices/SystemVersion.plist
Whether the last file's content is as expected can only be told by
its reader program, i guess:
  /var/guix/db/db.sqlite

So for now it indeed looks like plain truncation and not like a hickup
somewhere in the middle of ISO writing.
  

Several distros use xorriso to build their 32 bit ISOs. No complaints.
So i asked on debian-cd and debian-live mailing lists whether the ISOs
for 32-bit systems are indeed made on 32-bit systems. The answer is
  "All our images have been made on amd64 for years now."

So i need a 32-bit GNU/Linux VM for regression tests.

Being an untalented sysadmin, this can last a while. (First searching
for old cheat sheets and then stepping into any possible puddle ...)


I would still appreciate a test with minmally sized fifo. Its outcome would
be a strong indication whether the Guix problem is related to the fifo
at all. The result can be checked by executing

  xorriso -indev ...path.to.iso...

and looking for message
  libisoburn: WARNING : ISO image size ...s larger than readable size ...s
If the difference is in the range of only 32s, then the fifo stays
main suspect.

Also, the xorriso messages of a run with grub-mkrescue add-on arguments

  -- -- -report_about all

would be very welcome.

--------------------------------------------------------------------------
(Be invited to stop reading here. Only code musings follow.)

I reviewed the fifo code in libisofs and found no obvious opportunity for
a bug that would drop the final fifo content rather than offering it to
libburn:
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/buffer.c

(iso_ring_buffer_read() is exposed to libburn via libisofs/ecma119.c
 function bs_read() which serves as struct burn_source member (*read)()
 as defined in libburn/libburn.h.)

The condition for end of reading is a combination of
- no data are available in the ring buffer
- the writer has set the flag for having ended its work

        while (buf->size == 0) {
            ...
            if (buf->wend) {

The member buf->size is of type size_t. I.e. good for at least 4 GiB - 1
before it rolls over. Neither the fifo size nor the transaction size come
near to that number.
buf->wend is unsigned int :2 with defined values
  0 not finished, 1 finished ok, 2 finish with error


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

just to report that i did not forget this problem:

I have now a qemu-system-i386 VM with Debian GNU/Linux from
debian-9.6.0-i386-netinst.iso without desktop environment and reachable
via SSH. Very minimal. (I only did "apt-get install build-essential" to
feel not lonely without C compiler and friends.)

Then i followed the instructions of
  https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
with
  https://alpha.gnu.org/gnu/guix/guix-binary-0.16.0.i686-linux.tar.xz
  https://alpha.gnu.org/gnu/guix/guix-binary-0.16.0.i686-linux.tar.xz.sig
up to step 7 ("guix archive --authorize ...").

Then i made the mistake to do the proposed

  guix package -i hello

It downloads and builds and blows away the free space on the virtual 8 GB
disk ... /gnu is growing steadily and /tmp breathes between 50 MB and 2 GB.
I abort this after 100 minutes before the virtual disk gets too full and
my CPU melts.

"guix pull" happily begins to build that gcc-5.5.0 which is too much for my
feeble VM.

Back to step 0 ("rm -r /gnu /var/guix") and again to step 7.
(A small fight starts between me and systemd, to get guix-daemon running.
 "start" did not help. It had to be "restart".)

Then

  # guix system disk-image --file-system-type=iso9660 \
  >   -s i686-linux \
  >   ~/.config/guix/current/share/guile/site/2.2/gnu/system/install.scm

and the activities to build the world start again. Extra verbose.
This time i abort after 30 minutes.

Everything i do ends up in enormous production of gcc-5.5.0 related
software.

-------------------------------------------------------------------------

So for xorriso and a 32-bit system:

  # apt-get install xorriso
  ...
  # xorriso -version
  xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
  ...

I try what happens if i pack up the /gnu tree:

  # xorriso -as mkisofs -o /tmp/test.iso -J /gnu
  ...
  ISO image produced: 643046 sectors
  Written to medium : 643046 sectors at LBA 0
  Writing to 'stdio:/tmp/test.iso' completed successfully.

Inspection shows that the size ideas of xorriso match the image file size:

  # xorriso -indev /tmp/test.iso
  ... no warning about size mismatch ...
  Media summary: 1 session, 643046 data blocks, 1256m data, 3234m free

  # ls -l /tmp/test.iso
  -rw-r--r-- 1 root root 1316958208 Dec 15 19:17 /tmp/test.iso

  # expr 1316958208 / 2048
  643046

Now with GNU xorriso 1.5.0:

  $ wget https://www.gnu.org/software/xorriso/xorriso-1.5.0.tar.gz
  ...
  $ tar xzf xorriso-1.5.0.tar.gz
  $ cd xorriso-1.5.0
  $ ./configure && make
  ...
  $ xorriso/xorriso -version
  GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.
  ...

  # rm /tmp/test.iso
  # xorriso/xorriso -as mkisofs -o /tmp/test.iso -J /gnu
  GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.
  ...
  ISO image produced: 643046 sectors
  Written to medium : 643046 sectors at LBA 0
  ...

Inspection yields the same result. No truncation.

-------------------------------------------------------------------------

If i shall try again with "guix system disk-image", then i need more
guidance. E.g. about the required disk size and ways to curb the build
effort.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

it comes to me that i can get nearer to the Guix ISO production:

  # apt-get install grub-pc grub-efi-amd64-bin grub-efi-ia32-bin mtools
  ...
  # grub-mkrescue -o /tmp/test.iso /gnu
  xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
  ...
  ISO image produced: 652920 sectors
  Written to medium : 652920 sectors at LBA 0

  # ls -l /tmp/test.iso
  -rw-r--r-- 1 root root 1337180160 Dec 15 20:09 /tmp/test.iso

  # expr 1337180160 / 2048
  652920

  # xorriso -indev /tmp/test.iso
  ... no complaints ...

And with GNU xorriso 1.5.0 :

  # rm /tmp/test.iso
  # grub-mkrescue --xorriso=/home/thomas/xorriso-1.5.0/xorriso/xorriso \
  >               -o /tmp/test.iso /gnu
  GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.
  ...
  ISO image produced: 652920 sectors
  Written to medium : 652920 sectors at LBA 0

  # ls -l /tmp/test.iso
  -rw-r--r-- 1 root root 1337180160 Dec 15 20:15 /tmp/test.iso

  # xorriso -indev /tmp/test.iso
  ... no complaints ...

All looks well.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> I have now a qemu-system-i386 VM with Debian GNU/Linux from
> debian-9.6.0-i386-netinst.iso without desktop environment and reachable
> via SSH. Very minimal. (I only did "apt-get install build-essential" to
> feel not lonely without C compiler and friends.)

If you’re testing in a VM you might just as well download the GuixSD VM
image from <https://www.gnu.org/software/guix/download/>.  It should be
simpler than installing Debian and then installing Guix on top of
Debian.

> Then i followed the instructions of
>   https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
> with
>   https://alpha.gnu.org/gnu/guix/guix-binary-0.16.0.i686-linux.tar.xz
>   https://alpha.gnu.org/gnu/guix/guix-binary-0.16.0.i686-linux.tar.xz.sig
> up to step 7 ("guix archive --authorize ...").
>
> Then i made the mistake to do the proposed
>
>   guix package -i hello
>
> It downloads and builds and blows away the free space on the virtual 8 GB
> disk ... /gnu is growing steadily and /tmp breathes between 50 MB and 2 GB.
> I abort this after 100 minutes before the virtual disk gets too full and
> my CPU melts.

Did you actually run “guix archive --authorize < …/ci.guix.info.pub”?

  https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html

If you didn’t, then you are not getting pre-built binaries and thus you
end up building the world.

HTH,
Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> If you’re testing in a VM you might just as well download the GuixSD VM
> image from <https://www.gnu.org/software/guix/download/>.

There i only see only "x86_64" for QEMU, not "i686" like with ISO or Binary.


> Did you actually run “guix archive --authorize < …/ci.guix.info.pub”?

I did step 7 of Binary-Installation.html:

   guix archive --authorize < \
     ~root/.config/guix/current/share/guix/hydra.gnu.org.pub

The text "ci.guix.info.pub" does not appear in
  https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html

Looking at the existing state:

  # ls -l ~root/.config/guix/current/share/guix/
  total 12
  -r--r--r-- 1 root root  118 Jan  1  1970 berlin.guixsd.org.pub
  -r--r--r-- 1 root root  118 Jan  1  1970 ci.guix.info.pub
  -r--r--r-- 1 root root 1083 Jan  1  1970 hydra.gnu.org.pub

Shall i authorize the others too ?
If so: Is there need for clean-up actions after the aborted build runs ?


(If you find a bit of time, please run grub-mkrescue with some arbitrary
 input tree of about the size of the Guix ISO and check whether it gets
 truncated. If so, the messages from xorriso would be very interesting.)


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Ludovic Courtès wrote:
>> If you’re testing in a VM you might just as well download the GuixSD VM
>> image from <https://www.gnu.org/software/guix/download/>.
>
> There i only see only "x86_64" for QEMU, not "i686" like with ISO or Binary.

You’re right, my bad.

>> Did you actually run “guix archive --authorize < …/ci.guix.info.pub”?
>
> I did step 7 of Binary-Installation.html:
>
>    guix archive --authorize < \
>      ~root/.config/guix/current/share/guix/hydra.gnu.org.pub
>
> The text "ci.guix.info.pub" does not appear in
>   https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html

Oops, that was an omission that I’ve just fixed.

So yes, please authorize “ci.guix.info.pub” since https://ci.guix.info
is now the default substitute server.

HTH!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> Oops, that was an omission that I’ve just fixed.

Sometimes you need a clueless test user to clean the pipes. 

I now succeeded in running the ISO production command, but the truncation
problem is not reproducible here.

Please re-consider local reasons ... yada yada ... my main suspect would
be the immediate end of VM after the xorriso run. Maybe some buffers don't
get flushed down to the real disk ?

------------------------------------------------------------------------
What i did in detail:

I removed /gnu and /var/guix to get to a halfways clean state for
repeating steps 2 and 7 of
  https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
I.e. i unpacked the tarball, moved the trees to /gnu and /var/guix,
and authorized ci.guix.info.pub.

Then i did step 8
  guix package -i glibc-locales
This lasted 12 minutes (mainly with building 7 packages).

Now the proposed command to "confirm that Guix is working":
  guix package -i hello
lasted only about 30 seconds.

Scrolling back in my mailbox to
  Date: Thu, 06 Dec 2018 16:34:02 +0100
  Message-ID: <87va46is9h.fsf@gnu.org>

> Then you would need to run “guix pull” to get a current Guix (0.15.0
> itself didn’t have this bug.)

Do i still need this ? My tarball was already "0.16.0":
  guix-binary-0.16.0.i686-linux.tar.xz

I bet on omitting this step and go on with:

>   guix system disk-image --file-system-type=iso9660 \
>    -s i686-linux \
>    ~/.config/guix/current/share/guile/site/2.2/gnu/system/install.scm

After 5 minutes i see boot messages of a Linux kernel.
Oh. Qemu running on qemu. (The local power plant shifts one gear up.)

12 minutes elapsed and xorriso has started. Sloowly adding files:

  registering 302 items
  GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.
  ...
  45981 files added in 94 seconds
  ...
  xorriso : UPDATE : Thank you for being patient. Working since 265 seconds.
  ISO image produced: 500069 sectors
  Written to medium : 500069 sectors at LBA 0
  Writing to 'stdio:/xchg/guixsd.iso' completed successfully.

So far the xorriso run looks ok.
  ...
  /gnu/store/a8wwjfihb161maww0c8x4r797prdn8rr-image.iso

So this is where the ISO ended up.

  # ls -l /gnu/store/a8wwjfihb161maww0c8x4r797prdn8rr-image.iso
  -r--r--r-- 2 root root 1024141312 Jan  1  1970 /gnu/store/a8wwjfihb161maww0c8x4r797prdn8rr-image.iso

  # expr 1024141312 / 2048
  500069

  # xorriso -indev /gnu/store/a8wwjfihb161maww0c8x4r797prdn8rr-image.iso
  ... no complaints about size mismatch ...
  Media summary: 1 session, 500069 data blocks,  977m data, 3052m free

Well, then with
  guix pull
and then again
  guix system disk-image ...
lasts 30 minutes,

  # time guix system disk-image --file-system-type=iso9660 \
    -s i686-linux \
    ~/.config/guix/current/share/guile/site/2.2/gnu/system/install.scm
  ...
    GUILEC   gnu/packages/emacs.go
  GC Warning: Failed to expand heap by 8388608 bytes
  ...
  GC Warning: Out of Memory! Heap size: 943 MiB. Returning NULL!
  ...
  guix system: error: build failed: build of `/gnu/store/vr5mhnh430qabrrc1a82pv954b89axws-guix-0.16.0-4.60b0402.drv' failed
  
  real    21m55.875s
  user    0m5.816s
  sys     0m1.384s
  #

Looks like my VM needs more memory for that stunt.
So again with 2 GiB.

... it seems that "guix pull" brought back the addiction to world building.
I abort after 50 minutes while it is doing some qemu tests.

------------------------------------------------------------------------

Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hello,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>> Oops, that was an omission that I’ve just fixed.
>
> Sometimes you need a clueless test user to clean the pipes. 
>
> I now succeeded in running the ISO production command, but the truncation
> problem is not reproducible here.

It’s not reproducible because I “fixed” it:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=178be030c0e4fdeac5e1c968b5c99d84bb4691db

You should be able to reproduce it by running Guix from the parent
commit:

  guix pull --commit=676c3adc14f63df0f7a549e518ac87481c0f3e37

‘guix pull’ populates ~/.config/guix/current/bin/guix so you’ll have to
make sure this is the one you’re running when you try to reproduce the
issue.

Thanks for your help and perseverance!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> It’s not reproducible because I “fixed” it:
>  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=178be030c0e4fdeac5e1c968b5c99d84bb4691db
(This adds "-padding 10m" to the run of xorriso.)

No. The padding only moves the missing end piece into a region of the
image file where it does not matter for the filesystem payload files.
The ISO filesystem's meta data and the partition tables would still claim
the missing bytes of the image file, if the problem occured.

E.g. xorriso notices the mismatch in the ISO to which you pointed
me for download and which was most probably produced with -padding 10m:

  $ xorriso -indev guixsd-install-0.16.0.i686-linux.iso
  ...
  libisoburn: WARNING : ISO image size 481129s larger than readable size 479184s
  ...
  libburn : SORRY : Read start address 481128s larger than number of readable blocks 479184
  ...

The GPT in the ISO says that its backup header is at 512-byte block
1,924,515 = block 481,128.75 in units of 2048 bytes.

Highest file block is 475879 + 87 = 475966
   File data lba:  0 ,   475879 ,       88 ,   180224 , '/var/guix/db/db.sqlite'
which is a bit more than than 10 MiB before the expected image file end.
Given the lack of 1945 blocks at the image file end, the payload file end
is still more than 6 MB away from the escarpment.

-----------------------------------------------------------------------

But the ISO which i produced myself is healthy in that aspect.
The used software version is obviously before the 10 MiB padding.

The ISO contains as many bytes

  -r--r--r-- 2 root root 1024141312 Jan  1  1970 before_guix_pull.iso

as the ISO filesystem believes to cover, including the padding:

  Media summary: 1 session, 500069 data blocks,  977m data, 2187m free

Highest data file block is 499788 + 87 = 499875 :

  File data lba:  0 ,   499788 ,       88 ,   180224 , '/var/guix/db/db.sqlite'

which means that at most 194 blocks are expected to follow the end of
this file, not 10 MiB.
The GPT in the image says that its backup header block is at 512-byte
address 2,000,275 which is 500,068.75 in blocks of 2048 bytes.

So the inner size counters and image file size do match exactly.

This was done with guix from
  guix-binary-0.16.0.i686-linux.tar.xz
and with authorized ci.guix.info.pub.


>   guix pull --commit=676c3adc14f63df0f7a549e518ac87481c0f3e37

After "guix pull" the ISO production command indulges in building and
testing endlessly.
You will have to give me instructions how to get back to the ~ 12 minutes
of ISO production time which i had before trying "guix pull".


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

aside from my problems with the building and testing after "guix pull"
i also stand puzzled in front of the 8 files named "/gnu/.../build/vm.scm"
which all start grub-mkrescue.

If i'd succeed in reproducing the ISO image file truncation:
Which vm.scm file would i have to modify in order to report the size of
the freshly emerged ISO image in the filesystem of the upper VM ?
(I would suspect that this size is still untruncated and that the file
 in the underlying VM's filesystem is then truncated.)

And how to say "ls -l $target" in Guile ?


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> aside from my problems with the building and testing after "guix pull"
> i also stand puzzled in front of the 8 files named "/gnu/.../build/vm.scm"
> which all start grub-mkrescue.
>
> If i'd succeed in reproducing the ISO image file truncation:
> Which vm.scm file would i have to modify in order to report the size of
> the freshly emerged ISO image in the filesystem of the upper VM ?

None of those under /gnu/store.  /gnu/store is explicitly read-only.
The actual source code you’d edit is a checkout of Guix.  See
<https://www.gnu.org/software/guix/manual/en/html_node/Building-from-Git.html>.

> And how to say "ls -l $target" in Guile ?

In Scheme?  You could use ‘scandir’:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use (ice-9 ftw)
scheme@(guile-user)> (scandir "/")
$2 = ("." ".." "bin" "boot" "data" "dev" "etc" "gnu" "home" "lost+found" "mnt" "proc" "root" "run" "sys" "tmp" "var")
--8<---------------cut here---------------end--------------->8---

and also ‘lstat’, etc., but that’s not quite a “shell”.

HTH,
Ludo’.

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> ‘lstat’

Probably this.


> but that’s not quite a “shell”.

If i could reproduce the problem then i would want a long time visible
message about how large the ISO image file is after grub-mkrescue has
ended successfully.
This would give an opportunity to compare the size as produced in the VM
with the size later perceived on the host machine (which is a VM, too,
in my case).
If the sizes differ, then the VM contraption is to blame.
If the size is too small already in the VM that ran grub-mkrescue, then
xorriso or the VM operating system are to blame.

Since i am not yet able to reproduce the problem, i propose that you add
the necessary code to then end of make-iso9660-image in gnu/build/vm.scm.
Such a report message cannot harm, given the existing verbosity of the
ISO build command.

Next time you make an ISO, retrieve the last size messages of xorriso:
  ISO image produced: 500069 sectors
  Written to medium : 500069 sectors at LBA 0
the new message about the ISO image file size in bytes, and the size of
the ISO image file size when it is finally ready for exposure in the web.

(I have to stress that the problem is not fixed but only got a band aid
 of which it is not known whether its size will always be large enough.)


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

I have what may be the same problem on my x86_64 machine building for
x86_64 when creating an ISO install image by running

guix system disk-image --file-system-type=iso9660 gnu/system/install.scm

Since commit 45c0d1d790f01ebc020fc4b2787a6abcdaa3f383 increased the
RAM for the VM that builds the iso image from 256 to 512, iso files
consistently were corrupt, until I added an lstat call, see below.  On
a second and third attempt to build with lstat I got a corrupt image
again.  Guix install iso files I tested from before that commit were
fine.


florian@florianmacbook ~$ fdisk /gnu/store/4nrwajlpab4s8pdph4d77ww7716sa3ir-image.iso
[…]
GPT PMBR size mismatch (3231107 != 3200391) will be corrected by write.

xorriso is sorry exactly like in Ludo’s message from December 06.  The
numbers reported and file sizes are not consistent between corrupt
rebuilds.



On Fri, Dec 21, 2018 at 10:42:14PM +0100, Thomas Schmitt wrote:
> […]
> Next time you make an ISO, retrieve the last size messages of xorriso:
>   ISO image produced: 500069 sectors
>   Written to medium : 500069 sectors at LBA 0

For the corrupt iso with lstat call:

ISO image produced: 807777 sectors
Written to medium : 807777 sectors at LBA 0



> the new message about the ISO image file size in bytes,

Within the VM lstat consistently reports 1654327296 for non-corrupt
and corrupt images alike.



> and the size of
> the ISO image file size when it is finally ready for exposure in the web.
>

ls -l on the result reports 1638600704.

On the non-corrupt image after adding the lstat call, both lstat
within the VM and ls -l outside the VM print the same size: 1654327296
in this case, i.e. the same as lstat reported on the corrupt images
within the VM.


(To be precise, for lstat I added the following local git commit to my
copy of the Guix repo at the end of the G-expression executed by the
VM:

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index db9b1707d7..18ccb8970e 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -309,7 +309,8 @@ INPUTS is a list of inputs (as for packages)."
                                  #:closures graphs
                                  #:volume-id #$file-system-label
                                  #:volume-uuid #$(and=> file-system-uuid
-                                                        uuid-bytevector))))))
+                                                        uuid-bytevector))
+             (error (lstat "/xchg/guixsd.iso"))))))
    #:system system
 
    ;; Keep a local file system for /tmp so that we can populate it directly as



and then reconfigured the system after customizing the guix package to
use said commit and disabling tests on the guix package.  This
reported an lstat Scheme object as an error.  Note that the error
procedure does not cause a failed build.)

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> fdisk /gnu/store/4nrwajlpab4s8pdph4d77ww7716sa3ir-image.iso
> [...]
> GPT PMBR size mismatch (3231107 != 3200391) will be corrected by write.
> For the corrupt iso with lstat call:
> and corrupt images alike.

The GPT Protective MBR counts with block size 512 up to the GPT backup
header block, not counting itself at block 0. So in blocks of 2048, the
expected size is
  3231108 / 4 = 807777 ISO 9660 blocks
But the perceived size is
  3200392 / 4 = 800098 ISO 9660 blocks

I wrote:
> > retrieve the last size messages of xorriso:

> For the corrupt iso with lstat call:
> ISO image produced: 807777 sectors
> Written to medium : 807777 sectors at LBA 0
> Within the VM lstat consistently reports 1654327296 for non-corrupt
> and corrupt images alike.

  1654327296 / 2048 = 807777
So from the view of the VM the ISO is as large as xorriso believes to have
written and as the GPT announces as position of the backup header block.


> > and the size of
> > the ISO image file size when it is finally ready for exposure in the web.

> ls -l on the result reports 1638600704.

  1638600704 / 2048 = 800098
This matches the perceived size from the fdisk complaint.


> On the non-corrupt image after adding the lstat call, both lstat
> within the VM and ls -l outside the VM print the same size: 1654327296

The fact that the VM always sees the expected size but the host sees varying
sizes supports the suspicion that at the end of the VM its i/o buffers or
virtual disk are not always properly flushed to the i/o system of the host.
The varying success smells like a race condition.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

Hello,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> The fact that the VM always sees the expected size but the host sees varying
> sizes supports the suspicion that at the end of the VM its i/o buffers or
> virtual disk are not always properly flushed to the i/o system of the host.
> The varying success smells like a race condition.

Indeed, that rings a bell: I fixed a similar issue in commit
0dc7d298a33f83d5f02a962b5f1bd24ee0e8ef07.

Florian: could you check whether the patch below solves the problem for
you?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1340 bytes --]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index db9b1707d7..3ee03c84a0 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -240,7 +240,11 @@ made available under the /xchg CIFS share."
                                   #:target-arm32? #$(target-arm32?)
                                   #:disk-image-format #$disk-image-format
                                   #:disk-image-size size
-                                  #:references-graphs graphs))))))
+                                  #:references-graphs graphs)
+
+                ;; Make sure I/O buffers get flushed.  This is particularly
+                ;; important when MAKE-DISK-IMAGE? is true.
+                (sync))))))
 
     (gexp->derivation name builder
                       ;; TODO: Require the "kvm" feature.
@@ -530,10 +534,7 @@ should set REGISTER-CLOSURES? to #f."
                  #$os
                  #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
                  #:creation-time (make-time time-utc 0 1)
-                 #:transformations `((,root-directory -> "")))
-
-                ;; Make sure the tarball is fully written before rebooting.
-                (sync))))))
+                 #:transformations `((,root-directory -> ""))))))))
     (expression->derivation-in-linux-vm
      name build
      #:make-disk-image? #f

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Mon, Apr 08, 2019 at 10:50:29AM +0200, Ludovic Courtès wrote:
> Hello,
> 
> "Thomas Schmitt" <scdbackup@gmx.net> skribis:
> 
> > The fact that the VM always sees the expected size but the host sees varying
> > sizes supports the suspicion that at the end of the VM its i/o buffers or
> > virtual disk are not always properly flushed to the i/o system of the host.
> > The varying success smells like a race condition.
> 
> Indeed, that rings a bell: I fixed a similar issue in commit
> 0dc7d298a33f83d5f02a962b5f1bd24ee0e8ef07.
> 
> Florian: could you check whether the patch below solves the problem for
> you?
> 
> Thanks,
> Ludo’.
> 

No, sadly not.  I reconfigured to a commit with the Guix package
changed to use your patch and I again got this:

GPT PMBR size mismatch (3231103 != 3187775) will be corrected by write.
libburn : SORRY : Read start address 807775s larger than number of readable blocks 796944

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> > Florian: could you check whether the patch below solves the problem for
> > you?

Florian Pelz wrote:
> No, sadly not.

Given the smell of a race condition, i would next try to let the VM
wait 10 or 15 seconds after xorriso is finished and before it shuts down.

Not as a final remedy but just as proof that the VM end is really the
culprit. (It could also be an i/o problem between VM and host which
is unrelated to the VM end.)


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Wed, Apr 10, 2019 at 01:17:14PM +0200, Thomas Schmitt wrote:
> Given the smell of a race condition, i would next try to let the VM
> wait 10 or 15 seconds after xorriso is finished and before it shuts down.
> 

I added a (sleep 15) after ludo’s (sync).  The first image worked but
now I got

libburn : SORRY : Read start address 807777s larger than number of readable blocks 798640

again.

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

Hello Florian & Thomas,

I was able to reproduce the issue: ‘guix system disk-image
--file-system-format=iso9660’ would create partly unreadable images.

Since this was pretty much like the issue I had encountered with ‘guix
system docker-image’, which would produce truncated tarballs, and since
calling ‘sync’ wasn’t enough, I looked at our file system mount options…

The attached patch fixes the problem for me.  In hindsight, it’s not
surprising that “cache=loose” on the /xchg mount point (used to exchange
data between the host and the guest) would have this effect.

Florian, it would be great if you could confirm.  Just apply it on
‘master’, and then run:

  ./pre-inst-env guix system disk-image --file-system-format=iso9660 \
     gnu/system/install.scm

Thanks, and apologies for blaming Xorriso, which presumably never had
anything to do with it!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2008 bytes --]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index db9b1707d7..22e3fcc522 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -94,6 +94,12 @@
 (define %linux-vm-file-systems
   ;; File systems mounted for 'derivation-in-linux-vm'.  These are shared with
   ;; the host over 9p.
+  ;;
+  ;; The 9p documentation says that cache=loose is "intended for exclusive,
+  ;; read-only mounts", without additional details.  It's much faster than the
+  ;; default cache=none, especially when copying and registering store items.
+  ;; Thus, use cache=loose, except for /xchg where we want to ensure
+  ;; consistency.
   (list (file-system
           (mount-point (%store-prefix))
           (device "store")
@@ -102,18 +108,12 @@
           (flags '(read-only))
           (options "trans=virtio,cache=loose")
           (check? #f))
-
-        ;; The 9p documentation says that cache=loose is "intended for
-        ;; exclusive, read-only mounts", without additional details.  In
-        ;; practice it seems to work well for these, and it's much faster than
-        ;; the default cache=none, especially when copying and registering
-        ;; store items.
         (file-system
           (mount-point "/xchg")
           (device "xchg")
           (type "9p")
           (needed-for-boot? #t)
-          (options "trans=virtio,cache=loose")
+          (options "trans=virtio")
           (check? #f))
         (file-system
           (mount-point "/tmp")
@@ -530,10 +530,7 @@ should set REGISTER-CLOSURES? to #f."
                  #$os
                  #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
                  #:creation-time (make-time time-utc 0 1)
-                 #:transformations `((,root-directory -> "")))
-
-                ;; Make sure the tarball is fully written before rebooting.
-                (sync))))))
+                 #:transformations `((,root-directory -> ""))))))))
     (expression->derivation-in-linux-vm
      name build
      #:make-disk-image? #f

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

> apologies for blaming Xorriso, which presumably never had
> anything to do with it!

I will not complain that this time it was not my fault.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Fri, Apr 12, 2019 at 11:26:28PM +0200, Ludovic Courtès wrote:
> Florian, it would be great if you could confirm.  Just apply it on
> ‘master’, and then run:
> 
>   ./pre-inst-env guix system disk-image --file-system-format=iso9660 \
>      gnu/system/install.scm
> 

Yes, it seems fixed, I can confirm.  Four rebuilds seem fine and are
bootable in QEMU.  They have the same size and `xorriso -indev` is
happy.  The content is different at the beginning of the ISO image
(maybe padding or timestamps in the file system) and in the EFI
partition at the very end of the ISO, but this seems insignificant.

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Yes, it seems fixed, I can confirm.

Way back in december, Ludovic Courtès wrote:
>...> Based on this and on a suggestion Ricardo made on IRC, I passed
>...> -padding 10m and that solved the problem.  \o/

Please do not forget to remove this -padding command.


Florian Pelz wrote:
>  The content is different at the beginning of the ISO image
> (maybe padding or timestamps in the file system)

That's to expect if not environment SOURCE_DATE_EPOCH is set and exported.

SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It
is supposed to be either undefined or to contain a decimal number which
tells the seconds since january 1st 1970. If it contains a number, then
it is used for all timestamps and as seed of pseudo-random numbers like
MBR id or GPT UUIDs.

If all files and directories have the same names and the same content,
then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
value are supposed to create byte-identical result ISOs.

In december, i wrote:
>...> >   Creation Time: 1970010119010649
Ludovic Courtès wrote:
>...> For reproducibility purposes we set timestamps and related things
>...> to the Epoch.

Is this independent of SOURCE_DATE_EPOCH ?


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Fri, Apr 12, 2019 at 11:26:28PM +0200, Ludovic Courtès wrote:
>> Florian, it would be great if you could confirm.  Just apply it on
>> ‘master’, and then run:
>> 
>>   ./pre-inst-env guix system disk-image --file-system-format=iso9660 \
>>      gnu/system/install.scm
>> 
>
> Yes, it seems fixed, I can confirm.  Four rebuilds seem fine and are
> bootable in QEMU.

This is a happy end.  :-)
Committed as 66ec389580d4f1e4b81e1c72afe2749a547a0e7c.

Thank you!

Ludo’.

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Florian Pelz wrote:
>> Yes, it seems fixed, I can confirm.
>
> Way back in december, Ludovic Courtès wrote:
>>...> Based on this and on a suggestion Ricardo made on IRC, I passed
>>...> -padding 10m and that solved the problem.  \o/
>
> Please do not forget to remove this -padding command.

Done in f6e3f0f9b1287eca120517a0161e3d0b1ed6ed44.

> If all files and directories have the same names and the same content,
> then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
> value are supposed to create byte-identical result ISOs.

I’ve tried setting it but that doesn’t make any difference.

How did you visualize differences, Florian?  Diffoscope fails for me
here (missing tools and scalability issue.)

> In december, i wrote:
>>...> >   Creation Time: 1970010119010649
> Ludovic Courtès wrote:
>>...> For reproducibility purposes we set timestamps and related things
>>...> to the Epoch.
>
> Is this independent of SOURCE_DATE_EPOCH ?

Yes.

Thanks,
Ludo’.

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Sun, Apr 14, 2019 at 11:43:54PM +0200, Ludovic Courtès wrote:
> How did you visualize differences, Florian?  Diffoscope fails for me
> here (missing tools and scalability issue.)
> 

For me diffoscope failed too.  I used cmp as described here:

https://superuser.com/questions/125376/how-do-i-compare-binary-files-in-linux

and then looked at the addresses in ghex. It is not a nice method.
Sorry.  It works though.

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

I wrote:
> > If all files and directories have the same names and the same content,
> > then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
> > value are supposed to create byte-identical result ISOs.

Ludovic Courtès wrote:
> I’ve tried setting it but that doesn’t make any difference.

We should investigate this ...
... yes, there is some problem. But not always.

Timestamps of the root directory differ after mapping to an address
that is not the ISO root directory (here: /x):

  xorriso -outdev test.iso -map x /x
  xorriso -outdev test2.iso -map x /x

but not after mapping to the root directory:

  xorriso -outdev test.iso -map x /
  xorriso -outdev test2.iso -map x /

This would explain why my tests for Debian ISOs do not show this problem.

Do i get it right that gnu/build/vm.scm maps no files to "/" but all to
deeper paths:
                "etc=/tmp/root/etc"
                "var=/tmp/root/var"
                "run=/tmp/root/run"
I am unsure about
                "-path-list" "-"


I will now dig into the source to find the reason and maybe a preliminary
remedy.


> How did you visualize differences, Florian?

(I'm aware that i am not Florian.)

I made myself a little program "hxd" for combined hex-cleartext-decimal dump,
positional diff, and (not to be focused too much) CD-Text decoding.

===========================================================================

$ export SOURCE_DATE_EPOCH=$(date +%s)
$ xorriso -outdev test.iso -map x /x
...
xorriso : NOTE : Environment variable SOURCE_DATE_EPOCH encountered with value 1555311212
...
$ xorriso -outdev test2.iso -map x /x
...
xorriso : NOTE : Environment variable SOURCE_DATE_EPOCH encountered with value 1555311212
...
$ hxd -diff test.iso test2.iso

   32944 :    15   7  38  43   0   2   0   0   1   0   0   1   1   0  32  32
                       &   +
000080b0 :    0f  07  26  2b  00  02  00  00  01  00  00  01  01  00  20  20
                         ###
000080b0 :    0f  07  26  36  00  02  00  00  01  00  00  01  01  00  20  20
                       &   6
   32944 :    15   7  38  54   0   2   0   0   1   0   0   1   1   0  32  32

... more differences ...

===========================================================================

It looks like the root directory got the current timestamp. The other
differences are with the ".." directory entries of the directories in
the first level under "/".


The source of "hxd" is pure C, no special dependencies, 8141 bytes.
Shall i upload it somewhere ?


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

it seems to help if you explicitely set the timestamps of the "/" directory

  export SOURCE_DATE_EPOCH=1555311212

  xorriso -outdev test.iso -map x /x \
          -alter_date b-c 1970010100000000 / -- \
          -alter_date c 1970010100000000 / --

ISOs made with these xorriso commands match perfectly.

A bit more elegant than 1970 would be to use the seconds value from
SOURCE_DATE_EPOCH (prefix "=" announces date +%s format):

          -alter_date b-c =$SOURCE_DATE_EPOCH / -- \
          -alter_date c   =$SOURCE_DATE_EPOCH / --

The -alter_date commands should be performed after all -map commands,
just to make sure that the timestamps do not get changed again.

I still need to find out where the current time sneaks in.
But this workaround should not do harm after the bug was corrected.


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Sat, Apr 13, 2019 at 03:46:09PM +0200, pelzflorian (Florian Pelz) wrote:
> Yes, it seems fixed, I can confirm.

Well this is strange.  I got fine ISO images each time (fine with no
complaints from xorriso or fdisk and bootable in QEMU without errors),
but after dd’ing them to different USB flash drives each time I get
kernel output when inserting the flash drive:

[   10.025223] GPT:Primary header thinks Alt. header is not at the end of the disk.
[   10.026735] GPT:3220583 != 7831551
[   10.028235] GPT:Alternate GPT header not at the end of the disk.
[   10.029764] GPT:3220583 != 7831551
[   10.031290] GPT: Use GNU Parted to correct GPT errors.


Having such a USB flash drive inside my computer makes UEFI get stuck
on some computers but not on others.

Why is this?  Are all my USB drives bad?  I presume this is a
different bug, or is it?

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Well this is strange.  I got fine ISO images each time (fine with no
> complaints from xorriso or fdisk and bootable in QEMU without errors),
> but after dd’ing them to different USB flash drives each time I get
> kernel output when inserting the flash drive:
> [   10.025223] GPT:Primary header thinks Alt. header is not at the end of
> the disk.

The alternative/backup header is a property of GPT which makes it
rather unsuitable for disk images. xorriso puts it correctly into the
last 512-byte block of the image. But when copied to a storage device,
it should move up to the last block of the device.
Even worse, the main GPT header at 512-byte LBA 1 needs to learn the
new address.

So i would rather advise to use a MBR partition table. Wonderfully dumb
and open ended.

I see from
  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/vm.scm#n462
that program grub-mkrescue is in control of xorrisofs boot options.
Vladimir Serbinenko decided for GPT with no mountable ISO partition.

The libisoburn repo and tarball have a wrapper script by which other
boot layouts can be derived from the options which grub-mkrescue hands
over to xorrisofs:

  https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

To get MBR instead of GPT do:

  export MKRESCUE_SED_MODE=mbr_only
  export MKRESCUE_SED_PROTECTIVE=""

and maybe

  export MKRESCUE_SED_XORRISO=/...path/to/the/xorriso/binary/if/exotic...

Then start grub-mkrescue with the wrapper in the role of "xorriso":

  grub-mkrescue --xorriso=...path/to/grub-mkrescue-sed.sh \
                -partition_offset 16 \
                -iso_mbr_part_type 0x83 \
                \
                ...all.other.usual.arguments...

The mode "mbr_only" will move the EFI partition image out of the ISO
filesystem and rather append it after the ISO's end.

The option
  -partition_offset 16
costs the space of a second superblock and directory tree. But it brings
as benefits:
- More normal partition layout with partition 1 starting at block 64
  rather than at block 0.
- Nevertheless the partition 1 is mountable and shows the ISO content.
- The base device is mountable as the the same ISO too.
  (The ISO superblock of the base device also serves on CD or DVD.)
- Th base device superblock claims not only the ISO in partition 1 but
  also the EFI partition 2. So "/sbin/isosize" will tell the size of the
  image file, not only of the ISO filesystem.

Option
  -iso_mbr_part_type 0x83
chooses for partition 1 the MBR partitions type "Linux". (This is
purely ornamental. Nobody cares. But it looks good in partition editors.)

The partition layout of above wrapper run's output ISO will look like:

  $ /sbin/fdisk -l output.iso
  Disk output.iso: 16.5 MiB, 17338368 bytes, 33864 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0x00000000

  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28103   28040 13.7M 83 Linux
  output.iso2      28104 33863    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  33864


Have a nice day :)

Thomas

bug#33639: ISO installer image is broken on i686

[Gábor Boskovits]

Hello people,

Thomas Schmitt <scdbackup@gmx.net> ezt írta (időpont: 2019. ápr. 15., H, 19:54):
>
> Hi,
>
> Florian Pelz wrote:
> > Well this is strange.  I got fine ISO images each time (fine with no
> > complaints from xorriso or fdisk and bootable in QEMU without errors),
> > but after dd’ing them to different USB flash drives each time I get
> > kernel output when inserting the flash drive:
> > [   10.025223] GPT:Primary header thinks Alt. header is not at the end of
> > the disk.
>
> The alternative/backup header is a property of GPT which makes it
> rather unsuitable for disk images. xorriso puts it correctly into the
> last 512-byte block of the image. But when copied to a storage device,
> it should move up to the last block of the device.
> Even worse, the main GPT header at 512-byte LBA 1 needs to learn the
> new address.
>

Yes, this is a really painful point.

Could we create a simple tool to write the disk images to a disk
correcting this problem?

Does not look too hard?

I am also forwarding this to guix devel. I removed the xorriso bug
list, as I feel this does not belong there.

Best regards,
g_bor

Re: bug#33639: ISO installer image is broken on i686

[Gábor Boskovits]

Hello people,

Thomas Schmitt <scdbackup@gmx.net> ezt írta (időpont: 2019. ápr. 15., H, 19:54):
>
> Hi,
>
> Florian Pelz wrote:
> > Well this is strange.  I got fine ISO images each time (fine with no
> > complaints from xorriso or fdisk and bootable in QEMU without errors),
> > but after dd’ing them to different USB flash drives each time I get
> > kernel output when inserting the flash drive:
> > [   10.025223] GPT:Primary header thinks Alt. header is not at the end of
> > the disk.
>
> The alternative/backup header is a property of GPT which makes it
> rather unsuitable for disk images. xorriso puts it correctly into the
> last 512-byte block of the image. But when copied to a storage device,
> it should move up to the last block of the device.
> Even worse, the main GPT header at 512-byte LBA 1 needs to learn the
> new address.
>

Yes, this is a really painful point.

Could we create a simple tool to write the disk images to a disk
correcting this problem?

Does not look too hard?

I am also forwarding this to guix devel. I removed the xorriso bug
list, as I feel this does not belong there.

Best regards,
g_bor

ISO installer image: GPT versus MBR partitions (was bug#33639)

[Thomas Schmitt]

Hi,

Gábor Boskovits wrote:
> I removed the xorriso bug list, as I feel this does not belong there.

As its admin i say that everything xorriso related belongs there. :))

But maybe we should remove  33639@debbugs.gnu.org  from the Cc ?
I herby give this a try.


i wrote:
> > The alternative/backup header is a property of GPT which makes it
> > rather unsuitable for disk images.

Gábor Boskovits wrote:
> Could we create a simple tool to write the disk images to a disk
> correcting this problem?

There are GPT partition editors which offer to repair the backup block
problem. gdisk has in its expert's menu 'x':
  e       relocate backup data structures to the end of the disk

The job is indeed not very difficult. Backup header block and backup
table entries array need to be moved up. Then a few block addresses
in both header blocks need to be updated. Finally new CRCs need to
be computed and put into the headers.

More headache is in the wish to have a fool-proof tool for copying
the image file to USB stick.
Compared with the risks of burning a DVD, the risks of writing a disk
image are quite unbearable. Covering all pitfalls might be a larger
adventure. (Ubuntu has mkusb: https://help.ubuntu.com/community/mkusb)

A problem with GPT relocation is that checksums of the image cannot be
verified on USB stick any more.
An MBR partitioned image would not have to be altered. /sbin/isosize could
tell the number of bytes which should be processed when comparing with
published checksums.

Further an MBR partitioned image would have no non-mountable partitions.

  $ /sbin/gdisk guixsd-install-0.15.0.i686-linux.iso
  ...
  Command (? for help): p
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     1              64           34107   16.6 MiB    0700  Gap0
     2           34108           39867   2.8 MiB     EF00  EFI boot partition
     3           39868         1815727   867.1 MiB   AF00  HFSPLUS
     4         1815728         1816327   300.0 KiB   0700  Gap1

Gap0 and Gap1 are not mountable. One can get access to the ISO's files
only via the base device. The HFS+ partition publishes the same files,
but you need to find a HFS+ driver which works with block size 2048.
(Even more difficult could be to find any machine that boots via
 APM and HFS+ and can make use of the binaries in ISO and HFS+.)

Repeating my example of yesterday of a grub-mkrescue ISO with MBR
partition table (and no APM, which could be added if really of interest):

  $ /sbin/fdisk -l output.iso
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28103   28040 13.7M 83 Linux
  output.iso2      28104 33863    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  33864

Of course, ISO and partition 1 would be much larger in a Guix ISO.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions (was bug#33639)

[pelzflorian (Florian Pelz)]

I would prefer an image that can be written just with dd, because then
it would also be easy to use another tool like gnome-multi-writer or
to use a not UNIX-like OS to create the install USB and give
GNU Guix System a try.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I would prefer an image that can be written just with dd,

A broken backup GPT is not supposed to stop EFI from finding its partition
and running its architecture specific start-up program.

But it becomes annoying when another partition shall be created for
a read-write filesystem to carry the user's extra data.

(With Debian and Fedora ISOs it's even worse. They carry a MBR partition
 table and GPT header blocks, which are tehrfore invalid. One has to zap
 them before letting fdisk create a new MBR partition.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 16, 2019 at 07:57:20PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I would prefer an image that can be written just with dd,
> 
> A broken backup GPT is not supposed to stop EFI from finding its partition
> and running its architecture specific start-up program.
> 

On the 2011 Macbook I am using, the boot process gets stuck before I
can select the drive to boot from, as long as a USB flash drive with
this GPT error is in a USB port.  The few other computers I tried have
no problem with booting from the flash drive.

This means I cannot install the Guix distribution on this Macbook
using the Guix ISO image.

(This is the default UEFI boot process that came with the Macbook.
I have never tried setting up Libreboot or anything similar.)

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
> On the 2011 Macbook I am using, the boot process gets stuck before I
> can select the drive to boot from, as long as a USB flash drive with
> this GPT error is in a USB port.  The few other computers I tried have
> no problem with booting from the flash drive.
> 

Actually it still gets stuck after I used fdisk to repair the drive’s
GPT.  I will check again with a Debian ISO tomorrow.

bug#33639: ISO installer image is broken on i686

[Ludovic Courtès]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Sat, Apr 13, 2019 at 03:46:09PM +0200, pelzflorian (Florian Pelz) wrote:
>> Yes, it seems fixed, I can confirm.
>
> Well this is strange.  I got fine ISO images each time (fine with no
> complaints from xorriso or fdisk and bootable in QEMU without errors),
> but after dd’ing them to different USB flash drives each time I get
> kernel output when inserting the flash drive:
>
> [   10.025223] GPT:Primary header thinks Alt. header is not at the end of the disk.
> [   10.026735] GPT:3220583 != 7831551
> [   10.028235] GPT:Alternate GPT header not at the end of the disk.
> [   10.029764] GPT:3220583 != 7831551
> [   10.031290] GPT: Use GNU Parted to correct GPT errors.

Could it be simply due to the incorrect location of the GPT backup as
Thomas explained?

> Having such a USB flash drive inside my computer makes UEFI get stuck
> on some computers but not on others.

So you cannot boot from these USB drives at all?

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > A broken backup GPT is not supposed to stop EFI from finding its partition

Florian Pelz wrote:
> On the 2011 Macbook I am using, the boot process gets stuck before I
> can select the drive to boot from, as long as a USB flash drive with
> this GPT error is in a USB port.

Did you already test whether it works after you treated the stick
with gdisk expert option 'x' 'e' ?

If it really does work, then i am glad to have written "supposed".

It would then be interesting to see whether this picky machine boots
from a MBR partition table.
Is
  guixsd-install-0.15.0.i686-linux.iso
suitable for that machine (modulo GPT backup position) ?
If so, i will use my copy of this ISO to develop a xorriso run on
the mounted ISO which produces a new ISO with MBR-only layout.


(next mail from Florian):
> Actually it still gets stuck after I used fdisk to repair the drive’s
> GPT.  I will check again with a Debian ISO tomorrow.

Did the GPT repair really work ?
I.e. do other partition editors or kernels have no complaints about the
stick any more ?


Another try would be to deface the Apple Partition Map. It's a Mac.
So APM and HFS+ might have some impact. But other than expected by
grub-mkrescue.

With APM the first bytes of the ISO are
  45  52  08  00  eb  02  ff  ff
(which is the APM magic number and also x86 code without effect.)

Without APM they are other intentionally effectless x86 code:
  eb  63  90  90  90  90  90  90

If you write the latter 8 bytes over the start of your USB stick, then
APM should not be recognizable any more.


> I will check again with a Debian ISO tomorrow.

Debian installation ISOs have APM but no HFS+. I.e. their APM is quite
useless. These are: "netinst" ISOs, and the ISO sets for CD, DVD, BD, BDDL.

Debian 9 Live ISOs have no APM (and no HFS+).
Version 8 has no EFI equipment but only for legacy BIOS.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

The Parabola ISO parabola-systemd-cli-i686-netinstall-2018.06.02.iso
does not inhibit the boot process but also contains no EFI partition
so I cannot boot from it.

Debian 9.8.0 works, see the end of this mail.







On Tue, Apr 16, 2019 at 11:15:56PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > A broken backup GPT is not supposed to stop EFI from finding its partition
> 
> Florian Pelz wrote:
> > On the 2011 Macbook I am using, the boot process gets stuck before I
> > can select the drive to boot from, as long as a USB flash drive with
> > this GPT error is in a USB port.
> 
> Did you already test whether it works after you treated the stick
> with gdisk expert option 'x' 'e' ?
> 

Nope, it does not.




> If it really does work, then i am glad to have written "supposed".
> 
> It would then be interesting to see whether this picky machine boots
> from a MBR partition table.
> Is
>   guixsd-install-0.15.0.i686-linux.iso
> suitable for that machine (modulo GPT backup position) ?

Without changing the ISO’s partition tables, it does not work.

What should I change?







> If so, i will use my copy of this ISO to develop a xorriso run on
> the mounted ISO which produces a new ISO with MBR-only layout.
> 
> 
> (next mail from Florian):
> > Actually it still gets stuck after I used fdisk to repair the drive’s
> > GPT.  I will check again with a Debian ISO tomorrow.
> 
> Did the GPT repair really work ?
> I.e. do other partition editors or kernels have no complaints about the
> stick any more ?
> 
> 

I have overwritten the fdisk-corrected one, but the gdisk-corrected
flash drive is considered correct by fdisk.



> Another try would be to deface the Apple Partition Map. It's a Mac.
> So APM and HFS+ might have some impact. But other than expected by
> grub-mkrescue.
> 
> With APM the first bytes of the ISO are
>   45  52  08  00  eb  02  ff  ff
> (which is the APM magic number and also x86 code without effect.)
> 
> Without APM they are other intentionally effectless x86 code:
>   eb  63  90  90  90  90  90  90
> 
> If you write the latter 8 bytes over the start of your USB stick, then
> APM should not be recognizable any more.
> 
>

I inserted the gdisk-corrected USB flash drive and did:

$ ghex bytes  # put in eb  63  90  90  90  90  90  90
$ sudo dd if=bytes of=/dev/sdc
Passwort: 
0+1 Datensätze ein
0+1 Datensätze aus
8 bytes copied, 0,030945 s, 0,3 kB/s

It still does not work.




> > I will check again with a Debian ISO tomorrow.
> 
> Debian installation ISOs have APM but no HFS+. I.e. their APM is quite
> useless. These are: "netinst" ISOs, and the ISO sets for CD, DVD, BD, BDDL.
> 
> Debian 9 Live ISOs have no APM (and no HFS+).
> Version 8 has no EFI equipment but only for legacy BIOS.
> 
> 
> Have a nice day :)
> 
> Thomas
> 

The Debian ISO works fine and is bootable from its EFI partition.

gdisk prints:

$ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: present

Found valid MBR and GPT. Which do you want to use?
 1 - MBR
 2 - GPT
 3 - Create blank GPT

Your answer: 2
Using GPT and creating fresh protective MBR.
Warning! Main partition table overlaps the first partition by 64 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help): p
Disk Downloads/debian-live-9.8.0-amd64-xfce.iso: 3811392 sectors, 1.8 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): C05B92F2-4B16-4592-BA53-A2025E420B60
Partition table holds up to 248 entries
Main partition table begins at sector 2 and ends at sector 63
First usable sector is 64, last usable sector is 3811328
Partitions will be aligned on 8-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   2            1432            2263   416.0 KiB   0700  ISOHybrid1

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Tue, Apr 16, 2019 at 11:01:45PM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> > Having such a USB flash drive inside my computer makes UEFI get stuck
> > on some computers but not on others.
> 
> So you cannot boot from these USB drives at all?
> 

No, I cannot boot from them on this Macbook.  I wonder how I installed
Guix System here; it may have been on a Debian ISO.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> $ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso
> ...
> Found valid MBR and GPT. Which do you want to use?
> 1 - MBR
> 2 - GPT
> ...
> Your answer: 2
> Using GPT and creating fresh protective MBR.
> Warning! Main partition table overlaps the first partition by 64 blocks!
> You will need to delete this partition or resize it in another utility.
> ...
> Number  Start (sector)    End (sector)  Size       Code  Name
>    2            1432            2263   416.0 KiB   0700  ISOHybrid1

This operation has killed the MBR partition table with the entry for
the EFI partition.
Due to the odd layout introduced for Fedora ISOs in SYSLINUX program
"isohybrid", there is a GPT partition entry which marks the EFI FAT image.
But it does not have the EFI type GUID. So EFI won't recognize it.

(This layout has been adopted by Debian and others who use ISOLINUX
 for BIOS and GRUB for EFI. It needs MBR partition type 0x00 for the
 ISO partition. The GPT ISO partition is madly starting at block 0.
 Partition editors are entitled to hate it.
 But well, the user decides what to do. I can only give advise.)

There remains the boot path of legacy BIOS via the MBR x86 code.


> The Debian ISO works fine and is bootable from its EFI partition.

Did you try booting before or after the gdisk run ?


> > Is guixsd-install-0.15.0.i686-linux.iso
> > suitable for that machine (modulo GPT backup position) ?

> Without changing the ISO’s partition tables, it does not work.
> What should I change?

I will develop and test a xorriso repack run for the mounted ISO
which produces MBR partitioning without GPT and APM.
Maybe this evening.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 12:23:56PM +0200, Thomas Schmitt wrote:
> This operation has killed the MBR partition table with the entry for
> the EFI partition.
> […]
> Did you try booting before or after the gdisk run ?
> 
>

Before.  I also did not write out.  Note that I also called gdisk on
the iso image and not on the USB disk.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
>> On the 2011 Macbook I am using, the boot process gets stuck before I
>> can select the drive to boot from, as long as a USB flash drive with
>> this GPT error is in a USB port.  The few other computers I tried have
>> no problem with booting from the flash drive.
>> 
>
> Actually it still gets stuck after I used fdisk to repair the drive’s
> GPT.  I will check again with a Debian ISO tomorrow.

Does it boot the 0.16.0 installation image?  It was built in exactly the
same way, with the misplaced GPT backup.

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 02:24:12PM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:
> 
> > On Tue, Apr 16, 2019 at 09:55:10PM +0200, pelzflorian (Florian Pelz) wrote:
> >> On the 2011 Macbook I am using, the boot process gets stuck before I
> >> can select the drive to boot from, as long as a USB flash drive with
> >> this GPT error is in a USB port.  The few other computers I tried have
> >> no problem with booting from the flash drive.
> >> 
> >
> > Actually it still gets stuck after I used fdisk to repair the drive’s
> > GPT.  I will check again with a Debian ISO tomorrow.
> 
> Does it boot the 0.16.0 installation image?  It was built in exactly the
> same way, with the misplaced GPT backup.
> 

No, it does not boot.  I believe when I installed this Guix System, I
used `guix system disk-image gnu/system/install.scm` without
specifying --file-system-type=iso9660.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

here are instructions to repack a Guix ISO with EFI support (via GPT)
to an ISO with MBR-only partition table.

Tested with guixsd-install-0.15.0.i686-linux.iso, because my copy
of 0.16.0 is one of the victims of bug#33639.

It becomes 130 MiB larger than the original, probably because there
were hardlinks in the original disk tree, which share their content in
the original ISO but which the Linux isofs driver cannot expose as
hardlink siblings when the ISO gets mounted.
E.g.
  /gnu/store/0br6mgbnc8b0myvvgpihhry9lfb67pdv-gmp-6.1.2/share/info/gmp.info-1.gz
  /gnu/store/2078rjc6rlj5yran6sx9kd5k4281j0vm-gmp-6.1.2/share/info/gmp.info-1.gz  /gnu/store/v78qymi7sxywv8l6kk5iydvvpdpncrsv-gmp-6.1.2/share/info/gmp.info-1.gz
have all the same LBA in the original ISO. But Linux shows them with
three different inode numbers, which it derives from the byte addresses
of the files' directory records. Shwoops ... 190 KiB more.
A few hundred such duplications would cause 130 MiB of growth.

About 17 MiB of the size difference are due to the second directory
tree which is needed to make partition 1 mountable despite not
beginning at block address 0.
Guix has a generously sized directory tree in its ISO. More than 30,000
data files. But i think a nice partition table is worth even 17 MiB.

=========================================================================

# Some paths which one may choose differently
BASE_ISO=guixsd-install-0.15.0.i686-linux.iso
MOUNT_POINT=/mnt/iso
EXTRACTED_MBR_IMG=extracted_mbr.img
REPACKED_ISO=guix-0.15.0.i686-mbr-only.iso

# (I myself prefer to become real superuser for mounting. But sudo is modern.)
PRIV_CMD=sudo

# Now for the work:

# Extract the MBR x86 code part from the original ISO
dd if="$BASE_ISO" bs=1 count=446 of="$EXTRACTED_MBR_IMG"

# The mounted original ISO will donate all its files to the repacked one
$PRIV_CMD mount -o loop "$BASE_ISO" "$MOUNT_POINT"

# The follow xorriso run builds the repacked ISO by native xorriso commands.
# Other than the mkisofs emulation it would not truncate an existing
# ISO image but rather complain about an unsuitabe multi-session setup.
# So remove an older image, if it exists:
test -e "$REPACKED_ISO" && rm "$REPACKED_ISO"

# Map the ISO data files into the new ISO, but delete /efi.img, which is the
# FAT filesystem image of the EFI partition.
# Install the extracted MBR with GRUB specific adjustments and start of
# partition 1 at byte 32768.
# Append the EFI partition image after the ISO filesystem (that is why the file
# is not needed inside the filesystem and then makes a much nicer partition).
# Set up the first El Torito catalog entry for PC-BIOS.
# Set up the second  El Torito catalog entry for EFI, using the appended
# partition.
xorriso \
   -outdev "$REPACKED_ISO" \
   -volid 'GUIXSD_IMAGE' \
   \
   -map "$MOUNT_POINT" "/" \
   -rm /efi.img -- \
   \
   -boot_image grub grub2_mbr="$EXTRACTED_MBR_IMG" \
   -boot_image any mbr_force_bootable=on \
   -boot_image any partition_table=on \
   -boot_image any iso_mbr_part_type=0x83 \
   -boot_image any partition_offset=16 \
   \
   -append_partition 2 0xef "$MOUNT_POINT"/efi.img \
   \
   -boot_image any cat_path='/boot.catalog' \
   \
   -boot_image grub bin_path='/boot/grub/i386-pc/eltorito.img' \
   -boot_image grub grub2_boot_info=on \
   -boot_image any emul_type=no_emulation \
   -boot_image any load_size=2048 \
   -boot_image any boot_info_table=on \
   \
   -boot_image any next \
   \
   -boot_image any efi_path='--interval:appended_partition_2:all::' \
   -boot_image any emul_type=no_emulation

=========================================================================

The result may be inspected by

  xorriso \
     -indev "$REPACKED_ISO" \
     -report_system_area plain \
     -report_el_torito plain

which should say

  ...
  Volume id    : 'GUIXSD_IMAGE'
  System area options: 0x00004a00
  System area summary: MBR grub2-mbr cyl-align-off
  ISO image size/512 : 2071064
  Partition offset   : 16
  MBR heads per cyl  : 64
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x83           64      2065240
  MBR partition      :   2   0x00  0xef      2065304         5760
  El Torito catalog  : 17333  1
  El Torito cat path : /boot.catalog
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
  El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       17334
  El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      516326
  El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
  El Torito img opts :   1  boot-info-table grub2-boot-info
  El Torito img blks :   2  1440

=========================================================================

For eyes which are not accustomed to xorriso's native command set, here is
the same with mkisofs emulation options (without trying to exclude
/mnt/iso/efi.img from the filesystem):

xorriso -as mkisofs \
   -o "$REPACKED_ISO" \
   -V 'GUIXSD_IMAGE' \
   \
   --grub2-mbr "$EXTRACTED_MBR_IMG" \
   --mbr-force-bootable \
   -iso_mbr_part_type 0x83 \
   -partition_offset 16 \
   \
   -append_partition 2 0xef "$MOUNT_POINT"/efi.img \
   \
   -c '/boot.catalog' \
   \
   -b '/boot/grub/i386-pc/eltorito.img' \
   --grub2-boot-info \
   -boot-info-table \
   -no-emul-boot \
   -boot-load-size 4 \
   \
   -eltorito-alt-boot \
   \
   -e '--interval:appended_partition_2:all::' \
   -no-emul-boot \
   \
   "$MOUNT_POINT"

Most users, including grub-mkrescue, prefer the mkisofs emulation because
they can recycle old knowledge from El-Torito-BIOS-only times.
The appeal of the native command set mostly shows up when the ISO content
is not shaped in a tree on hard disk but collected and arranged from
various files and trees.

Main difference is that sequence does strictly matter with the native set.
E.g. i cannot set boot attributes to files which are not mapped in yet.
The mkisofs emulation sets up a production job and only then assesses
the input file objects. (Imagine a shell which takes a dozen commands
from you and re-arranges them before executing them. Ewww ...)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 17, 2019 at 05:51:41PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> here are instructions to repack a Guix ISO with EFI support (via GPT)
> to an ISO with MBR-only partition table.
> 

Thank you!  However, even though Linux is happy about the repacked
ISO, my Macbook still does not like it.  I wonder what makes Debian so
likable.  I will investigate tomorrow.

(The ISO image file size is reproducable, the content is not, of
course, as before.)




> The result may be inspected by
> 
>   xorriso \
>      -indev "$REPACKED_ISO" \
>      -report_system_area plain \
>      -report_el_torito plain
> 
> which should say
> 
>   ...
>   Volume id    : 'GUIXSD_IMAGE'
>   System area options: 0x00004a00
>   System area summary: MBR grub2-mbr cyl-align-off
>   ISO image size/512 : 2071064
>   Partition offset   : 16
>   MBR heads per cyl  : 64
>   MBR secs per head  : 32
>   MBR partition table:   N Status  Type        Start       Blocks
>   MBR partition      :   1   0x80  0x83           64      2065240
>   MBR partition      :   2   0x00  0xef      2065304         5760
>   El Torito catalog  : 17333  1
>   El Torito cat path : /boot.catalog
>   El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
>   El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       17334
>   El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      516326
>   El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
>   El Torito img opts :   1  boot-info-table grub2-boot-info
>   El Torito img blks :   2  1440
> 

Looks good, I believe:

florian@florianmacbook ~$   xorriso \
>      -indev "$REPACKED_ISO" \
>      -report_system_area plain \
>      -report_el_torito plain
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : ISO image bears MBR with  -boot_image any partition_offset=16
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE :   32980 nodes read in 1 seconds
xorriso : UPDATE :   56154 nodes read in 2 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 742333 * 2048 , 5760 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'g7.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record  : El Torito , MBR grub2-mbr cyl-align-off
Media summary: 1 session, 743725 data blocks, 1453m data, 58.4g free
Volume id    : 'GUIXSD_IMAGE'
System area options: 0x00004a00
System area summary: MBR grub2-mbr cyl-align-off
ISO image size/512 : 2975092
Partition offset   : 16
MBR heads per cyl  : 91
MBR secs per head  : 32
MBR partition table:   N Status  Type        Start       Blocks
MBR partition      :   1   0x80  0x83           64      2969268
MBR partition      :   2   0x00  0xef      2969332         5760
El Torito catalog  : 21669  1
El Torito cat path : /boot.catalog
El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       21670
El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      742333
El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
El Torito img opts :   1  boot-info-table grub2-boot-info
El Torito img blks :   2  1440







> For eyes which are not accustomed to xorriso's native command set, here is
> the same with mkisofs emulation options (without trying to exclude
> /mnt/iso/efi.img from the filesystem):
> 

This yields a differently sized ISO (perhaps because of not excluding
/mnt/iso/efi.img) which too is now liked by Linux but still not by my
Macbook:

$ ls -l g7.iso g8.iso
-rw-r--r-- 1 florian users 1523580928 17. Apr 21:26 g7.iso
-rw-r--r-- 1 florian users 1525852160 17. Apr 22:03 g8.iso



$ xorriso      -indev "$REPACKED_ISO"      -report_system_area plain      -report_el_torito plain
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : ISO image bears MBR with  -boot_image any partition_offset=16
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE :   33890 nodes read in 1 seconds
xorriso : UPDATE :   56155 nodes read in 2 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 743455 * 2048 , 5760 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'g8.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record  : El Torito , MBR grub2-mbr cyl-align-off
Media summary: 1 session, 745045 data blocks, 1455m data, 57.0g free
Volume id    : 'GUIXSD_IMAGE'
System area options: 0x00004a00
System area summary: MBR grub2-mbr cyl-align-off
ISO image size/512 : 2980180
Partition offset   : 16
MBR heads per cyl  : 0
MBR secs per head  : 0
MBR partition table:   N Status  Type        Start       Blocks
MBR partition      :   1   0x80  0x83           64      2973756
MBR partition      :   2   0x00  0xef      2973820         5760
El Torito catalog  : 21351  1
El Torito cat path : /boot.catalog
El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz         LBA
El Torito boot img :   1  BIOS  y   none  0x0000  0x00      4       21352
El Torito boot img :   2  UEFI  y   none  0x0000  0x00   5760      743455
El Torito img path :   1  /boot/grub/i386-pc/eltorito.img
El Torito img opts :   1  boot-info-table grub2-boot-info
El Torito img blks :   2  1440

Again, thank you!  (Even though I would prefer if it worked on
Macbooks.  Many people I know own one of those.  I do not know if
newer Macbooks are as picky as mine.)

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> my Macbook still does not like it.

The content of the EFI partition of guixsd-install-0.15.0.i686-linux.iso
does not look like it is ready for 64-bit x86:

  $ mount guixsd-install-0.15.0.i686-linux.iso /mnt/iso
  $ mount /mnt/iso/efi.img /mnt/fat
  $ find /mnt/fat -exec ls -ld '{}' ';'
  drwxr-xr-x 3 root root 7680 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 1024 Jul  6  2018 /mnt/fat/efi
  drwxr-xr-x 2 root root 1024 Jul  6  2018 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 96256 Jul  6  2018 /mnt/fat/efi/boot/boot.efi
  -rwxr-xr-x 1 root root 96256 Jul  6  2018 /mnt/fat/efi/boot/bootia32.efi

According to UEFI specs, bootia32.efi is the start program for 32-bit x86
systems.
The name for 64-bit x86 systems would be bootx64.efi like in the EFI
partition of debian-9.3.0-amd64-netinst.iso :

  drwxr-xr-x 3 root root 16384 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 2048 Dec  9  2017 /mnt/fat/efi
  drwxr-xr-x 2 root root 2048 Dec  9  2017 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 390656 Dec  9  2017 /mnt/fat/efi/boot/bootx64.efi

(Afaik, the EFI partition image is created by grub-mkimage. But i may
 be wrong. Better check source of debian-cd or ask Steve McIntyre.)

Was GRUB configured to produce 64-bit x86 code ?


I only know how to do it with Debian. There i have to install the
packages "grub-pc" for BIOS, "grub-efi-ia32-bin" for 32 bit x86 EFI,
"grub-efi-amd64-bin" for 64 bit x86 EFI.

The EFI partition then has:

  drwxr-xr-x 3 root root 7680 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 1024 Feb  6  2016 /mnt/fat/efi
  drwxr-xr-x 2 root root 1024 Feb  6  2016 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 132608 Feb  6  2016 /mnt/fat/efi/boot/bootx64.efi
  -rwxr-xr-x 1 root root 95744 Feb  6  2016 /mnt/fat/efi/boot/bootia32.efi
  -rwxr-xr-x 1 root root 95744 Feb  6  2016 /mnt/fat/efi/boot/boot.efi
  -rwxr-xr-x 1 root root 1759 Dec 10  2017 /mnt/fat/NvVars

("grub-pc" caused bootable MBR x85 code.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

meanwhile i learned that "i686" means 32 bit.

So i downloaded
  guixsd-install-0.16.0.x86_64-linux.iso
and see indeed the boot program for 64 bit x86 in the EFI partition:

  -rwxr-xr-x 1 root root 132608 Dec  5 16:59 /mnt/fat/efi/boot/bootx64.efi

From Florian's reports i cannot tell which ISO he used as base for
repacking. Maybe one should distinguish the ISOs by their Volume Id. Like:
  GUIXSD_IMAGE_I686
  GUIXSD_IMAGE_X86_64


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 09:00:23AM +0200, Thomas Schmitt wrote:
> From Florian's reports i cannot tell which ISO he used as base for
> repacking. Maybe one should distinguish the ISOs by their Volume Id. Like:
>   GUIXSD_IMAGE_I686
>   GUIXSD_IMAGE_X86_64

The current git master for x86_64.  I can try another one this weekend.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > From Florian's reports i cannot tell which ISO he used

Florian Pelz wrote:
> The current git master for x86_64.

I begin to remember discussions about 64-bit Macs having 32-bit EFI
firmware.
Googling brought me to
  https://everymac.com/mac-answers/snow-leopard-mac-os-x-faq/mac-os-x-snow-leopard-64-bit-macs-64-bit-efi-boot-in-64-bit-mode.html
Note the combinations of "Processor 64" and "EFI 32".
(I think the "Boot" column is about MacOS, not the firmware.)


> ... > On the 2011 Macbook I am using

The rumors rather talk of the years up to 2010. But well ...


Yesterday:
> The Debian ISO works fine
> $ gdisk Downloads/debian-live-9.8.0-amd64-xfce.iso

Let's see what's in there ...

  drwxr-xr-x 3 root root 16384 Jan  1  1970 /mnt/fat
  drwxr-xr-x 3 root root 2048 Feb 11 20:17 /mnt/fat/efi
  drwxr-xr-x 2 root root 2048 Feb 11 20:17 /mnt/fat/efi/boot
  -rwxr-xr-x 1 root root 390656 Feb 11 20:17 /mnt/fat/efi/boot/bootx64.efi

No 32-bit start program. Grrr.


> I can try another one this weekend.

I am nearly out of ideas if not by surprise a test with Guix "i686" ISO
works.

The last resort would be to try the weird "isohybrid --uefi" layout of
questionable MBR partition table and invalid GPT.
(This might answer the riddle why Matthew Garrett kept the GPT data
 after he obviously decided for MBR partitioning.)

But before we do that, we should make absolutely sure that EFI does
not start the bootx64.efi of the Guix ISO. Only then, the partitioning
is to blame. After bootx64.efi was started, GRUB is in charge.
So its modules in both ISOs might make the difference.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 10:13:33AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > ... > On the 2011 Macbook I am using
> 
> The rumors rather talk of the years up to 2010. But well ...
> 
> 

Next week I can ask others who own newer Macbooks to try out the ISO.
If this only affects old Macbooks, fixing it might not be that
important.



> I am nearly out of ideas if not by surprise a test with Guix "i686" ISO
> works.
> 

I just tried guix-0.15.0.i686-mbr-only.iso and boot gets stuck too.
Remarkably, when using the original
guixsd-install-0.15.0.i686-linux.iso on the same x86_64 2011 Macbook,
I can at least still boot from other partitions, the Guix USB drive
merely does not show up (for Debian, it shows up as an EFI partition
in the boot menu).





> The last resort would be to try the weird "isohybrid --uefi" layout of
> questionable MBR partition table and invalid GPT.
> (This might answer the riddle why Matthew Garrett kept the GPT data
>  after he obviously decided for MBR partitioning.)
> 
> But before we do that, we should make absolutely sure that EFI does
> not start the bootx64.efi of the Guix ISO. Only then, the partitioning
> is to blame. After bootx64.efi was started, GRUB is in charge.
> So its modules in both ISOs might make the difference.
> 
> 

I find a efi/boot/bootx64.efi PE executable inside efi.img on the Guix
ISO.  On Debian the long name is EFI/boot/bootx64.iso in capital
letters; I do not know if this can make a difference.  How would I
check if it gets run?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I find a efi/boot/bootx64.efi PE executable inside efi.img on the Guix
> ISO. On Debian the long name is EFI/boot/bootx64.iso in capital letters

(I assume s/bootx64.iso/bootx64.efi/)

It's FAT filesystem. Upper case or lower case does not matter.
Most probably it is a matter of mount options whether you see capital
or small letters.
The image files seem to contain the names in capital letters:

  # mount guixsd-install-0.16.0.x86_64-linux.iso /mnt/iso

  $ strings /mnt/iso/efi.img | head -9
  MTOO4021
  Hf3NO NAME    FAT12
  EFI
  .
  ..
  BOOT
  .
  ..
  BOOTX64 EFI

  # umount /mnt/iso
  # mount debian-live-9.8.0-amd64-xfce.iso /mnt/iso

  $ strings /mnt/iso/boot/grub/efi.img | head -17
  mkfs.fat
  dNO NAME    FAT12
  This is not a bootable disk.  Please insert a bootable floppy and
  press any key to try again ...
  EFI
  KNKN
  .
  KNKN
  ..
  KNKN
  BOOT
  KNKN
  .
  KNKN
  ..
  KNKN
  BOOTX64 EFI


> How would I check if it gets run?

Interesting question. If it says "GRUB" or "Guix", then it runs. But up
to then it may start and fail silently.


> the Guix USB drive
> merely does not show up (for Debian, it shows up as an EFI partition
> in the boot menu).

So probably it is not run.

I'll develop a xorriso repack run for the layout of Debian ISOs, as
strange as it is.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

this is the xorriso run for producing a layout like "isohybrid --uefi"
with GRUB equipment for BIOS instead of ISOLINUX.

I prepended command -stdio_sync "off" in order to make use of my generous
amount of RAM for write buffering. The default syncs to disk every 16 MiB.
Good with low RAM or fast SSD.

xorriso treats its output as an optical drive.
Without sync, the speed reports on image files are rather funny:
  xorriso : UPDATE : Writing:   262432s  43.6%  fifo 88%  buf  50%  1169.9xD

===========================================================================

# Layout of isohybrid --uefi. Invented by Matthew J. Garrett for Fedora.
# The EFI partition is a file inside the ISO. The BIOS equipment is ISOLINUX
# instead of GRUB. But the EFI equipment is GRUB.
# Used by Fedora, Debian, Ubuntu, Mint, Arch, and more.
# (SuSE is different by using a pure MBR partition table.)

# Additionally to a Guix ISO, we need one of those isohybrid ISOs.
# The result will not be usable for booting via BIOS, because only the first
# stages of ISOLINUX are extracted. The rest would not fit Guix anyways.
ISOHYBRID_ISO=debian-live-9.8.0-amd64-xfce.iso
ISOLINUX_BIN_ISO=/isolinux/isolinux.bin
EXTRACTED_ISOLINUX_BIN=isolinux.bin
EXTRACTED_MBR_IMG=isohybrid_mbr.img

# Guix ISO related names
BASE_ISO=guixsd-install-0.16.0.x86_64-linux.iso
MOUNT_POINT=/mnt/iso
REPACKED_ISO=guix-0.16.0.x86_64-mjg.iso

PRIV_CMD=sudo

# Extract the ISOLINUX isohybrid MBR x86 code part from the Debian ISO
dd if="$ISOHYBRID_ISO" bs=1 count=446 of="$EXTRACTED_MBR_IMG"

# Extract the ISOLINUX boot image for El Torito
xorriso \
  -report_about warning \
  -osirrox on \
  -indev "$ISOHYBRID_ISO" \
  -report_about note \
  -extract "$ISOLINUX_BIN_ISO" "$EXTRACTED_ISOLINUX_BIN"

# The mounted original ISO will donate all its files to the repacked one
$PRIV_CMD mount -o loop "$BASE_ISO" "$MOUNT_POINT"

# Pack it all up
xorriso \
   -stdio_sync off \
   -outdev "$REPACKED_ISO" \
   -volid 'GUIXSD_IMAGE' \
   \
   -map "$MOUNT_POINT" "/" \
   -map "$EXTRACTED_ISOLINUX_BIN" /isolinux/isolinux.bin \
   \
   -boot_image isolinux system_area="$EXTRACTED_MBR_IMG" \
   -boot_image any mbr_force_bootable=on \
   -boot_image any partition_offset=0 \
   \
   -boot_image any cat_path='/isolinux/boot.cat' \
   -boot_image isolinux bin_path='/isolinux/isolinux.bin'  \
   -boot_image any emul_type=no_emulation \
   -boot_image any load_size=2048 \
   -boot_image any boot_info_table=on \
   \
   -boot_image any next \
   \
   -boot_image any efi_path=/efi.img \
   -boot_image any emul_type=no_emulation \
   -boot_image isolinux partition_entry=gpt_basdat

#Clean up disk
rm "$EXTRACTED_MBR_IMG" "$EXTRACTED_ISOLINUX_BIN"

===========================================================================

Expected inspection result:

  $ xorriso -indev "$REPACKED_ISO" -report_system_area plain
  ...
  Volume id    : 'GUIXSD_IMAGE'
  System area options: 0x00000102
  System area summary: MBR isohybrid cyl-align-on GPT
  ISO image size/512 : 2405888
  Partition offset   : 0
  MBR heads per cyl  : 74
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x00            0      2405888
  MBR partition      :   2   0x00  0xef        35132         2880
  MBR partition path :   2  /efi.img
  GPT                :   N  Info
  GPT disk GUID      :      451fe71ed0ff5d48a1c144a0316dd901
  GPT entry array    :      2  248  overlapping
  GPT lba range      :      64  2405824  2405887
  GPT partition name :   1  490053004f00480079006200720069006400
  GPT partname local :   1  ISOHybrid
  GPT partition GUID :   1  451fe71ed0ff5d48a1c044a0316dd901
  GPT type GUID      :   1  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   1  0x1000000000000001
  GPT start and size :   1  0  2405824
  GPT partition name :   2  490053004f004800790062007200690064003100
  GPT partname local :   2  ISOHybrid1
  GPT partition GUID :   2  451fe71ed0ff5d48a1c344a0316dd901
  GPT type GUID      :   2  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   2  0x1000000000000001
  GPT start and size :   2  35132  2880
  GPT partition path :   2  /efi.img

For comparison, the output with debian-live-9.8.0-amd64-xfce.iso :

  Volume id    : 'd-live 9.8.0 xf amd64'
  System area options: 0x00000102
  System area summary: MBR isohybrid cyl-align-on GPT
  ISO image size/512 : 3811392
  Partition offset   : 0
  MBR heads per cyl  : 117
  MBR secs per head  : 32
  MBR partition table:   N Status  Type        Start       Blocks
  MBR partition      :   1   0x80  0x00            0      3811392
  MBR partition      :   2   0x00  0xef         1432          832
  MBR partition path :   2  /boot/grub/efi.img
  GPT                :   N  Info
  GPT disk GUID      :      f2925bc0164b9245ba53a2025e420b60
  GPT entry array    :      2  248  overlapping
  GPT lba range      :      64  3811328  3811391
  GPT partition name :   1  490053004f00480079006200720069006400
  GPT partname local :   1  ISOHybrid
  GPT partition GUID :   1  f2925bc0164b9245ba52a2025e420b60
  GPT type GUID      :   1  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   1  0x1000000000000001
  GPT start and size :   1  0  3811328
  GPT partition name :   2  490053004f004800790062007200690064003100
  GPT partname local :   2  ISOHybrid1
  GPT partition GUID :   2  f2925bc0164b9245ba51a2025e420b60
  GPT type GUID      :   2  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   2  0x1000000000000001
  GPT start and size :   2  1432  832
  GPT partition path :   2  /boot/grub/efi.img


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 11:28:13PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> this is the xorriso run for producing a layout like "isohybrid --uefi"
> with GRUB equipment for BIOS instead of ISOLINUX.
> 

No, it still gets stuck. :(


> Expected inspection result:
> 

The GPT partition GUID is different each time I repeat the process.
Otherwise I get the same result.



> For comparison, the output with debian-live-9.8.0-amd64-xfce.iso :
> 

The debian output is fully identical.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 18, 2019 at 03:50:59PM +0200, Thomas Schmitt wrote:
> > the Guix USB drive
> > merely does not show up (for Debian, it shows up as an EFI partition
> > in the boot menu).
> 
> So probably it is not run.
> 

Boot gets stuck on x86_64 Guix images only, but the one i686 I tried
did not show up in the boot menu.  I believe bootx64.efi is run for
x86_64 and is the culprit.

I could try burning a DVD and attempt to boot from it.

It would also be interesting to find out how to change the
bootx64.efi’s code and make it display debug information in some way.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > this is the xorriso run for producing a layout like "isohybrid --uefi"

Florian Pelz wrote:
> No, it still gets stuck. :(

Then it must be something inside the EFI partition which makes the
Macbook firmware dismiss the partition for booting.


> The GPT partition GUID is different each time I repeat the process.

This can be avoided by setting environment variable SOURCE_DATE_EPOCH
to the same value in each production run.
(I just tested that this works as expected.)


Well, i am out of ideas how the Guix ISO layout or content differs from
the stuff in Debian ISOs, so that an EFI implementation dismisses the
former and accepts the latter.

It really does not look like an issue with partitioning.
I don't think that it goes deeper into the boot equipment than the
files in the EFI partition.

Maybe GRUB experts have ideas what to examine there. After all it is
grub-mkrescue which fails to produce a recognizable partition, whereas
Debian's effort with (most probably) grub-mkimage can do the trick.
grub-mkrescue uses grub-mkimage, too, afaik. So it probably is about
some magic ingredient to the production of the EFI start programs or
the FAT image file.

We could ask at debian-cd mailing list whether the problem rings a bell
from a past problem fix.
(I am subscribed there and ready to assist with technical details about
 our findings.)


Next mail from Florian:
> Boot gets stuck on x86_64 Guix images only, but the one i686 I tried
> did not show up in the boot menu.  I believe bootx64.efi is run for
> x86_64 and is the culprit.

This fits into my findings if EFI looks into the EFI partitions
and dismisses those without its CPU specific flavor of start program.

Can you get EFI to confess that the x86_64 USB stick indeed has a
recognizable EFI partition ?


> I could try burning a DVD and attempt to boot from it.

On DVD the El Torito boot equipment should lead EFI to the same partition
image as the partition table should do on USB stick.
But a try cannot harm.


> It would also be interesting to find out how to change the
> bootx64.efi’s code and make it display debug information in some way.

The highest control layer is
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c
The name "BOOTX64.EFI" appears in
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-install.c
Somewhere inbetween must be its source. :o)

------------------------------------------------------------------------

For now i say thanks for three found bugs in my own software, during
this discussion:

- Environment variable SOURCE_DATE_EPOCH did not affect synthetic files.
  Thanks to Florian and Ludovic for pointing out the non-reproducibility
  issue.
  Fixed by libisoburn 5b62c55
  Workaround for Guix and currently released versions of xorriso:
  Give the ISO root directory fixed atime, mtime, and ctime by xorriso
  command -alter_date. (Would work with future xorriso versions, too.)

- SIGSEGV happened if "-boot_image grub grub2_mbr=" was used and no
  El Torito boot image was defined.
  This one i found by a bad backslash while testing my xorriso run
  proposals to Florian.
  Fixed by libisofs 4b21386
  No workaround needed because no usable result can emerge anyways.

- I was unable yesterday to get "isohybrid --uefi" layout when using GRUB
  equipment for BIOS. I thought it was possible. My grub-mkrescue argument
  filter script falsely promises to do it.
  Still unresolved.
  Workaround was to use body parts of a real isohybrid ISO.

------------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 11:01:51AM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > this is the xorriso run for producing a layout like "isohybrid --uefi"
> 
> Florian Pelz wrote:
> > No, it still gets stuck. :(
> 
> Then it must be something inside the EFI partition which makes the
> Macbook firmware dismiss the partition for booting.
> 
> 

For x86_64 iso images, the Macbook does *not dismiss*.  It gets stuck
in the lightgray startup screen forever without showing a menu with
the drives to boot from, except for the Debian ISO image, for which a
menu is shown with an »EFI partition« that runs a bootloader when
chosen.






> Can you get EFI to confess that the x86_64 USB stick indeed has a
> recognizable EFI partition ?
> 
> 

I do not currently know how.  Maybe by changing the bootx64.efi on its
EFI partition?



> > I could try burning a DVD and attempt to boot from it.
> 
> On DVD the El Torito boot equipment should lead EFI to the same partition
> image as the partition table should do on USB stick.
> But a try cannot harm.
> 
>

Will try now.




> For now i say thanks for three found bugs in my own software, during
> this discussion:
> 

I am happy something good came of it.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> For x86_64 iso images, the Macbook does *not dismiss*.  It gets stuck
> in the lightgray startup screen forever without showing a menu with
> the drives to boot from, except for the Debian ISO image, for which a
> menu is shown with an »EFI partition« that runs a bootloader when
> chosen.

So the problem might be during the inspection of the EFI partition whether
it is worth to be offered for booting.

Comparing the output of program "strings" from both bootx64.efi files,
i see in the Debian file text about Secure Boot, which are not in the
Guix file:

  SecureBoot
  SetupMode
  ...
  Secure Boot forbids loading module from %s
  ...
  grub_efi_secure_boot

Older Debian netinst maybe ? ... digging in my collection ...

The youngest without "Secure" in its bootx64.efi is
  debian-7.7.0-amd64-netinst.iso

You could check whether this hangs like Guix.

Debian's own ISO archive of 7.7.0 only offers CD-1 as jigdo.
Probably with the same EFI equipment.
But let's google ...

  http://ftp.riken.go.jp/Linux/debian/debian-cdimage/archive/7.7.0/amd64/iso-cd/debian-7.7.0-amd64-netinst.iso

It matches my old copy. 232,783,872 bytes.
SHA512:
  5cb6e4fea55fbb5173f90c3a545b843c6c193e29c3aa32b3306c9bbdfb1ad6a6a36ae8be50e91af9d03d5f21c472bd05d04d3508172e0b519e76714333c7c74b


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 11:01:51AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > I could try burning a DVD and attempt to boot from it.
> 
> On DVD the El Torito boot equipment should lead EFI to the same partition
> image as the partition table should do on USB stick.
> But a try cannot harm.
> 
>

The DVD works and is bootable. :)  It displays twice as “GRUB 2.02”
and “Windows”.  I presume one of them is booted in legacy BIOS mode
and one is booted in UEFI mode.

Regards,
Florian

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi,

(Moving discussion to <https://issues.guix.info/issue/35283>, which is
specifically about ISO image reproducibility issues.)

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Florian Pelz wrote:
>>  The content is different at the beginning of the ISO image
>> (maybe padding or timestamps in the file system)
>
> That's to expect if not environment SOURCE_DATE_EPOCH is set and exported.
>
> SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It
> is supposed to be either undefined or to contain a decimal number which
> tells the seconds since january 1st 1970. If it contains a number, then
> it is used for all timestamps and as seed of pseudo-random numbers like
> MBR id or GPT UUIDs.
>
> If all files and directories have the same names and the same content,
> then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
> value are supposed to create byte-identical result ISOs.

By mounting the ISO image, I found that some files didn’t have their
timestamp reset: some files in /var/guix (easily fixed), but more
importantly those added by GRUB in /boot and /System.

Files added by ‘grub-mkrescue’ are “out of our control” so we would need
to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

However, after rereading the Xorriso manual, it seemed to me that if we
set SOURCE_DATE_EPOCH and pass:

  -volume_date all_file_dates set_to_mtime

then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
which would solve the problem.

I tried it, but that’s not what happened.  What am I missing, Thomas?

Thanks,
Ludo’.

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

> Files added by ‘grub-mkrescue’ are “out of our control” so we would need
> to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

Google shows that patches have been proposed. But they seem not to
have made it into the source.

Vladimir Serbinko's answer here
  https://lists.gnu.org/archive/html/grub-devel/2015-12/msg00046.html
might be the reason. I understand that he demands uniqueness of UUIDs.

But that's not really a problem with reproducible ISOs. If pseudo-random
UUIDs depend deterministically on SOURCE_DATE_EPOCH, then collisions are
only to expect between ISOs made with the same seconds value.
This can also happen if non-reproducible ISOs are made while their
systems' clocks show the same time by mere incident.

So one should use SOURCE_DATE_EPOCH values with best possible entropy.
Not one humanly invented lucky number for all ISOs of a distro.

If ever two identical ISOs are offered to GRUB at boot time, it needs
some imagination to construct a problem if GRUB operates on the one
which was not used by the EFI firmware to start GRUB.


So when a reproducible ISO is made for the first time, its SOURCE_DATE_EPOCH
should be taken from "date +%s" and recorded for further runs.
The ISO will bear it as "Creation Time", like "2019021612165300".
The last two digits "00" are centiseconds and should be ignored even
if not "00".
If decoding that time back to seconds-since-1970 is cumbersome, one may
store the seconds value in a data file in the input tree of the ISO
before packing up by a xorriso run with SOURCE_DATE_EPOCH having that
value.


> after rereading the Xorriso manual, it seemed to me that if we
> set SOURCE_DATE_EPOCH and pass:
>   -volume_date all_file_dates set_to_mtime
> then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
> which would solve the problem.

This is the support for ignoring atime and ctime changes of input files
but respecting their mtime changes.

If you want a fixed time for all three timestamps in all files, do:

  -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

The "=" announces seconds-since-1970 as time format. See -alter_date.

Note that in this proposal $SOURCE_DATE_EPOCH is evaluated by the shell,
not by xorriso. Depending on the way how xorriso is started, you need to
insert the actual number.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 19, 2019 at 12:58:58PM +0200, Thomas Schmitt wrote:
> The youngest without "Secure" in its bootx64.efi is
>   debian-7.7.0-amd64-netinst.iso
> 
> You could check whether this hangs like Guix.
> 

No, it does not hang, it is bootable.


> SHA512:
>   5cb6e4fea55fbb5173f90c3a545b843c6c193e29c3aa32b3306c9bbdfb1ad6a6a36ae8be50e91af9d03d5f21c472bd05d04d3508172e0b519e76714333c7c74b
> 

Yes, for me the sha512sum is the same.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

What is this --hfs-bless option of mkisofs?  This is not related by
any chance?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> The DVD works and is bootable. :)

Hrmpf. I hate it when success spoils a theory.


I wrote:
> > The youngest without "Secure" in its bootx64.efi is
> >   debian-7.7.0-amd64-netinst.iso

> No, it does not hang, it is bootable.

It is about time for black candles and the toad mummy resurrection ritual.


Summary:

Something brings the firmware of this Macbook to a halt, probably when
it assesses the offers of bootable partitions.
It cannot entirely be about the partition table, because it even happens
when we achieve the table type and partition entries of Debian ISOs.
But it cannot entirely be about the content of the EFI partition, because
the same EFI partition is recognized and boots if it is accessed via the
El Torito boot catalog from DVD (*).

(*) Assumed that the Macbook EFI halfways obeys UEFI specs about which
    table to interpret depending on the medium type.

---------------------------------------------------------------------

Next mail:
> What is this --hfs-bless option of mkisofs?  This is not related by
> any chance?

Rather not. (Unless the toad becomes mother and the candles turn white.)

"Blessing" in HFS or HFS+ is a way to mark a file or directory for a
special role in the filesystem. For the firmware and maybe for MacOS.

In guixsd-install-0.16.0.x86_64-linux.iso (which should be re-generated
for the download site) there is one blessed file:

  $ xorriso -hfsplus on -indev guixsd-install-0.16.0.x86_64-linux.iso \
            -find / -has_hfs_bless any -exec get_hfs_bless --
  ...
  intel_bootfile   '/System/Library/CoreServices/boot.efi'

Other possible blessings are "ppc_bootdir", "show_folder", "os9_folder",
and "osx_folder".

But in Debian Live there is no HFS+ and thus no opportunity for blessing.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive and
mounted it.  I am trying to make the boot not get stuck.

I replaced efi/boot/bootx64.efi by Debian’s bootx64.efi.  Still stuck.

I deleted efi/boot/bootx64.efi.  Still stuck.

I changed the EFI partition’s GPT type code with gdisk from EF00 to
0700.  Still stuck.

I deleted the EFI partition.  Now I can boot (but not from the USB
drive, of course).

I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
Boot gets stuck again.

I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
partition.  Now I can boot again (not from the Guix USB drive, of
course).

I wonder, what kind of FAT is the EFI partition using?  Could it be
the wrong kind of FAT?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I wonder, what kind of FAT is the EFI partition using?  Could it be
> the wrong kind of FAT?

I got a mail from a bystander, Ady Ady whom i know from SYSLINUX/ISOLINUX
mailing list. He proposes the experiment to exchange the whole EFI partition
of the Guix ISO by the one of Debian Live 9.

This will not be able to really boot Guix, of course. But if the Mac's
boot manager shows it and is willing to start the bootx64.efi program,
then the FAT filesystem's meta data become prime suspect.


The EFI partition of Debian Live 9 is smaller than the one of Guix 0.16.0.
So after these steps :

> I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
> Boot gets stuck again.
>
> I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
> partition.

mount the Debian Live 9 ISO (e.g. at /mnt/iso) and do

  dd if=/mnt/iso/boot/grub/efi.img of=/dev/sdc2


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i now look at the FAT filesystem images of Debian Live 9 and Guix 0.16.0:

Guix:

- There is an MBR partition table in the image file:
    Device            Boot Start   End Sectors  Size Id Type
    /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12

- Program "file" says:

  DOS/MBR boot sector, code offset 0x3c+2,
  OEM-ID "MTOO4021",
  root entries 224, sectors 2880 (volumes <=32 MB) ,
  sectors/FAT 9, sectors/track 18,
  serial number 0x336648a7, unlabeled,
  FAT (12 bit), followed by FAT

Debian:

- No partition table in image.

- Program "file" says:
  DOS/MBR boot sector, code offset 0x3c+2,
  OEM-ID "mkfs.fat",
  sectors/cluster 4, root entries 512, sectors 832 (volumes <=32 MB) ,
  Media descriptor 0xf8,
  sectors/FAT 1, sectors/track 32, heads 64,
  serial number 0x64c6c435, unlabeled, FAT (12 bit)

---------------------------------------------------------------------

My favorite suspect would be the partition table in Guix /efi.img.

To kill table entry 1 after having put the Guix EFI image back into
partition 2 of the USB stick:

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

---------------------------------------------------------------------

If this does not help, then create a new empty FAT filesystem image,
mount it, and copy the tree from the mounted Guix /efi.img into it.
The new filesystem should have between 200 KiB and 1.4 MiB.
(The Guix image of 1.4 MiB still has 90 % of its capacity free.)

Then put the new filesystem into /dev/sdc2 and see what happens.

---------------------------------------------------------------------

If this does not help, then copy again the Debian efi.img into partition 2,
mount it, and replace the file bootx64.efi by the one of the mounted
Guix efi.img.

---------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

... and if the erasure of the partition table entry helps, would it then
also help if that entry's partition does not start at LBA 0 ?

Change bytes 446 (dec) to 461 (dec) from (in hex):

  80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00

to

  80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

"02" = Sector 2 = LBA 1.
"01 00 00 00" = LBA 1.
"3f 0b 00 00" = bloc count 2879, rather than 2880.


Reasoning:

It would make a nice endless loop if EFI dives into any partition
and looks for sub partition tables. The one in GRUB's efi.img refers
to itself.
DVD might be immune because EFI does not do this diving when it finds
the partition by the El Torito boot catalog.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> To kill table entry 1 after having put the Guix EFI image back into
> partition 2 of the USB stick:
>
>   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

Of course i meant  of=/dev/sdc2

The 16 bytes written to /dev/sdc1 would deface the GPT's "protective" MBR
and thus make the GPT invalid.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

It works, see end of this e-mail.

On Sat, Apr 20, 2019 at 12:50:49PM +0200, Thomas Schmitt wrote:
> So after these steps :
> 
> > I wrote guixsd-install-0.16.0.x86_64-linux.iso to my USB drive again.
> > Boot gets stuck again.
> >
> > I did `sudo dd if=/dev/zero of=/dev/sdc2` to zero out the EFI
> > partition.
> 
> mount the Debian Live 9 ISO (e.g. at /mnt/iso) and do
> 
>   dd if=/mnt/iso/boot/grub/efi.img of=/dev/sdc2
> 
> 

It still does not get stuck. :)



On Sat, Apr 20, 2019 at 01:16:14PM +0200, Thomas Schmitt wrote:
> My favorite suspect would be the partition table in Guix /efi.img.
> 
> To kill table entry 1 after having put the Guix EFI image back into
> partition 2 of the USB stick:
> 
>    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1
>

I did

sudo dd if=Downloads/guixsd-install-0.16.0.x86_64-linux.iso of=/dev/sdc
sudo dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc1

Boot gets stuck.

On Sat, Apr 20, 2019 at 04:23:49PM +0200, Thomas Schmitt wrote:
> Of course i meant  of=/dev/sdc2
> 

Oops, too late.


> If this does not help, then create a new empty FAT filesystem image,
> mount it, and copy the tree from the mounted Guix /efi.img into it.
> The new filesystem should have between 200 KiB and 1.4 MiB.
> (The Guix image of 1.4 MiB still has 90 % of its capacity free.)
> 

I do not recreate the USB drive.  I do:

qemu-img create -f raw fatfs.img 1M
mkfs.fat fatfs.img # from package dosfstools like the EFI system
                   # partition on Arch wiki
sudo mount Downloads/guixsd-install-0.16.0.x86_64-linux.iso /mnt/iso
sudo mount fatfs.img /mnt/img
sudo mount /mnt/iso/efi.img /mnt/efiimg
sudo cp -r /mnt/efiimg/efi /mnt/img/
sudo umount /mnt/img

> Then put the new filesystem into /dev/sdc2 and see what happens.
>

sudo dd if=fatfs.img of=/dev/sdc2
sync

Hooray!  It boots!  Thank you so much for your patience.

I remember back when I used Parabola GNU/Linux-libre, I had to make
sure I used mkfs.fat instead of mkfs.msdos or else the EFI partition
did not work, I think.

But…  Guix’ gnu/build/vm.scm already uses mkfs.fat.  Strange…

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sat, Apr 20, 2019 at 04:54:24PM +0200, pelzflorian (Florian Pelz) wrote:
> sudo dd if=fatfs.img of=/dev/sdc2
> sync
> 

I can copy the same fatfs.img from Guix System 0.16.0 onto a current
Guix git master installer iso and it works too.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sat, Apr 20, 2019 at 04:54:24PM +0200, pelzflorian (Florian Pelz) wrote:
> But…  Guix’ gnu/build/vm.scm already uses mkfs.fat.  Strange…
> 

It uses mkfs.fat for creating FAT partitions, but maybe this
particular EFI partition is created by grub-mkrescue?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> mkfs.fat fatfs.img
> sudo mount fatfs.img /mnt/img
> sudo cp -r /mnt/efiimg/efi /mnt/img/
> Hooray!  It boots!

So it is indeed the filesystem hull, which is to blame.

We still need to find out whether the partition entry is the culprit.
So please do the experiment with the Guix EFI partition image in
partition 2 of the USB stick and

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

The partition entry in the Guix EFI partition image has all the
properties of the thing we are looking for:
- Not in the FAT filesystem, but also not outside of it.
- Plausible to be differently approached from USB stick and DVD.
- Self referring enough for an endless inspection loop in EFI.


> It uses mkfs.fat for creating FAT partitions, but maybe this
> particular EFI partition is created by grub-mkrescue?

Yes.
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812
has:
      rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
            efiimgfat, "::", NULL });

This run produces an image file with partition 1, starting at LBA 0.
A remark below
  https://sources.debian.org/src/mtools/4.0.23-1/mformat.c/#L1373
says:
      /* install fake partition table pointing to itself */

(But why does grub-mkrescue.c create a 2.8 MB floppy where Guix has a
 1.4 MB floppy image ? Does Guix use a patched version of grub-mkrescue ?)

---------------------------------------------------------------------------
Under the assumption that it is really the partition entry, i snooped ahead:

mformat options -2 and -k can suppress the production of the partition
entry. Option -2 is for some exotic floppy format from the 1990s.

Option -k seems better, but if i use it, program "file" becomes sparing
with words:

  $ file /u/test/mformat.fat
  /u/test/mformat.fat: DOS floppy 1440k
  $

It can be mounted by Linux and offers free space, though.

More similar to the current FAT image hull is the result of

   dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

I actually did with a copy of partition 2:
   dd if=/dev/zero bs=1 count=16 seek=446 conv=notrunc of=test.img
and the report of "file" did not change. fdisk reports no partitions
in test.img any more.


Assuming that we cannot easily convince GRUB to use mformat -k or to
switch to some other FAT creator, i currently ponder the opportunity
to let my man-in-the-middle script do this dd patching of the EFI image
before it gets copied into the ISO.

-------------------------------------------------------------------------
Second correction of my statement about zapping the partition entry:

i wrote:
> The 16 bytes written to /dev/sdc1 would deface the GPT's "protective" MBR
> and thus make the GPT invalid.

Wrong again. Because it is a grub-mkrescue ISO, it hit byte 446 ff of
the ISO Primary Volume Descriptor. Funnily this is the place where
xorriso placed its hallmark:
   XORRISO-1.5.0 2018...
So the misplaced dd was effectively without effect.

-------------------------------------------------------------------------

Have a nice day :)

Thomas

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> If you want a fixed time for all three timestamps in all files, do:
>
>   -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

Thanks, that’s what I was missing.

It was still not the end of the story, but I have some good news: the
series of commits below allow me to build ISO images reproducibly!  \o/

  1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing it.
  5502fbd7fd gnu: valgrind: Add 3.15.0.
  605815023c vm: Use a fixed FAT serial number for 'efi.img' in ISO images.
  52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
  833480cc1f vm: Reset file timestamps in ISO images.

To check by yourself you can do, say:

  guix system disk-image --file-system-type=iso9660 \
     gnu/system/examples/bare-bones.tmpl

and then check the ISO derivation that was built as the last step above:

  guix build --check -K /gnu/store/…-image.iso.drv

Ludo’.

bug#35283: [PATCH] mformat: initialize boot sector before writing it

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 303 bytes --]

Hello,

While investigating reproducible ISO images for Guix¹, I found that
‘mformat’ would not initialize the boot sector before writing it.  This
led to non-deterministic FAT image contents.

The attached patch fixes that.

Thanks,
Ludo’.

¹ https://issues.guix.info/issue/35283


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 593 bytes --]

Fix a bug whereby 'mformat' could end up passing uninitialized bytes
to write(2).  This could be reproduced with:

  mformat -C -f 1440 -L 16  -N 77777777 -i  /tmp/x ::

where the output of /tmp/x would be non-deterministic.

Patch by Ludovic Courtès <ludo@gnu.org>.

--- mtools-4.0.23/mformat.c	2019-04-21 00:12:01.496116195 +0200
+++ mtools-4.0.23/mformat.c	2019-04-21 00:12:36.675967157 +0200
@@ -927,6 +927,7 @@ void mformat(int argc, char **argv, int
 	
 	char *endptr;
 
+	memset(&boot.bytes, '\0', sizeof boot);
 	hs = hs_set = 0;
 	argtracks = 0;
 	argheads = 0;

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Sat, Apr 20, 2019 at 06:32:07PM +0200, Thomas Schmitt wrote:
> We still need to find out whether the partition entry is the culprit.
> So please do the experiment with the Guix EFI partition image in
> partition 2 of the USB stick and
> 
>    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2
> 

Yes, this fixes the issue.


I tried current Git guix master with ludo's reproducibility patches.
Of course with the reproducibility patches the Macbook still gets
stuck before booting.

> Yes.
>   http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812
> has:
>       rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
>             efiimgfat, "::", NULL });
> 

I added "-k" to it (see attached ugly Guix patch).  This fixes the
issue for me.  (I do not know if this has other unintended side
effects and I can only try on other Macbooks next week.)

Regards,
Florian

[-- Attachment #2: use-gnu-packages-patches-grub-make-mkrescue-use-k-op.patch --]
[-- Type: text/plain, Size: 2073 bytes --]

From 0d266ebe33d03816d200e285d172e76ff641dca6 Mon Sep 17 00:00:00 2001
From: Florian Pelz <pelzflorian@pelzflorian.de>
Date: Sun, 21 Apr 2019 08:34:17 +0200
Subject: [PATCH] use
 gnu/packages/patches/grub-make-mkrescue-use-k-option.patch

---
 gnu/packages/bootloaders.scm                          |  3 ++-
 .../patches/grub-make-mkrescue-use-k-option.patch     | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/grub-make-mkrescue-use-k-option.patch

diff --git a/gnu/packages/bootloaders.scm b/gnu/packages/bootloaders.scm
index b4eabaea48..013b112592 100644
--- a/gnu/packages/bootloaders.scm
+++ b/gnu/packages/bootloaders.scm
@@ -91,7 +91,8 @@
                "03vvdfhdmf16121v7xs8is2krwnv15wpkhkf16a4yf8nsfc3f2w1"))
              (patches (search-patches "grub-check-error-efibootmgr.patch"
                                       "grub-binutils-compat.patch"
-                                      "grub-efi-fat-serial-number.patch"))))
+                                      "grub-efi-fat-serial-number.patch"
+                                      "grub-make-mkrescue-use-k-option.patch"))))
     (build-system gnu-build-system)
     (arguments
      `(#:phases (modify-phases %standard-phases
diff --git a/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch b/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch
new file mode 100644
index 0000000000..fbb4c5f596
--- /dev/null
+++ b/gnu/packages/patches/grub-make-mkrescue-use-k-option.patch
@@ -0,0 +1,11 @@
+--- grub-2.02/util/grub-mkrescue.c	2019-04-21 08:17:18.880000000 +0200
++++ grub-2.02/util/grub-mkrescue.c	2019-04-21 08:17:52.588000000 +0200
+@@ -791,7 +791,7 @@
+ 
+       const char *fat_serial_number = getenv ("GRUB_FAT_SERIAL_NUMBER");
+       const char *mformat_args[] =
+-	{ "mformat", "-C", "-f", "2880", "-L", "16",
++	{ "mformat", "-k", "-C", "-f", "2880", "-L", "16",
+ 	  fat_serial_number != NULL ? "-N" : "-C",
+ 	  fat_serial_number != NULL ? fat_serial_number : "-C",
+ 	  "-i", efiimgfat, "::", NULL };
-- 
2.21.0

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

>   833480cc1f vm: Reset file timestamps in ISO images.

That's also a big solution for the problem of timestamps of synthetic files.

I understand that your plan for reproducibility is to make timestamps
completely insignificant. Radical but effective.

But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
use the same seconds value for the ISO file objects ?


>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.

Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
not only influences mformat underneath grub-mkrescue, but also the run
of xorriso, where it determines volume date timestamps and GPT individual
UUIDs.
(Other impacts of the variable get overridden by the
   -volume_date "all_file_dates"
 command in commit 833480cc1f.)


>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.

I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
a 2.8 MB FAT image.


>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing

How good are chances to bring such changes into upstream ?

I ask in the advance assumption that we find a way to make the mformat
image digestible for Florian's Macbook.
(It is clear now that the difference between failure and success is in
 mformat versus mkfs.fat. But the exact point of failure is not found yet.
 I place my bet on the partition entry with start LBA 0.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > with the Guix EFI partition image in
> > partition 2 of the USB stick and
> >    dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Florian Pelz wrote:
> Yes, this fixes the issue.

I really had a good feeling with that theory. :))

-----------------------------------------------------------------------

Ok, now for the last experiment. Just for curiosity, as i believe it
is not a good solution (and mformat -k isn't either):
Does start LBA 1 of partition 1 prevent the endless loop, too ?

So please write to byte 446 (dec) to 461 (dec) of the FAT image:

  80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

In the FAT image treated by the "dd" above, these bytes are all 00.
In the original FAT image they are:

  80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00
          ==                      ==              ==  ==

If this works, then we can blame a bug in the Macbook's EFI which is
too stupid to avoid endless recursion when exploring a potential
tree of partition tables.
(If it does not work, then we only have a heuristic indication that
 a partition table entry in the EFI partition is bad for the Macbook.)


Nevertheless, start LBA 1 is less desirable than no partition table
entry at all, because LBA 1 causes a partition that is not mountable.
So an option for mformat would be of interest, which writes an MBR
(other than with -k) but does not create a partition entry.

Alternatively, the Guix ISO production could apply above zeroizing dd
run to the EFI image by help of a pseudo-xorriso wrapper under
grub-mkrescue. This wrapper would learn the EFI image file address from
the xorriso options, patch that file, and then run xorriso with the
arguments given by grub-mkrescue.

(I will give my grub-mkrescue-sed.sh wrapper such a feature as addon-on
 to its other manipulation offers.)

----------------------------------------------------------------------

Reading UEFI 2.4, 12.3.1 "System Partition" i do not find statements
about partition tables inside that partition, but only:

  "A System Partition supports backward compatibility with legacy systems
   by reserving the first block (sector) of the partition for compatibility
   code. On legacy systems, the first block (sector) of a partition is
   loaded into memory and execution is transferred to this code.
   EFI firmware does not execute the code in the MBR. The EFI firmware
   contains knowledge about the partition structure of various devices,
   and can understand legacy MBR, GPT, and “El Torito.”

So an MBR-like start block of the partition is explicitely encouraged.
"understand"ing partitioning offers a wide range of interpretation.
So the behavior of the Macbook is not totally against specs.
It is just not overly stable and helpful.

"I am not crazy. Don't call me crazy. I...I'm just not user friendly."
Roberto, Futurama.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 11:35:49AM +0200, Thomas Schmitt wrote:
> Ok, now for the last experiment. Just for curiosity, as i believe it
> is not a good solution (and mformat -k isn't either):
> Does start LBA 1 of partition 1 prevent the endless loop, too ?
> 
> So please write to byte 446 (dec) to 461 (dec) of the FAT image:
> 
>   80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> 
> In the FAT image treated by the "dd" above, these bytes are all 00.
> In the original FAT image they are:
> 
>   80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00
>           ==                      ==              ==  ==
> 

Why is 0b underlined?

OK, so I just wrote the Guix git master with ludo’s reproducibility
patches to a USB drive (boot gets stuck again) and then did:

sudo dd if=/dev/sdc2 of=mysdc2.img

When I open it with ghex, I find at position 446:

    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

Strange.  I write the guixsd-install-0.16.0.x86_64-linux.iso to
/dev/sdc (boot gets stuck) and repeat

sudo dd if=/dev/sdc2 of=mysdc2.img

I find:

    80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00

This matches what you wrote.  I change it to

    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00

I dd the changed mysdc2.img back to /dev/sdc2.

Now it boots. :)  Strangely, I now have only one entry “GRUB 2.02” in
the boot selection, but “EFI Boot” (or what it was called) is gone.



> Nevertheless, start LBA 1 is less desirable than no partition table
> entry at all, because LBA 1 causes a partition that is not mountable.
> So an option for mformat would be of interest, which writes an MBR
> (other than with -k) but does not create a partition entry.
> 

This raises the question what should be done in Guix.  Is it a good
idea to add -k to mformat in grub-mkrescue for the upcoming Guix 1.0
release (even though you don’t like it)?  Could it break install on
other machines?  Or should mformat be patched instead?  Could any of
this be upstreamed?  What about your MBR repacking?  I’m sure Ludo has
an opinion on these things too.  I just doing what I’m told, but I
don’t quite understand what I’m doing here.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i plan to ask at gru-devel about mformat, its MBR, and the partition entry,
as soon as the last experiment is made (LBA 1 rather than LBA 0 as start
of the inner partition).

For that discussion i could also need the exact model name of the Macbook.
If its EFI has a name, then this too.


Florian Pelz wrote:
> +--- grub-2.02/util/grub-mkrescue.c     2019-04-21 08:17:18.880000000 +0200
> ++++ grub-2.02/util/grub-mkrescue.c     2019-04-21 08:17:52.588000000 +0200
> [...]
> +-      { "mformat", "-C", "-f", "2880", "-L", "16",
> ++      { "mformat", "-k", "-C", "-f", "2880", "-L", "16",

The difference between the contents of the first image block with and
without option -k is significant, at least if we ask program "file".

If a special version of gub-mkrescue is acceptable, then i would rather
propose to apply the dd treatment to   efiimgfat  after mformat created
it without option -k.

We could ask at grub-devel for advise how to write 16 bytes to that file
without needing another shell command. (It's C language, after all.)

The binary differences between both blocks 0, except partition entry, are:

/efi.img from Guix 0.16.0
#########################
Image from mformat -k

00000000 :    eb  3c  90  4d  54  4f  4f  34  30  32  31  00  02  01  01  00
             ### ### ### ### ### ### ### ### ### ### ###
00000000 :    00  00  00  00  00  00  00  00  00  00  00  00  02  01  01  00

00000020 :    00  00  00  00  00  00  29  a7  48  66  33  4e  4f  20  4e  41
                                         ### ### ### ###
00000020 :    00  00  00  00  00  00  29  42  87  4b  7d  4e  4f  20  4e  41

00000030 :    4d  45  20  20  20  20  46  41  54  31  32  20  20  20  fa  31
                                                                     ### ###
00000030 :    4d  45  20  20  20  20  46  41  54  31  32  20  20  20  00  00

00000040 :    c0  8e  d8  8e  c0  fc  b9  00  01  be  00  7c  bf  00  80  f3
             ### ### ### ### ### ### ###     ### ###     ### ###     ### ###
00000040 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

00000050 :    a5  ea  56  00  00  08  b8  01  02  bb  00  7c  ba  80  00  b9
             ### ### ###         ### ### ### ### ###     ### ### ###     ###
00000050 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

00000060 :    01  00  cd  13  72  05  ea  00  7c  00  00  cd  19  00  00  00
             ###     ### ### ### ### ###     ###         ### ###
00000060 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

...

000001f0 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  55  aa
                                                                     ### ###
000001f0 :    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 01:16:14PM +0200, Thomas Schmitt wrote:
> For that discussion i could also need the exact model name of the Macbook.
> If its EFI has a name, then this too.
> 

I booted my ancient version of macOS Yosemite and clicked on the apple
symbol in the top left corner and About this Mac.

MacBook Pro (13-inch, Mid 2010)
[...]
Serial Number  WQ04501UATM

When I click on System Report, I see:

Model Name:            MacBook Pro
Model Identifier:      MacBookPro7,1
Boot ROM Version:      MBP71.003F.B00
SMC Version (system):  1.62f7

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > ... 3f  0b
> > ... 40  0b
> > ... ==  ==

Florian Pelz wrote:
> Why is 0b underlined?

Too much enthusiasm on my side.


> OK, so I just wrote the Guix git master with ludo’s reproducibility
> patches to a USB drive (boot gets stuck again) and then did:
> sudo dd if=/dev/sdc2 of=mysdc2.img
> When I open it with ghex, I find at position 446:
>    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

Pinching eyes ...

Bootable flag is set.
Start C/H/S is 4/0/0. (x/y/1 is usual)
Partition type is 1.
End C/H/S is 1/36/15.
Start LBA is 0x80000000 = 2,147,483,648.
Block count is 0x16 = 22.

So what ever did you do to this mysdc2.img ?


> I change it to
>    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> I dd the changed mysdc2.img back to /dev/sdc2.
>
> Now it boots. :)

Start C/H/S is 0/0/2.
Partition type is 1.
End C/H/S ...  i'm getting too old for decoding MBR C/H/S bit fields.
Start LBA is 1.
Block count is 0x0b3f = 2879.

So Start LBA 0 is indeed the trigger of the problem.
As soon as the partition entry does not point to its hosting block any
more, the (now very probable) loop in EFI cannot happen.


> Strangely, I now have only one entry “GRUB 2.02” in
> the boot selection, but “EFI Boot” (or what it was called) is gone.

What do you see if the partition entry is zeroized entirely ?

(Elsewise i refer to the Futurama quote in my previous mail.)


> Is it a good
> idea to add -k to mformat in grub-mkrescue for the upcoming Guix 1.0
> release (even though you don’t like it)?

We need to ask at grub-devel. I have begun to compose a mail.
You will be Cc-ed.
Does guix-devel want to be Cc-ed ?

New mail:
> MacBook Pro (13-inch, Mid 2010)
> Model Name:            MacBook Pro
> Model Identifier:      MacBookPro7,1
> Boot ROM Version:      MBP71.003F.B00
> SMC Version (system):  1.62f7

I will put this into my mail to grub-devel.

> Serial Number  ...

... but this only if asked for.


Back to old mail:
> Or should mformat be patched instead? Could any of
> this be upstreamed?

That's why i asked Ludovic about our chances with mtools upstream.
I would propose an option to write the usual pseudo-MBR but without that
partition entry.
(Well, maybe somebody there can even remember why a partition entry
 is made by default.)


> What about your MBR repacking?

I will create an option for zeroizing bytes 446 to 461 of the EFI image.

When the dust has settled here, i will ask Ludovic for preparing the
ISO production code for a run with libisoburn's wrapper script
frontend/grub-mkrescue-sed.sh for MBR-only.

At that occasion, the EFI image repairer could be tested, too.

The script has a mode "original" which does not change the xorriso
options submitted by grub-mkrescue. So it produces original GPT.
(It's original purpose is spying on grub-mkrescue's xorriso options.)


> I just doing what I’m told, but I
> don’t quite understand what I’m doing here.

We had a wild ride over boot sectors and partition tables.
I collect my knowledge in libisofs file doc/boot_sectors.txt .

At days when our web certificates are valid, it is available at
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/doc/boot_sectors.txt

Hah. Bug #4. No doc/boot_sectors.txt in libisofs tarball.
But it is in GNU xorriso's tarball:
  https://fossies.org/linux/misc/xorriso-1.5.1.tar.gz/xorriso-1.5.1/doc/boot_sectors.txt

For completeness, a reserve address for my grub-mkrescue-xorriso wrapper:
  https://sources.debian.org/src/libisoburn/1.5.0-1/frontend/grub-mkrescue-sed.sh


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Sun, Apr 21, 2019 at 02:27:10PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> i wrote:
> > > ... 3f  0b
> > > ... 40  0b
> > > ... ==  ==
> 
> Florian Pelz wrote:
> > Why is 0b underlined?
> 
> Too much enthusiasm on my side.
> 
> 
> > OK, so I just wrote the Guix git master with ludo’s reproducibility
> > patches to a USB drive (boot gets stuck again) and then did:
> > sudo dd if=/dev/sdc2 of=mysdc2.img
> > When I open it with ghex, I find at position 446:
> >    80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00
> 
> Pinching eyes ...
> 
> Bootable flag is set.
> Start C/H/S is 4/0/0. (x/y/1 is usual)
> Partition type is 1.
> End C/H/S is 1/36/15.
> Start LBA is 0x80000000 = 2,147,483,648.
> Block count is 0x16 = 22.
> 
> So what ever did you do to this mysdc2.img ?
> 
> 

Ooops I am sorry, it is not

80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

but

80  00  01  00  04  01  24  4f  00  00  00  00  80  16  00  00  00



> We need to ask at grub-devel. I have begun to compose a mail.
> You will be Cc-ed.
> Does guix-devel want to be Cc-ed ?
> 

Thank you for sending a mail at
https://lists.gnu.org/archive/html/grub-devel/2019-04/msg00099.html

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> it is not
> 80  00  00  04  01  24  4f  00  00  00  00  80  16  00  00  00
> but
> 80  00  01  00  04  01  24  4f  00  00  00  00  80  16  00  00  00

This makes more sense as an MBR partition table entry:
Start C/H/S is 0/0/1.
Start LBA is 0.
Block count is 0x1680 = 5760 = 2.8 MB floppy.
The last 00 belongs to the next partition entry slot.

Overall it looks like the output of mformat -f 2880 without -k.


Have a nice day :)

Thomas

bug#35283: [PATCH] grub-mkrescue: Allow users to specify a FAT serial number

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Hello,

While investigating reproducible ISO images for Guix¹, I found that
‘grub-mkrescue’ would invoke ’mformat’ without the ‘-N’ option.
Consequently, ‘mformat’ would pick a random serial number, thereby
making the ‘efi.img’ build process non-deterministic.

I came up with the gross hack attached: the ‘grub-mkrescue’ caller can
set the ‘GRUB_FAT_SERIAL_NUMBER’ environment variable, which
‘grub-mkrescue’ translates into a ‘-N’ flag for ‘mformat’.

We could perhaps achieve the same result differently, for instance by
adding an option to ‘grub-mkrescue’.

WDYT?

Thanks,
Ludo’.

¹ https://issues.guix.info/issue/35283


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 1262 bytes --]

Change 'grub-mkrescue' to honor the 'GRUB_FAT_SERIAL_NUMBER'
environment variable.  That way, the caller can specify a fixed
serial number (instead of the randomly chosen one) to create EFI
images (the 'efi.img' file) that are reproducible bit-for-bit.

Patch by Ludovic Courtès <ludo@gnu.org>.

--- grub-2.02/util/grub-mkrescue.c	2019-04-20 19:15:26.180242812 +0200
+++ grub-2.02/util/grub-mkrescue.c	2019-04-20 21:56:34.672370849 +0200
@@ -788,8 +788,15 @@ main (int argc, char *argv[])
 
       efiimgfat = grub_util_path_concat (2, iso9660_dir, "efi.img");
       int rv;
-      rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
-	    efiimgfat, "::", NULL });
+
+      const char *fat_serial_number = getenv ("GRUB_FAT_SERIAL_NUMBER");
+      const char *mformat_args[] =
+	{ "mformat", "-C", "-f", "2880", "-L", "16",
+	  fat_serial_number != NULL ? "-N" : "-C",
+	  fat_serial_number != NULL ? fat_serial_number : "-C",
+	  "-i", efiimgfat, "::", NULL };
+
+      rv = grub_util_exec (mformat_args);
       if (rv != 0)
 	grub_util_error ("`%s` invocation failed\n", "mformat");
       rv = grub_util_exec ((const char * []) { "mcopy", "-s", "-i", efiimgfat, efidir_efi, "::/", NULL });

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>>   833480cc1f vm: Reset file timestamps in ISO images.
>
> That's also a big solution for the problem of timestamps of synthetic files.
>
> I understand that your plan for reproducibility is to make timestamps
> completely insignificant. Radical but effective.
>
> But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
> use the same seconds value for the ISO file objects ?

Files in /gnu/store, by convention, all have their mtime set to 1 (one
second after the epoch).

>>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
>
> Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
> not only influences mformat underneath grub-mkrescue, but also the run
> of xorriso, where it determines volume date timestamps and GPT individual
> UUIDs.
> (Other impacts of the variable get overridden by the
>    -volume_date "all_file_dates"
>  command in commit 833480cc1f.)

AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
Xorriso, or at least it was overridden by the “-volume_date” options
that I pass.

It’s crucial for me to have the mtime set to 1 for all the files on the
ISO; I wanted the 1980 setting to apply only to ‘efi.img’.

>>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
>
> I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
> but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
> a 2.8 MB FAT image.

I haven’t dig deep enough to provide a satisfactory answer.  :-)

>>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing
>
> How good are chances to bring such changes into upstream ?

I’ve emailed them (actually tried to, their mailing list rejected my
message.)  We’ll see!

Ludo’.

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
> Xorriso, or at least it was overridden by the “-volume_date” options
> that I pass.

Probably. Among the automatic grub-mkrescue options for xorriso's mkisofs
emulation is

  --modification-date=2019042117165600

The equivalent native command is

  -volume_date uuid 2019042117165600

So you indeed have to override this by an own subsequent command.

(SOURCE_DATE_EPOCH overrides defaults of xorriso. But commands or options
 override the overridden defaults.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ady Ady gave me a hint about why the EFI partition in Guix 0.16.0 ISO
has 1.4 MB of size, despite grub-mkrescue asks mformat for 2.8 MB.

Obviously Guix uses mtools 4.0.21 (image OEM-ID "MTOO4021") which is one
version before this bug fix:
  "Fixed -f flag for mformat (size is KBytes, rather than sectors)"
  http://lists.gnu.org/archive/html/info-gnu/2018-12/msg00000.html

(Actually this size is quite some waste. Even 1.4 MB is generously
 oversized for the three directories and one or two boot*.efi.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

A friend’s “early 2014” MacBookAir6,2 does not have issues with the
old iso image.  It seems likely that only old Macbooks are affected.

On Sun, Apr 21, 2019 at 02:27:10PM +0200, Thomas Schmitt wrote:
> > I change it to
> >    80  00  02  00  01  01  12  4f  01  00  00  00  3f  0b  00  00
> > I dd the changed mysdc2.img back to /dev/sdc2.
> >
> > Now it boots. :)
> > […]
> > Strangely, I now have only one entry “GRUB 2.02” in
> > the boot selection, but “EFI Boot” (or what it was called) is gone.
> 
> What do you see if the partition entry is zeroized entirely ?
>

I see both “GRUB 2.02” and “EFI Boot” when I zero it (I have not tried
on Guix 0.16.0 this time though).

So the options are:

· Update mtools and then somehow patch mformat to zero out this
  region.

· Patch grub-mkrescue instead to make it use mformat -k.  Vladimir
  Serbinenko on the grub-devel list said this removes only information
  that is “Probably” not useful anyway, see
  https://lists.gnu.org/archive/html/grub-devel/2019-04/msg00100.html

· Ignore the issue.  The only affected machines we know are old Macbooks.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian pelaz wrote:
> So the options are:
> · Update mtools and then somehow patch mformat to zero out this
>  region.

I will hopefully tonight post a proposal for

- Leave grub-mkrescue and mformat as they are. Rather do the partition
  entry removal in the course of Guix ISO production.

It will be windfall profit of my detailed proposal for creating ISOs
with MBR partition table and Linux mountable ISO partition.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

after the adventure of diagnosing Macbook EFI and mformat, i want to
return to the original topic of this thread:
I deem it worthwhile to offer the opportunity to create the Guix ISO
with MBR partition table rather than with GPT.

If you just want to see my technical proposal, then hop to the next
section titled "How to do it".

The advantages of my proposal are:

- No GPT backup which needs to be relocated after the ISO was put
  onto USB stick.

- Mountable ISO 9660 partition. (The GPT of a pure grub-mkrescue ISO
  offers a HFS+ partition for mounting. But afaik, Linux hates the
  block size 2048.)

A disadvantage is that for best overall properties, this layout should
have the ISO partition start at 512-byte block 64. This needs a second
superblock and directory tree. About 17 MiB with Guix 0.16.0.

If the size increase matters, then i propose to sacrifice the HFS+
filesystem tree as compensation.
It is for Macs even older than Florian's.

Without that non-zero partition start, the ISO superblock (PVD) of the
overall image cannot claim the appended EFI partition 2 as part of its
filesystem. That's because the superblock is mountable as partition 1
and a partition filesystem should not be larger than its partition.

But with partition start at LBA 64, the overall superblock is not
mountable via a partition and thus free to claim as much space as it
wants.
So programs like /sbin/isosize can tell the image size even when the
ISO has already been copied to USB stick.

Partition editors, too, will love a partition start after the partition
table rather than at block 0.

----------------------------------------------------------------------

How to do it:

We can achieve the MBR partition table based layout by a script which
is part of libisoburn. It acts as pseudo-xorriso between grub-mkrescue
and real xorriso.
Yesterday i improved it by a partition table killer for the EFI image:
  MKRESCUE_SED_IN_EFI_NO_PT=yes.

For experiments it is best to download it from upstream.
Currently the https certificate is not valid. So for now unsafe:

  wget --no-check-certificate \
       https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

Size of today's version is 11077 bytes.
SHA512 is
  23cbc25aac8e03de82bd6e76b00a6ad249629c19822728616d6144f075dbb78f69811648559846ba33319c38baa21515e4a3a0f7bb3ca86f2427e887592101cf
(Ask if you get a different checksum. Maybe i'll make more changes.)

------------------------------------------------------------------------

In a shell script it would be used like this:

  export MKRESCUE_SED_MODE=mbr_hfs
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ... \
                ... with or without a -- switch to native command mode ... \
                -- \
                -boot_image any partition_offset=16 \
                -boot_image any iso_mbr_part_type=0x83

The result (with nearly no payload) looks like

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  34456

The HFS+ superblock and directory tree will be marked by a partition
in an Apple Partition Map. From xorriso -report_system_area :

  APM                :   N  Info
  APM block size     :      2048
  APM gap fillers    :      1
  APM partition name :   1  Gap0
  APM partition type :   1  ISO9660_data
  APM start and size :   1  16  150
  APM partition name :   2  HFSPLUS_Hybrid
  APM partition type :   2  Apple_HFS
  APM start and size :   2  166  7008

------------------------------------------------------------------------

If HFS+ shall be omitted for size reduction or other reasons, use
a different mode of grub-mkrescue-sed.sh :

  export MKRESCUE_SED_MODE="mbr_only"

rather than MKRESCUE_SED_MODE="mbr_hfs".
The result will be smaller by the size of the additional HFS+ tree.

  $ expr $(/sbin/isosize output.iso) / 512
  33864

(The true cost of HFS+ and second ISO directory tree will show up only
 with much more files in the trees.)

------------------------------------------------------------------------

The original GPT layout with additionally zeroed inner EFI partition
table can be achieved by:

  export MKRESCUE_SED_MODE=original
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ...

(Partition offset and MBR partition type are of no use with "original".)

The result is

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: gpt
  ...
  Device      Start   End Sectors  Size Type
  output.iso1    64   339     276  138K Microsoft basic data
  output.iso2   340  6099    5760  2.8M EFI System
  output.iso3  6100 34131   28032 13.7M Apple HFS/HFS+
  output.iso4 34132 34731     600  300K Microsoft basic data

  $ expr $(/sbin/isosize output.iso) / 512
  34780

------------------------------------------------------------------------

Another interesting layout is "gpt_appended". It moves the EFI partition
out of the ISO filesystem, as does "mbr_only". But it marks the partitions
in GPT. The ISO partition is mountable. Two directory trees are created
automatically.

  export MKRESCUE_SED_MODE=gpt_appended
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                ... the usual arguments for grub-mkrescue ...

Result:

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: gpt
  ...
  Device      Start   End Sectors  Size Type
  output.iso1    64 28103   28040 13.7M Microsoft basic data
  output.iso2 28104 33863    5760  2.8M EFI System

  $ expr $(/sbin/isosize output.iso) / 512
  33928

(With the next version of xorriso it will be possible to choose
   -boot_image any iso_mbr_part_type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
 for "Linux filesystem data".)

------------------------------------------------------------------------

As can be seen with MKRESCUE_SED_IN_EFI_NO_PT, such a pseudo-xorriso
program between grub-mkrescue and xorriso gives the caller control
over the stage between GRUB file preparation and ISO production run.

For example, it would be possible to make a smaller FAT image, copy
the files from grub-mkrescue's EFI images over to the small one,
and then use that as EFI partition.

It would of course be possible to create such a program in Guile.
Be invited to pick from grub-mkrescue-sed.sh what you need.

--------------------------------------------------------------------
Somewhat off topic:

The script also has a killer for the MBR signature in the EFI image
file. It gets activated together with partition table erasure by
  export MKRESCUE_SED_IN_EFI_NO_PT=extra
It is meant for those users who are curious whether block 0 of an EFI
partition really must look like an MBR to be accepted by EFI firmware.

It would be interesting to learn about EFIs which fail without MBR
signature.

The specs say that there may be "compatibility code" in a EFI System
Partition. They talk of "MBR" and understanding partitions in the EFI
partition. I understand it as "may be there, but does not have to".

Minimal test outside of Guix ISO production:

- Install GRUB and configure it for EFI for 32 or 64 bit as your firmware
  needs. (Both is combinable, as Debian GRUB packages demonstrate.)

- Create a minimal bootable ISO. It will not offer more than a GRUB
  command prompt after it was successfully booted:

    mkdir minimal
    touch minimal/empty-file.txt

    export MKRESCUE_SED_MODE=original
    export MKRESCUE_SED_IN_EFI_NO_PT=extra

    grub-mkrescue -o output.iso minimal \
                  --xorriso=./grub-mkrescue-sed.sh

- Put the resulting output.iso on USB stick and offer it to your firmware
  at boot time.
  Is it not found by the EFI boot manager ?
  If you let it boot, does the machine not show a GRUB prompt in the end ?

- Negative results with other mode settings in MKRESCUE_SED_MODE would be
  interesting, too.

--------------------------------------------------------------------

Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

This sounds nice and clean.  Thank you!

On Tue, Apr 23, 2019 at 08:14:48PM +0200, Thomas Schmitt wrote:
> A disadvantage is that for best overall properties, this layout should
> have the ISO partition start at 512-byte block 64. This needs a second
> superblock and directory tree. About 17 MiB with Guix 0.16.0.
> 

The ISO files are offered on alpha.gnu.org only with xz compression.
I suppose most of the 17 MiB would disappear with compression.


> So programs like /sbin/isosize can tell the image size even when the
> ISO has already been copied to USB stick.
> 

Sounds like this could be used to get a checksum or maybe GPG
verification of a USB drive that supposedly contains the iso.



> Somewhat off topic:
> 
> The script also has a killer for the MBR signature in the EFI image
> file. It gets activated together with partition table erasure by
>   export MKRESCUE_SED_IN_EFI_NO_PT=extra
> It is meant for those users who are curious whether block 0 of an EFI
> partition really must look like an MBR to be accepted by EFI firmware.
> 
> It would be interesting to learn about EFIs which fail without MBR
> signature.
> 
> The specs say that there may be "compatibility code" in a EFI System
> Partition. They talk of "MBR" and understanding partitions in the EFI
> partition. I understand it as "may be there, but does not have to".
> 
> Minimal test outside of Guix ISO production:
> 
> - Install GRUB and configure it for EFI for 32 or 64 bit as your firmware
>   needs. (Both is combinable, as Debian GRUB packages demonstrate.)
> 
> - Create a minimal bootable ISO. It will not offer more than a GRUB
>   command prompt after it was successfully booted:
> 
>     mkdir minimal
>     touch minimal/empty-file.txt
> 
>     export MKRESCUE_SED_MODE=original
>     export MKRESCUE_SED_IN_EFI_NO_PT=extra
> 
>     grub-mkrescue -o output.iso minimal \
>                   --xorriso=./grub-mkrescue-sed.sh
>

This prints

florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal --xorriso=./grub-mkrescue-sed.sh
grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'. Some features are disabled. Please use xorriso 1.2.9 or later..

and yields no output.iso for me.  Note that xorriso actually has
version 1.5.0.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal
> --xorriso=./grub-mkrescue-sed.sh
> grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'.
> Some features are disabled. Please use xorriso 1.2.9 or later..
>
> and yields no output.iso for me.  Note that xorriso actually has
> version 1.5.0.

Does the script offer x-permission ?
(Shame on me for not thinking of this in my download instructions.)

If so, what do you get from

  ./grub-mkrescue-sed.sh -as mkisofs -help 2>&1 | fgrep grub2-boot-info

If it says

  --grub2-boot-info           Patch boot image at byte 2548

then i wonder why grub-mkrescue is not satisfied in
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n608


I wrote:
> > So programs like /sbin/isosize can tell the image size even when the
> > ISO has already been copied to USB stick.

> Sounds like this could be used to get a checksum or maybe GPG
> verification of a USB drive that supposedly contains the iso.

That's what this property is advised for in Debian's CD FAQ.
One can read trailing garbage not only from USB sticks but also from
most DVD types.
(The layout with nested partitions obsoletes the cleanliness
 considerations about partition start at LBA 0.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Tue, Apr 23, 2019 at 10:18:10PM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote:
> > florian@florianmacbook ~ [env]$ grub-mkrescue -o output.iso minimal
> > --xorriso=./grub-mkrescue-sed.sh
> > grub-mkrescue: warning: Your xorriso doesn't support `--grub2-boot-info'.
> > Some features are disabled. Please use xorriso 1.2.9 or later..
> >
> > and yields no output.iso for me.  Note that xorriso actually has
> > version 1.5.0.
> 
> Does the script offer x-permission ?
> (Shame on me for not thinking of this in my download instructions.)
> 

This was the error.  It works after `chmod +x grub-mkrescue-sed.sh`.


On Tue, Apr 23, 2019 at 08:14:48PM +0200, Thomas Schmitt wrote:
> It would be interesting to learn about EFIs which fail without MBR
> signature.
> […]
> - Put the resulting output.iso on USB stick and offer it to your firmware
>   at boot time.
>   Is it not found by the EFI boot manager ?
>   If you let it boot, does the machine not show a GRUB prompt in the end ?
> 

It shows up twice in the boot selection screen as “GRUB 2.02” and “EFI
Boot” just like the Guix USB drive before.  It is bootable and shows a
GRUB prompt on my Macbook.



> - Negative results with other mode settings in MKRESCUE_SED_MODE would be
>   interesting, too.
> 

I can try tomorrow.

By the way, someone I know has a most peculiar machine called Lenovo
Ideapad 100S which does not boot Guix, but apparently is very picky in
general (cf. <https://forums.linuxmint.com/viewtopic.php?t=216476>).
Its CPU is claimed to support 64-bit, but its boot firmware rejects
most 64-bit isos.  I wonder if we could get that machine to boot Guix
(at least it currently cannot boot
guixsd-install-0.15.0.i686-linux.iso), but that may well be a waste of
time.  It might just not be feasible to please every EFI system.



> I wrote:
> > > So programs like /sbin/isosize can tell the image size even when the
> > > ISO has already been copied to USB stick.
> 
> > Sounds like this could be used to get a checksum or maybe GPG
> > verification of a USB drive that supposedly contains the iso.
> 
> That's what this property is advised for in Debian's CD FAQ.
> One can read trailing garbage not only from USB sticks but also from
> most DVD types.
> (The layout with nested partitions obsoletes the cleanliness
>  considerations about partition start at LBA 0.)
>

So… GPT is wrong for some optical media too?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> > > > So programs like /sbin/isosize can tell the image size

> > (The [Debian] layout with nested partitions obsoletes the cleanliness
> >  considerations about partition start at LBA 0.)

Florian Pelz wrote
> So… GPT is wrong for some optical media too?

Not in this aspect. GPT partitions must not start at block 0.
So the overall ISO superblock is not mountable as partition and thus
free to claim the full image size as filesystem size.

The problem of restriction to partition size arises with MBR and
start block 0 of partition 1, when the nested partition mess of
"isohybrid -u" is untangled.


> By the way, someone I know has a most peculiar machine called Lenovo
> Ideapad 100S which does not boot Guix, but apparently is very picky in
> general (cf. <https://forums.linuxmint.com/viewtopic.php?t=216476>).
> Its CPU is claimed to support 64-bit, but its boot firmware rejects
> most 64-bit isos.

If Guix does not show up in the boot manager, try a plain grub-mkrescue
image that was made when GRUB was configured for both EFI variations:
32 bit x86 and 64 x86.
My experiments are made when it is configured for those two plus PC-BIOS.
The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.


> It might just not be feasible to please every EFI system.

Let's see how far we get. This research not only helps Guix but also
GRUB. (And it has a tendency to reveal my bugs in xorriso.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 24, 2019 at 08:56:57AM +0200, Thomas Schmitt wrote:
> Florian Pelz wrote
> > So… GPT is wrong for some optical media too?
> 
> Not in this aspect. GPT partitions must not start at block 0.
> So the overall ISO superblock is not mountable as partition and thus
> free to claim the full image size as filesystem size.
>

Sorry, I meant to quote this aspect:

> > One can read trailing garbage not only from USB sticks but also from
> > most DVD types.
> > […]
> >
> 
> So… GPT is wrong for some optical media too?
> 


> > By the way, someone I know has a most peculiar machine called Lenovo
> > Ideapad 100S which does not boot Guix, […]
> 
> If Guix does not show up in the boot manager, try a plain grub-mkrescue
> image that was made when GRUB was configured for both EFI variations:
> 32 bit x86 and 64 x86.
> My experiments are made when it is configured for those two plus PC-BIOS.
> The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.
> 

Will do.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Sorry, I meant to quote this aspect:

I wrote:
> > > One can read trailing garbage not only from USB sticks but also from
> > > most DVD types.

Florian Pelz wrote:
> > So… GPT is wrong for some optical media too?

I wrote:
> Not in this aspect. GPT partitions must not start at block 0.
> So the overall ISO superblock is not mountable as partition and thus
> free to claim the full image size as filesystem size.

I already understood you correctly ... i think. :))

GPT partitioned ISO are not hampered from telling the full image
size as size of the ISO 9660 filesystem.

  $ /sbin/gdisk -l guixsd-install-0.15.0.i686-linux.iso
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     1              64           34107   16.6 MiB    0700  Gap0
     2           34108           39867   2.8 MiB     EF00  EFI boot partition
     3           39868         1815727   867.1 MiB   AF00  HFSPLUS
     4         1815728         1816327   300.0 KiB   0700  Gap1

  $ expr $(/sbin/isosize guixsd-install-0.15.0.i686-linux.iso) / 512
  1816376

The isosize is a bit larger than the end of partition 4, because after
this partition comes the GPT backup.
The isosize matches the image file size:

  $ expr $(ls -l guixsd-install-0.15.0.i686-linux.iso | awk '{print $5}') / 512
  1816376


But an MBR partition which begins by block 0 and is mountable as ISO 9660
filesystem keeps its ISO superblock from telling a size larger than the
partition. So it cannot claim an appended partition 2 as part of its range.

  $ export MKRESCUE_SED_MODE=mbr_hfs
  $ grub-mkrescue --xorriso=./grub-mkrescue-sed.sh -o output.iso minimal \
                  --iso_mbr_part_type 0x83
  ...

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *        0 28371   28372 13.9M 83 Linux
  output.iso2      28372 34131    5760  2.8M ef EFI (FAT-12/16/32)

  $ expr $(/sbin/isosize output.iso) / 512
  28372

This discrepancy of image file size and isosize is undesirable when the
image file itself cannot say its size any more, because it is on USB
stick or a DVD which delivers more bytes than were written as image.

So i propose partition offset 16 to get the partition superblock away
from being mountable by the base device.

  $ export MKRESCUE_SED_MODE=mbr_hfs
  $ grub-mkrescue --xorriso=./grub-mkrescue-sed.sh -o output.iso minimal \
                  -partition_offset 16 --iso_mbr_part_type 0x83

This yields

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

The ISO filesystem of the overall image then claims the image file size

  $ expr $(/sbin/isosize output.iso) / 512
  34456

With the partition's ISO filesystem we only get the partition size:

  $ dd if=output.iso bs=512 skip=64 of=partition1.iso

  $ expr $(/sbin/isosize partition1.iso) / 512
  28632

This is how it should be.


The ISOs of Fedora, Debian, Ubuntu, and others are MBR partitioned and
most of them have no partition offset 16. But despite that fact they can
claim the full image size as ISO 9660 filesystem size.

This is possible because partition 1 covers the whole image and partition 2
is located inside partition 1. Strictly illegal in UEFI specs, unless
the outer partition has MBR partition type 0x00 and thus does not exist
for EFI and some partition editors. (I.e. this is a really dirty hack.)

  $ /sbin/fdisk -l debian-live-9.8.0-amd64-xfce.iso
  ...
  Disklabel type: dos
  ...
  Device                            Boot Start     End Sectors  Size Id Type
  debian-live-9.8.0-amd64-xfce.iso1 *        0 3811391 3811392  1.8G  0 Empty
  debian-live-9.8.0-amd64-xfce.iso2       1432    2263     832  416K ef EFI (FAT-1

  $ expr $(/sbin/isosize debian-live-9.8.0-amd64-xfce.iso) / 512
  3811392

Old /sbin/gdisk only reports partition 2:

  $ /sbin/gdisk -l debian-live-9.8.0-amd64-xfce.iso
  ...
  Found valid MBR and GPT. Which do you want to use?
  ...
  Your answer: 1
  ...
  Number  Start (sector)    End (sector)  Size       Code  Name
     2            1432            2263   416.0 KiB   EF00  EFI System



Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Hi Thomas,

thanks for the investigative work you are doing.

If someone is testing this with Guix, make sure to actually try to boot Guix
with it (until the root is mounted).  Back in the day I changed the root
discovery of Guix to make it also consider using the entire disk (as opposed to
a partition on it) as a viable root device (see guix master,
commit 9833bcfc08ef009b9e8b4398baa481ef65c80ad7).
That could maybe make it choke if there are multiple partitions/disks with
equal labels (or whatever field it's searching for) on it (though probably not--
as long as its immaterial which of the partitions/disks with the equal field
values is mounted).

FWIW, I'm all for having a MBR-only Guix installer disk as you suggest.
But I'd rather not do the hack the others are doing.  It's hard enough to
debug this stuff when everyone is playing nice.

How should we mount the root when the right way (with two non-overlapping
partitions) boots from DVD ?

(I'm surprised that it needs so much manual intervention to make grub-mkrescue
do the right thing.  One of the reasons I chose grub-mkrescue was that I
assumed that its output would boot fine on many different system types--
it being a rescue disk creator and all.  I guess there are tradeoffs to be
made...)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Danny Milosavljevic wrte:
> If someone is testing this with Guix, make sure to actually try to boot Guix
> with it (until the root is mounted). Back in the day I changed the root
> discovery of Guix to make it also consider using the entire disk (as opposed
> to a partition on it) as a viable root device (see guix master,
> commit 9833bcfc08ef009b9e8b4398baa481ef65c80ad7).
> That could maybe make it choke if there are multiple partitions/disks with
> equal labels (or whatever field it's searching for) on it (though probably
> not--
> as long as its immaterial which of the partitions/disks with the equal field
> values is mounted).

Volume Id and other superblock properties of the two possible ISO 9660 mount
candidates are supposed to be identical. So on USB stick there is indeed
an ambiguity, which superblock to choose (/dev/sdc versus /dev/sdc1).

But both candidates offer identical directory trees with identical file
properties and content. So it should make no difference which one gets
mounted.
The difference is in the block addresses by which superblock and directory
tree refer to file objects. They have to differ by 16. (Block size 2048.)

Unaware interpreters of hard disk partitioning will hardly consider
the base device for mounting but rather resort to the partitions.
Aware readers, like software in the ISO's initrd, will normally ignore
the partition table and act like on DVD.
But as said, in the end it does not matter which ISO gets mounted.


> FWIW, I'm all for having a MBR-only Guix installer disk as you suggest.

It should at least be tested.
Best with an option of the build process by which the user can choose
between "original", "mbr_hfs" + partition offset 16, "mbr_only" + offset,
and "gpt_appended".

Default should be "original" for now, but with the goal to switch to
"mbr_hfs" or "mbr_only" when enough courageous testers have reported
success.


> But I'd rather not do the hack the others are doing.

I am nagging against the Fedora-inspired layout since years. :))


> How should we mount the root when the right way (with two non-overlapping
> partitions) boots from DVD ?

On DVD there is no public offer of the partition's filesystem.
At least the Linux kernel is not supposed to look for partition tables
on optical media.
(Actually ioctl BLKRRPART is so CD-phobic that it does not even refresh
 the size perception of the medium but simply bails out.
 Else it would be a nice gesture after DVD burning to do
   hdparm -z /dev/sr0
 growisofs emulates this by putting out the tray and pulling it in again,
 if there is a tray motor. Watch your fingers.)


> I'm surprised that it needs so much manual intervention to make
> grub-mkrescue do the right thing.

It does the right thing for the main purpose of creating a widely bootable
ISO image. (The fact that Florian's Macbook takes offense from the EFI
partition image is independent of the partition layout.)

The complaints of Linux at partition assessment time are ugly but harmless.
The lack of a Linux-mountable ISO partition among the four GPT partitions
is ugly, too. But the user can at any time mount the base device of the
USB stick.

grub-mkrescue's partition layout becomes really suboptimal when the user
wants to use the remaining capacity of the USB stick for one or more
read-write data partitions.
Partition editors righteously complain about the misplaced GPT backup,
which cannot be avoided because the image does not get produced for a
storage device of a particular size.

Similarly suboptimal for partition editors is the Fedora-inspired layout
as produced by "isohybrid -u" or "xorrisofs ... -isohybrid-gpt-basdat".
Other than grub-mkrescue's GPT, this layout is barely legal.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

So what’s the plan?

Wait for another responce from the grub-devel mailing list?

Package grub-mkrescue-sed.sh and use
--xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?

Add an option to “guix system disk-image” to select which
grub-mkrescue-sed.sh environment variables to enable?

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Wed, Apr 24, 2019 at 11:13:23AM +0200, pelzflorian (Florian Pelz) wrote:
> On Wed, Apr 24, 2019 at 08:56:57AM +0200, Thomas Schmitt wrote:
> > Florian Pelz wrote
> > > By the way, someone I know has a most peculiar machine called Lenovo
> > > Ideapad 100S which does not boot Guix, […]
> > 
> > If Guix does not show up in the boot manager, try a plain grub-mkrescue
> > image that was made when GRUB was configured for both EFI variations:
> > 32 bit x86 and 64 x86.
> > My experiments are made when it is configured for those two plus PC-BIOS.
> > The EFI partition then has bootx64.efi and bootia32.efi in /efi/boot/.
> > 
> 
> Will do.
> 

The Lenovo Ideapad 100S’s owner tried a USB drive prepared with a
purely 32-bit GRUB as per

On Thu, Apr 25, 2019 at 10:18:08AM +0200, Thomas Schmitt wrote:
>   wget https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
>   chmod u+x grub-mkrescue-sed.sh
> 
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
> 
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh  \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \

-o mbrhfs.iso

and it boots fine.

I do not know and did not try yet how to configure GRUB to produce
both 32-bit and 64-bit EFI.  (Would putting both 32-bit and 64-bit
bootefi files in EFI/Boot/ be a good idea for Guix ISOs in general, or
could this lead to problems with 64-bit machines or lead to confusing
boot selection screens?)

A Debian 9.8.0 32-bit also boots fine and a DVD with an untreated Guix
git master could not be booted but very likely due to an error with my
external DVD drive (it often does not work) and not due to an error
with the DVD.

I will try a current git master Guix iso built for 32-bit next.  If it
does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
will try the same with

dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> So what’s the plan?
> Wait for another responce from the grub-devel mailing list?

They need time to to make up their mind. Maybe other incidents are needed to
push towards some change in grub-mkrescue.
We have a halfways positive response from the old owner and a somewhat
uninformed response from the new manager, indicating between the lines
that he is too busy with other things currently.


> Package grub-mkrescue-sed.sh and use
> --xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?

It is part of the libisoburn release tarball, from where Guix gets its
xorriso, afaik. It is also part of the GNU xorriso tarball, which builds
a static compilation of libburn, libisofs, libisoburn, and libjte.

Problem is that for the EFI partition fix, you need the current development
version. The partition definitions should be ok with the released version
1.5.0.
I plan a new release this year, but not right now.

How comfortable is the Guix patching system ? :))

  "New control variable for grub-mkrescue-sed.sh: MKRESCUE_SED_IN_EFI_NO_PT"
  https://dev.lovelyhq.com/libburnia/libisoburn/commit/3a2a3ba737a06162c22ace0ae09d33ba97aa2673

  "grub-mkrescue-sed.sh: MKRESCUE_SED_IN_EFI_NO_PT="extra"
   to erase MBR signature"
  https://dev.lovelyhq.com/libburnia/libisoburn/commit/1eb51f44dadb8b6c5f87533ca357186cdc1ac625

(The holidays are over. Our certificate works again.)


Looking at a commit from 2 years ago (i.e. in released version)
  "Made -iso_mbr_part_type 0x00 default of grub-mkrescue-sed.sh
   mode "mbr_only""
i remember that there was an EFI reported on grub-devel which strictly
refused on anything but a single MBR partition for EFI. So the camouflage
trick of partition type 0x0 was necessary for the ISO partition.

This is quite contrary to the expectations of partition editors, though.
I deem this EFI's behavior a much clearer bug than the Macbook EFI's.
So i stay with my proposal of native xorriso command
  -boot_image any iso_mbr_part_type=0x83
or mkisofs emulation option
  -iso_mbr_part_type 0x83

(To my knowledge, the Guix xorriso run switches from mkisofs emulation
 to native commands by "--". So -boot_image would be the one to use
 if it gets appended to the other arguments.)


> Add an option to “guix system disk-image” to select which
> grub-mkrescue-sed.sh environment variables to enable?

This would be great.
(I was brought up with HP BASIC and never was able to solve the Lisp
 puzzles in german magazine Bild der Wissenschaft of the 1980s.
 So i cannot help much with translating the usage gestures or even
 the script itself from shell to Guile.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4507 bytes --]

> How comfortable is the Guix patching system ? :))

Very comfortable.  But we'd like not to diverge from upstream too much.
We are all in this together, so I'd like upstream to eventually converge on
a solution.  I'm fine with carrying a patch if it's coordinated by people with
knowledge about the matter at hand.  For multiple projects, we already carry
patches while they are being actively reviewed upstream in order to fix bugs.

If the problem is not too bad, I'd like the change to take the normal path
instead of us interfering with upstream.  But here, you *are* upstream so
it's really up to you whether we should patch xorriso or not.

As for people testing possible avenues by patches in Guix, that's easy to do.

> This is quite contrary to the expectations of partition editors, though.
> I deem this EFI's behavior a much clearer bug than the Macbook EFI's.
> So i stay with my proposal of native xorriso command
>   -boot_image any iso_mbr_part_type=0x83
> or mkisofs emulation option
>   -iso_mbr_part_type 0x83
> 
> (To my knowledge, the Guix xorriso run switches from mkisofs emulation
>  to native commands by "--". So -boot_image would be the one to use
>  if it gets appended to the other arguments.)
> 
> 
> > Add an option to “guix system disk-image” to select which
> > grub-mkrescue-sed.sh environment variables to enable?  
> 
> This would be great.

Since grub-mkrescue-sed.sh is part of xorriso, it's easy to get it
to where it needs to be (gnu/build/vm.scm) from where xorriso is
refererred to (gnu/system/vm.scm).  (Just pass the package as argument)
I can do that part.

The patches would then go into the xorriso package definition we carry
(gnu/packages/cdrom.scm) and modify grub-mkrescue-sed.sh in its output
(the patching part would require no Guile knowledge, they are just
".patch" files we distribute and automatically apply).

I'm not sure about amending "guix system disk-image" by options that
aren't really end user options, but I guess it can't be helped.

Basically they would just be something that we'd tell users which
have problems booting the Guix iso to use.  Even that is a lot to ask
from normal users--building a new installer ISO for the sake of
installing Guix.

> (I was brought up with HP BASIC and never was able to solve the Lisp
>  puzzles in german magazine Bild der Wissenschaft of the 1980s.
>  So i cannot help much with translating the usage gestures or even
>  the script itself from shell to Guile.)

Oh, I see.  It's fine if it's in a shell script.

We aren't opposed to shell scripts--sometimes (in initrd etc) we already have
Guile there and don't want to bloat the image by shells and shell utils.

But in this case, we would be patching the iso generator which requires
a lot of the system anyway, so who cares if we pull in the shell and 20
small utils?

Also, we try very hard to keep packages modular.

But if no care is taken, shell scripts take stuff from the global environment
and thus are very difficult to predict and isolate for modularity.

Think of what happens when someone replaces the "test" binary
in $PATH, or "sed".  It will unintentionally potentially completely
change the output depending on what the user has installed (it's definitely
not what the user intended to happen).

As a first countermeasure, we build inside isolated build containers, which
makes the problem less bad.  But then grub-mkrescue will invoke
grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
as the sed to use?

(For that matter, think of what happens when another xorriso is somewhere
in $PATH and we use --xorriso=.../grub-mkrescue-sed.sh).  (I see that
you already take care to use a "closer" xorriso--but these kind of problems
pop up in shell scripts for almost every term used in there.  It's kinda
difficult to get them to behave)

What's more, shell scripts don't fail on error.  For example in
grub-mkrescue-sed.sh there's no "-e" in the shebang, which means that if
some command in there has an error, the shell will happily continue anyway.
I'm not blaming you, that's just the culture with shell scripts.

Even with "-e", if a command in a pipeline fails, it will sometimes still
not fail the shell script.  For that, you need "set -o pipefail" which is
a bash feature and not necessarily available on other shells (dash etc).

Guile, on the other hand, will just fail the script on error (usually).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I do not know and did not try yet how to configure GRUB to produce
> both 32-bit and 64-bit EFI.

As Debian user i am in the comfortable position to install the
three binary packages for BIOS, x86 EFI 32 bit, and x86 EFI 64 bit.
Then grub-mkrescue knows what to do.

Some digging in the source of grub-mkrescue and util/grub-install-common.c
reveils that it is about the existence of $cpu-$firmware directories
with lots of .mod files in them:

  $ ls /usr/lib/grub
  grub-mkconfig_lib  i386-efi  i386-pc  x86_64-efi

The binary Debian packages nearly entirely consist of one of these
directories, each:
  https://packages.debian.org/sid/amd64/grub-efi-amd64-bin/filelist
  https://packages.debian.org/sid/amd64/grub-efi-ia32-bin/filelist
  https://packages.debian.org/sid/amd64/grub-pc-bin/filelist

But i don't know the tool which is to be used for creating those
directories and for filling them with .mod files.

Maybe you find some GRUB tool usage in the Guix ISO production which
uses parameters like "x86_64" "efi".
Or the name "grub-mkconfig_lib" gives a hint ? man grub-mkconfig and
info chapter are quite sparse.


> (Would putting both 32-bit and 64-bit
> bootefi files in EFI/Boot/ be a good idea for Guix ISOs in general, or
> could this lead to problems with 64-bit machines or lead to confusing
> boot selection screens?)

It should be perfectly ok to combine several EFI start programs in
one EFI partition. The firmware is supposed to pick what it deems
suitable.
UEFI 2.4 3.4.1.1 "Removable Media Boot Behavior":

  "[...] the system firmware will attempt to boot from a removable media
   FilePathList[0] by adding a default file name in the form
     \EFI\BOOT\BOOT{machine type short-name}.EFI.
   Each file only contains one UEFI image type, and a system may
   support booting from one or more images types."

You could even combine "IA32", "X64", "ARM", and "AA64" programs if you
manage to keep all CPU specific files apart in the course of the further
operating system run.


> The Lenovo Ideapad 100S’s owner tried a USB drive prepared with a
> purely 32-bit GRUB [...]
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh  \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \

> and it boots fine.

So that's not the mad EFI which hates all MBR partitions except one 0xef.


> a DVD with an untreated Guix
> git master could not be booted but very likely due to an error with my
> external DVD drive (it often does not work)

We could start a separate endeavor in private for diagnosing that.
(My other sport besides ISO 9660 is burning flat round things.)


> I will try a current git master Guix iso built for 32-bit next.  If it
> does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
> will try the same with
> dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2

Was the machine similarly stuck as the Macbook ?
That would probably be the partition problem, indeed.

But failure when GRUB or a Linux kernel have taken over would be a
different problem. The minimal grub-mkrescue ISO would then possibly
work because it prompts the user before the problem can occur.
Or maybe because it has a .mod file that is needed for that machine ?


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 25, 2019 at 05:09:05PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I do not know and did not try yet how to configure GRUB to produce
> > both 32-bit and 64-bit EFI.
> […]
> Some digging in the source of grub-mkrescue and util/grub-install-common.c
> reveils that it is about the existence of $cpu-$firmware directories
> with lots of .mod files in them:
> 

This is interesting, but I am not familiar enough to try to quickly
change the guix package and I am not going to invest time in that.
The reason I was interested is that the CPU supposedly can run 64-bit
code, it is apparently just the UEFI that refuses 64-bit.  But one can
install 32-bit and then reconfigure Guix later on to use 64-bit, so it
is not important.



> > a DVD with an untreated Guix
> > git master could not be booted but very likely due to an error with my
> > external DVD drive (it often does not work)
> 
> We could start a separate endeavor in private for diagnosing that.
> (My other sport besides ISO 9660 is burning flat round things.)
> 
>

:D Burning is not the problem. ;)  This slim laptop has no DVD drive
itself and my external USB DVD drive works only sometimes and might
have required even more patience.  Booting got stuck waiting for the
DVD drive which continued to make noises, as it often does on other
machines when it has a bad day.  This was not a problem with the DVD
or the image, but with that DVD drive of mine.

I will test Danny’s patches on my Macbook and on the Ideapad.


> > I will try a current git master Guix iso built for 32-bit next.  If it
> > does not work (guixsd-install-0.15.0.i686-linux.iso did not work), I
> > will try the same with
> > dd if=/dev/zero bs=1 count=16 seek=446 of=/dev/sdc2
> 
> Was the machine similarly stuck as the Macbook ?
> That would probably be the partition problem, indeed.
> 

With the USB flash drive booting was not stuck, it just did not appear
in the boot device selection.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Danny Milosavljevic wrote:
> you *are* upstream so
> it's really up to you whether we should patch xorriso or not.

Then do if you can make use of those two git commits.
They are the only ones to that file since release 1.5.0.


> I'm not sure about amending "guix system disk-image" by options that
> aren't really end user options, but I guess it can't be helped.

How about

  --file-system-type=iso9660[_$variation]

with $variation being "gpt_hfs" (= "original"), "mbr_hfs", "mbr_only",
and "gpt_appended", defaulting to "gpt_hfs" ?

This translates quite straightforward to the proposed variations of
grub-mkrescue-sed.sh in
  http://lists.gnu.org/archive/html/guix-devel/2019-04/msg00420.html

The partition offset 16 and MBR partition type 0x83 would be ordered
only with "mbr_*" variations.

The setting
  export MKRESCUE_SED_IN_EFI_NO_PT=yes
should be used unconditionally. It gets applied to the EFI partition
block 0, only if it is marked with an MBR signature 55 aa. Then we may
safely assume that bytes 446 to 509 are MBR partition table, which is
of no use and maybe harmfull. Only then they get zeroed.

The experimental setting
  export MKRESCUE_SED_IN_EFI_NO_PT=extra
is not yet sufficiently motivatable for production ISOs. It exists for
getting an impression whether mformat option -k would be safe in
grub-mkrescue. (I.e. it is research for general GRUB enlightenment.)


> Basically they would just be something that we'd tell users which
> have problems booting the Guix iso to use.

If the Guix download servers can afford two more ISOs, how about offering
"gpt_hfs" and "mbr_hfs" ISOs with the explanation that both should work
for booting and installing, but that "mbr_hfs" is more convenient when
the remaining capacity of a Guix ISO USB stick shall be used as partition
for a read-write filesystem.
(Moving the backup GPT is an expert option in fdisk. But without moving
 it there is no room for a new partition.)

On the long run and if no negative results arise, i propose to switch the
default to "mbr_hfs".


> But if no care is taken, shell scripts take stuff from the global
> environment and thus are very difficult to predict and isolate for
> modularity.

I tried hard to keep it conservative and am open for proposals to
make it more portable to older shells.


> But then grub-mkrescue will invoke
> grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
> as the sed to use?

grub-mkrescue runs the mformat program and the mcopy program before
it runs xorriso.


> (For that matter, think of what happens when another xorriso is somewhere
> in $PATH and we use --xorriso=.../grub-mkrescue-sed.sh).

Should not matter much, as long as this xorriso is young enough.
grub-mkrescue will ask by running effectively
  ./grub-mkrescue-sed.sh -as mkisofs -help 2>&1
and inspecting the options list of the real xorriso involved.

The only difference will be two message lines (one empty)
  frontend/grub-mkrescue-sed.sh manipulating xorriso arguments

but then it will say
  xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

  Usage: xorriso -as mkisofs [options] file...
  ...

If you can tell the absolute path to a good xorriso, then set

  export MKRESCUE_SED_XORRISO=...path.to.real.xorriso.binary...


> For example in grub-mkrescue-sed.sh there's no "-e" in the shebang

Is it conservative enough to add it ?


> Guile, on the other hand, will just fail the script on error (usually).

That's why i brought a Guile re-write into consideration.
Firstly it would be the language of choice and secondly it would make
Guix independent of my ideas how it should be done. As said, the
script-in-the-middle gives substantial control over the stage of
grub-mkrescue when the prepared files are still not packed up in the ISO.

(Or maybe it's time to re-write the script in C ... as was done with
 grub-mkrescue when it became too complicated.)


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> So what’s the plan?
>
> Wait for another responce from the grub-devel mailing list?
>
> Package grub-mkrescue-sed.sh and use
> --xorriso=/path/to/grub-mkrescue-sed.sh when calling grub-mkrescue?
>
> Add an option to “guix system disk-image” to select which
> grub-mkrescue-sed.sh environment variables to enable?

I lost track of the discussion.  You’re looking at a MacBook-specific
issue, right?

(The underlying question being: can this wait after 1.0 is out?  :-)
If it’s MacBook-specific, I would answer “yes.”)

Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i worte:
> > (My other sport besides ISO 9660 is burning flat round things.)

Florian Pelz wrote:
> :D Burning is not the problem. ;)

We could let xorriso put some read load on it and watch out for error
messages.


> With the USB flash drive booting was not stuck, it just did not appear
> in the boot device selection.

Then it is undecided yet. Either it dislikes the EFI partition's content
or it avoided the endless loop by discarding the iwhole partition.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4253 bytes --]

Hi Thomas,

On Thu, 25 Apr 2019 18:22:11 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Then do if you can make use of those two git commits.
> They are the only ones to that file since release 1.5.0.

Okay, I've picked them up.

> > I'm not sure about amending "guix system disk-image" by options that
> > aren't really end user options, but I guess it can't be helped.  
> 
> How about
> 
>   --file-system-type=iso9660[_$variation]
> 
> with $variation being "gpt_hfs" (= "original"), "mbr_hfs", "mbr_only",
> and "gpt_appended", defaulting to "gpt_hfs" ?

I'm not sure yet.  We have a lot of special-casing for iso9660
already.  If anything, at that point, we could pass an arbitrary list
of options or something (an "environment" if you will.  Hah).

> This translates quite straightforward to the proposed variations of
> grub-mkrescue-sed.sh in
>   http://lists.gnu.org/archive/html/guix-devel/2019-04/msg00420.html
> 
> The partition offset 16 and MBR partition type 0x83 would be ordered
> only with "mbr_*" variations.
> 
> The setting
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
> should be used unconditionally. It gets applied to the EFI partition
> block 0, only if it is marked with an MBR signature 55 aa. Then we may
> safely assume that bytes 446 to 509 are MBR partition table, which is
> of no use and maybe harmfull. Only then they get zeroed.

Okay, that makes sense.

> If the Guix download servers can afford two more ISOs, how about offering
> "gpt_hfs" and "mbr_hfs" ISOs with the explanation that both should work
> for booting and installing, but that "mbr_hfs" is more convenient when
> the remaining capacity of a Guix ISO USB stick shall be used as partition
> for a read-write filesystem.
> (Moving the backup GPT is an expert option in fdisk. But without moving
>  it there is no room for a new partition.)

I don't know these things.  But someone on the list will know.

> On the long run and if no negative results arise, i propose to switch the
> default to "mbr_hfs".

Yeah, I agree.

> I tried hard to keep it conservative and am open for proposals to
> make it more portable to older shells.

The problem is that there were horrible security problems in bash and that
made a lot of distributions switch to a more minimal shell ("dash") for
scripts.  I agree that that was a good move, but error handling is even
worse there.

> > But then grub-mkrescue will invoke
> > grub-mkrescue-sed.sh at runtime, and who knows what the latter will pick up
> > as the sed to use?  
> 
> grub-mkrescue runs the mformat program and the mcopy program before
> it runs xorriso.

Yeah, we patch grub-mkrescue to use an absolute path for mformat and mcopy
in order to make them predictable.

> > For example in grub-mkrescue-sed.sh there's no "-e" in the shebang  
> 
> Is it conservative enough to add it ?

It will exit the shell when one of the toplevel commands has an
exit status != 0.
So as long as your script doesn't have toplevel commands return error status
in normal operation it should be fine.

Toplevel:

$ grep foo bar

Not toplevel:

$ if grep foo bar

(that won't fail the script even with "-e")

Questionable:

$ foo | bar

If foo fails, should the script fail?  Spoiler: It won't fail the script even with "-e".
If bar fails, should the script fail?  Spoiler: It will fail the script with "-e".

But:

$ set -e
$ set -o pipefail
$ foo | bar

Now if foo or bar fails, the script will fail.

> > Guile, on the other hand, will just fail the script on error (usually).  
> 
> That's why i brought a Guile re-write into consideration.
> Firstly it would be the language of choice and secondly it would make
> Guix independent of my ideas how it should be done. 

We're not trying to become iso bootloader experts ourselves :)

>As said, the
> script-in-the-middle gives substantial control over the stage of
> grub-mkrescue when the prepared files are still not packed up in the ISO.
> 
> (Or maybe it's time to re-write the script in C ... as was done with
>  grub-mkrescue when it became too complicated.)

Yeah, we'll see.  For now, let's try the shell script and make it more
paranoid.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

i wrote:
> >   --file-system-type=iso9660[_$variation]

Danny Milosavljevic wrote:
> I'm not sure yet.  We have a lot of special-casing for iso9660
> already.  If anything, at that point, we could pass an arbitrary list
> of options or something (an "environment" if you will.  Hah).

How ever the options are brought to the grub-mkrescue run and the script,
i strongly urge not to expose all possible combinations of MKRESCUE_SED*
variables to the ISO production process.
Four consistent variations will be enough and avoid riddling failures.


> The problem is that there were horrible security problems in bash and that
> made a lot of distributions switch to a more minimal shell ("dash") for
> scripts.

It is supposed to run on dash, ksh, and BSD sh versions.


> we patch grub-mkrescue to use an absolute path for mformat and mcopy
> in order to make them predictable.

How deep shall this go ? dd, awk, od, head, find, test, mv, rm, ...


> > > For example in grub-mkrescue-sed.sh there's no "-e" in the shebang

> > Is it conservative enough to add it ?

> It will exit the shell when one of the toplevel commands has an
> exit status != 0.

But do all reasonable shells support -e ?
(I must dig in man ksh down to builtin command -set to get confirmation.)


> > That's why i brought a Guile re-write into consideration.
> > Firstly it would be the language of choice and secondly it would make
> > Guix independent of my ideas how it should be done.

> We're not trying to become iso bootloader experts ourselves :)

Currently Guix ISO production is not enough boss over what gets into the
ISO image. Neither with EFI partition properties nor with partition table
type.


> For now, let's try the shell script and make it more paranoid.

Please report all changes (and be ready to motivate them).


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

> How deep shall this go ? dd, awk, od, head, find, test, mv, rm, ...

Ideally everything.  That's why we don't use shell scripts that much:  They
are annoying to make behave.  Compare with C which will link a library and
with option "-Wl,-rpath" (which we use everywhere) will embed the absolute
path to the library in the executable and NEVER use another one than the
one it was compiled with.

Usually, we just wrap the entire shell script in a wrapper that sets
up PATH first.  As long as there are no naming conflicts in the
dependencies, it should be fine (since PATH just lists directories, not
regular files, it could still be bad).

I'm thinking of doing this PATH wrapper here, too.  It's easily automated on
the Guix side.

The end goal (for the most part reached) is to have a package be a
module that doesn't affect the rest of the system except when it is
supposed to.  Other distributions are spectacularily bad at this--to
say nothing about Windows where it's normal that installing one
application can break a totally unrelated application that had already
been there.

> But do all reasonable shells support -e ?
> (I must dig in man ksh down to builtin command -set to get confirmation.)

I don't know.

> > For now, let's try the shell script and make it more paranoid.  
> 
> Please report all changes (and be ready to motivate them).

Okay!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Thu, Apr 25, 2019 at 06:34:16PM +0200, Ludovic Courtès wrote:
> I lost track of the discussion.  You’re looking at a MacBook-specific
> issue, right?
> 
> (The underlying question being: can this wait after 1.0 is out?  :-)
> If it’s MacBook-specific, I would answer “yes.”)
> 

As I understand it, Thomas’ work is about
- making a correct ISO image and
- making it work on more machines.

I would have hoped the 1.0 ISO could already be made correct.
Older ISOs work on cooperative motherboards as well though, it appears
only old Macs are affected.  A more recent 2014 Macbook Air boots fine
though.

As for the latter, here are my findings when booting the USB
installer:


|                     | Nvidia PC with        | AMD GPU PC with                      |
|                     | Asus P8H67-V board    | Asus P8H67-?                         |
|---------------------+-----------------------+--------------------------------------|
| x86_64              | boots with UEFI       | boots but graphical installer        |
| unpatched master    | and legacy            | only works with UEFI                 |
|---------------------+-----------------------+--------------------------------------|
| x86_64 with Danny's | boots with UEFI       | boots but graphical installer        |
| patches (wip6)      | and legacy            | only works with UEFI                 |
|---------------------+-----------------------+--------------------------------------|
| i686                | boots graphical       | boots in legacy mode w/o             |
| unpatched master    | installer in legacy   | graphical installer, boots from      |
|                     | mode, boots from      | harddisk instead when selecting UEFI |
|                     | harddisk instead when |                                      |
|                     | selecting UEFI        |                                      |
|---------------------+-----------------------+--------------------------------------|
| i686 with Danny's   | boots graphical       | boots in legacy mode w/o             |
| patches (wip6)      | installer in legacy   | graphical installer, boots from      |
|                     | mode, boots from      | harddisk instead when selecting UEFI |
|                     | harddisk instead when |                                      |
|                     | selecting UEFI        |                                      |
|---------------------+-----------------------+--------------------------------------|

|                     | 2010 Macbook       | Lenovo Ideapad 100S |
|---------------------+--------------------+---------------------|
| x86_64              | boot gets stuck    | does not appear     |
| unpatched master    | before listing     | in the boot         |
|                     | devices to boot    | selection           |
|                     | from               |                     |
|---------------------+--------------------+---------------------|
| x86_64 with Danny's | boots fine with a  | does not appear     |
| patches (wip6)      | selection of “EFI  | in the boot         |
|                     | Boot” and “GRUB    | selection           |
|                     | 2.02”; both appear |                     |
|                     | to boot in UEFI    |                     |
|                     | mode though        |                     |
|---------------------+--------------------+---------------------|
| i686                | boot gets stuck    | boots fine (is      |
| unpatched master    | before listing     | only shown as EFI   |
|                     | devices to boot    | device)             |
|                     | from               |                     |
|---------------------+--------------------+---------------------|
| i686 with Danny's   | only “GRUB 2.02”   | boots fine (is      |
| patches (wip6)      | can be selected,   | only shown as EFI   |
|                     | but boots from     | device)             |
|                     | harddisk instead   |                     |
|---------------------+--------------------+---------------------|

I wonder why guixsd-install-0.15.0.i686-linux.iso did not boot on the
Lenovo Ideapad 100S before.  Windows claims the Lenovo is a 32-bit
machine only btw.

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> > I lost track of the discussion.  You’re looking at a MacBook-specific
> > issue, right?

Florian Pelz wrote:
> As I understand it, Thomas’ work is about
> - making a correct ISO image and

The grub-mkrescue ISOs are correct but also sometimes inconvenient on
USB stick. So rather:
- Making ISO images which are more digestible for partition interpreters
  when presented unchanged on real storage devices. Especially for
  partition editors by which the user wants to add a read-write data
  partition to the USB stick.

> - making it work on more machines.

In this case it's about the Macbook EFI and mformat's partition table
entry. Exotic but also typical for firmware quirks.

----------------------------------------------------------------------

In general my proposal is about getting control over the GRUB-produced
files before xorriso packs them up, and over the arguments of the xorriso
run.

Danny pointed out that Guix ISO production software does not aim for
becomming expert for iso bootloaders.
But the Macbook EFI behavior and the Linux partition table complaints
do not demand deep knowledge of bootloaders. The remedies just need the
opportunity to make small adjustments to the work of grub-mkrescue.

Where such remedies come from, is a different question.
Count on frustration-tolerant users and curious bystanders.

The shell script implementation is somewhat questionable. We could use
it for exploring the best variation and then implement that variation
in a less tolerant programming language with less outer influences.

I asked Danny, whether the download servers could offer two partition
table variations of the ISOs: GPT and MBR (both with APM/HFS+).
Goal would be to gain experience with MBR and to finally replace GPT
by that layout.
He expressed the hope that other members of Guix could answer this.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Thu, Apr 25, 2019 at 06:34:16PM +0200, Ludovic Courtès wrote:
>> I lost track of the discussion.  You’re looking at a MacBook-specific
>> issue, right?
>> 
>> (The underlying question being: can this wait after 1.0 is out?  :-)
>> If it’s MacBook-specific, I would answer “yes.”)
>> 
>
> As I understand it, Thomas’ work is about
> - making a correct ISO image and
> - making it work on more machines.
>
> I would have hoped the 1.0 ISO could already be made correct.
> Older ISOs work on cooperative motherboards as well though, it appears
> only old Macs are affected.  A more recent 2014 Macbook Air boots fine
> though.

So the ISO only “fails” (somehow?) on old Macs, is that correct?

If it’s “just” about old Macs, that’s not a 1.0 blocker for me (though
it’s something we should fix.)

If the issue might be broader, we need more data to estimate the scope
of the problem (your table mentioned UEFI vs. “legacy” boot and Lenovo
laptops, which I’m not sure how to interpret.)

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[pelzflorian (Florian Pelz)]

On Fri, Apr 26, 2019 at 04:41:27PM +0200, Ludovic Courtès wrote:
> So the ISO only “fails” (somehow?) on old Macs, is that correct?
> 
> If it’s “just” about old Macs, that’s not a 1.0 blocker for me (though
> it’s something we should fix.)
>

Old Macs are the only machines we know of that otherwise don’t work.
Lenovo Ideapad 100S had problems with Guix 0.15.0, but it seems
already fixed and unrelated.  There may be other affected machines.
As Thomas Schmitt wrote in his last e-mail, there are other advantages
when users want or need to modify an install USB drive.

If I understand correctly, Thomas Schmitt and Danny Milosavljevic
would rather work on a Guile implementation instead of submitting the
current patch to use a shell script from xorriso for
the 1.0 release (?).

Regards,
Florian

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> I asked Danny, whether the download servers could offer two partition
> table variations of the ISOs: GPT and MBR (both with APM/HFS+).
> Goal would be to gain experience with MBR and to finally replace GPT
> by that layout.
> He expressed the hope that other members of Guix could answer this.

I don’t feel like distributing two variants of the ISO per architecture.
In addition to the fact that it would take up more space on the GNU
servers, I suspect few users would know which one to choose (I
wouldn’t.)  It’s also something other distros don’t do: you get only one
ISO per architecture.

Thanks,
Ludo’.

Re: ISO installer image: GPT versus MBR partitions

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> I don’t feel like distributing two variants of the ISO per architecture.

Then i guess the most early opportunity for my proposal to get
into production is after the upcomming Guix release.


> It’s also something other distros don’t do: you get only one
> ISO per architecture.

Not really. :))
Debian offers a wide range of sizes from mini.iso (32 MB) to BDDL (50 GB)
and two flavors for x86: Installer and Live (with several desktop flavors).
I have to acknowledge, though, that their boot equipment at most differs by
lagging behind debian-cd evolution.

If it's about what other distros do: They do MBR partition table.


Have a nice day :)

Thomas

Re: ISO installer image: GPT versus MBR partitions

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Ludovic Courtès wrote:
>> I don’t feel like distributing two variants of the ISO per architecture.
>
> Then i guess the most early opportunity for my proposal to get
> into production is after the upcomming Guix release.

Sounds good to me.  Thanks again for helping out!

Ludo’.