bug#35283: ISO images are not reproducible

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

As noted by Florian in <https://issues.guix.info/issue/33639>, images
created by ‘guix system disk-image --file-system-type=iso9660’ are not
bit-reproducible.

Ludo’.

bug#33639: ISO installer image is broken on i686

[pelzflorian (Florian Pelz)]

On Fri, Apr 12, 2019 at 11:26:28PM +0200, Ludovic Courtès wrote:
> Florian, it would be great if you could confirm.  Just apply it on
> ‘master’, and then run:
> 
>   ./pre-inst-env guix system disk-image --file-system-format=iso9660 \
>      gnu/system/install.scm
> 

Yes, it seems fixed, I can confirm.  Four rebuilds seem fine and are
bootable in QEMU.  They have the same size and `xorriso -indev` is
happy.  The content is different at the beginning of the ISO image
(maybe padding or timestamps in the file system) and in the EFI
partition at the very end of the ISO, but this seems insignificant.

Regards,
Florian

bug#33639: ISO installer image is broken on i686

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Yes, it seems fixed, I can confirm.

Way back in december, Ludovic Courtès wrote:
>...> Based on this and on a suggestion Ricardo made on IRC, I passed
>...> -padding 10m and that solved the problem.  \o/

Please do not forget to remove this -padding command.


Florian Pelz wrote:
>  The content is different at the beginning of the ISO image
> (maybe padding or timestamps in the file system)

That's to expect if not environment SOURCE_DATE_EPOCH is set and exported.

SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It
is supposed to be either undefined or to contain a decimal number which
tells the seconds since january 1st 1970. If it contains a number, then
it is used for all timestamps and as seed of pseudo-random numbers like
MBR id or GPT UUIDs.

If all files and directories have the same names and the same content,
then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
value are supposed to create byte-identical result ISOs.

In december, i wrote:
>...> >   Creation Time: 1970010119010649
Ludovic Courtès wrote:
>...> For reproducibility purposes we set timestamps and related things
>...> to the Epoch.

Is this independent of SOURCE_DATE_EPOCH ?


Have a nice day :)

Thomas

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi,

(Moving discussion to <https://issues.guix.info/issue/35283>, which is
specifically about ISO image reproducibility issues.)

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> Florian Pelz wrote:
>>  The content is different at the beginning of the ISO image
>> (maybe padding or timestamps in the file system)
>
> That's to expect if not environment SOURCE_DATE_EPOCH is set and exported.
>
> SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It
> is supposed to be either undefined or to contain a decimal number which
> tells the seconds since january 1st 1970. If it contains a number, then
> it is used for all timestamps and as seed of pseudo-random numbers like
> MBR id or GPT UUIDs.
>
> If all files and directories have the same names and the same content,
> then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
> value are supposed to create byte-identical result ISOs.

By mounting the ISO image, I found that some files didn’t have their
timestamp reset: some files in /var/guix (easily fixed), but more
importantly those added by GRUB in /boot and /System.

Files added by ‘grub-mkrescue’ are “out of our control” so we would need
to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

However, after rereading the Xorriso manual, it seemed to me that if we
set SOURCE_DATE_EPOCH and pass:

  -volume_date all_file_dates set_to_mtime

then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
which would solve the problem.

I tried it, but that’s not what happened.  What am I missing, Thomas?

Thanks,
Ludo’.

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

> Files added by ‘grub-mkrescue’ are “out of our control” so we would need
> to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

Google shows that patches have been proposed. But they seem not to
have made it into the source.

Vladimir Serbinko's answer here
  https://lists.gnu.org/archive/html/grub-devel/2015-12/msg00046.html
might be the reason. I understand that he demands uniqueness of UUIDs.

But that's not really a problem with reproducible ISOs. If pseudo-random
UUIDs depend deterministically on SOURCE_DATE_EPOCH, then collisions are
only to expect between ISOs made with the same seconds value.
This can also happen if non-reproducible ISOs are made while their
systems' clocks show the same time by mere incident.

So one should use SOURCE_DATE_EPOCH values with best possible entropy.
Not one humanly invented lucky number for all ISOs of a distro.

If ever two identical ISOs are offered to GRUB at boot time, it needs
some imagination to construct a problem if GRUB operates on the one
which was not used by the EFI firmware to start GRUB.


So when a reproducible ISO is made for the first time, its SOURCE_DATE_EPOCH
should be taken from "date +%s" and recorded for further runs.
The ISO will bear it as "Creation Time", like "2019021612165300".
The last two digits "00" are centiseconds and should be ignored even
if not "00".
If decoding that time back to seconds-since-1970 is cumbersome, one may
store the seconds value in a data file in the input tree of the ISO
before packing up by a xorriso run with SOURCE_DATE_EPOCH having that
value.


> after rereading the Xorriso manual, it seemed to me that if we
> set SOURCE_DATE_EPOCH and pass:
>   -volume_date all_file_dates set_to_mtime
> then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
> which would solve the problem.

This is the support for ignoring atime and ctime changes of input files
but respecting their mtime changes.

If you want a fixed time for all three timestamps in all files, do:

  -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

The "=" announces seconds-since-1970 as time format. See -alter_date.

Note that in this proposal $SOURCE_DATE_EPOCH is evaluated by the shell,
not by xorriso. Depending on the way how xorriso is started, you need to
insert the actual number.


Have a nice day :)

Thomas

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi Thomas,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

> If you want a fixed time for all three timestamps in all files, do:
>
>   -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

Thanks, that’s what I was missing.

It was still not the end of the story, but I have some good news: the
series of commits below allow me to build ISO images reproducibly!  \o/

  1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing it.
  5502fbd7fd gnu: valgrind: Add 3.15.0.
  605815023c vm: Use a fixed FAT serial number for 'efi.img' in ISO images.
  52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
  833480cc1f vm: Reset file timestamps in ISO images.

To check by yourself you can do, say:

  guix system disk-image --file-system-type=iso9660 \
     gnu/system/examples/bare-bones.tmpl

and then check the ISO derivation that was built as the last step above:

  guix build --check -K /gnu/store/…-image.iso.drv

Ludo’.

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

>   833480cc1f vm: Reset file timestamps in ISO images.

That's also a big solution for the problem of timestamps of synthetic files.

I understand that your plan for reproducibility is to make timestamps
completely insignificant. Radical but effective.

But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
use the same seconds value for the ISO file objects ?


>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.

Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
not only influences mformat underneath grub-mkrescue, but also the run
of xorriso, where it determines volume date timestamps and GPT individual
UUIDs.
(Other impacts of the variable get overridden by the
   -volume_date "all_file_dates"
 command in commit 833480cc1f.)


>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.

I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
a 2.8 MB FAT image.


>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing

How good are chances to bring such changes into upstream ?

I ask in the advance assumption that we find a way to make the mformat
image digestible for Florian's Macbook.
(It is clear now that the difference between failure and success is in
 mformat versus mkfs.fat. But the exact point of failure is not found yet.
 I place my bet on the partition entry with start LBA 0.)


Have a nice day :)

Thomas

bug#35283: ISO images are not reproducible

[Ludovic Courtès]

Hi,

"Thomas Schmitt" <scdbackup@gmx.net> skribis:

>>   833480cc1f vm: Reset file timestamps in ISO images.
>
> That's also a big solution for the problem of timestamps of synthetic files.
>
> I understand that your plan for reproducibility is to make timestamps
> completely insignificant. Radical but effective.
>
> But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
> use the same seconds value for the ISO file objects ?

Files in /gnu/store, by convention, all have their mtime set to 1 (one
second after the epoch).

>>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
>
> Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
> not only influences mformat underneath grub-mkrescue, but also the run
> of xorriso, where it determines volume date timestamps and GPT individual
> UUIDs.
> (Other impacts of the variable get overridden by the
>    -volume_date "all_file_dates"
>  command in commit 833480cc1f.)

AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
Xorriso, or at least it was overridden by the “-volume_date” options
that I pass.

It’s crucial for me to have the mtime set to 1 for all the files on the
ISO; I wanted the 1980 setting to apply only to ‘efi.img’.

>>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
>
> I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
> but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
> a 2.8 MB FAT image.

I haven’t dig deep enough to provide a satisfactory answer.  :-)

>>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing
>
> How good are chances to bring such changes into upstream ?

I’ve emailed them (actually tried to, their mailing list rejected my
message.)  We’ll see!

Ludo’.

bug#35283: ISO images are not reproducible

[Thomas Schmitt]

Hi,

Ludovic Courtès wrote:
> AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
> Xorriso, or at least it was overridden by the “-volume_date” options
> that I pass.

Probably. Among the automatic grub-mkrescue options for xorriso's mkisofs
emulation is

  --modification-date=2019042117165600

The equivalent native command is

  -volume_date uuid 2019042117165600

So you indeed have to override this by an own subsequent command.

(SOURCE_DATE_EPOCH overrides defaults of xorriso. But commands or options
 override the overridden defaults.)


Have a nice day :)

Thomas