Installing via iso.xz, really best idea? -- was Re: Gnome Boxes

[Bengt Richter <bokr@bokr.com>]

Hi,

On +2021-02-22 00:39:36 -0500, Evan Rowley wrote:
> FreeBSD also provides .iso.xz. Some examples here:
> http://ftp-archive.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/13.0/
>

Looking there for an example:
--8<---------------cut here---------------start------------->8---
FreeBSD-13.0-BETA2-amd64-bootonly.iso       363927552  2021-Feb-12 07:47
FreeBSD-13.0-BETA2-amd64-bootonly.iso.xz     82351604  2021-Feb-12 07:47
--8<---------------cut here---------------end--------------->8---

That seems like a really worthwhile compression! (ratio 4.4:1)

However, I wonder if big .iso's are really the best way
to take advantage of the xz (or any) compression for installation purposes.

If the point of having the information in an .iso is to have it be bootable
with everything (drivers, UI, etc) needed to install to any raw disk anywhere,
ISTM that is wastefully general for my usual case:

I have several laptops all able to dd to a USB-attached SSD or thumbdrive.
I trust their impure apps to do the xfr and dd, if I can verify result hashes independently.
I don't need to boot anything but my laptop for these apps: they're on my laptops.
I don't intend to use my foreign laptop to synthesize content of the files to xfr by dd,
  so there is only dependency on the pure guix versions of fdisk, mkfs, etc used on
  the machine that executed
      guix system -make-custom-bootstrap-scripted-tarball my-neat-system.scm
  (in my dreams :) (maybe a modded version of guix pack could do it, but sizes?)
┌─────────────────────────────────────────────────────────────────────────────────┐
│ I don't want to modify the laptop I'm using (in fact, I want a guarantee        │
│ that it WON'T be modified as I use it to dd-write to the attached USB storage). │
└─────────────────────────────────────────────────────────────────────────────────┘
(boxed for emphasis :)

All I really need is a tarball with a bootstrap script that essentially
does a series of dd file transfers to selected block sequences on the target disk(s).

I am thinking this could be smaller and easier to use than using guix pack
to concoct the same functionality.

As the script starts, IWBN to be able to select target disk(s) interactively, probably using
lsblk to probe and display, but I want to avoid partitioning target disk(s) with foreign tools
(i.e. writing to the medium with other than dd).

I think read-only access to get existing (if any) partition locations and sizes and types is ok,
so the script can potentially choose between sparse files to dd into common sizes of disk and
partition spaces where e.g. just padding with zeroes is not workable. IWBN to be able to target
a toy 32-gb USB as well as the front end of a 500GB SSD with the same script/tarball.

Those files for dd can contain ready-made images of GPT pieces -- superblock, VFAT fs,
legacy-bootable grub blocks, /boot and / and maybe other specialized block-sequence images
(with mkfs and transfers into the populated fs images already done on a pure guix system
using loopback-mounted files for fs images), or whatever else may be needed.
(caveat hunch: subject to some init-hook binary patching on first boot (hopefully no grub-launched
or BIOS-validated thing like firmware update necessary if kernel init can do it ;/ ).

For the case where I want to modify my own laptop, e.g. to boot from the USB-attached
storage I just prepared, that depends on what its BIOS does for booting, but I guess
that is commonly just a matter of updating grub's .cfg.

For the case where I want to install to my own laptop's disk(s), I still don't like
booting a monster iso. If my laptop's existing OS can't go into a safe maintenace mode,
where the target disk is synced and unmounted, then I will have to reboot.

If that is from a bootable installer USB, I want a rescue-size bootable, not
the monster iso including everything (nor the job and wait of writing the iso image
to a USB just so I can boot it once).

ISTM a manifest for the installation job could be a file of symbolic by-hash links
or equivalent sha256sum listing, and a list of mirrors, specifying
directories where the by-hash links are findable, e.g. like
    https://example.com/pub/filedir/by-hash/
and links in that directory looking like e.g.,
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -> ../zerolengthfile
(which are easy to generate selectively or wholesale for a filedir/ directory
just creating the ./by-hash/ subdirectory and making the links))

ISTM like a custom bootstrap.sh could safely wget each file,
piping directly to dd writing these chunks to raw disk slots,
afterwards checking the sha256 of every block sequence it wrote.
Done simply --  and irrespective of whether the by-hash links
and files were stored in a git repo or plain file directories.

I think "guix system" could automate the production of customized
my-neat-system-boostrap.sh scripts containing the right wget and dd commands
and sha256 verifications, while providing safe interactive choice
of target disk(s) at the start of running it.

Such a script, built to run on a specific but non-guile/guix-dependent
"foreign" platform like Linux x86_64 would be a nice way to share a
"my-neat-system" ... or a Guix release.

So it seems to me, anyway :)

(I know you can do all kinds of things with laptops already running guix.
A main point here is to be able to use a (borrowed even) foreign, non-guile/guix
laptop to install something onto raw USB-attached storage without transferring
anything from the state of the laptop doing the work (i.e., xfr only, no synthesis).

I.e., the system or data being transferred
could be for a totally different architecture or purpose.

For the paranoid, a first-boot (assuming new thing is bootable)
hook could re-check all the hashes on the new system.

(Please excuse all the handwaving :)

> On Thu, Feb 18, 2021 at 4:23 PM Julien Lepiller <julien@lepiller.eu> wrote:
> 
> > Sorry, a compressed .iso is probably common, a .iso.xz is very uncommon
> > :). We even have had some reports of people trying to copy that directly to
> > an installation media.
> >
> > Le 18 février 2021 14:56:44 GMT-05:00, Tobias Geerinckx-Rice <me@tobias.gr>
> > a écrit :
> >>
> >> Julien Lepiller 写道：
> >>
> >>> Usually compression is provided by the webserver, but maybe ours
> >>> is not configured to do that.  I think we're the only distro to
> >>> provide compressed isos.
> >>>
> >>
> >> Really?  Most distribution ISOs use squashfs or similar with
> >> XZ/LZMA compression.  It doesn't make sense to compress that over
> >> the wire.
> >>
> >> That said: XZ compression currently saves 27% (559M -> 405M).
> >> Transparently serving pre-compressed ISOs with nginx (gzip level
> >> 9) would save about 25% (559M -> 415M), which is surprisingly
> >> similar.
> >>
> >> Kind regards,
> >>
> >> T G-R
> >>
> >>
> 
> -- 
>  - EJR

Hm, I wonder how large a tarball "guix pack" as it works now  would create
to create something runnable with the my-neat-system-bootstrap.sh
functionality described above, vs depending on the foreign system's
bash, wget, dd, etc.


-- 
Regards,
Bengt Richter