Speed up package installation by using images instead of archives (like distri)?

Speed up package installation by using images instead of archives (like distri)?

[Mekeor Melire]

Hello Guix!

On the Guix-blog¹, Ludo already mentioned the lightning talk of Michael 
Stapelberg on the “distri” package manager which installs packages 
faster than common package managers by an order of magnitude.

Common package managers install a package by downloading it as an 
(compressed) archive and extracting the archive. But this extraction 
takes very long as it requires to create dozens of files on the file 
system which is an expensive IO operation.

In contrast, distri downloads the package as an (squashfs) image. That's 
it. The image will then be mounted (which is not an expensive IO 
operation) as a read-only virtual-file-system. (But as it's not 
recommendable to mount as many images as there are installed packages, 
distri mounts the package-images lazily, i.e. right before they are 
being used.)

(At least, the above explanation reflects my understanding.)

Thus, I wonder: (0) Does it make sense to make Guix follow this idea as 
well? I.e. should we make Guix install packages by downloading images 
and mounting those instead of extracting archives? (1) Does this even 
fit into Guix technically? I.e. how hard would it be to implement this? 
(2) And would it even be worth it? I.e. how much faster would Guix 
become?

I hope for an exciting discussion!


mekeor

--

Footnotes:

   1: 
https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

Re: Speed up package installation by using images instead of archives (like distri)?

[Ludovic Courtès]

Hi!

Mekeor Melire <mekeor@posteo.de> skribis:

> In contrast, distri downloads the package as an (squashfs)
> image. That's it. The image will then be mounted (which is not an
> expensive IO operation) as a read-only virtual-file-system. (But as
> it's not recommendable to mount as many images as there are installed
> packages, distri mounts the package-images lazily, i.e. right before
> they are being used.)
>
> (At least, the above explanation reflects my understanding.)
>
> Thus, I wonder: (0) Does it make sense to make Guix follow this idea
> as well? I.e. should we make Guix install packages by downloading
> images and mounting those instead of extracting archives? (1) Does
> this even fit into Guix technically? I.e. how hard would it be to
> implement this? (2) And would it even be worth it? I.e. how much
> faster would Guix become?

The Guix blog post mentions squashfs images, suggesting it’s not
transposable to Guix.  :-)

Specifically, there are several practical issues if you think about what
it would take to mount squashfs images instead of extracting things:
you’d have one mount point per store item? how does that interact with
GC?  what about GNU/Hurd support?  I view it as a can of worms, if it’s
workable at all.

Would it be faster?  Like other proposals in the distri talk, it’s all
about doing things lazily instead of eagerly.  In this case,
decompression happens lazily instead of eagerly.  I think that could
make application startup on a cold cache slower, precisely because that
work has to happen anyway.  More importantly, mounting all these
squashfs images at boot time could be costly.

Looking at the big picture, with zstd substitutes available, the main
speed issues in Guix are elsewhere.  What would make a big difference is
if we didn’t have to download (or build) that much so often.  This is
what the end of the blog post hints at.

HTH!

Ludo’.