linux-libre source tarballs

[Vagrant Cascadian <vagrant@debian.org>]

[-- Attachment #1: Type: text/plain, Size: 3984 bytes --]

So, for better or worse, I do a lot with guix on aarch64 platforms and
sometimes even armhf ones. And these platforms are...  often...
quite...  slow...

The way guix does updates of linux-libre sources by downloading upstream
linux tarballs and applying the linux-libre deblobbing and related
scripts to them is really technically excellent.

Pragmatically speaking, on slower platforms this is a huge resource
overhead. So much so that ci.guix.gnu.org *usually* times out when
generating the linux-libre aarch64 tarballs:

  https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic

Same story for "linux-libre".

This means that platforms that are already fairly slow have to build one
or more source tarballs every linux-libre update, which can take several
hours to build the source for a kernel that takes only an hour or less
of compilation time.


I explored some of the alternatives last year:

  https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00089.html

Some of the surrounding threads also go into various issues around this
topic, but I'll summarize what I tried:

* Using linux-libre's git repository. A *bit* slow to download and
  consumed ~1GB of space in /gnu/store, though much faster than running
  the deblobbing scripts. Probably would deduplicate fairly well between
  different versions, but would require re-downloading a lot of
  content. Maybe a more savvy method to make use of this would be
  possible. If a method somewhat like the mirror:// syntax for git
  repositories allowed for a local git cache or a local git proxy, this
  could save a lot of downloading and still get some benefits of
  git. Needs someone to actually do the work, of course...
 
* Using the linux-libre tarballs. May have some issues with long-term
  availability, but fast to download and the deblobbing scripts have
  effectively already been run, saving a lot of local work. There is
  basically support for this in guix already, and was the method used in
  guix before july 2019 1ad9c105c208caa9059924cbfbe4759c8101f6c9.


Recently I've been thinking more about this for aarch64 and armhf; there
are a few things that might help considerably:

* Bump the timeouts on ci.guix.gnu.org for linux-libre so that the
  deblob scripts actually get a chance to finish. Doesn't require much
  change in the guix infrastructure to significantly improve the status
  quo for slower architectures. Might take some tweaking over time to
  find the right timeouts.

* Default to using the fsfla linux-libre tarballs on aarch64 and armhf,
  while doing the full local deblobbing process for other faster
  architectures. This keeps the infrastructure of both methods
  up-to-date and tested, and can be used to do interesting things like
  comparing to make sure the linux-libre tarballs guix generates and the
  fsfla linux-libre tarballs do not contain significant
  differences. This may very well require maintenance overhead by
  generating two different hashes (one for the guix deblobbed tarball,
  and one for the linux-libre deblobbed tarball), and doing things a
  little differently by architecture is a little awkward.

* Produce the guix tarballs in a way that architectures could share the
  resulting tarball. I'm not sure what hoops this would require jumping
  through, something like generate the tarball, upload somewhere, update
  hash. The tarball produced by running the deblob scripts really
  shouldn't be architecture-dependent; re-doing this work on each
  architecture seems suboptimal (though I get is also somewhat a result
  of the functional package management model).


I like the idea of keeping all the infrastructure there to handle what
is currently done now, but would really like to figure out alternatives
at least for aarch64 and armhf.


Thanks for reading so far, please let me know what you think!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]