Speed up grafts by storing reference offset in index

Speed up grafts by storing reference offset in index

[Ricardo Wurmus]

Hello Guix,

grafts can be a little slow for a number of reasons:

- they are not substituted, because the assumption is that it is
  preferrable to rewrite references locally instead of downloading a big
  archive with the modified file.  Local computations on x86_64 are
  often acceptable, but on aarch64 systems they can be very slow indeed.

- when a lot of grafts need to be applied, many files need to be
  rewritten

- big files take longer to read and thus to rewrite

Since it is December and I'm in a silly mood here is a silly idea: would
it make sense to shift parts of the grafting work to an offloadable
build?  Here's what I imagine:

- on the build farms build an additional derivation for a references
  file.  The references file is an S-expression containing a list of
  tuples of the form (FILE-NAME OFFSET).  Each of these tuples
  identifies the location of a single reference at the recorded byte
  OFFSET in FILE-NAME.

- when computing grafts, don't search the local files sequentially for
  references but look them up in the references file.  Instead of
  computing the reference file substitute it from a build server.

Alternatively, change the format for substitutes and record reference
locations there, so that the local store database can also store
reference locations.  It already stores references (for things like
"guix gc -R"), so maybe we could store just a little extra information
to allow us to seek to the offset directly.

As you can tell, I haven't thought this through, and there are a number
of different places where a feature like this could live.

What do you think?

-- 
Ricardo