[bug#47336] Disarchive as a fallback for downloads

["Ludovic Courtès" <ludo@gnu.org>]

Hi!

Timothy Sample <samplet@ngyro.com> skribis:

> This patch series adds Disarchive assembly (backed by SWH lookup) as a
> fallback for downloads.
>
> To try it, make sure you are running the daemon in an environment with
> Disarchive available:
>
>     $ ./pre-inst-env guix environment --ad-hoc guile disarchive
>     # ./pre-inst-env guix-daemon --build-users-group=guixbuild
>
> Don’t forget to stop your existing Guix Daemon.  :)
>
> You also need to make sure that regular downloads are unavailable.  I do
> this by adjusting the “try” loop at the end of “url-fetch” in
> “guix/build/download.scm”.  I replace the usual list of URLs with ‘()’:
>
>     (let try ((uri (append uri content-addressed-uris)))
>       (match '() ; uri
>         ...))
>
> Now you can ask Guix for a recent .tar.gz source package:
>
>     $ ./pre-inst-env guix build --no-substitutes -S python-httpretty
>
> You should see:
>
>     Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
>     Assembling the directory httpretty-1.0.5
>     Downloading from Software Heritage...
>     7903d608efc89c14afb4d692a3721156e31a43e2/
>     7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
>     7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
>     [...]
>     Checking httpretty-1.0.5 digest... ok
>     Assembling the tarball httpretty-1.0.5.tar
>     Checking httpretty-1.0.5.tar digest... ok
>     Assembling the Gzip file httpretty-1.0.5.tar.gz
>     Checking httpretty-1.0.5.tar.gz digest... ok
>     Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
>     successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv

Commits 67bf61255414115ffae0141df9dd3623bc742bff and
0b1f70d1a792af40aa0d13b3d227fde88f02d061 add the dependency on
Disarchive, so this fallback path is now enabled!

> There’s lots to talk about though....
>
> First, it looks up the metadata on my server.  This is fine for a demo,
> but not what we want forever.  The patch series supports adding several
> mirrors for looking up the metadata.  In the past, we talked about
> putting everything on one or a few of the big Git hosting platforms like
> GitHub or Gitlab.  That way, it would be easily picked up by SWH and
> archived “forever”.  Right now, I have Cuirass set up to build the
> metadata, and a little script that moves it from the build server to my
> Web server.  It would be simple enough to adjust that script to push it
> to a remote Git repo.  (Of course, the next step is to move this setup
> to Guix infrastructure.)  Thoughts?

We should talk to SWH, giving them the figures you gave earlier in this
thread.  But yeah, a Git repo looks best to me (it would be useful to
keep track of changes, for example if we eventually update metadata to a
new format) and it simplifies archival to SWH.

Second thing we need to figure out if where to create this database.  If
you have a Cuirass job already, we should run it on ci.guix.  WDYT?

> On the code level, there were two things I couldn’t figure out for
> myself.
>
> I made the mirror list just simple strings.  AIUI, the client and the
> daemon have to agree about the format of the mirror list.  Given that
> running old daemons is common, changing the format is difficult.  Is it
> worth it to copy the more flexible interface used by the content
> addressed mirrors?  If yes, do I have to do the same ‘module-autoload!’
> dance to use ‘bytevector->base16-string’?  :)  (I probably would have
> just copied it, but that part confused me a bit.)

I had overlooked this suggestion of yours.  Yes, I think it’s best to
copy the SWH scheme.  Don’t worry about ‘module-autoload!’: nowadays we
can safely assume (guix base16) is available.

When we change from list-of-strings to list-of-procedures, we’ll have to
adjust the (guix build download) code so that it can deal with both.

> I imported some modules from “guix/build/download.scm” (well, just
> “base16” and “swh”).  It feels weird to use a bunch of host-side modules
> from what’s nominally a “guix/build” module.  This is okay because
> “guix/build/download.scm” is not /really/ build-side code.  It’s more
> like daemon (-ish) code that just happens to live in “guix/build”, which
> is why importing host-side modules is OK... right?

Yup.  :-)  In the end, the whole point is to reuse code on both sides,
and that’s what’s being done here.

Thanks,
Ludo’.