‘guix publish’ as a content-addressed file server

‘guix publish’ as a content-addressed file server

[Ludovic Courtès]

Hi!

Until now, if hydra.gnu.org had a source file in its store (the result
of an ‘origin’), you could get it via substitutes.  However, with
substitutes disabled, hydra.gnu.org was of no help, even though it did
have the source file.

Commit ff6638d112d794c9c433731643711932452fd2ff helps address that: it
augments ‘guix publish’ such that it can be used as a content-addressed
mirror for source files¹.

If you run ‘guix publish -p 9999’ on your machine, and if
hello-2.10.tar.gz is in the store, then this URL:

  http://localhost:9999/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

… gives you hello-2.10.tar.gz.

This works by constructing the corresponding store file name, assuming
the file being requested results from a fixed-output derivation using
the same hash algorithm (SHA256 here) or was added with ‘add-to-store’.
That the hash algorithm must match is a limitation, but it’s probably
“good enough” for our purposes.

Eventually I think we’ll run it on our build farm and add the URL to
‘%content-addressed-mirrors’ in (guix download).

Feedback welcome.  :-)

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00450.html

Re: ???guix publish??? as a content-addressed file server

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]

It sounds like a usefull capability.

But presumably it depends on your localhost having a TCP/IP stack
and a http server installed.

It would be better if it could just use a socket or even just the filesystem.

J'

On Wed, Jul 20, 2016 at 05:25:30PM +0200, Ludovic Court??s wrote:
     Hi!
     
     Until now, if hydra.gnu.org had a source file in its store (the result
     of an ???origin???), you could get it via substitutes.  However, with
     substitutes disabled, hydra.gnu.org was of no help, even though it did
     have the source file.
     
     Commit ff6638d112d794c9c433731643711932452fd2ff helps address that: it
     augments ???guix publish??? such that it can be used as a content-addressed
     mirror for source files??.
     
     If you run ???guix publish -p 9999??? on your machine, and if
     hello-2.10.tar.gz is in the store, then this URL:
     
       http://localhost:9999/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
     
     ??? gives you hello-2.10.tar.gz.
     
     This works by constructing the corresponding store file name, assuming
     the file being requested results from a fixed-output derivation using
     the same hash algorithm (SHA256 here) or was added with ???add-to-store???.
     That the hash algorithm must match is a limitation, but it???s probably
     ???good enough??? for our purposes.
     
     Eventually I think we???ll run it on our build farm and add the URL to
     ???%content-addressed-mirrors??? in (guix download).
     
     Feedback welcome.  :-)
     
     Ludo???.
     
     ?? https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00450.html
     

-- 
Avoid eavesdropping.  Send strong encryted email.
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://sks-keyservers.net or any PGP keyserver for public key.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: ???guix publish??? as a content-addressed file server

[Ludovic Courtès]

John Darrington <john@darrington.wattle.id.au> skribis:

> It sounds like a usefull capability.
>
> But presumably it depends on your localhost having a TCP/IP stack
> and a http server installed.

You need an OS with a TCP/IP stack, but ‘guix publish’ is the web server
(it uses the web server that is part of Guile.)

> It would be better if it could just use a socket or even just the filesystem.

It’s meant as a way to download sources remotely, so HTTP seems well
suited, especially since it usually passes firewalls.

Ludo’.

Re: ‘guix publish’ as a content-addressed file server

[Ludovic Courtès]

Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

> Until now, if hydra.gnu.org had a source file in its store (the result
> of an ‘origin’), you could get it via substitutes.  However, with
> substitutes disabled, hydra.gnu.org was of no help, even though it did
> have the source file.
>
> Commit ff6638d112d794c9c433731643711932452fd2ff helps address that: it
> augments ‘guix publish’ such that it can be used as a content-addressed
> mirror for source files¹.
>
> If you run ‘guix publish -p 9999’ on your machine, and if
> hello-2.10.tar.gz is in the store, then this URL:
>
>   http://localhost:9999/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
>
> … gives you hello-2.10.tar.gz.

guix-maintenance commit 7f59985566b384e31da7e6f1a36744e9edfba54f adds
nginx proxying for those /file URLs (‘guix publish’ is now running on
hydra.gnu.org, though we only use it for those /file URLs;
mirror.hydra.gnu.org proxies and caches those requests):

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://mirror.hydra.gnu.org/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i | guix hash /dev/stdin
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---

Guix commit 40f788b9f6184436d9cc36a4dd8e7d101cd2f0ba adds
mirror.hydra.gnu.org as a content-addressable mirror.

Ludo’.