Static hosting of substitutes

Static hosting of substitutes

[Ian Eure]

Hi folks,

I’d like to provide substitutes for packages in my personal 
channel.  The ideal setup for this would be for a machine on my 
internal net to perform the builds, then upload the results to 
another system on the open internet.  That could be a machine 
running a web server pointed at a directory where the NARs get 
uploaded, or an S3-like object store, or something like that -- 
dirt simple, just shifting bytes off disk and out a socket.

It seems that nothing like this exists, all the public substitute 
servers appear to use `guix publish'.  That’s not an option for 
me, since it requires significantly more disk and compute than I 
have on any public-facing system, and I can’t justify the cost of 
bigger machines.

What would it take to make a system like this work?

Thanks,

  — Ian

Re: Static hosting of substitutes

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

Ian Eure <ian@retrospec.tv> writes:

> I’d like to provide substitutes for packages in my personal channel.
> The ideal setup for this would be for a machine on my internal net to
> perform the builds, then upload the results to another system on the
> open internet.  That could be a machine running a web server pointed
> at a directory where the NARs get uploaded, or an S3-like object
> store, or something like that -- dirt simple, just shifting bytes off
> disk and out a socket.
>
> It seems that nothing like this exists, all the public substitute
> servers appear to use `guix publish'.  That’s not an option for me,
> since it requires significantly more disk and compute than I have on
> any public-facing system, and I can’t justify the cost of bigger
> machines.
>
> What would it take to make a system like this work?

I've run a few substitute servers like this, the required code is
actually quite simple and the build coordinator includes the necessary
bits in the form of some included hooks [1].

1: build-success-publish-hook and build-success-s3-publish-hook in
https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/tree/guix-build-coordinator/hooks.scm

bordeaux.guix.gnu.org used to use the build-success-publish-hook to
populate a directory with the narinfo and nar files, and NGinx simply
served this directory, although now it uses the nar-herder to manage the
nars (it still doesn't use guix publish).

Maybe in the future the nar-herder could provide an easy utility for
people to serve and manage substitutes like this.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Static hosting of substitutes

[Ian Eure]

Christopher Baines <mail@cbaines.net> writes:

> [[PGP Signed Part:Undecided]]
> Ian Eure <ian@retrospec.tv> writes:
>
>> I’d like to provide substitutes for packages in my personal 
>> channel.
>> The ideal setup for this would be for a machine on my internal 
>> net to
>> perform the builds, then upload the results to another system 
>> on the
>> open internet.  That could be a machine running a web server 
>> pointed
>> at a directory where the NARs get uploaded, or an S3-like 
>> object
>> store, or something like that -- dirt simple, just shifting 
>> bytes off
>> disk and out a socket.
>>
>> It seems that nothing like this exists, all the public 
>> substitute
>> servers appear to use `guix publish'.  That’s not an option for 
>> me,
>> since it requires significantly more disk and compute than I 
>> have on
>> any public-facing system, and I can’t justify the cost of 
>> bigger
>> machines.
>>
>> What would it take to make a system like this work?
>
> I've run a few substitute servers like this, the required code 
> is
> actually quite simple and the build coordinator includes the 
> necessary
> bits in the form of some included hooks [1].
>
> 1: build-success-publish-hook and build-success-s3-publish-hook 
> in
> https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/tree/guix-build-coordinator/hooks.scm
>
> bordeaux.guix.gnu.org used to use the build-success-publish-hook 
> to
> populate a directory with the narinfo and nar files, and NGinx 
> simply
> served this directory, although now it uses the nar-herder to 
> manage the
> nars (it still doesn't use guix publish).
>

Thank you for the pointer.  Does this integrate with Cuirass at 
all?  I have a box running cuirass and guix publish which I’m 
using for my internal builds & substitutes -- ideally, I’d like 
NARs to get uploaded to a public host when Cuirass completes a 
build.

Thanks,

  — Ian