Re: Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 5199 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> Thanks for the update.  And for the all work. :-)
>
>
> On Wed, 15 Dec 2021 at 16:48, Christopher Baines <mail@cbaines.net> wrote:
>
>> In summary, the space issue I mentioned in the previous update has
>> effectively been addressed. All the paused agents are now unpaused and
>> builds are happening again.
>
> The timing had almost been perfect. ;-)
>
>
> Well, as discussed on Sept., one concern I have is about “long-term
> storage” – where long-term is not well-defined and storage either.
>
> Do you think that Bordeaux could run
>
>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>

The Guix Build Coordinator just builds derivations. I haven't got it to
build a manifest before, but that's possible I guess.

I think it's unnecessary though, since I believe derivations for all
origins of all packages are already being built, that happens through
just asking the coordinator to build derivations for all packages, you
don't need to specify "source" derivations separately.

> ?  Having a redundancy about all origins would avoid breakage.  For
> instance, because Berlin was down yesterday morning, “guix pull” was
> broken because the missing ’datefuge’ package – disappeared upstream.

I would hope that bordeaux.guix.gnu.org has a substitute for that, could
you check the derivation against data.guix.gnu.org, and see if there's a
build? Use a URL like:

  https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv

There is one issue though, bordeaux.guix.gnu.org doesn't provide content
addressed files in the same way guix publish does. I hope to add that
through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
hopefully be added to the list of content addressed mirrors:

  https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368

That would mean that the bytes for a tar archive for example would be
available by the sha256 hash, not just as a nar. I'm not sure to what
extent this would help, but it's probably useful.

>> In general this is an important step in being more flexible where the
>> nars are stored. There's still a reliance on storing pretty much all the
>> nars on a single machine, but which machine has this role is more
>> flexible now. I think this architecture also makes it easier to break
>> the "all nars on a single machine" restriction in the future as well.
>
> IIUC the design, if the proxy server is lost, then it is easy to replace
> it.  Right?

I guess so, the nar-herder helps with managing the data at least which
makes setting up new or replacement servers easier.

> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
> solves the issue but from my understanding, it will improve at least
> performance delivery.  Well, it appears to me worth to give a try.
>
>
> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
> 3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
> 4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
> 6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>

Effectively this is moving towards building a CDN. With the nar-herder,
you could deploy reverse proxies (or edge nodes) in various
locations. Then the issue just becomes how to have users use the ones
that are best for them. This might require doing some fancy stuff with
GeoIP based DNS, and somehow sharing TLS certificates between the
machines, but I think it's quite feasible.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>
> To me, one first general question about backup coordination is to define
> a window for time:
>
>  - source: forever until the complete fallback to SWH is robust;
>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>    for any commit reachable by inferior: forever;
>  - package substitute: rule something.

The idea I've been working with so far is simply to store everything
that's built, forever.

Currently, that amounts to 561,043 nars totaling ~2.5TB's.

How feasible this is depends on a number of factors, but I don't have
any reason to think it's not feasible yet.

> Thanks for taking care about redundancy and reliance of CI.

There's not a relationship to continuous integration yet, although I am
hoping if the building and serving substitutes stuff stabilises,
bordeaux.guix.gnu.org might be able to play a part in testing patches
and branches (as discussed in [1]).

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00001.html

Thanks for all your comments and questions!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]