Re: Mid-December update on bordeaux.guix.gnu.org

[zimoun <zimon.toutoune@gmail.com>]

Hi Chris,

On Thu, 16 Dec 2021 at 00:20, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> Do you think that Bordeaux could run
>>
>>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>
>
> The Guix Build Coordinator just builds derivations. I haven't got it to
> build a manifest before, but that's possible I guess.

I am not sure to understand since Cuirass also builds derivations and
the purpose of this source-manifest.scm is to let Cuirass ingest all
sources,

    <https://ci.guix.gnu.org/jobset/source>


> I think it's unnecessary though, since I believe derivations for all
> origins of all packages are already being built, that happens through
> just asking the coordinator to build derivations for all packages, you
> don't need to specify "source" derivations separately.

Your assumption is wrong, IMHO.  We have many failed examples and at the
end Ludo wrote this source-manifest.scm to be sure that all is ingested
for sure.


>> ?  Having a redundancy about all origins would avoid breakage.  For
>> instance, because Berlin was down yesterday morning, “guix pull” was
>> broken because the missing ’datefuge’ package – disappeared upstream.
>
> I would hope that bordeaux.guix.gnu.org has a substitute for that, could
> you check the derivation against data.guix.gnu.org, and see if there's a
> build? Use a URL like:
>
>   https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv
>
> There is one issue though, bordeaux.guix.gnu.org doesn't provide content
> addressed files in the same way guix publish does. I hope to add that
> through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
> hopefully be added to the list of content addressed mirrors:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368
>
> That would mean that the bytes for a tar archive for example would be
> available by the sha256 hash, not just as a nar. I'm not sure to what
> extent this would help, but it's probably useful.

Thanks for explaining the details.  From a pragmatical point of view as
end-user, “guix pull” must Just Work whatever the plumbing.

For instance, some of us spent energy to “evangelize” in scientific
communities how Guix is awesome.  Next morning, people give a look and
bang!  Because a tiny and short outage.  Obviously, “shit happens”(*)
and it is really really really sparse but too late the damage is there.

My feeling here is that both build farms work independently instead of
coordinate the usage of resources and exploit this strength to have two.


(*) quoting Forest Gump: :-)
  – whoa man, you just ran through a big pile of dog shit!
  – it happens
  – what? shit?
  – sometimes


>> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
>> solves the issue but from my understanding, it will improve at least
>> performance delivery.  Well, it appears to me worth to give a try.
>>
>> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
>> 3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
>> 4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
>> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
>> 6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>
>
> Effectively this is moving towards building a CDN. With the nar-herder,
> you could deploy reverse proxies (or edge nodes) in various
> locations. Then the issue just becomes how to have users use the ones
> that are best for them. This might require doing some fancy stuff with
> GeoIP based DNS, and somehow sharing TLS certificates between the
> machines, but I think it's quite feasible.

Considering the human resource vs the money resource, it appears to me
better to invest the human energy into things that do not exist and rely,
for now, on existing solutions; even if the project has to pay money.

For what my opinion is worth here.


>> To me, one first general question about backup coordination is to define
>> a window for time:
>>
>>  - source: forever until the complete fallback to SWH is robust;
>>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>>    for any commit reachable by inferior: forever;
>>  - package substitute: rule something.
>
> The idea I've been working with so far is simply to store everything
> that's built, forever.

For sure, anyone wants that at the end.  My point is raising the
priority of the intermediary steps.


> Currently, that amounts to 561,043 nars totaling ~2.5TB's.
>
> How feasible this is depends on a number of factors, but I don't have
> any reason to think it's not feasible yet.

That’s exactly my point!  It is not about feasibility – all is doable
with enough time and energy – but instead it is about controlling the
factors to “robustify“ what the project consider highly important – as
keep all sources or never break ‘guix pull’ – whatever the status of
infrastructure.


Again, thanks for all the work.  Becausse, in any case, for sure, the
situation is daily improving. :-)

Cheers,
simon