Re: Guix infrastructure

[ludo@gnu.org (Ludovic Courtès)]

Hello Guix!

Leo Famulari <leo@famulari.name> skribis:

> On Thu, Jul 06, 2017 at 08:09:17PM -0400, myglc2 wrote:
>> On 07/01/2017 at 14:01 Leo Famulari writes:
>> > ... Bayfront is still not fully operational, so hydra.gnu.org is still
>> > serving as the front-end of the build farm. We are still relying on the
>> > Hydra software. That is, the situation is basically the same as before.
>> > Adding build machines will not help very much until the front-end
>> > hardware gets faster.
>> 
>> This leaves me wondering ...
>> 
>> Is the hydra/front-end hardware going to be upgraded?
>
> Yes...
>
>> Is bayfront/cuirass intended to replace hydra?
>
> ... and yes.
>
>> The bayfront hardware described here ...
>> 
>> https://www.gnu.org/software/guix/news/growing-our-build-farm.html
>> 
>> ... seems weak to me. Is there a plan to scale it up and make it redundant?
>
> It will be a lot more powerful than the current Hydra system. As for
> specific plans, I'll let those administering the system chime in.

That machine is super powerful… but alas, it has also been terribly
unstable.  Vikings has been kind enough to assist us; they’ve notably
provided us with CPU replacements once already.  Despite these efforts,
the machine is still crashing.  We’re investigating with them what to
do next.

On top of that, all the testing and all the back and forth takes an
awful lot of time, which is in part due to hardware problems being hard
to pinpoint and debug in general, and in part due to us here in Bordeaux
(where the machine is hosted) being unable to scale up.


Infrastructure has been the project’s Achilles’ heel since we run the
crowdfunding campaign in Dec. 2015 (!).  Now it’s becoming detrimental
to the project.  Our initial plan was to buy more Libreboot-based
machines like the one above once the first one has proved to work well.
However, given the situation, we’ve been discussing on guix-sysadmin a
change in strategy, at least in the short term; Ricardo has been working
on re-purposing used hardware for our needs, and that may well be our
short- to medium-term solution.

For now, we prefer not to entrust the binaries we deliver to commercial
VPS providers.  I think we owe it to our users, but it undoubtedly has a
cost in terms of system maintenance.

I not only sympathize with your frustration, ng0, I feel it even ten
times more.  ;-)  Several of us are determined to come up with a
solution quickly, so I hope that will materialize soon!

Thanks,
Ludo’.