From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: plz is there a roadmap for a more resilient substitutes infrastructure? Date: Tue, 06 Nov 2018 12:23:05 +0100 Message-ID: <87tvku5u1y.fsf@gnu.org> References: <87wopv7jzw.fsf@roquette.mug.biscuolo.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:54505) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gJzS7-000208-MU for guix-devel@gnu.org; Tue, 06 Nov 2018 06:23:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gJzS3-00008k-Hp for guix-devel@gnu.org; Tue, 06 Nov 2018 06:23:21 -0500 In-Reply-To: <87wopv7jzw.fsf@roquette.mug.biscuolo.net> (Giovanni Biscuolo's message of "Fri, 02 Nov 2018 13:16:03 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Giovanni Biscuolo Cc: guix-devel@gnu.org Ciao Giovanni, Giovanni Biscuolo skribis: > recently users and developers are facing hard to manage problems due to > the maintainance of hydra.gnu.org and its proxy mirror.hydra.gnu.org [1] > since 23 Oct 2018 We Guix developers don=E2=80=99t have control over the physical hardware be= hind hydra.gnu.org; for this machine, we rely on the work of the FSF sysadmins for all things hardware/networking. Unfortunately in this case, this maintenance period was rather unprepared: it wasn=E2=80=99t supposed to last a whole week, rather a few h= ours or a day at most. Most of the time it took was about copying data to a new disk (!). Had this been prepared, we could have arranged to keep hydra.gnu.org up until the replacement was ready. We Guix developers didn=E2=80=99t have much visibility over what was going on though, and we j= ust didn=E2=80=99t anticipate this. It is clear that this prolonged downtime was harmful to many users and to the project=E2=80=99s reputation. What to do from here? Our main focus is on making berlin.guixsd.org the primary build farm of the project. It has the advantage that one Guix dev has physical access to it (Ricardo); it=E2=80=99s also much more powerful than hydra.gnu.org an= d the associated build machines. Yet, there=E2=80=99s more work to do: berlin has just 1T of disk space. Ri= cardo started looking on growing it but was stuck on software issues IIRC. I think fixing this should be a priority, so I think we should help Ricardo fix the software issues as much as we can. That alone doesn=E2=80=99t fix the resilience issue: berlin.guixsd.org coul= d go down at some point for some time. To address that, a possibility that was discussed recently on guix-sysadmin is use bayfront.guixsd.org has a separate build farm and/or mirror of berlin. On top of that we could have a service like httpredir.debian.org, or maybe even a CDN where we=E2=80=99d replicate substitutes, or torrents (loo= king at you, Julien ;-)). At this point, all these options are still on the table, and anyone with expertise in these areas is very much welcome! > given the prolonged issue, please also consider writing an *official* > blog post explaining the current situation and steps adopted to prevent > similar issues in the future We set up the info-guix mailing list with that in mind (but too late for this incident). Posting blog posts is also a good idea; we should have done that, with instructions on how to switch to berlin.guixsd.org. > 1. is there a method to "replicate the whole store of an official server > (e.g. hydra.gnu.org once healed)" so we can just "guix publish" a > *complete* mirror? In this case a ready to use official > mirror-config.scm could be useful mirror.hydra.gnu.org is a simple nginx proxy to hydra.gnu.org. You can find its config here: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/m= irror.conf In the past a few people set up their own mirrors using a similar configuration. > 2. is there an official mirrors directory users can look at when needed? No. > 3. is there a plan to build a service similar to > http://httpredir.debian.org/? (I looked on the web but did not find any > reference to such plan) Like I wrote, there=E2=80=99s no concrete plan at this point, which means i= t=E2=80=99s an opportunity for you and anyone else to chime in and give a hand! Thanks, Ludo=E2=80=99.