Ludovic Courtès writes: >> However, due to the time spent not building things, the backlog is >> longer than usual, and the substitute availability (especially for >> x86_64-linux and i686-linux) is lower than usual. > > Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now. I've been so slow in replying to this email that the situation has actually improved now, substitute availability for x86_64-linux is around ~70% and still slowly rising. >> ** Space issues and the nar-herder >> >> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on >> bayfront was getting a little scarce. This week I started rolling out >> the nar-herder [2], a utility I've been thinking about for a while. This >> has enabled moving nars off of bayfront on to another machine which I've >> confusingly named lakefront. > > Woow, neat! > > Regarding nar-herder, I think it’d be nice to have a solution to > mirroring in Guix proper, developed similarly to other components, > because it could be a fairly central tool. > > ‘guix publish’ is probably not extensible enough to support that, but we > could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command. I think having something in Guix would be good too. I still view the nar-herder as a bit wierd in terms of the collection of concerns, mixing mirroring in with moving nars around between machines and I'm hoping to add some metrics functionality soon as well. >> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of >> storage across 2 hard drives. When a nar is requested from bayfront, it >> will check it's local storage, and serve it from there if it exists, >> otherwise it will forward the request to lakefront. There might be a >> slight drop in the speed you can download nars, but apart from that this >> change shouldn't be visible. > > Excellent, thanks for acting this promptly as problems crop up! > > I think we should make sure this is funded by the project and that > credentials are shared. As discussed before, I think it’s best to keep > things tidy in that respect, in the spirit of building and maintaining > this collectively. That would be good, I can start a thread on guix-sysadmin about this. >> Going forward, it would be good to have an additional full backup of the >> nars that bayfront can serve things from, to provide more >> redundancy. I'm hoping the nar-herder will also enable setting up >> geographically distributed mirrors, which will hopefully improve >> redundancy further, and maybe performance of fetching nars too. >> >> Once I've had a chance to neaten up the code a little, I'll get a Guix >> package and service written, plus I'll draft a Guix blog post about the >> nar-herder. > > Usually I’m the one asking for blog posts :-), but I’d really like us as > a project to collectively engage on the topic before we publicize this > specific approach. Sure, I also still need to post patches for the Guix service, which I'll try to do soon. >> That means there's the following currently running build agents (by >> architecture): >> >> - x86_64-linux + i686-linux (3 machines): >> - 4 core Intel NUC (giedi) >> - Max 16 cores for 1 concurrent build on bayfront >> - 32 cores on milano-guix-1 (slow storage though) >> - aarch64-linux + armhf-linux (2 machines) >> - 16 core HoneyComb LX2 (hatysa) >> - 4 core Overdrive (monokuma) >> - powerpc64le-linux (1 machine) >> - 64 core machine (polaris) > > This is looking pretty nice! I’m also all for streamlining machine > handling, both administratively (in maintenance.git) and financially. I think there was some discussion previously about guix-europe buying the HoneyComb board, which I can probably restart. It would be good to also sort out better hosting for the powerpc64le-linux machine. I'll try to put some time in to organising things in maintenance.git. >> Ironically, I think that the most under-resourced area is x86_64-linux + >> i686-linux. bayfront is restricted in what it can do since it also runs >> the coordinator, and things go badly if the machine gets >> overloaded. bayfront and milano-guix-1 both have hard drive storage, >> which also can slow them down when building things (especially >> milano-guix-1). >> >> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to >> keep up, it would be good to make a plan to add capacity. I think even a >> single high core count x86_64-linux machine with performant storage >> would make a big difference. > > Yes, we should do that, we still have funds for it. Great :) >> ** IPv6 not supported (yet) >> >> I was slow to notice, but bordeaux.guix.gnu.org isn't available over >> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want >> to address this, but I haven't worked out quite how to yet. > > Should be almost done now, as discussed on IRC today. \o/ Indeed, I think this is working now. Thanks, Chris