all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Update on bordeaux.guix.gnu.org
@ 2021-11-24  8:52 Christopher Baines
  2021-11-28 17:26 ` Ludovic Courtès
  0 siblings, 1 reply; 13+ messages in thread
From: Christopher Baines @ 2021-11-24  8:52 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 3436 bytes --]

Hey!

It's been 3 months since I sent the last update [1]. This email was
meant to go out on Friday, but it seems opensmtpd was broken on my
machine, so it got stuck.

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00075.html

First, some good things:

I've been doing some performance tuning, submitting builds is now more
parallelised, a source of slowness when fetching builds has been
addressed, and one of the long queries involved in allocating builds has
been removed, which also improved handling of the WAL (Sqlite write
ahead log).

There's also a few new features. Agents can be deactivated which means
they won't get any builds allocated. The coordinator now checks the
hashes of outputs which are submitted, a safeguard which I added because
the coordinator now also supports resuming the uploads of outputs. This
is particularly important when trying to upload large (> 1GiB) outputs
over slow connections.

I also added a new x86_64 build machine. It's a 4 core Intel NUC that I
had sitting around, but I cleaned it up and got it building things. This
was particularly useful as I was able to use it to retry building
guile@3.0.7, which is extremely hard to build [2]. This was blocking
building the channel instance derivations for x86_64-linux.

2: https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-guile-3.0.7.drv

On the related subject of data.guix.gnu.org (which is the source of
derivations for bordeaux.guix.gnu.org, as well as a recipient of build
information), there have been a couple of changes. There was some web
crawler activity that was slowing data.guix.gnu.org down significantly,
NGinx now has some rate limiting configuration to prevent crawlers
abusing the service. The other change is that substitutes for the latest
processed revision of master will be queried on a regular basis, so this
page [3] should be roughly up to date, including for ci.guix.gnu.org.

3: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability

Now for some not so good things:

Submitting builds wasn't working quite right for around a month, one of
the changes I made to speed things up led to some builds being
missed. This is now fixed, and all the missed builds have been
submitted, but this was more than 50,000 builds. This, along with all
the channel instance derivation builds that can now proceed mean that
there's a very large backlog of x86 and ARM builds which will probably
take at least another week to clear. While this backlog exists,
substitute availability for x86_64-linux will be lower than usual.

Space is running out on bayfront, the machine that runs the coordinator,
stores all the nars and build logs, and serves the substitutes. I knew
this was probably going to be an issue, bayfront didn't have much space
to begin with, but I had hoped I'd be further forward in developing some
way to allow moving the nars around between multiple machines, to remove
the need to store all of them on bayfront. I have got a plan, there's
some ideas I mentioned back in February [4], but I haven't got around to
implementing anything yet. The disk space usage trend is pretty much
linear, so if things continue without any change, I think it will be
necessary to pause the agents within a month, to avoid filling up
bayfront entirely.

4: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread
* Update on bordeaux.guix.gnu.org
@ 2021-08-18 11:36 Christopher Baines
  2021-08-18 16:07 ` Vagrant Cascadian
  0 siblings, 1 reply; 13+ messages in thread
From: Christopher Baines @ 2021-08-18 11:36 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2562 bytes --]

Hey!

Around 2 months ago, bordeaux.guix.gnu.org came in to existence [1][2].

1: https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/
2: https://lists.gnu.org/archive/html/guix-devel/2021-06/msg00044.html

This took work I'd done on providing substitutes back in 2020 and
attempted to bring benefits from that to normal users of Guix.

Unfortunately, I don't really know if this has been much of a
success. While it should be possible to track requests for substitutes
to roughly see if anyone is making use of them, this is something I
haven't been doing yet.

In terms of the substitute availability stats, I think it's delivered
the expected benefits. I recently enabled armhf-linux builds, so now
substitute availability for the following 5 architectures should be
good:

 - x86_64-linux
 - i686-linux
 - aarch64-linux
 - armhf-linux
 - powerpc64le-linux

You can use guix weather to check the stats yourself, or look at [3] for
an overview (ignore the ci.guix.gnu.org numbers, as they're not
currently up to date).

3: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability

There's still some issues holding substitute availability back. The Guix
Build Coordinator still has issues building things that it can't garbage
collect. The majority of the issues though are actual problems, like
broken fixed output derivations, or generally broken packages.

The next steps in my mind remain roughly the same as they were 2 months
ago:

 - It would be good to have something to provide more visibility in to
   the Guix Build Coordinator as well as the submitting of the builds

 - Supporting performant mirroring would be great, and I have some ideas
   of how to go about this

 - I did previously have some success building things for the Hurd [4],
   and it would be great to try and replicate this on
   bordeaux.guix.gnu.org

 - data.guix.gnu.org performance in processing new revisions is a
   limiting factor, so improving this would be helpful

 - Having aggregate statistics on use of substitutes (splitting out
   machines in the build farm) would be good for assessing use and
   changes in use

 - More hardware would be good for build throughput and redundancy. For
   example, there's currently only two ARM build machines linked up, and
   I host both of them.

4: https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00074.html

If you're interested in getting involved, or have any comments or
questions, please just let me know!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-12-07 19:27 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-24  8:52 Update on bordeaux.guix.gnu.org Christopher Baines
2021-11-28 17:26 ` Ludovic Courtès
2021-11-28 19:54   ` Ricardo Wurmus
2021-12-01 17:42     ` Ludovic Courtès
2021-12-01 22:04       ` Ricardo Wurmus
2021-12-06 12:51         ` Upgrading storage " Ludovic Courtès
2021-12-07 19:26           ` Maxim Cournoyer
2021-12-03 10:17     ` Update " Christopher Baines
2021-12-03 11:18       ` Ricardo Wurmus
2021-12-03  9:39   ` Christopher Baines
  -- strict thread matches above, loose matches on Subject: below --
2021-08-18 11:36 Christopher Baines
2021-08-18 16:07 ` Vagrant Cascadian
2021-08-18 16:39   ` Christopher Baines

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.