From: Christopher Baines <mail@cbaines.net>
To: guix-devel@gnu.org
Subject: January (2023) update on bordeaux.guix.gnu.org
Date: Tue, 17 Jan 2023 10:00:13 +0000 [thread overview]
Message-ID: <87sfg92ukt.fsf@cbaines.net> (raw)
[-- Attachment #1: Type: text/plain, Size: 3341 bytes --]
Hey!
The last update was sent out in September [1], so 4 and a bit months
ago.
1: https://lists.gnu.org/archive/html/guix-devel/2022-09/msg00038.html
## Numbers
bordeaux.guix.gnu.org currently provides ~2 million nars, which take up
~8.3TiB to store.
Substitute availability is still reasonable I think. Additionally,
there's some substitutes for i586-gnu (the Hurd) now.
## Non master branch substitutes
One gradual change that's been happening over the last few months is
that there's more builds/substitutes on bordeaux.guix.gnu.org for non
master branch things (so for patches and branches like staging).
## Mirrors
No progress has happened in terms of mirrors.
There's still the US mirror I host [2] plus bishan effectively acts as a
mirror [3].
2: https://bordeaux-us-east-mirror.cbaines.net/
3: https://bishan.guix.gnu.org/
I think there's still the opportunity to sort out the DNS and
certificates so that bordeaux.guix.gnu.org refers to multiple servers,
increasing reliability and hopefully performance as well.
## Serving fixed output files by hash
This has progressed recently as Guile Fibers now supports streaming
responses without storing the whole response in memory [4].
4: https://github.com/wingo/fibers/pull/63
I've enabled this in the bayfront NGinx config, so you can now make
requests like [5]
5: https://bordeaux.guix.gnu.org/file/texlive-20210325-texmf.tar.xz/sha256/070gczcm1h9rx29w2f02xd3nhd84c4k28nfmm8qgp69yq8vd84pz
This seems to work for me on a fast connection, but the requests drops
part way through on a slower connection. I'm not quite sure why yet but
it's probably some NGinx configuration issue. You can also try making
these requests directly against bishan.
## Next steps
There's still a whole lot to improve on in multiple areas.
While bordeaux.guix.gnu.org is now a default substitute server, and
there's been a release since this change was made, the substitute
servers list is ordered, so the obvious question is what needs to change
so that bordeaux.guix.gnu.org is the substitute server that users should
try first? In my mind, the main blocking issue is the current lack of
zstd substitutes, but that's something that's being addressed now.
There's also the content addressed mirror issue (discussed above as
serving fixed output files by hash). I think a little more testing is
required, but then it should be possible to add bordeaux.guix.gnu.org to
the list of content addressed mirrors.
In terms of resource requirements, especially now that builds for
patches and non-master branches are happening, there's a use for more
build hardware. The more pressing issue though is storage
space. Currently the two machines responsible for storing all the nars
(bishan and hatysa) each have around 10TiB of storage for this purpose,
which will probably be all used up by July this year. It should be
possible to extend the storage in hatysa, but bishan will probably need
replacing.
There's also improvements to and around the build coordinator,
particularly in the area of observability (seeing what builds are
happening and queued).
If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.
Let me know if you have any comments or questions!
Thanks,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]
reply other threads:[~2023-01-17 12:52 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sfg92ukt.fsf@cbaines.net \
--to=mail@cbaines.net \
--cc=guix-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.