unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Christopher Baines <mail@cbaines.net>
To: guix-devel@gnu.org
Subject: Handling nars/narinfos at scale, some ideas...
Date: Sat, 06 Feb 2021 22:02:50 +0000	[thread overview]
Message-ID: <87h7moyhv9.fsf@cbaines.net> (raw)

[-- Attachment #1: Type: text/plain, Size: 3292 bytes --]

Hey,

This is something I've been thinking about for a while, I also ended up
setting out some of these ideas on IRC a few days ago [1].

1: https://logs.guix.gnu.org/guix/2021-02-01.log#222156

While I think the approach taken in the Guix Build Coordinator for
serving substitutes for built outputs, generating the nar+narinfo files
upfront and storing them is the way to go when you're trying to serve
lots of substitutes, there's some areas for improvement on this
approach.

When serving from a store, you can use guix gc to remove items, and gc
roots to protect the items you want to keep. I'm not aware of similar
tooling when you just have a bunch of nars+narinfo files. This means you
either just delete files based on when you generated them, or don't
delete anything and potentially have an ever growing collection of nars.

When serving the substitutes, there's advantages to having low latency
access to the narinfo files, since they're very small. If you're trying
to serve the whole world. one way of doing this would be to store the
narinfos on several machines around the world, and direct requests for
them to a machine that's close in terms of network latency. The relevant
bit here is storing the narinfos on multiple machines, and keeping them
in sync. This also may improve resilience if through this there's not a
single point of failure with the one machine storing the narinfo files.

I think these needs: doing garbage collection across narinfo data and
storing narinfo data on multiple machines can be met with one
approach. I'm also thinking this might be a good place to try and store
analytics about the fetching of nars+narinfos.

This new tool/service would be a standalone thing, but I'm very much
thinking about deploying it alongside a Guix Build Coordinator
instance. Again, while the Guix Build Coordinator can help with serving
substitutes, that approach doesn't stretch yet to doing the things
above.

Note that while this does similar things to guix publish, it's not
designed to replace it. This approach is probably only worth it if you
want to store/serve nars+narinfos on from more than one machine.

I also don't see this as something to do instead of things like IPFS
distribution for substitutes, but I do think it would be good to have a
way of providing substitutes over HTTP which is reliable and works at a
global scale.

The architecture I'm currently thinking about for this is to store the
narinfo data in a PostgreSQL database. This will allow for storing the
equivalent of "roots" in the graph, using SQL queries to traverse the
graph to find the "garbage" and using logical replication to sync the
data between multiple machines. Additionally, I'm thinking that the
narinfo's can be served directly from the database, and maybe analytics
data (counts of narinfo requests) can be saved back to the database.

My testbed for this will probably be guix.cbaines.net, so I'll probably
need to look at doing something to direct requests to different servers
(maybe GeoIP with knot) and getting Letsencrypt to work across multiple
servers, but that can come later.

Anyway, I haven't actually implemented this yet, but maybe after sending
this email I'll be one step closer...

Please let me know if you have any thoughts or questions!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

             reply	other threads:[~2021-02-06 22:03 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-06 22:02 Christopher Baines [this message]
2021-02-10 21:04 ` Handling nars/narinfos at scale, some ideas Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87h7moyhv9.fsf@cbaines.net \
    --to=mail@cbaines.net \
    --cc=guix-devel@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).