unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* March update on bordeaux.guix.gnu.org
@ 2023-03-16 19:29 Christopher Baines
  2023-03-21  2:49 ` Maxim Cournoyer
  2023-03-22 14:27 ` Ludovic Courtès
  0 siblings, 2 replies; 4+ messages in thread
From: Christopher Baines @ 2023-03-16 19:29 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 4620 bytes --]

Hey!

The last update was sent out back in January [1], while things are
stable there has been some activity in the last couple of months.

1: https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00199.html


## Numbers

bordeaux.guix.gnu.org currently provides ~2.2 million nars, which take
up ~9.2TiB to store.

Substitute availability is still reasonable, although recent
availability for i686-linux has been lower than it has been previously.


## Mirrors

Still no progress has happened in terms of mirrors. This would still be
nice to get setup, but it's probably not the priority in terms of
administration and infrastructure.


## Serving fixed output files by hash

bordeaux.guix.gnu.org is now in the list of content addressed mirrors
[2].

2: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=7d0ebe040d80adcf143656e754a82b569243568c

In addition to the above change, some fixed output derivations use
download-nar rather than content addressed mirrors, so I've now updated
that to also fetch from bordeaux.guix.gnu.org [3].

3: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b59f89cf18fbad9ee95521c4cadc6642c580feb8


## Referential integrity

This is usually something I think about in the context of databases, but
nars/narinfos have references, and I've been working to ensure that for
any nar provided by bordeaux.guix.gnu.org, you're also able to fetch any
nar it references.

I noticed something was lacking here a while ago when I went to
substitute an output for guix pull, but guix wasn't able to since some
referenced outputs weren't available. These missing outputs turned out
to be files that guix inserts in to the store to use as sources in the
derivations computed by guix pull. The build coordinator now knows about
this and will publish these as nars by default [4]. This was never an
issue when actually using guix pull, since it would compute the
derivations and create the missing outputs locally, so they aren't
usually substituted. To backfill the missing items, I wrote a script to
fetch the nars from data.guix.gnu.org, since it has them as part of
providing substitutes for the derivations.

4: https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/commit/?id=751910162c54d0bf85fa5a21c25ad229cb12828d

There's still a bit more work to do in terms of ensuring outputs aren't
missing when nars are inserted and removed, but this is nearly there.


## Offering zstd compressed nars

bordeaux.guix.gnu.org uses lzip to compress the nars, which is good
choice in terms of minimising the storage requirements and size of
downloads. It's been known for a while though that this might not suit
all users, as those with fast network connections may benefit from nars
that can be decompressed much quicker even if there's more bytes to
download.

5: https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

To enable providing zstd compressed nars, without having to store every
nar as an lzip compressed file and zstd compressed file, the nar-herder
now supports generating cached nars with different compressions.

The nar-herder running on bayfront (which serves bordeaux.guix.gnu.org)
is now configured to generate and cache zstd nars [6]. Currently there
are ~25,000 cached nars taking up about 86GiB of space.

6: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=c4778aa2689e0e73e77cadb71a04dc2dfee4ea27


## Next steps

Last time I said that I thought the main blocking issue in making
bordeaux.guix.gnu.org the default substitute server to try first was the
lack of zstd nars. Some nars are now available with zstd compression, so
there's progress been made on this at least. I think a good argument can
be made for trying bordeaux.guix.gnu.org being better for users than
trying ci.guix.gnu.org first, but it's not clear cut.

On the issue of expanding the storage for nars, I've submitted the
request to purchase an additional hard drive for hatysa to expand the
storage there. There is still a need to come up with a plan to replace
bishan when it runs out of space (ideally before!).

Now that bordeaux.guix.gnu.org is building things for the master branch,
plus qa.guix.gnu.org is submitting lots of builds for patches and
branches, there's more going on at varying priorities. This is really
good, but there's a greater need now to expose what's being built and
what the backlog looks like.

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: March update on bordeaux.guix.gnu.org
  2023-03-16 19:29 March update on bordeaux.guix.gnu.org Christopher Baines
@ 2023-03-21  2:49 ` Maxim Cournoyer
  2023-03-22 14:27 ` Ludovic Courtès
  1 sibling, 0 replies; 4+ messages in thread
From: Maxim Cournoyer @ 2023-03-21  2:49 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hi Chris,

Thanks for writing the update; it's come a long way :-).

-- 
Thanks,
Maxim


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: March update on bordeaux.guix.gnu.org
  2023-03-16 19:29 March update on bordeaux.guix.gnu.org Christopher Baines
  2023-03-21  2:49 ` Maxim Cournoyer
@ 2023-03-22 14:27 ` Ludovic Courtès
  1 sibling, 0 replies; 4+ messages in thread
From: Ludovic Courtès @ 2023-03-22 14:27 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> ## Serving fixed output files by hash
>
> bordeaux.guix.gnu.org is now in the list of content addressed mirrors
> [2].

Yay!

> ## Referential integrity
>
> This is usually something I think about in the context of databases, but
> nars/narinfos have references, and I've been working to ensure that for
> any nar provided by bordeaux.guix.gnu.org, you're also able to fetch any
> nar it references.

Great that you looked into it.  That’s something we did not explicitly
address until now, on the grounds that ‘guix substitute’ would typically
look up both a store item and its dependencies, which in practice means
that the TTL of a store item and that of its dependencies would likely
be synchronized.  But clearly, it’s best to ensure that by construction.

> ## Offering zstd compressed nars

Yay!

> ## Next steps
>
> Last time I said that I thought the main blocking issue in making
> bordeaux.guix.gnu.org the default substitute server to try first was the
> lack of zstd nars. Some nars are now available with zstd compression, so
> there's progress been made on this at least. I think a good argument can
> be made for trying bordeaux.guix.gnu.org being better for users than
> trying ci.guix.gnu.org first, but it's not clear cut.

I think keeping both is ideal, notably from a fault-tolerance viewpoint.

> On the issue of expanding the storage for nars, I've submitted the
> request to purchase an additional hard drive for hatysa to expand the
> storage there. There is still a need to come up with a plan to replace
> bishan when it runs out of space (ideally before!).

Definitely; this should be high-priority for the project.

Thanks!

Ludo’.


^ permalink raw reply	[flat|nested] 4+ messages in thread

* March update on bordeaux.guix.gnu.org
@ 2024-03-27 13:49 Christopher Baines
  0 siblings, 0 replies; 4+ messages in thread
From: Christopher Baines @ 2024-03-27 13:49 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

Hey!

I'm think the last update I sent out was back in April 2023 [1], but
it's coming up to 3 years since bordeaux was added as a default
substitute server [2]. It's scary how much time has passed!

1: https://lists.gnu.org/archive/html/guix-devel/2023-04/msg00319.html
2: https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

I've finally got around to starting to address the problems with
disappearing nars discussed in [3]. The nar-herder now schedules nars
which it's generated for removal and the time for removal is based on
the TTL in use.

3: https://issues.guix.gnu.org/63634

Related to this, I've added options to the nar-herder to help change the
TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
minutes (from 180 days) [4]. This will at least mean that in the future,
the nar-herder on bordeaux will be able to delete zstd compressed nars
that it's generated more quickly.

4: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=1fc5a1af488cb8838c2f196e07a43f37f2fbf509

I'm really unsure about the need/usefulness of narinfo caching in
general, the cost of storing all these narinfos locally is quite high I
think and I don't really know why it's done.

The mishandling of these zstd nars available from bordeaux was the last
thing on my mind blocking proposing to switch the default substitute
server ordering, so I've now gone ahead and done that in [5].

5: https://issues.guix.gnu.org/70028

Obviously the differences to having ci.guix.gnu.org first in the list of
substitute URLs is subtle, but I'm hoping this will help with substitute
speed issues (as discussed in [6]) plus increase the impact of mirroring
work for the bordeaux substitutes in the future.

6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html

Apart from that, the main thing on my mind for the next year regarding
bordeaux substitutes specifically is storage space. We're at 90%
capacity on hatysa (one of the two machines storing all the nars) so
this will need looking at shortly. I'd also like to finally get removing
nars that don't relate to the guix master branch happening, as that
should free up a little bit of space at least.

Let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2024-03-27 16:58 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-16 19:29 March update on bordeaux.guix.gnu.org Christopher Baines
2023-03-21  2:49 ` Maxim Cournoyer
2023-03-22 14:27 ` Ludovic Courtès
  -- strict thread matches above, loose matches on Subject: below --
2024-03-27 13:49 Christopher Baines

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).