March update on bordeaux.guix.gnu.org

March update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

Hey!

I'm think the last update I sent out was back in April 2023 [1], but
it's coming up to 3 years since bordeaux was added as a default
substitute server [2]. It's scary how much time has passed!

1: https://lists.gnu.org/archive/html/guix-devel/2023-04/msg00319.html
2: https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

I've finally got around to starting to address the problems with
disappearing nars discussed in [3]. The nar-herder now schedules nars
which it's generated for removal and the time for removal is based on
the TTL in use.

3: https://issues.guix.gnu.org/63634

Related to this, I've added options to the nar-herder to help change the
TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
minutes (from 180 days) [4]. This will at least mean that in the future,
the nar-herder on bordeaux will be able to delete zstd compressed nars
that it's generated more quickly.

4: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=1fc5a1af488cb8838c2f196e07a43f37f2fbf509

I'm really unsure about the need/usefulness of narinfo caching in
general, the cost of storing all these narinfos locally is quite high I
think and I don't really know why it's done.

The mishandling of these zstd nars available from bordeaux was the last
thing on my mind blocking proposing to switch the default substitute
server ordering, so I've now gone ahead and done that in [5].

5: https://issues.guix.gnu.org/70028

Obviously the differences to having ci.guix.gnu.org first in the list of
substitute URLs is subtle, but I'm hoping this will help with substitute
speed issues (as discussed in [6]) plus increase the impact of mirroring
work for the bordeaux substitutes in the future.

6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html

Apart from that, the main thing on my mind for the next year regarding
bordeaux substitutes specifically is storage space. We're at 90%
capacity on hatysa (one of the two machines storing all the nars) so
this will need looking at shortly. I'd also like to finally get removing
nars that don't relate to the guix master branch happening, as that
should free up a little bit of space at least.

Let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: March update on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> I've finally got around to starting to address the problems with
> disappearing nars discussed in [3]. The nar-herder now schedules nars
> which it's generated for removal and the time for removal is based on
> the TTL in use.
>
> 3: https://issues.guix.gnu.org/63634

Neat!  Yeah it’s important to honor the TTL that’s advertised in
narinfos.

> Related to this, I've added options to the nar-herder to help change the
> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
> minutes (from 180 days) [4]. This will at least mean that in the future,
> the nar-herder on bordeaux will be able to delete zstd compressed nars
> that it's generated more quickly.

It’s not 10mn right now:

--8<---------------cut here---------------start------------->8---
$ wget --debug -qO/dev/null https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 |grep Cache
Cache-Control: max-age=15502374
--8<---------------cut here---------------end--------------->8---

Or maybe that’s just for newly created nars?

But then again, that’s the advertised TTL; the real TTL is still
infinite, right?

> I'm really unsure about the need/usefulness of narinfo caching in
> general, the cost of storing all these narinfos locally is quite high I
> think and I don't really know why it's done.

It’s a cache.  It’s useful to have this cache because in “typical” Guix
usage you’re likely to ask repeatedly for the same substitutes.

Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
more reasonable by putting a higher bound on narinfo retention.  On my
laptop, I have:

--8<---------------cut here---------------start------------->8---
$ ls -lrt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ |wc -l
11549
$ du -h /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ 
50M     /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
--8<---------------cut here---------------end--------------->8---

Maybe that’s still excessive and we could further reduce the maximum
caching time.

> The mishandling of these zstd nars available from bordeaux was the last
> thing on my mind blocking proposing to switch the default substitute
> server ordering, so I've now gone ahead and done that in [5].
>
> 5: https://issues.guix.gnu.org/70028
>
> Obviously the differences to having ci.guix.gnu.org first in the list of
> substitute URLs is subtle, but I'm hoping this will help with substitute
> speed issues (as discussed in [6]) plus increase the impact of mirroring
> work for the bordeaux substitutes in the future.

Good!

> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html

BTW, should we document this mirror somewhere (and also ensure that Guix
Foundation pays the bills), or do you view it more as an experiment for
now?

> Apart from that, the main thing on my mind for the next year regarding
> bordeaux substitutes specifically is storage space. We're at 90%
> capacity on hatysa (one of the two machines storing all the nars) so
> this will need looking at shortly. I'd also like to finally get removing
> nars that don't relate to the guix master branch happening, as that
> should free up a little bit of space at least.

Nice (the difficulty, I guess, is that some substitutes that we not
initially for ‘master’ eventually get used on ‘master’).

On this issue, I think we should learn from fellow NixOS hackers.  They
kept substitutes for ~20 years I think and are now in a difficult
position because they cannot afford, financially, to keep that.  So one
of the solutions envisioned was to figure out which of these millions of
substitutes were “important” (for instance, source code), which turns
out to be very hard if you don’t have that info already at hand.

  https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/36493

Do you think the Data Service or another source of info would let us
make such decisions?

If we take it to the extreme, we could have a sophisticated retention
policy like: drop all fixed-output derivations known to be available
from disarchive.guix + SWH, drop substitutes for packages that have less
than 100 dependents, etc.

Thanks for the update!

Ludo’.

Re: March update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 6583 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Related to this, I've added options to the nar-herder to help change the
>> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
>> minutes (from 180 days) [4]. This will at least mean that in the future,
>> the nar-herder on bordeaux will be able to delete zstd compressed nars
>> that it's generated more quickly.
>
> It’s not 10mn right now:
>
> $ wget --debug -qO/dev/null https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 |grep Cache
> Cache-Control: max-age=15502374
>
>
> Or maybe that’s just for newly created nars?

The max-age of that narinfo is currently based on the scheduled removal
of the zstd compressed nar, which is going to happen quite far in the
future.

I did think of a number of ways to approach this, and I'm not sure I've
settled on the right one yet. Maybe the TTL should be capped at 600, and
then drop to 0 as the time to remove the zstd nar approaches?

This narinfo for example currently has a max-age of 600:

  https://bordeaux.guix.gnu.org/ganfjbgy75r31bwzgddpnpswwjrrffvj.narinfo

> But then again, that’s the advertised TTL; the real TTL is still
> infinite, right?

As you probably know, the situation is more complex.

The problems caused when the nar-herder started removing zstd compressed
nars shows the difference between retention of the nar in some form, and
whether a cached narinfo response can be considered fresh or stale.

Users might also not notice the availability of zstd nars if they cache
responses forever, since currently there will be a lag between the nar
becoming available, and a zstd compression being created (although we
could generate zstd compressed nars for everything).

As described below, I also do want to start removing some nars, and that
requires not having an infinite TTL.

>> I'm really unsure about the need/usefulness of narinfo caching in
>> general, the cost of storing all these narinfos locally is quite high I
>> think and I don't really know why it's done.
>
> It’s a cache.  It’s useful to have this cache because in “typical” Guix
> usage you’re likely to ask repeatedly for the same substitutes.
>
> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
> more reasonable by putting a higher bound on narinfo retention.  On my
> laptop, I have:
>
> $ ls -lrt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ |wc -l
> 11549
> $ du -h /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ 
> 50M     /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>
> Maybe that’s still excessive and we could further reduce the maximum
> caching time.

Having played around with this a bit (e.g. hacking guix weather not to
cache), I'm a bit sceptical. Given maintaining the cache takes time that
could be spent doing network I/O, and does potentially slow disk I/O, I
think it would be interesting to try and work out in what situations the
cache speeds things up overall, and in what situations it slows things
down overall..

>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html
>
> BTW, should we document this mirror somewhere (and also ensure that Guix
> Foundation pays the bills), or do you view it more as an experiment for
> now?

If the project does want to provide mirrors, I think that would be
great. From this experiment, I think we have some evidence that there
are people using Guix outside of Europe, and in some cases they struggle
with the European based infrastructure. It also seems like these mirrors
do help, and the monetary cost isn't too high in my view.

I think we should probably wait until the project starts managing them
before documenting/advertising them more widely though.

>> Apart from that, the main thing on my mind for the next year regarding
>> bordeaux substitutes specifically is storage space. We're at 90%
>> capacity on hatysa (one of the two machines storing all the nars) so
>> this will need looking at shortly. I'd also like to finally get removing
>> nars that don't relate to the guix master branch happening, as that
>> should free up a little bit of space at least.
>
> Nice (the difficulty, I guess, is that some substitutes that we not
> initially for ‘master’ eventually get used on ‘master’).

Yep, my plan is to wait some long amount of time (say 6 months) before
scheduling things for removal, to check that they haven't started being
used by the master branch in this time.

We could also add other criteria as well, like tracking which nars are
generated by fixed output derivations and never removing them.

> On this issue, I think we should learn from fellow NixOS hackers.  They
> kept substitutes for ~20 years I think and are now in a difficult
> position because they cannot afford, financially, to keep that.  So one
> of the solutions envisioned was to figure out which of these millions of
> substitutes were “important” (for instance, source code), which turns
> out to be very hard if you don’t have that info already at hand.
>
>   https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/36493
>
> Do you think the Data Service or another source of info would let us
> make such decisions?
>
> If we take it to the extreme, we could have a sophisticated retention
> policy like: drop all fixed-output derivations known to be available
> from disarchive.guix + SWH, drop substitutes for packages that have less
> than 100 dependents, etc.

I think the Data Service (specifically data.guix.gnu.org) might be
really helpful here, as it speeds up being able to work out what a nar
or derivation relates to.

Additionally, the nar-herder can tag narinfos (associate key=value pairs
with them), and that's intended to help you manage the nars. So we
should probably start tagging the nars with potentially useful
information now, so that we can use that data later to make desicions.

We're storing 17.5TiB of nars currently, and this increases linearly, so
it would be good to understand how this can be broken down. The
nar-herder should help here as well, as providing you can download the
11G database, that should contain all the information you need to start
digging in to this.

  wget https://bordeaux.guix.gnu.org/latest-database-dump -O bordeaux.db

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: March update on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> The max-age of that narinfo is currently based on the scheduled removal
> of the zstd compressed nar, which is going to happen quite far in the
> future.
>
> I did think of a number of ways to approach this, and I'm not sure I've
> settled on the right one yet. Maybe the TTL should be capped at 600, and
> then drop to 0 as the time to remove the zstd nar approaches?

Not sure what you mean by removal.  To me, it’s the other way around:
when the server advertises a TTL, it must guarantee that the associated
nars remain available until that TTL has expired.  (It can also keep it
longer.)

The way ‘guix publish’ honors that commitment is by keeping track of the
last time each narinfo was published and protecting the narinfo and nars
from deletion until the TTL has expired.

>> But then again, that’s the advertised TTL; the real TTL is still
>> infinite, right?
>
> As you probably know, the situation is more complex.
>
> The problems caused when the nar-herder started removing zstd compressed
> nars shows the difference between retention of the nar in some form, and
> whether a cached narinfo response can be considered fresh or stale.
>
> Users might also not notice the availability of zstd nars if they cache
> responses forever, since currently there will be a lag between the nar
> becoming available, and a zstd compression being created (although we
> could generate zstd compressed nars for everything).

I think we should arrange to never advertise nars that are not actually
available, if that’s what you meant.

>> It’s a cache.  It’s useful to have this cache because in “typical” Guix
>> usage you’re likely to ask repeatedly for the same substitutes.
>>
>> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
>> more reasonable by putting a higher bound on narinfo retention.  On my
>> laptop, I have:
>>
>> $ ls -lrt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ |wc -l
>> 11549
>> $ du -h /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/ 
>> 50M     /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>>
>> Maybe that’s still excessive and we could further reduce the maximum
>> caching time.
>
> Having played around with this a bit (e.g. hacking guix weather not to
> cache), I'm a bit sceptical. Given maintaining the cache takes time that
> could be spent doing network I/O, and does potentially slow disk I/O, I
> think it would be interesting to try and work out in what situations the
> cache speeds things up overall, and in what situations it slows things
> down overall..

Yes, we can surely fine-tune that to make sure it remains beneficial.

>>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html
>>
>> BTW, should we document this mirror somewhere (and also ensure that Guix
>> Foundation pays the bills), or do you view it more as an experiment for
>> now?
>
> If the project does want to provide mirrors, I think that would be
> great. From this experiment, I think we have some evidence that there
> are people using Guix outside of Europe, and in some cases they struggle
> with the European based infrastructure. It also seems like these mirrors
> do help, and the monetary cost isn't too high in my view.
>
> I think we should probably wait until the project starts managing them
> before documenting/advertising them more widely though.

*We* are “the project”.  :-)

By that I mean that we should discuss it with people at the Guix
Foundation and with the broader community, but it seems pretty clear to
me that there’s interest in having mirrors up and running, especially
outside Europe.  Then of course we need to be able to scale it according
to available funds, but at least the goal itself is clear.

[...]

>> Do you think the Data Service or another source of info would let us
>> make such decisions?
>>
>> If we take it to the extreme, we could have a sophisticated retention
>> policy like: drop all fixed-output derivations known to be available
>> from disarchive.guix + SWH, drop substitutes for packages that have less
>> than 100 dependents, etc.
>
> I think the Data Service (specifically data.guix.gnu.org) might be
> really helpful here, as it speeds up being able to work out what a nar
> or derivation relates to.
>
> Additionally, the nar-herder can tag narinfos (associate key=value pairs
> with them), and that's intended to help you manage the nars. So we
> should probably start tagging the nars with potentially useful
> information now, so that we can use that data later to make desicions.

Really good.

> We're storing 17.5TiB of nars currently, and this increases linearly, so
> it would be good to understand how this can be broken down. The
> nar-herder should help here as well, as providing you can download the
> 11G database, that should contain all the information you need to start
> digging in to this.
>
>   wget https://bordeaux.guix.gnu.org/latest-database-dump -O bordeaux.db

Neat, thanks!

Ludo’.

March update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4620 bytes --]

Hey!

The last update was sent out back in January [1], while things are
stable there has been some activity in the last couple of months.

1: https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00199.html


## Numbers

bordeaux.guix.gnu.org currently provides ~2.2 million nars, which take
up ~9.2TiB to store.

Substitute availability is still reasonable, although recent
availability for i686-linux has been lower than it has been previously.


## Mirrors

Still no progress has happened in terms of mirrors. This would still be
nice to get setup, but it's probably not the priority in terms of
administration and infrastructure.


## Serving fixed output files by hash

bordeaux.guix.gnu.org is now in the list of content addressed mirrors
[2].

2: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=7d0ebe040d80adcf143656e754a82b569243568c

In addition to the above change, some fixed output derivations use
download-nar rather than content addressed mirrors, so I've now updated
that to also fetch from bordeaux.guix.gnu.org [3].

3: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b59f89cf18fbad9ee95521c4cadc6642c580feb8


## Referential integrity

This is usually something I think about in the context of databases, but
nars/narinfos have references, and I've been working to ensure that for
any nar provided by bordeaux.guix.gnu.org, you're also able to fetch any
nar it references.

I noticed something was lacking here a while ago when I went to
substitute an output for guix pull, but guix wasn't able to since some
referenced outputs weren't available. These missing outputs turned out
to be files that guix inserts in to the store to use as sources in the
derivations computed by guix pull. The build coordinator now knows about
this and will publish these as nars by default [4]. This was never an
issue when actually using guix pull, since it would compute the
derivations and create the missing outputs locally, so they aren't
usually substituted. To backfill the missing items, I wrote a script to
fetch the nars from data.guix.gnu.org, since it has them as part of
providing substitutes for the derivations.

4: https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/commit/?id=751910162c54d0bf85fa5a21c25ad229cb12828d

There's still a bit more work to do in terms of ensuring outputs aren't
missing when nars are inserted and removed, but this is nearly there.


## Offering zstd compressed nars

bordeaux.guix.gnu.org uses lzip to compress the nars, which is good
choice in terms of minimising the storage requirements and size of
downloads. It's been known for a while though that this might not suit
all users, as those with fast network connections may benefit from nars
that can be decompressed much quicker even if there's more bytes to
download.

5: https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

To enable providing zstd compressed nars, without having to store every
nar as an lzip compressed file and zstd compressed file, the nar-herder
now supports generating cached nars with different compressions.

The nar-herder running on bayfront (which serves bordeaux.guix.gnu.org)
is now configured to generate and cache zstd nars [6]. Currently there
are ~25,000 cached nars taking up about 86GiB of space.

6: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=c4778aa2689e0e73e77cadb71a04dc2dfee4ea27


## Next steps

Last time I said that I thought the main blocking issue in making
bordeaux.guix.gnu.org the default substitute server to try first was the
lack of zstd nars. Some nars are now available with zstd compression, so
there's progress been made on this at least. I think a good argument can
be made for trying bordeaux.guix.gnu.org being better for users than
trying ci.guix.gnu.org first, but it's not clear cut.

On the issue of expanding the storage for nars, I've submitted the
request to purchase an additional hard drive for hatysa to expand the
storage there. There is still a need to come up with a plan to replace
bishan when it runs out of space (ideally before!).

Now that bordeaux.guix.gnu.org is building things for the master branch,
plus qa.guix.gnu.org is submitting lots of builds for patches and
branches, there's more going on at varying priorities. This is really
good, but there's a greater need now to expose what's being built and
what the backlog looks like.

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: March update on bordeaux.guix.gnu.org

[Maxim Cournoyer]

Hi Chris,

Thanks for writing the update; it's come a long way :-).

-- 
Thanks,
Maxim

Re: March update on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> ## Serving fixed output files by hash
>
> bordeaux.guix.gnu.org is now in the list of content addressed mirrors
> [2].

Yay!

> ## Referential integrity
>
> This is usually something I think about in the context of databases, but
> nars/narinfos have references, and I've been working to ensure that for
> any nar provided by bordeaux.guix.gnu.org, you're also able to fetch any
> nar it references.

Great that you looked into it.  That’s something we did not explicitly
address until now, on the grounds that ‘guix substitute’ would typically
look up both a store item and its dependencies, which in practice means
that the TTL of a store item and that of its dependencies would likely
be synchronized.  But clearly, it’s best to ensure that by construction.

> ## Offering zstd compressed nars

Yay!

> ## Next steps
>
> Last time I said that I thought the main blocking issue in making
> bordeaux.guix.gnu.org the default substitute server to try first was the
> lack of zstd nars. Some nars are now available with zstd compression, so
> there's progress been made on this at least. I think a good argument can
> be made for trying bordeaux.guix.gnu.org being better for users than
> trying ci.guix.gnu.org first, but it's not clear cut.

I think keeping both is ideal, notably from a fault-tolerance viewpoint.

> On the issue of expanding the storage for nars, I've submitted the
> request to purchase an additional hard drive for hatysa to expand the
> storage there. There is still a need to come up with a plan to replace
> bishan when it runs out of space (ideally before!).

Definitely; this should be high-priority for the project.

Thanks!

Ludo’.