Mid-December update on bordeaux.guix.gnu.org

Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4517 bytes --]

Hey!

I sent out the last update 3 weeks ago [1].

1: https://lists.gnu.org/archive/html/guix-devel/2021-11/msg00154.html

In summary, the space issue I mentioned in the previous update has
effectively been addressed. All the paused agents are now unpaused and
builds are happening again.

However, due to the time spent not building things, the backlog is
longer than usual, and the substitute availability (especially for
x86_64-linux and i686-linux) is lower than usual.

I've also noticed that bordeaux.guix.gnu.org doesn't work over IPv6, and
I want to fix that soon.

** Space issues and the nar-herder

bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
bayfront was getting a little scarce. This week I started rolling out
the nar-herder [2], a utility I've been thinking about for a while. This
has enabled moving nars off of bayfront on to another machine which I've
confusingly named lakefront.

The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
storage across 2 hard drives. When a nar is requested from bayfront, it
will check it's local storage, and serve it from there if it exists,
otherwise it will forward the request to lakefront. There might be a
slight drop in the speed you can download nars, but apart from that this
change shouldn't be visible.

The nar-herder is now busy deleting nars on bayfront which are available
on lakefront. Once it's got through the backlog, I'll enable NGinx
caching for the nars on bayfront, which should help improve
performance. I've tested downloading the largest nars (~2GB) though, and
it seems to work fine.

In addition to lakefront, I've also added a 6TB hard drive to hatysa,
the HoneyComb LX2 machine that I host. Like lakefront, it's busy
downloading the nars from bayfront. This will act as a backup in case
lakefront is lost.

In general this is an important step in being more flexible where the
nars are stored. There's still a reliance on storing pretty much all the
nars on a single machine, but which machine has this role is more
flexible now. I think this architecture also makes it easier to break
the "all nars on a single machine" restriction in the future as well.

Going forward, it would be good to have an additional full backup of the
nars that bayfront can serve things from, to provide more
redundancy. I'm hoping the nar-herder will also enable setting up
geographically distributed mirrors, which will hopefully improve
redundancy further, and maybe performance of fetching nars too.

Once I've had a chance to neaten up the code a little, I'll get a Guix
package and service written, plus I'll draft a Guix blog post about the
nar-herder.

2: https://git.cbaines.net/guix/nar-herder/about/

** Build machines and backlog

Because of the 2 weeks of not building anything, there's a significant
backlog of builds to get through, and I'm not including the new builds
from the core-updates-frozen merge here.

As for build machines, milano-guix-1 came back online today, which is
great. I believe harbourfront is still unusable through (broken hard
drive).

That means there's the following currently running build agents (by
architecture):

 - x86_64-linux + i686-linux (3 machines):
   - 4 core Intel NUC (giedi)
   - Max 16 cores for 1 concurrent build on bayfront
   - 32 cores on milano-guix-1 (slow storage though)
 - aarch64-linux + armhf-linux (2 machines)
   - 16 core HoneyComb LX2 (hatysa)
   - 4 core Overdrive (monokuma)
 - powerpc64le-linux (1 machine)
   - 64 core machine (polaris)

Ironically, I think that the most under-resourced area is x86_64-linux +
i686-linux. bayfront is restricted in what it can do since it also runs
the coordinator, and things go badly if the machine gets
overloaded. bayfront and milano-guix-1 both have hard drive storage,
which also can slow them down when building things (especially
milano-guix-1).

If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
keep up, it would be good to make a plan to add capacity. I think even a
single high core count x86_64-linux machine with performant storage
would make a big difference.

** IPv6 not supported (yet)

I was slow to notice, but bordeaux.guix.gnu.org isn't available over
IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
to address this, but I haven't worked out quite how to yet.

Please let me know if you have any comments or questions!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Mid-December update on bordeaux.guix.gnu.org

[zimoun]

Hi Chris,

Thanks for the update.  And for the all work. :-)


On Wed, 15 Dec 2021 at 16:48, Christopher Baines <mail@cbaines.net> wrote:

> In summary, the space issue I mentioned in the previous update has
> effectively been addressed. All the paused agents are now unpaused and
> builds are happening again.

The timing had almost been perfect. ;-)


Well, as discussed on Sept., one concern I have is about “long-term
storage” – where long-term is not well-defined and storage either.

Do you think that Bordeaux could run

   <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>

?  Having a redundancy about all origins would avoid breakage.  For
instance, because Berlin was down yesterday morning, “guix pull” was
broken because the missing ’datefuge’ package – disappeared upstream.

Today, Guix is well covered for package using ’git-fetch’ but not for
all the others methods.  The situation is improving to have a complete
fallback using Software Heritage via Disarchive.  Not ready yet [1]. :-)

This redundancy about all sources appears to me vitally important.
Because if Berlin is totally lost for whatever reason, it is game over
for preserving Guix – well, recover from scattered with people’s store.

Other said, in term of capacity and priority, it appears to me worse if
0.01% source (or even just one) is missing than if some substitutes are
missing.  Because I can always locally burn some CPU, but I cannot
create source code. :-)

1: <https://ngyro.com/pog-reports/2021-12-06/>


> In addition to lakefront, I've also added a 6TB hard drive to hatysa,
> the HoneyComb LX2 machine that I host. Like lakefront, it's busy
> downloading the nars from bayfront. This will act as a backup in case
> lakefront is lost.

Cool!  Thanks.


> In general this is an important step in being more flexible where the
> nars are stored. There's still a reliance on storing pretty much all the
> nars on a single machine, but which machine has this role is more
> flexible now. I think this architecture also makes it easier to break
> the "all nars on a single machine" restriction in the future as well.

IIUC the design, if the proxy server is lost, then it is easy to replace
it.  Right?

I remember discussions about CDN [2,3,4,5,6].  I do not know if it
solves the issue but from my understanding, it will improve at least
performance delivery.  Well, it appears to me worth to give a try.


2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>


> Going forward, it would be good to have an additional full backup of the
> nars that bayfront can serve things from, to provide more
> redundancy. I'm hoping the nar-herder will also enable setting up
> geographically distributed mirrors, which will hopefully improve
> redundancy further, and maybe performance of fetching nars too.

To me, one first general question about backup coordination is to define
a window for time:

 - source: forever until the complete fallback to SWH is robust;
 - all the substitutes to run “guix time-machine --commit=<> -- help ”
   for any commit reachable by inferior: forever;
 - package substitute: rule something.


Thanks for taking care about redundancy and reliance of CI.


Cheers,
simon

Re: Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 5199 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> Thanks for the update.  And for the all work. :-)
>
>
> On Wed, 15 Dec 2021 at 16:48, Christopher Baines <mail@cbaines.net> wrote:
>
>> In summary, the space issue I mentioned in the previous update has
>> effectively been addressed. All the paused agents are now unpaused and
>> builds are happening again.
>
> The timing had almost been perfect. ;-)
>
>
> Well, as discussed on Sept., one concern I have is about “long-term
> storage” – where long-term is not well-defined and storage either.
>
> Do you think that Bordeaux could run
>
>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>

The Guix Build Coordinator just builds derivations. I haven't got it to
build a manifest before, but that's possible I guess.

I think it's unnecessary though, since I believe derivations for all
origins of all packages are already being built, that happens through
just asking the coordinator to build derivations for all packages, you
don't need to specify "source" derivations separately.

> ?  Having a redundancy about all origins would avoid breakage.  For
> instance, because Berlin was down yesterday morning, “guix pull” was
> broken because the missing ’datefuge’ package – disappeared upstream.

I would hope that bordeaux.guix.gnu.org has a substitute for that, could
you check the derivation against data.guix.gnu.org, and see if there's a
build? Use a URL like:

  https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv

There is one issue though, bordeaux.guix.gnu.org doesn't provide content
addressed files in the same way guix publish does. I hope to add that
through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
hopefully be added to the list of content addressed mirrors:

  https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368

That would mean that the bytes for a tar archive for example would be
available by the sha256 hash, not just as a nar. I'm not sure to what
extent this would help, but it's probably useful.

>> In general this is an important step in being more flexible where the
>> nars are stored. There's still a reliance on storing pretty much all the
>> nars on a single machine, but which machine has this role is more
>> flexible now. I think this architecture also makes it easier to break
>> the "all nars on a single machine" restriction in the future as well.
>
> IIUC the design, if the proxy server is lost, then it is easy to replace
> it.  Right?

I guess so, the nar-herder helps with managing the data at least which
makes setting up new or replacement servers easier.

> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
> solves the issue but from my understanding, it will improve at least
> performance delivery.  Well, it appears to me worth to give a try.
>
>
> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
> 3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
> 4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
> 6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>

Effectively this is moving towards building a CDN. With the nar-herder,
you could deploy reverse proxies (or edge nodes) in various
locations. Then the issue just becomes how to have users use the ones
that are best for them. This might require doing some fancy stuff with
GeoIP based DNS, and somehow sharing TLS certificates between the
machines, but I think it's quite feasible.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>
> To me, one first general question about backup coordination is to define
> a window for time:
>
>  - source: forever until the complete fallback to SWH is robust;
>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>    for any commit reachable by inferior: forever;
>  - package substitute: rule something.

The idea I've been working with so far is simply to store everything
that's built, forever.

Currently, that amounts to 561,043 nars totaling ~2.5TB's.

How feasible this is depends on a number of factors, but I don't have
any reason to think it's not feasible yet.

> Thanks for taking care about redundancy and reliance of CI.

There's not a relationship to continuous integration yet, although I am
hoping if the building and serving substitutes stuff stabilises,
bordeaux.guix.gnu.org might be able to play a part in testing patches
and branches (as discussed in [1]).

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00001.html

Thanks for all your comments and questions!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Mid-December update on bordeaux.guix.gnu.org

[zimoun]

Hi Chris,

On Thu, 16 Dec 2021 at 00:20, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> Do you think that Bordeaux could run
>>
>>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>
>
> The Guix Build Coordinator just builds derivations. I haven't got it to
> build a manifest before, but that's possible I guess.

I am not sure to understand since Cuirass also builds derivations and
the purpose of this source-manifest.scm is to let Cuirass ingest all
sources,

    <https://ci.guix.gnu.org/jobset/source>


> I think it's unnecessary though, since I believe derivations for all
> origins of all packages are already being built, that happens through
> just asking the coordinator to build derivations for all packages, you
> don't need to specify "source" derivations separately.

Your assumption is wrong, IMHO.  We have many failed examples and at the
end Ludo wrote this source-manifest.scm to be sure that all is ingested
for sure.


>> ?  Having a redundancy about all origins would avoid breakage.  For
>> instance, because Berlin was down yesterday morning, “guix pull” was
>> broken because the missing ’datefuge’ package – disappeared upstream.
>
> I would hope that bordeaux.guix.gnu.org has a substitute for that, could
> you check the derivation against data.guix.gnu.org, and see if there's a
> build? Use a URL like:
>
>   https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv
>
> There is one issue though, bordeaux.guix.gnu.org doesn't provide content
> addressed files in the same way guix publish does. I hope to add that
> through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
> hopefully be added to the list of content addressed mirrors:
>
>   https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368
>
> That would mean that the bytes for a tar archive for example would be
> available by the sha256 hash, not just as a nar. I'm not sure to what
> extent this would help, but it's probably useful.

Thanks for explaining the details.  From a pragmatical point of view as
end-user, “guix pull” must Just Work whatever the plumbing.

For instance, some of us spent energy to “evangelize” in scientific
communities how Guix is awesome.  Next morning, people give a look and
bang!  Because a tiny and short outage.  Obviously, “shit happens”(*)
and it is really really really sparse but too late the damage is there.

My feeling here is that both build farms work independently instead of
coordinate the usage of resources and exploit this strength to have two.


(*) quoting Forest Gump: :-)
  – whoa man, you just ran through a big pile of dog shit!
  – it happens
  – what? shit?
  – sometimes


>> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
>> solves the issue but from my understanding, it will improve at least
>> performance delivery.  Well, it appears to me worth to give a try.
>>
>> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
>> 3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
>> 4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
>> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
>> 6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>
>
> Effectively this is moving towards building a CDN. With the nar-herder,
> you could deploy reverse proxies (or edge nodes) in various
> locations. Then the issue just becomes how to have users use the ones
> that are best for them. This might require doing some fancy stuff with
> GeoIP based DNS, and somehow sharing TLS certificates between the
> machines, but I think it's quite feasible.

Considering the human resource vs the money resource, it appears to me
better to invest the human energy into things that do not exist and rely,
for now, on existing solutions; even if the project has to pay money.

For what my opinion is worth here.


>> To me, one first general question about backup coordination is to define
>> a window for time:
>>
>>  - source: forever until the complete fallback to SWH is robust;
>>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>>    for any commit reachable by inferior: forever;
>>  - package substitute: rule something.
>
> The idea I've been working with so far is simply to store everything
> that's built, forever.

For sure, anyone wants that at the end.  My point is raising the
priority of the intermediary steps.


> Currently, that amounts to 561,043 nars totaling ~2.5TB's.
>
> How feasible this is depends on a number of factors, but I don't have
> any reason to think it's not feasible yet.

That’s exactly my point!  It is not about feasibility – all is doable
with enough time and energy – but instead it is about controlling the
factors to “robustify“ what the project consider highly important – as
keep all sources or never break ‘guix pull’ – whatever the status of
infrastructure.


Again, thanks for all the work.  Becausse, in any case, for sure, the
situation is daily improving. :-)

Cheers,
simon

Re: Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 6151 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Thu, 16 Dec 2021 at 00:20, Christopher Baines <mail@cbaines.net> wrote:
>> zimoun <zimon.toutoune@gmail.com> writes:
>
>>> Do you think that Bordeaux could run
>>>
>>>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>
>>
>> The Guix Build Coordinator just builds derivations. I haven't got it to
>> build a manifest before, but that's possible I guess.
>
> I am not sure to understand since Cuirass also builds derivations and
> the purpose of this source-manifest.scm is to let Cuirass ingest all
> sources,
>
>     <https://ci.guix.gnu.org/jobset/source>
>
>> I think it's unnecessary though, since I believe derivations for all
>> origins of all packages are already being built, that happens through
>> just asking the coordinator to build derivations for all packages, you
>> don't need to specify "source" derivations separately.
>
> Your assumption is wrong, IMHO.  We have many failed examples and at the
> end Ludo wrote this source-manifest.scm to be sure that all is ingested
> for sure.

What assumption?

I believe the reason the source-manifest.scm thing is useful to use with
Cuirass is that it has something to do with Cuirass copying the outputs
from those builds back to where they're served from, and maybe
registering GC roots as well. I could be wrong though.

I'm saying that this additional thing is unnecessary when using the Guix
Build Coordinator to build packages, since at least with the
configuration for bordeaux.guix.gnu.org, it'll build the sources, store
and serve them without any extra effort.

>>> ?  Having a redundancy about all origins would avoid breakage.  For
>>> instance, because Berlin was down yesterday morning, “guix pull” was
>>> broken because the missing ’datefuge’ package – disappeared upstream.
>>
>> I would hope that bordeaux.guix.gnu.org has a substitute for that, could
>> you check the derivation against data.guix.gnu.org, and see if there's a
>> build? Use a URL like:
>>
>>   https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv
>>
>> There is one issue though, bordeaux.guix.gnu.org doesn't provide content
>> addressed files in the same way guix publish does. I hope to add that
>> through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
>> hopefully be added to the list of content addressed mirrors:
>>
>>   https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368
>>
>> That would mean that the bytes for a tar archive for example would be
>> available by the sha256 hash, not just as a nar. I'm not sure to what
>> extent this would help, but it's probably useful.
>
> Thanks for explaining the details.  From a pragmatical point of view as
> end-user, “guix pull” must Just Work whatever the plumbing.
>
> For instance, some of us spent energy to “evangelize” in scientific
> communities how Guix is awesome.  Next morning, people give a look and
> bang!  Because a tiny and short outage.  Obviously, “shit happens”(*)
> and it is really really really sparse but too late the damage is there.
>
> My feeling here is that both build farms work independently instead of
> coordinate the usage of resources and exploit this strength to have two.

I too want to coordinate, although I think that having two independent
build farms is actually good for reliability.

>>> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
>>> solves the issue but from my understanding, it will improve at least
>>> performance delivery.  Well, it appears to me worth to give a try.
>>>
>>> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
>>> 3: <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
>>> 4: <https://yhetil.org/guix/87tvju6145.fsf@gnu.org/>
>>> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
>>> 6: <https://yhetil.org/guix/87d0my1380.fsf@gmail.com/>
>>
>> Effectively this is moving towards building a CDN. With the nar-herder,
>> you could deploy reverse proxies (or edge nodes) in various
>> locations. Then the issue just becomes how to have users use the ones
>> that are best for them. This might require doing some fancy stuff with
>> GeoIP based DNS, and somehow sharing TLS certificates between the
>> machines, but I think it's quite feasible.
>
> Considering the human resource vs the money resource, it appears to me
> better to invest the human energy into things that do not exist and rely,
> for now, on existing solutions; even if the project has to pay money.
>
> For what my opinion is worth here.

I'm not against paying for some CDN, although I'd prefer not to pay, or
pay less.

>>> To me, one first general question about backup coordination is to define
>>> a window for time:
>>>
>>>  - source: forever until the complete fallback to SWH is robust;
>>>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>>>    for any commit reachable by inferior: forever;
>>>  - package substitute: rule something.
>>
>> The idea I've been working with so far is simply to store everything
>> that's built, forever.
>
> For sure, anyone wants that at the end.  My point is raising the
> priority of the intermediary steps.
>
>
>> Currently, that amounts to 561,043 nars totaling ~2.5TB's.
>>
>> How feasible this is depends on a number of factors, but I don't have
>> any reason to think it's not feasible yet.
>
> That’s exactly my point!  It is not about feasibility – all is doable
> with enough time and energy – but instead it is about controlling the
> factors to “robustify“ what the project consider highly important – as
> keep all sources or never break ‘guix pull’ – whatever the status of
> infrastructure.
>
>
> Again, thanks for all the work.  Becausse, in any case, for sure, the
> situation is daily improving. :-)
>
> Cheers,
> simon


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Mid-December update on bordeaux.guix.gnu.org

[Andreas Enge]

Am Thu, Dec 16, 2021 at 12:48:34PM +0000 schrieb Christopher Baines:
> I too want to coordinate, although I think that having two independent
> build farms is actually good for reliability.

Indeed, if we have enough build power, that would be ideal.
It would also help to examine reproducibility.

Andreas

Redundancy for source code and Disarchive

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> zimoun <zimon.toutoune@gmail.com> writes:

[...]

>> Do you think that Bordeaux could run
>>
>>    <https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>
>
> The Guix Build Coordinator just builds derivations. I haven't got it to
> build a manifest before, but that's possible I guess.

It’d be great.  If you’re on-line today, I’d like to see how we can
achieve that.  (This manifest is not related to how Cuirass works; it’s
a way to make sure we keep source around and/or notice when it is
unavailable.)

> There is one issue though, bordeaux.guix.gnu.org doesn't provide content
> addressed files in the same way guix publish does.

That’s a problem I’d like to fix.  We put a lot of effort in preserving
source code through different means; it’d be ridiculous to be unable to
retrieve tarballs that *are* available on the project’s machines.

Can we run ‘guix publish’ or bordeaux, possibly with additional nginx
routes?  Let’s discuss the details on #guix.

The last thing I’d like to discuss is Disarchive database replication.
The strategy I had imagined doesn’t work here:

  https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00167.html

One approach would be to periodically build
‘etc/disarchive-manifest.scm’ and to copy the result to persistent
storage, as is done on berlin:

  https://lists.gnu.org/archive/html/guix-devel/2021-10/msg00080.html

Another one would be to rsync the whole directory from berlin.

Thoughts?

Ludo’.

Re: Mid-December update on bordeaux.guix.gnu.org

[Andreas Enge]

Hello!

Am Wed, Dec 15, 2021 at 04:48:21PM +0000 schrieb Christopher Baines:
> As for build machines, milano-guix-1 came back online today, which is
> great. I believe harbourfront is still unusable through (broken hard
> drive).

We spent an hour in the server room with a colleague and tried to get the
machine running again. I put a spare SATA III SSD into a hard drive slot
that was labelled SAS; when booting, the SSD was recognised with its brand,
so I suppose this should work.

When booting on the graphical installer USB key, it complains about missing
firmware; the last messages are:
[17.44] udevd[186]: no sender credentials received, message ignored
[17.96] Error: Driver 'pcspkr' is already registered, aborting...
[41.30] 0000:03:00.0 Missing free firmware (non-Free firmware loading is disabled)
[41.30] bnx2: Can't load firmware file "/*(DEBLOBBED)*/"
(The last two lines are repeated twice.)

It would be helpful if it told us which firmware it could not load; I would
not consider this advertising for the non-free firmware, but useful
information on which device is blocked...

A web search reveals this:
bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"

This is apparently a Broadcom network card:
   https://packages.debian.org/stretch/firmware-bnx2

The strange thing is that when installing harbourfront for the first time,
we exchanged its network card so that it would work with the Guix free
drivers, and until we took out the hard disk, it was running GuixSD just
fine.

Do you have any idea what could be wrong?

Andreas

Re: Mid-December update on bordeaux.guix.gnu.org

[Andreas Enge]

Sorry, this message was intended for the guix-sysadmin mailing list,
I did not want to bother all of Guix ;-)

Andreas

Re: Mid-December update on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> In summary, the space issue I mentioned in the previous update has
> effectively been addressed. All the paused agents are now unpaused and
> builds are happening again.

Yay!

> However, due to the time spent not building things, the backlog is
> longer than usual, and the substitute availability (especially for
> x86_64-linux and i686-linux) is lower than usual.

Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now.

> ** Space issues and the nar-herder
>
> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
> bayfront was getting a little scarce. This week I started rolling out
> the nar-herder [2], a utility I've been thinking about for a while. This
> has enabled moving nars off of bayfront on to another machine which I've
> confusingly named lakefront.

Woow, neat!

Regarding nar-herder, I think it’d be nice to have a solution to
mirroring in Guix proper, developed similarly to other components,
because it could be a fairly central tool.

‘guix publish’ is probably not extensible enough to support that, but we
could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.

> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
> storage across 2 hard drives. When a nar is requested from bayfront, it
> will check it's local storage, and serve it from there if it exists,
> otherwise it will forward the request to lakefront. There might be a
> slight drop in the speed you can download nars, but apart from that this
> change shouldn't be visible.

Excellent, thanks for acting this promptly as problems crop up!

I think we should make sure this is funded by the project and that
credentials are shared.  As discussed before, I think it’s best to keep
things tidy in that respect, in the spirit of building and maintaining
this collectively.

> The nar-herder is now busy deleting nars on bayfront which are available
> on lakefront. Once it's got through the backlog, I'll enable NGinx
> caching for the nars on bayfront, which should help improve
> performance. I've tested downloading the largest nars (~2GB) though, and
> it seems to work fine.
>
> In addition to lakefront, I've also added a 6TB hard drive to hatysa,
> the HoneyComb LX2 machine that I host. Like lakefront, it's busy
> downloading the nars from bayfront. This will act as a backup in case
> lakefront is lost.
>
> In general this is an important step in being more flexible where the
> nars are stored. There's still a reliance on storing pretty much all the
> nars on a single machine, but which machine has this role is more
> flexible now. I think this architecture also makes it easier to break
> the "all nars on a single machine" restriction in the future as well.

That’s really cool.

> Going forward, it would be good to have an additional full backup of the
> nars that bayfront can serve things from, to provide more
> redundancy. I'm hoping the nar-herder will also enable setting up
> geographically distributed mirrors, which will hopefully improve
> redundancy further, and maybe performance of fetching nars too.
>
> Once I've had a chance to neaten up the code a little, I'll get a Guix
> package and service written, plus I'll draft a Guix blog post about the
> nar-herder.

Usually I’m the one asking for blog posts :-), but I’d really like us as
a project to collectively engage on the topic before we publicize this
specific approach.

[...]

> That means there's the following currently running build agents (by
> architecture):
>
>  - x86_64-linux + i686-linux (3 machines):
>    - 4 core Intel NUC (giedi)
>    - Max 16 cores for 1 concurrent build on bayfront
>    - 32 cores on milano-guix-1 (slow storage though)
>  - aarch64-linux + armhf-linux (2 machines)
>    - 16 core HoneyComb LX2 (hatysa)
>    - 4 core Overdrive (monokuma)
>  - powerpc64le-linux (1 machine)
>    - 64 core machine (polaris)

This is looking pretty nice!  I’m also all for streamlining machine
handling, both administratively (in maintenance.git) and financially.

> Ironically, I think that the most under-resourced area is x86_64-linux +
> i686-linux. bayfront is restricted in what it can do since it also runs
> the coordinator, and things go badly if the machine gets
> overloaded. bayfront and milano-guix-1 both have hard drive storage,
> which also can slow them down when building things (especially
> milano-guix-1).
>
> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
> keep up, it would be good to make a plan to add capacity. I think even a
> single high core count x86_64-linux machine with performant storage
> would make a big difference.

Yes, we should do that, we still have funds for it.

> ** IPv6 not supported (yet)
>
> I was slow to notice, but bordeaux.guix.gnu.org isn't available over
> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
> to address this, but I haven't worked out quite how to yet.

Should be almost done now, as discussed on IRC today.  \o/

Thanks!

Ludo’.

extend ’guix archive’?

[zimoun]

Hi,

On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:

> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.
>
> ‘guix publish’ is probably not extensible enough to support that, but we
> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.

Why not extend “guix archive”?

However, without all the details so my remark is totally naive, I miss
what “nar-herder” is doing that “guix archive”+rsync+“guix publish” is
not doing – other said I miss why another SQL database is required to
serve stuff from one place to another.  I have read README but I did not
get the point.


Cheers,
simon

Re: extend ’guix archive’?

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

On Mon, 20 Dec 2021, zimoun wrote:

> Hi,
>
> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Regarding nar-herder, I think it’d be nice to have a solution to
>> mirroring in Guix proper, developed similarly to other components,
>> because it could be a fairly central tool.
>>
>> ‘guix publish’ is probably not extensible enough to support that, but we
>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>
> Why not extend “guix archive”?

Hi all,

I'm quite interested in learning more and potentially trying out the 
nar-herder! Some thoughts that I'd like to add to the design space:

I think it would be great if one of the pastures to which we herd the nars 
would be a free and open source software mirror site. In my experience, 
these are usually some static web hosting in front of a large disk with a 
place to run scripts to sync the content. A database server may not be 
available. I'd like to support this use case because I think it is a great 
way to build bridges to the communities who run or gather around these 
mirrors.

I'd also like the ability fetch nars directly from the local-to-me mirror 
rather than having them be proxied through a far way server.

One of the things that I really like and find empowering about Guix is 
that the developer/system administration tools are as available, easy to 
use, and convenient as the every day tooling. To the extent possible, I 
think that we should strive to make our syncing/mirroring solution 
practical to run for local, small setups, and not require project-scale 
infrastructure or coordination between many programs that are not captured 
in a Guix service.

Best,
Jack

Re: extend ’guix archive’?

[zimoun]

Hi,

On Tue, 21 Dec 2021 at 00:50, Jack Hill <jackhill@jackhill.us> wrote:

> I think it would be great if one of the pastures to which we herd the nars 
> would be a free and open source software mirror site. In my experience, 
> these are usually some static web hosting in front of a large disk with a 
> place to run scripts to sync the content. A database server may not be 
> available. I'd like to support this use case because I think it is a great 
> way to build bridges to the communities who run or gather around these 
> mirrors.

Well, from my understanding, it looks like P2P distribution. ;-)


> I'd also like the ability fetch nars directly from the local-to-me mirror 
> rather than having them be proxied through a far way server.

Well, from my understanding, it looks like CDN. :-)


> One of the things that I really like and find empowering about Guix is 
> that the developer/system administration tools are as available, easy to 
> use, and convenient as the every day tooling. To the extent possible, I 
> think that we should strive to make our syncing/mirroring solution 
> practical to run for local, small setups, and not require project-scale 
> infrastructure or coordination between many programs that are not captured 
> in a Guix service.

At some point I agree, but I would like to mitigate.  The issue is
scaling up.  It is not easy and it requires a lot of effort, solve a
large range of bugs or design issues, etc.  Scaling up comes with a
cost.  To be concrete, one example: Cuirass migrated from SQLite to
PostgreSQL.  And at some point, the project should rely on existing
bullet-proof solutions for scaling up instead of reinventing the wheel
with all what it implies and focus on the project strengths.  And
bullet-proof solutions able to scale often mean not-so-easy small
setups.  Well, hard to find the right balance. :-)


Cheers,
simon

Re: extend ’guix archive’?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]


Jack Hill <jackhill@jackhill.us> writes:

> On Mon, 20 Dec 2021, zimoun wrote:
>
>> Hi,
>>
>> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Regarding nar-herder, I think it’d be nice to have a solution to
>>> mirroring in Guix proper, developed similarly to other components,
>>> because it could be a fairly central tool.
>>>
>>> ‘guix publish’ is probably not extensible enough to support that, but we
>>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>>
>> Why not extend “guix archive”?
>
> I'm quite interested in learning more and potentially trying out the
> nar-herder! Some thoughts that I'd like to add to the design space:

Apologies for the slow reply, it's great that you're interested!

> I think it would be great if one of the pastures to which we herd the
> nars would be a free and open source software mirror site. In my
> experience, these are usually some static web hosting in front of a
> large disk with a place to run scripts to sync the content. A database
> server may not be available. I'd like to support this use case because
> I think it is a great way to build bridges to the communities who run
> or gather around these mirrors.

I think there's a general discrepancy between how Guix works and how
mirror sites generally work, but there are probably ways of bridging
that gap. Maybe all the nars for the latest release could be mirrored
for example, and the nar-herder could probably help with that.

> I'd also like the ability fetch nars directly from the local-to-me
> mirror rather than having them be proxied through a far way server.

I think setting up some mirrors closer to the people that use Guix is
now easier to do with the help of the nar-herder.

> One of the things that I really like and find empowering about Guix is
> that the developer/system administration tools are as available, easy
> to use, and convenient as the every day tooling. To the extent
> possible, I think that we should strive to make our syncing/mirroring
> solution practical to run for local, small setups, and not require
> project-scale infrastructure or coordination between many programs
> that are not captured in a Guix service.

Indeed, and this is something to strive for in the design.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: extend ’guix archive’?

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Regarding nar-herder, I think it’d be nice to have a solution to
>> mirroring in Guix proper, developed similarly to other components,
>> because it could be a fairly central tool.
>>
>> ‘guix publish’ is probably not extensible enough to support that, but we
>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>
> Why not extend “guix archive”?

‘guix archive’ is a low-level tool that does very little right now; it’s
pretty far from ‘guix publish’ and related concerns.

Ludo’.

Re: extend ’guix archive’?

[zimoun]

Hi,

On Tue, 21 Dec 2021 at 10:39, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
>> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Regarding nar-herder, I think it’d be nice to have a solution to
>>> mirroring in Guix proper, developed similarly to other components,
>>> because it could be a fairly central tool.
>>>
>>> ‘guix publish’ is probably not extensible enough to support that, but we
>>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>>
>> Why not extend “guix archive”?
>
> ‘guix archive’ is a low-level tool that does very little right now; it’s
> pretty far from ‘guix publish’ and related concerns.

Hum, that’s why “extend”. ;-)  But if you think another subcommand is
better, why not.

The core of my previous message is: «I miss why another SQL database is
required to serve stuff from one place to another».  Other said, what
are the features for “guix mirror” (or guix sync or whatever bikeshed)?


Cheers,
simon

Re: extend ’guix archive’?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2093 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Regarding nar-herder, I think it’d be nice to have a solution to
>> mirroring in Guix proper, developed similarly to other components,
>> because it could be a fairly central tool.
>>
>> ‘guix publish’ is probably not extensible enough to support that, but we
>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>
> Why not extend “guix archive”?
>
> However, without all the details so my remark is totally naive, I miss
> what “nar-herder” is doing that “guix archive”+rsync+“guix publish” is
> not doing – other said I miss why another SQL database is required to
> serve stuff from one place to another.  I have read README but I did not
> get the point.

Apologies, I missed replying to this at the time.

Using an SQL database (sqlite) isn't required in my opinion, but I do
like that aspect of the design.

You could for example keep the narinfo's as files on the disk, and then
use rsync say to copy them between machines to setup mirrors. I think
this would perform worse compared to storing the narinfos in a database
when initially setting up a mirror. With a database, you only have to
download one large file, whereas with individual narinfo files, you have
to download lots of individual small files. Storing all the narinfo
files individually also generally takes up more space than storing them
in a database, since there's an overhead associated with each file on
the filesystem.

Additionally, the database is used to do extra things on top of just
storing the narinfos. The references from the narinfos are stored
separately to facilitate doing GC like removal of the
nars. Additionally, there's a way to tag nars, something which I was
thinking of using to allow selectively removing some nars which only
need to be stored for a short time.

https://git.cbaines.net/guix/nar-herder/tree/nar-herder/database.scm#n74

I hope that makes some sense,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 5114 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>> However, due to the time spent not building things, the backlog is
>> longer than usual, and the substitute availability (especially for
>> x86_64-linux and i686-linux) is lower than usual.
>
> Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now.

I've been so slow in replying to this email that the situation has
actually improved now, substitute availability for x86_64-linux is
around ~70% and still slowly rising.

>> ** Space issues and the nar-herder
>>
>> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
>> bayfront was getting a little scarce. This week I started rolling out
>> the nar-herder [2], a utility I've been thinking about for a while. This
>> has enabled moving nars off of bayfront on to another machine which I've
>> confusingly named lakefront.
>
> Woow, neat!
>
> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.
>
> ‘guix publish’ is probably not extensible enough to support that, but we
> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.

I think having something in Guix would be good too.

I still view the nar-herder as a bit wierd in terms of the collection of
concerns, mixing mirroring in with moving nars around between machines
and I'm hoping to add some metrics functionality soon as well.

>> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
>> storage across 2 hard drives. When a nar is requested from bayfront, it
>> will check it's local storage, and serve it from there if it exists,
>> otherwise it will forward the request to lakefront. There might be a
>> slight drop in the speed you can download nars, but apart from that this
>> change shouldn't be visible.
>
> Excellent, thanks for acting this promptly as problems crop up!
>
> I think we should make sure this is funded by the project and that
> credentials are shared.  As discussed before, I think it’s best to keep
> things tidy in that respect, in the spirit of building and maintaining
> this collectively.

That would be good, I can start a thread on guix-sysadmin about this.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>>
>> Once I've had a chance to neaten up the code a little, I'll get a Guix
>> package and service written, plus I'll draft a Guix blog post about the
>> nar-herder.
>
> Usually I’m the one asking for blog posts :-), but I’d really like us as
> a project to collectively engage on the topic before we publicize this
> specific approach.

Sure, I also still need to post patches for the Guix service, which I'll
try to do soon.

>> That means there's the following currently running build agents (by
>> architecture):
>>
>>  - x86_64-linux + i686-linux (3 machines):
>>    - 4 core Intel NUC (giedi)
>>    - Max 16 cores for 1 concurrent build on bayfront
>>    - 32 cores on milano-guix-1 (slow storage though)
>>  - aarch64-linux + armhf-linux (2 machines)
>>    - 16 core HoneyComb LX2 (hatysa)
>>    - 4 core Overdrive (monokuma)
>>  - powerpc64le-linux (1 machine)
>>    - 64 core machine (polaris)
>
> This is looking pretty nice!  I’m also all for streamlining machine
> handling, both administratively (in maintenance.git) and financially.

I think there was some discussion previously about guix-europe buying
the HoneyComb board, which I can probably restart. It would be good to
also sort out better hosting for the powerpc64le-linux machine.

I'll try to put some time in to organising things in maintenance.git.

>> Ironically, I think that the most under-resourced area is x86_64-linux +
>> i686-linux. bayfront is restricted in what it can do since it also runs
>> the coordinator, and things go badly if the machine gets
>> overloaded. bayfront and milano-guix-1 both have hard drive storage,
>> which also can slow them down when building things (especially
>> milano-guix-1).
>>
>> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
>> keep up, it would be good to make a plan to add capacity. I think even a
>> single high core count x86_64-linux machine with performant storage
>> would make a big difference.
>
> Yes, we should do that, we still have funds for it.

Great :)

>> ** IPv6 not supported (yet)
>>
>> I was slow to notice, but bordeaux.guix.gnu.org isn't available over
>> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
>> to address this, but I haven't worked out quite how to yet.
>
> Should be almost done now, as discussed on IRC today.  \o/

Indeed, I think this is working now.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]