Bringing substitutes from the Guix Build Coordinator to users

Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]

Hey,

The Guix Build Coordinator has been around for a while, it's been
available in Guix for more than 6 months now. The setup I've been using
to test the Guix Build Coordinator for building things for substitutes
(guix.cbaines.net) has been running since mid 2020.

Anecdotally, the test setup I've been running (guix.cbaines.net) has for
a while now, generally had higher percentage substitute availability
compared to ci.guix.gnu.org, as reported by guix weather. I think it
might also be building more things since cross derivations are built
plus i586-gnu builds.

I think there are some benefits for using the Guix Build Coordinator to
build things for substitutes, and it would be good to work out how to
get those benefits to users of Guix generally.

More specifically, while the architecture is similar to daemon
offloading, there are some practical advantages. Since the switch to
Cuirass remote workers for ci.guix.gnu.org, the advantages of building
things across multiple machines in a coordinated manor have also become
clear. Additionally, there are things that the Guix Build Coordinator
enables, like automated retries, with the option to target specific
agents, which can help with avoiding issues with non-reproducible
failures, as well as helping to avoid issues when mixing QEMU emulated
and native builds.

Going back to discussions in 2020, I think there was a path for starting
to experiment with the Guix Build Coordinator, trying it on a few
machines in the berlin build farm, but as I say, this was last year and
before any work on implementing similar functionality in Cuirass as far
as I'm aware. Since then, I've also got an instance of the Guix Build
Coordinator running on bayfront with a couple of extra machines also
involved for performing builds, but this is just starting off, and
doesn't bring any benefit in terms of substitutes, at least not yet.

This is a small part of the wider puzzle, when I was designing the Guix
Build Coordinator, it was meant for much more than building things for
substitutes, and those things are also happening. The Guix Build
Coordinator has enabled things like building packages affected by
patches, building fixed output derivations regularly, and will hopefully
enable a whole load of quality assurance things. But my hope was also to
try and tackle building things for substitutes, such that there's not
more software to maintain, and also to inform future work on the
guix-daemon itself, like whether it would be good for it to have an
asynchronous API, and how that might work.

This is a topic I haven't got directly involved in, until now. I'm just
a volunteer, in some ways the most involved I am is that I host an idle
ARM build machine, I don't have any particular connection in the default
approach to substitutes for users, or the hardware currently
involved. However, I think some of the stuff I've been working on could
be helpful, as I say, I think the Guix Build Coordinator is a step
forward in terms of building things for substitutes, and that shows in
the substitute availability percentages.

Is there still a path to bring some of these benefits to users, and if
so, what things need doing?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hi Christopher,

Thanks for your message!  It’s great to see a summary of what’s been
cooking in the Coordinator and your vision around it.

Christopher Baines <mail@cbaines.net> skribis:

> More specifically, while the architecture is similar to daemon
> offloading, there are some practical advantages. Since the switch to
> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
> things across multiple machines in a coordinated manor have also become
> clear. Additionally, there are things that the Guix Build Coordinator
> enables, like automated retries, with the option to target specific
> agents, which can help with avoiding issues with non-reproducible
> failures, as well as helping to avoid issues when mixing QEMU emulated
> and native builds.

These are nice examples of current QA blind spots that would be nice to
address.

[...]

> This is a topic I haven't got directly involved in, until now. I'm just
> a volunteer, in some ways the most involved I am is that I host an idle
> ARM build machine, I don't have any particular connection in the default
> approach to substitutes for users, or the hardware currently
> involved.

It would be great to have that OverDrive plugged in behind ci.guix
because we’re still short on CPU power for ARM.  (Not great we had
forgotten about this one!)

It could be reconfigured with a config similar to
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
with a different host name and IP.  We can discuss the details in a
separate thread and/or on guix-sysadmin, let me know!

> However, I think some of the stuff I've been working on could be
> helpful, as I say, I think the Guix Build Coordinator is a step
> forward in terms of building things for substitutes, and that shows in
> the substitute availability percentages.
>
> Is there still a path to bring some of these benefits to users, and if
> so, what things need doing?

My understanding is that, to build substitutes, we need more than the
Coordinator; we also need something to perform evaluations, similar to
the “top half” of Cuirass and the Data Service, right?  How would you
envision setting it up?

To me, the way forward is not necessarily substitutes; it could be QA,
as you showed that it has helpful features.  You already have access to
bayfront and it’d be great to experiment there.  Do you have ideas on
how to do make progress on that front?


Besides, I feel that the Data Service has a huge role to play with a
variety of applications, including QA, but also way beyond as you showed
in the blog post¹.  I think it’s time take advantage of it.  :-)

Specifically, as far as QA is concerned, I think we should place the
Data Service at the center of the stage.  It already does things that
Cuirass cannot and will not do, notably: linting, challenging substitute
servers to estimate reproducibility, and recording the history of all
this.  Couldn’t it gather these other QA metrics mentioned above,
possibly from a Coordinator-powered bayfront?

What I’d love to see is a set of “skins” for the Data Service:
purpose-specific interfaces to the service.  There’s already the
reproducibility one, the one for package versions, etc.  It would be
great to have a qa.guix.gnu.org entry point with a simplified interface
that would only show (say) reproducibility data, lint warnings, and so
forth.  (Then we can also think of other skins, like Guix Weekly News, a
page dedicated to browsing package version history, a security-related
page, etc.  But now I feel like a spoiled kid asking for yet more
presents.  :-))

How does that sound?

Thanks!

Ludo’.

¹ https://guix.gnu.org/en/blog/2020/introduction-to-the-guix-data-service-the-missing-blog-post/

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 6452 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Thanks for your message!  It’s great to see a summary of what’s been
> cooking in the Coordinator and your vision around it.
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> More specifically, while the architecture is similar to daemon
>> offloading, there are some practical advantages. Since the switch to
>> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
>> things across multiple machines in a coordinated manor have also become
>> clear. Additionally, there are things that the Guix Build Coordinator
>> enables, like automated retries, with the option to target specific
>> agents, which can help with avoiding issues with non-reproducible
>> failures, as well as helping to avoid issues when mixing QEMU emulated
>> and native builds.
>
> These are nice examples of current QA blind spots that would be nice to
> address.
>
> [...]
>
>> This is a topic I haven't got directly involved in, until now. I'm just
>> a volunteer, in some ways the most involved I am is that I host an idle
>> ARM build machine, I don't have any particular connection in the default
>> approach to substitutes for users, or the hardware currently
>> involved.
>
> It would be great to have that OverDrive plugged in behind ci.guix
> because we’re still short on CPU power for ARM.  (Not great we had
> forgotten about this one!)
>
> It could be reconfigured with a config similar to
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
> with a different host name and IP.  We can discuss the details in a
> separate thread and/or on guix-sysadmin, let me know!

It was/is being discussed, the relevant thread is here:
  https://lists.gnu.org/mailman/private/guix-sysadmin/2021-March/003487.html

>> However, I think some of the stuff I've been working on could be
>> helpful, as I say, I think the Guix Build Coordinator is a step
>> forward in terms of building things for substitutes, and that shows in
>> the substitute availability percentages.
>>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> My understanding is that, to build substitutes, we need more than the
> Coordinator; we also need something to perform evaluations, similar to
> the “top half” of Cuirass and the Data Service, right?  How would you
> envision setting it up?

The queue builds script bundled in with the guix-build-coordinator seems
to work fine, that's how bayfront is currently set up:

  (service guix-build-coordinator-queue-builds-service-type
         (guix-build-coordinator-queue-builds-configuration
          (systems '("x86_64-linux"))))

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

> To me, the way forward is not necessarily substitutes; it could be QA,
> as you showed that it has helpful features.  You already have access to
> bayfront and it’d be great to experiment there.  Do you have ideas on
> how to do make progress on that front?

Ideas yes, and the two things are interrelated, but my perspective at
the moment is that it'll be harder to try and get some value from the
Guix Build Coordinator in terms of quality assurance, as compared to
substitutes.

> Besides, I feel that the Data Service has a huge role to play with a
> variety of applications, including QA, but also way beyond as you showed
> in the blog post¹.  I think it’s time take advantage of it.  :-)
>
> Specifically, as far as QA is concerned, I think we should place the
> Data Service at the center of the stage.  It already does things that
> Cuirass cannot and will not do, notably: linting, challenging substitute
> servers to estimate reproducibility, and recording the history of all
> this.  Couldn’t it gather these other QA metrics mentioned above,
> possibly from a Coordinator-powered bayfront?

To me, linting and substitute comparison are less important quality
assurance things. I think the key issues are around testing packages and
things like system tests.

There's been some progress, albeit slow progress with the patch testing
setup I've been working on, but I've been viewing that as a way to test
branches as well. Given changes happen through patches and branches, I
think this makes sense, and I think the Guix Data Service has some nice
features for helping with this, like comparing two revisions and working
out what breakages have occurred.

However, with patches and branches, this is very much a continuous
integration tooling issue, much more than substitutes. Cuirass is
already building branches, and it's been gaining features previously
found in the Guix Data Service, like tracking the history of which
derivations relate to which revision, and separating out packages from
system tests so that it's easier to look at them.

As people are going to Cuirass to look at the build status for packages,
system tests and various branches, the problem is similar to that of
substitutes. It doesn't matter if the Guix Build Coordinator seems to do
some things better, when the question comes up about whether something
has built yet, or whether a branch is ready to merge, Cuirass on
ci.guix.gnu.org is where people are going, and that seems unlikely to
change (at least less likely than the setup for providing substitutes).

The Guix Data Service in the center, making it easier to do various
things by providing the data, this was the idea when it started, but the
reality recently is that strategy hasn't been paying off.

> What I’d love to see is a set of “skins” for the Data Service:
> purpose-specific interfaces to the service.  There’s already the
> reproducibility one, the one for package versions, etc.  It would be
> great to have a qa.guix.gnu.org entry point with a simplified interface
> that would only show (say) reproducibility data, lint warnings, and so
> forth.  (Then we can also think of other skins, like Guix Weekly News, a
> page dedicated to browsing package version history, a security-related
> page, etc.  But now I feel like a spoiled kid asking for yet more
> presents.  :-))

Haha, some QA entry point has come up before [1], but I haven't got
around to looking at implementing anything yet.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> My understanding is that, to build substitutes, we need more than the
>> Coordinator; we also need something to perform evaluations, similar to
>> the “top half” of Cuirass and the Data Service, right?  How would you
>> envision setting it up?
>
> The queue builds script bundled in with the guix-build-coordinator seems
> to work fine, that's how bayfront is currently set up:
>
>   (service guix-build-coordinator-queue-builds-service-type
>          (guix-build-coordinator-queue-builds-configuration
>           (systems '("x86_64-linux"))))
>
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

Oh, right.

[...]

> There's been some progress, albeit slow progress with the patch testing
> setup I've been working on, but I've been viewing that as a way to test
> branches as well. Given changes happen through patches and branches, I
> think this makes sense, and I think the Guix Data Service has some nice
> features for helping with this, like comparing two revisions and working
> out what breakages have occurred.
>
> However, with patches and branches, this is very much a continuous
> integration tooling issue, much more than substitutes. Cuirass is
> already building branches, and it's been gaining features previously
> found in the Guix Data Service, like tracking the history of which
> derivations relate to which revision, and separating out packages from
> system tests so that it's easier to look at them.

Yes, though to be fair, the system test separation is a separate jobset,
not a “feature” of Cuirass.

From what I can see Cuirass is in the footsteps of Hydra in terms of
design, which was somewhat ad-hoc at times, focusing on answering
developer questions.

> As people are going to Cuirass to look at the build status for packages,
> system tests and various branches, the problem is similar to that of
> substitutes. It doesn't matter if the Guix Build Coordinator seems to do
> some things better, when the question comes up about whether something
> has built yet, or whether a branch is ready to merge, Cuirass on
> ci.guix.gnu.org is where people are going, and that seems unlikely to
> change (at least less likely than the setup for providing substitutes).
>
> The Guix Data Service in the center, making it easier to do various
> things by providing the data, this was the idea when it started, but the
> reality recently is that strategy hasn't been paying off.

The Data Service provides a wealth of information that’s underused!

I think addressing the interface issue (be it web UI or JSON + CLI) is
high priority so we can all start taking advantage of the Data Service.
The current interface is generic enough that it allows you to see
everything, from lint warnings to version changes to reproducitility
rates.

We could have interfaces that answer very specific needs:

  • Which packages are broken on x86_64?

  • How does branch X compare to branch Y in terms of build failures?

  • Which packages are not reproducible?

  • Which packages are “flaky”?

I know all this information is already available from the web UI, but
because it’s generic, it can be hard to find out how to answer very
concrete issues like this.

A QA entry point like you proposed in the thread you mentioned¹ could
certainly help.  A reproducibility entry point would be nice too.  A
package browser for guix.gnu.org like the one Danjela worked on would be
great too, possibly with version browsing facilities.  And Guix Weekly
News!  And the security tracker!  :-)

¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

It seems to me that all that hard work is already done and what I
describe above are rather low-hanging fruits.

Taking Conway’s law into account, we may find it easier to recruit if as
much as possible takes place here, things get deployed behind
*.guix.gnu.org, and relevant bits are made part of Guix proper.  And
also, we must regularly advertise progress; one blog post in all of the
Guix Data Service’s lifetime is not enough.  :-)

Thoughts?

Ludo’.

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4023 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
>> As people are going to Cuirass to look at the build status for packages,
>> system tests and various branches, the problem is similar to that of
>> substitutes. It doesn't matter if the Guix Build Coordinator seems to do
>> some things better, when the question comes up about whether something
>> has built yet, or whether a branch is ready to merge, Cuirass on
>> ci.guix.gnu.org is where people are going, and that seems unlikely to
>> change (at least less likely than the setup for providing substitutes).
>>
>> The Guix Data Service in the center, making it easier to do various
>> things by providing the data, this was the idea when it started, but the
>> reality recently is that strategy hasn't been paying off.
>
> The Data Service provides a wealth of information that’s underused!
>
> I think addressing the interface issue (be it web UI or JSON + CLI) is
> high priority so we can all start taking advantage of the Data Service.
> The current interface is generic enough that it allows you to see
> everything, from lint warnings to version changes to reproducitility
> rates.
>
> We could have interfaces that answer very specific needs:
>
>   • Which packages are broken on x86_64?

While the Guix Data Service can sort of do this (if it has builds data),
I think Cuirass has a page for that now, there's not a single URL for
the latest evaulation, but assuming you click on the latest one, this
page list the failed builds for x86_64:

  https://ci.guix.gnu.org/eval/30959/dashboard

>   • How does branch X compare to branch Y in terms of build failures?
>
>   • Which packages are not reproducible?
>
>   • Which packages are “flaky”?
>
> I know all this information is already available from the web UI, but
> because it’s generic, it can be hard to find out how to answer very
> concrete issues like this.
>
> A QA entry point like you proposed in the thread you mentioned¹ could
> certainly help.  A reproducibility entry point would be nice too.  A
> package browser for guix.gnu.org like the one Danjela worked on would be
> great too, possibly with version browsing facilities.  And Guix Weekly
> News!  And the security tracker!  :-)
>
> ¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html
>
> It seems to me that all that hard work is already done and what I
> describe above are rather low-hanging fruits.
>
> Taking Conway’s law into account, we may find it easier to recruit if as
> much as possible takes place here, things get deployed behind
> *.guix.gnu.org, and relevant bits are made part of Guix proper.  And
> also, we must regularly advertise progress; one blog post in all of the
> Guix Data Service’s lifetime is not enough.  :-)
>
> Thoughts?

This doesn't really relate to the subject of substitutes.

Some of the things you mention do relate to work I'm trying to progress
though. I'm still working on automated patch (plus branch) testing, and
I think having a simple overview of patch+branch states is hopefully
something that I'll get to at some point.

On the subject of the patch testing stuff, that isn't under
.guix.gnu.org and I haven't written a blog post yet. I can't see Cuirass
starting to test patches, but then I wouldn't have predicted it would be
managing builds across multiple machines. Maybe there are some risks
with the patch testing work that I haven't done enough/the right stuff
to mitigate.

I've also got too many things in progress at the moment, with the
combination of work on substitutes (I hope to implement things set out
in [1] some point soon), Outreachy, and the security related work that
I'm trying to start, I need to "finish" some things before starting new
ones or going back to unfinished things.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> A QA entry point like you proposed in the thread you mentioned¹ could
>> certainly help.  A reproducibility entry point would be nice too.  A
>> package browser for guix.gnu.org like the one Danjela worked on would be
>> great too, possibly with version browsing facilities.  And Guix Weekly
>> News!  And the security tracker!  :-)
>>
>> ¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html
>>
>> It seems to me that all that hard work is already done and what I
>> describe above are rather low-hanging fruits.
>>
>> Taking Conway’s law into account, we may find it easier to recruit if as
>> much as possible takes place here, things get deployed behind
>> *.guix.gnu.org, and relevant bits are made part of Guix proper.  And
>> also, we must regularly advertise progress; one blog post in all of the
>> Guix Data Service’s lifetime is not enough.  :-)
>>
>> Thoughts?
>
> This doesn't really relate to the subject of substitutes.

Indeed, I slid off-topic, but it’s all there in the big picture.  :-)

> Some of the things you mention do relate to work I'm trying to progress
> though. I'm still working on automated patch (plus branch) testing, and
> I think having a simple overview of patch+branch states is hopefully
> something that I'll get to at some point.
>
> On the subject of the patch testing stuff, that isn't under
> .guix.gnu.org and I haven't written a blog post yet. I can't see Cuirass
> starting to test patches, but then I wouldn't have predicted it would be
> managing builds across multiple machines. Maybe there are some risks
> with the patch testing work that I haven't done enough/the right stuff
> to mitigate.
>
> I've also got too many things in progress at the moment, with the
> combination of work on substitutes (I hope to implement things set out
> in [1] some point soon), Outreachy, and the security related work that
> I'm trying to start, I need to "finish" some things before starting new
> ones or going back to unfinished things.
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Yeah, I do think polishing, deploying, and advertising some of these
things that are already 80% ready would be a great booster for all of
us.  To help recruit people, we can also deploy at guix.gnu.org things
that are not complete yet (the package browser, Guix Weekly News, the QA
entry point, etc.); it’s always easier once the thing is palatable.

HTH!

Ludo’.

Re: Bringing substitutes from the Guix Build Coordinator to users

[Andreas Enge]

Hello Chris,

Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
> I think there are some benefits for using the Guix Build Coordinator to
> build things for substitutes, and it would be good to work out how to
> get those benefits to users of Guix generally.

my question is a bit tangential, but how can we get the substitutes that
the build coordinator puts on bayfront? I still have bayfront.guix.info
in my list of substitute servers, but does it use the same signing key
as before? It would be nice to document what a user should do. In my
opinion, a second build farm would be a useful thing, in case one server
goes down, or needs to be updated, for instance. (And then it allows for
reproducibility checks.)

Relatedly, and this may already be documented: How do I know where the
packages are downloaded from?
$ guix package -u
substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
Before the print format overhaul, the full URL would be printed.

Does the guix daemon complain when substitute URLs do not match any
authorised key?

I know this should probably go to help-guix, but my questions fit the thread ;-)

Andreas

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> Hello Chris,
>
> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>> I think there are some benefits for using the Guix Build Coordinator to
>> build things for substitutes, and it would be good to work out how to
>> get those benefits to users of Guix generally.
>
> my question is a bit tangential, but how can we get the substitutes that
> the build coordinator puts on bayfront? I still have bayfront.guix.info
> in my list of substitute servers, but does it use the same signing key
> as before? It would be nice to document what a user should do. In my
> opinion, a second build farm would be a useful thing, in case one server
> goes down, or needs to be updated, for instance. (And then it allows for
> reproducibility checks.)

So, bayfront is currently serving substitutes build through the Guix
Build Coordinator, the signing key is the same as it was before.

It's only been building things at pace since yesterday though, so it'll
be a few days at least until it's caught up.

I agree that having multiple independent substitute providers would be
nice, and getting bayfront working well is a step towards that.

> Relatedly, and this may already be documented: How do I know where the
> packages are downloaded from?
> $ guix package -u
> substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>  imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
> Before the print format overhaul, the full URL would be printed.

Maybe you can turn on more detailed output? I'm unsure.

> Does the guix daemon complain when substitute URLs do not match any
> authorised key?

I don't think so, it'll just ignore them.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3126 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Andreas Enge <andreas@enge.fr> writes:
>
>> Hello Chris,
>>
>> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>>> I think there are some benefits for using the Guix Build Coordinator to
>>> build things for substitutes, and it would be good to work out how to
>>> get those benefits to users of Guix generally.
>>
>> my question is a bit tangential, but how can we get the substitutes that
>> the build coordinator puts on bayfront? I still have bayfront.guix.info
>> in my list of substitute servers, but does it use the same signing key
>> as before? It would be nice to document what a user should do. In my
>> opinion, a second build farm would be a useful thing, in case one server
>> goes down, or needs to be updated, for instance. (And then it allows for
>> reproducibility checks.)
>
> So, bayfront is currently serving substitutes build through the Guix
> Build Coordinator, the signing key is the same as it was before.
>
> It's only been building things at pace since yesterday though, so it'll
> be a few days at least until it's caught up.
>
> I agree that having multiple independent substitute providers would be
> nice, and getting bayfront working well is a step towards that.

Continuing on the subject of bayfront, it seems like the most feasible
way to get substitutes build through the Guix Build Coordinator to users
might be through it. At least there hasn't been any other leads yet from
this thread.

Luckily, from a thread ("Machine things") on guix-sysadmin [1] a number
of months ago, the suggestion of using the Guix Build Coordinator on
bayfront already came up, so it's actually been running for around a
month, and making progress building things for part of that.

1: https://lists.gnu.org/mailman/private/guix-sysadmin/2021-February/003412.html

Getting some benefit from the substitutes will require a few things
though, firstly getting things in order such that a good proportion and
range of substitutes are available, and secondly amending the default
configuration to include bayfront.

On the default configuration point, what are the prerequsites for that?
Beyond having some substitutes to offer, is there any particular
criteria to consider, perhaps about the machines involved?

On the subject of the machines involved, currently there's:

 - x86_64-linux + i686-linux
   - bayfront (coordinator, agent and serving substitutes)
   - harbourfront
   - milano-guix-1
 - aarch64-linux + armhf-linux
   - monokuma

I'd like to get some childhurd VMs running, they can go on the x86_64
machines, and hopefully there will be enough resources to keep up with
x86_64-linux, i686-linux, while doing some i586-gnu builds. If I have
some luck, I might also be able to get some powerpc64 hardware, and I do
have two Raspberry Pi 4's, but they're not running Guix as the OS yet,
so it would be good to work out if they're suitable, or better left to
other tasks.

There's a few intertangled things, but the main question is, if bayfront
can be a source of substitutes, what's the path to actually have that
benefit users?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hello!

That all sounds like a plan!

Christopher Baines <mail@cbaines.net> skribis:

> There's a few intertangled things, but the main question is, if bayfront
> can be a source of substitutes, what's the path to actually have that
> benefit users?

One way it’ll benefit everyone is by feeding build/reproducibility data
to the Data Service.

Another way can be by advertising bayfront.guix.gnu.org to “insiders”
(that is, maybe we won’t put it in the manual, but we can suggest this
option to people reading this list etc.)

Thoughts?

BTW, we must make sure the load on bayfront itself remains under control
because it hosts <https://hpc.guix.info>.

Thanks for working on it!

Ludo’.

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> That all sounds like a plan!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> There's a few intertangled things, but the main question is, if bayfront
>> can be a source of substitutes, what's the path to actually have that
>> benefit users?
>
> One way it’ll benefit everyone is by feeding build/reproducibility data
> to the Data Service.
>
> Another way can be by advertising bayfront.guix.gnu.org to “insiders”
> (that is, maybe we won’t put it in the manual, but we can suggest this
> option to people reading this list etc.)
>
> Thoughts?

I guess my question was maybe too vague, I was thinking about getting
substitutes served from bayfront to users, and the benefit coming from
that.

In my mind, this would ideally include users who want to enable
substitutes, but don't know much more than that. I've started looking at
the changes this would involve [1].

1: https://issues.guix.gnu.org/48435

If there's effort put in to getting substitutes served from bayfront,
why do you suggest not documenting how to get those substitutes in the
manual?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> If there's effort put in to getting substitutes served from bayfront,
> why do you suggest not documenting how to get those substitutes in the
> manual?

Mostly because bayfront is not as powerful as the build farm behind
ci.guix.

Ludo’.

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> If there's effort put in to getting substitutes served from bayfront,
>> why do you suggest not documenting how to get those substitutes in the
>> manual?
>
> Mostly because bayfront is not as powerful as the build farm behind
> ci.guix.

I see that mostly as an argument for trying to bring the benefits of the
Guix Build Coordinator to the substitutes served by ci.guix.gnu.org.

Even with 3 machines for x86_64-linux (2.x given that bayfront is only
lightly used), bayfront has pretty much caught up with ci.guix.gnu.org
in terms of x86_64-linux substitutes, and I expect it to have a higher
substitute availability percentage in the coming days, since it's still
working through the backlog.

Once it is doing better, at least for x86_64-linux substitutes, I see
that as a strong argument to promote it to users (default config change,
blog post, ...).

1:
→ guix describe
Generation 310	May 16 2021 09:06:12	(current)
  guix 7c4c781
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 7c4c781aa40c42d4cd10b8d9482199f3db345e1b

→ guix weather --substitute-urls="https://bayfront.guix.gnu.org https://ci.guix.gnu.org"
computing 17,423 package derivations for x86_64-linux...
looking for 18,872 store items on https://bayfront.guix.gnu.org...
https://bayfront.guix.gnu.org
  91.6% substitutes available (17,293 out of 18,872)
  37,146.8 MiB of nars (compressed)
  143,978.2 MiB on disk (uncompressed)
  (continuous integration information unavailable)
looking for 18,872 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  92.1% substitutes available (17,389 out of 18,872)
  at least 139,781.5 MiB of nars (compressed)
  35,184,372,238,367.3 MiB on disk (uncompressed)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hallo!

Andreas Enge <andreas@enge.fr> skribis:

> Relatedly, and this may already be documented: How do I know where the
> packages are downloaded from?
> $ guix package -u
> substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>  imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
> Before the print format overhaul, the full URL would be printed.

You can use ‘-v2’ to see URLs:

  https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html

> Does the guix daemon complain when substitute URLs do not match any
> authorised key?

No, it just ignores it.

Ludo’.

[bug#48435] Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2179 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Is there still a path to bring some of these benefits to users, and if
> so, what things need doing?

It's been a few weeks now, so to summarise, I think only one path
emerged, and that is to get substitutes from bayfront to users.

Bayfront was already running the Guix Build Coordinator (although only
for the last month), and it's now caught up to the point where I'm
seeing similar or better substitute availability percentages for
x86_64-linux (and powerpc64le-linux) when compared to
ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things,
but they're still catching up.

Obviously just having the substitutes doesn't magically get them to
users, so I've started looking in to the changes to start making that
happen. Adding the signing key and changing the defaults in a few places
seems like a good step forward [1].

1: https://issues.guix.gnu.org/48435

I want to push on with this within the next couple of weeks, mostly so I
can shift focus to Outreachy and the security related tooling work, but
also because I still think this will be a good step forward in terms of
substitute availability for users. It's been over a year now since
implementation started, so it would be good to actually make a positive
difference.

There's a few issues still on my mind. Even though the substitute
availability percentages are good when compared to ci.guix.gnu.org, as
bayfront has much less compute power connected, it might not keep up as
well when big sets of changes are merged. I think that's just an
argument for using the build coordinator on berlin and the connected
machines though.

The other thing in comparison to ci.guix.gnu.org is that bayfront only
has ~4TB of storage rather than ~37TB, and given that currently none of
the generated nars are deleted, that will need thinking about in a few
months to avoid running out of space. I've had some plans around this
for a while [1], but they still require implementing.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Apart from merging the changes in [1], I guess a blog post might be
useful. Have I missed anything?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#48435] Bringing substitutes from the Guix Build Coordinator to users

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?

[...]

> Obviously just having the substitutes doesn't magically get them to
> users, so I've started looking in to the changes to start making that
> happen. Adding the signing key and changing the defaults in a few places
> seems like a good step forward [1].
>
> 1: https://issues.guix.gnu.org/48435
>
> I want to push on with this within the next couple of weeks, mostly so I
> can shift focus to Outreachy and the security related tooling work, but
> also because I still think this will be a good step forward in terms of
> substitute availability for users. It's been over a year now since
> implementation started, so it would be good to actually make a positive
> difference.

I’m fine with distributing an extra signing key alongside that of
ci.guix.gnu.org.

I’m unsure about having two substitute URLs by default since it adds a
bit of overhead, though that overhead is only upon cache misses (I have
that setup on my laptop actually).

It’s also a one-way change: people are likely to keep the defaults
“forever”.  So we can’t just “experiment” and change our mind later.
That means we should at least have a DNS entry that’s not tied to a
particular machine, like ci2.guix.gnu.org or whatever.

WDYT?

Now, what would be nice is to have a second build farm with the
K-out-of-N policy you mention in mind.

> There's a few issues still on my mind. Even though the substitute
> availability percentages are good when compared to ci.guix.gnu.org, as
> bayfront has much less compute power connected, it might not keep up as
> well when big sets of changes are merged. I think that's just an
> argument for using the build coordinator on berlin and the connected
> machines though.

As much as I’d have preferred a single solution in this area, fueling
competition between the Coordinator and Cuirass and their access to
official infrastructure doesn’t seem like a viable path to me.

I think the primary value in having a second build farm would be
reproducibility and doing away with the single point of failure.
Overall substitute coverage probably wouldn’t change much.

I agree with Mathieu that maintaining it has a cost, but maybe we can
try.

I realize I’m asking questions rather than providing answers, which may
be because I don’t see a clear path ahead.  :-)

Thanks!

Ludo’.

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2421 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>>> Is there still a path to bring some of these benefits to users, and if
>>> so, what things need doing?
>
> [...]
>
>> Obviously just having the substitutes doesn't magically get them to
>> users, so I've started looking in to the changes to start making that
>> happen. Adding the signing key and changing the defaults in a few places
>> seems like a good step forward [1].
>>
>> 1: https://issues.guix.gnu.org/48435
>>
>> I want to push on with this within the next couple of weeks, mostly so I
>> can shift focus to Outreachy and the security related tooling work, but
>> also because I still think this will be a good step forward in terms of
>> substitute availability for users. It's been over a year now since
>> implementation started, so it would be good to actually make a positive
>> difference.
>
> I’m fine with distributing an extra signing key alongside that of
> ci.guix.gnu.org.

Great.

> I’m unsure about having two substitute URLs by default since it adds a
> bit of overhead, though that overhead is only upon cache misses (I have
> that setup on my laptop actually).

All of this work has been built on the assumption that it's possible to
do better in providing substitutes, and anecdotally from the data I've
seen over the last year, that should be possible, even with the limited
hardware (compared to ci.guix.gnu.org) connected to bayfront.

So yes, that's a valid concern, but if all the addition of bayfront does
is make users wait a little longer because of cache misses, it's a sign
that the whole endeavour is not working out.

> It’s also a one-way change: people are likely to keep the defaults
> “forever”.  So we can’t just “experiment” and change our mind later.
> That means we should at least have a DNS entry that’s not tied to a
> particular machine, like ci2.guix.gnu.org or whatever.

That sounds sensible. On the specific name, given this is just about
substitutes, and at least in my opinion has nothing to do with
continuous integration, maybe picking just another word would avoid
thinking too much, it could be bordeaux, or hippo, or anything
really. As you say, stability and not being tied to a particular machine
is the important thing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Mathieu Othacehe]

Hey Chris,

> That sounds sensible. On the specific name, given this is just about
> substitutes, and at least in my opinion has nothing to do with
> continuous integration, maybe picking just another word would avoid
> thinking too much, it could be bordeaux, or hippo, or anything
> really. As you say, stability and not being tied to a particular machine
> is the important thing.

The substitutes coverage is one indicator to take into account but there
are many others. For instance, the evaluation speed, the failed
evaluation count, the average evaluation builds completion time, the
availability of the connected build machines between other things.

Deploying a solution that builds substitutes is fine, but as soon as it
is deployed and accessible to all Guix users, the system administrators
will have to monitor it and maintain it in the long run.

Having two heterogeneous build infrastructures on two sets of machines,
providing different metrics will make the update and maintenance of
those machines harder.

I hear your point about K-out-of-N policy and it also makes sense to
me. However, we should maybe consider doing it using two similar
infrastructures.

Thanks,

Mathieu

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1542 bytes --]


Mathieu Othacehe <othacehe@gnu.org> writes:

> Hey Chris,
>
>> That sounds sensible. On the specific name, given this is just about
>> substitutes, and at least in my opinion has nothing to do with
>> continuous integration, maybe picking just another word would avoid
>> thinking too much, it could be bordeaux, or hippo, or anything
>> really. As you say, stability and not being tied to a particular machine
>> is the important thing.
>
> The substitutes coverage is one indicator to take into account but there
> are many others. For instance, the evaluation speed, the failed
> evaluation count, the average evaluation builds completion time, the
> availability of the connected build machines between other things.

Indeed, and I'm aware that the Guix Data Service, which performs a
similar function to the evaluations in Cuirass, is much slower.

> Deploying a solution that builds substitutes is fine, but as soon as it
> is deployed and accessible to all Guix users, the system administrators
> will have to monitor it and maintain it in the long run.
>
> Having two heterogeneous build infrastructures on two sets of machines,
> providing different metrics will make the update and maintenance of
> those machines harder.
>
> I hear your point about K-out-of-N policy and it also makes sense to
> me. However, we should maybe consider doing it using two similar
> infrastructures.

Indeed. The reality though is that two different approaches have been in
development now for a little over a year, and this is a reflection of
that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Bringing substitutes from the Guix Build Coordinator to users

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> It's been a few weeks now, so to summarise, I think only one path
> emerged, and that is to get substitutes from bayfront to users.

More weeks have past, it's taking me longer to get things sorted out
that I'd like, but things are still moving forward.

> Bayfront was already running the Guix Build Coordinator (although only
> for the last month), and it's now caught up to the point where I'm
> seeing similar or better substitute availability percentages for
> x86_64-linux (and powerpc64le-linux) when compared to
> ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things,
> but they're still catching up.

Substitute availability for x86_64-linux and i686-linux should be
roughly comparable to ci.guix.gnu.org.  powerpc64le-linux substitute
availability is OK, and aarch64-linux might even be doing better than
ci.guix.gnu.org somehow.

Other things like armhf-linux and i586-gnu are still very much works in
progress.

> Obviously just having the substitutes doesn't magically get them to
> users, so I've started looking in to the changes to start making that
> happen. Adding the signing key and changing the defaults in a few places
> seems like a good step forward [1].
>
> 1: https://issues.guix.gnu.org/48435

I've gone ahead and put the key in to the Guix git repository [1] and
sent an updated patch for changing various bits of configuration [2].

1: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=26499816a973b3aab9aaf8e13b909d0bde4e2dd5
2: https://issues.guix.gnu.org/48435#8

I think the patch still needs a bit more work, mostly to update the
docs. I'll try to work out what needs tweaking in the docs and send a v3
ASAP.

In terms of what to initially change, I'm still not sure if there's
something that needs updating that I'm currently missing, or something
that I'm updating that can be done later.

> Apart from merging the changes in [1], I guess a blog post might be
> useful. Have I missed anything?

I'll start another thread on guix-devel to solicit feedback about
substitutes from bordeaux.guix.gnu.org, I'm not sure what specifically
about, but peoples observations might be helpful when writing a blog
post about this. I'll also try to start drafting a blog post.

What else needs doing to actually get these substitutes to users?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]