unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* Bringing substitutes from the Guix Build Coordinator to users
@ 2021-05-01 18:56 Christopher Baines
  2021-05-02 21:51 ` Ludovic Courtès
  2021-05-04 18:38 ` Andreas Enge
  0 siblings, 2 replies; 10+ messages in thread
From: Christopher Baines @ 2021-05-01 18:56 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]

Hey,

The Guix Build Coordinator has been around for a while, it's been
available in Guix for more than 6 months now. The setup I've been using
to test the Guix Build Coordinator for building things for substitutes
(guix.cbaines.net) has been running since mid 2020.

Anecdotally, the test setup I've been running (guix.cbaines.net) has for
a while now, generally had higher percentage substitute availability
compared to ci.guix.gnu.org, as reported by guix weather. I think it
might also be building more things since cross derivations are built
plus i586-gnu builds.

I think there are some benefits for using the Guix Build Coordinator to
build things for substitutes, and it would be good to work out how to
get those benefits to users of Guix generally.

More specifically, while the architecture is similar to daemon
offloading, there are some practical advantages. Since the switch to
Cuirass remote workers for ci.guix.gnu.org, the advantages of building
things across multiple machines in a coordinated manor have also become
clear. Additionally, there are things that the Guix Build Coordinator
enables, like automated retries, with the option to target specific
agents, which can help with avoiding issues with non-reproducible
failures, as well as helping to avoid issues when mixing QEMU emulated
and native builds.

Going back to discussions in 2020, I think there was a path for starting
to experiment with the Guix Build Coordinator, trying it on a few
machines in the berlin build farm, but as I say, this was last year and
before any work on implementing similar functionality in Cuirass as far
as I'm aware. Since then, I've also got an instance of the Guix Build
Coordinator running on bayfront with a couple of extra machines also
involved for performing builds, but this is just starting off, and
doesn't bring any benefit in terms of substitutes, at least not yet.

This is a small part of the wider puzzle, when I was designing the Guix
Build Coordinator, it was meant for much more than building things for
substitutes, and those things are also happening. The Guix Build
Coordinator has enabled things like building packages affected by
patches, building fixed output derivations regularly, and will hopefully
enable a whole load of quality assurance things. But my hope was also to
try and tackle building things for substitutes, such that there's not
more software to maintain, and also to inform future work on the
guix-daemon itself, like whether it would be good for it to have an
asynchronous API, and how that might work.

This is a topic I haven't got directly involved in, until now. I'm just
a volunteer, in some ways the most involved I am is that I host an idle
ARM build machine, I don't have any particular connection in the default
approach to substitutes for users, or the hardware currently
involved. However, I think some of the stuff I've been working on could
be helpful, as I say, I think the Guix Build Coordinator is a step
forward in terms of building things for substitutes, and that shows in
the substitute availability percentages.

Is there still a path to bring some of these benefits to users, and if
so, what things need doing?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-01 18:56 Bringing substitutes from the Guix Build Coordinator to users Christopher Baines
@ 2021-05-02 21:51 ` Ludovic Courtès
  2021-05-03 10:30   ` Christopher Baines
  2021-05-04 18:38 ` Andreas Enge
  1 sibling, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2021-05-02 21:51 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hi Christopher,

Thanks for your message!  It’s great to see a summary of what’s been
cooking in the Coordinator and your vision around it.

Christopher Baines <mail@cbaines.net> skribis:

> More specifically, while the architecture is similar to daemon
> offloading, there are some practical advantages. Since the switch to
> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
> things across multiple machines in a coordinated manor have also become
> clear. Additionally, there are things that the Guix Build Coordinator
> enables, like automated retries, with the option to target specific
> agents, which can help with avoiding issues with non-reproducible
> failures, as well as helping to avoid issues when mixing QEMU emulated
> and native builds.

These are nice examples of current QA blind spots that would be nice to
address.

[...]

> This is a topic I haven't got directly involved in, until now. I'm just
> a volunteer, in some ways the most involved I am is that I host an idle
> ARM build machine, I don't have any particular connection in the default
> approach to substitutes for users, or the hardware currently
> involved.

It would be great to have that OverDrive plugged in behind ci.guix
because we’re still short on CPU power for ARM.  (Not great we had
forgotten about this one!)

It could be reconfigured with a config similar to
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
with a different host name and IP.  We can discuss the details in a
separate thread and/or on guix-sysadmin, let me know!

> However, I think some of the stuff I've been working on could be
> helpful, as I say, I think the Guix Build Coordinator is a step
> forward in terms of building things for substitutes, and that shows in
> the substitute availability percentages.
>
> Is there still a path to bring some of these benefits to users, and if
> so, what things need doing?

My understanding is that, to build substitutes, we need more than the
Coordinator; we also need something to perform evaluations, similar to
the “top half” of Cuirass and the Data Service, right?  How would you
envision setting it up?

To me, the way forward is not necessarily substitutes; it could be QA,
as you showed that it has helpful features.  You already have access to
bayfront and it’d be great to experiment there.  Do you have ideas on
how to do make progress on that front?


Besides, I feel that the Data Service has a huge role to play with a
variety of applications, including QA, but also way beyond as you showed
in the blog post¹.  I think it’s time take advantage of it.  :-)

Specifically, as far as QA is concerned, I think we should place the
Data Service at the center of the stage.  It already does things that
Cuirass cannot and will not do, notably: linting, challenging substitute
servers to estimate reproducibility, and recording the history of all
this.  Couldn’t it gather these other QA metrics mentioned above,
possibly from a Coordinator-powered bayfront?

What I’d love to see is a set of “skins” for the Data Service:
purpose-specific interfaces to the service.  There’s already the
reproducibility one, the one for package versions, etc.  It would be
great to have a qa.guix.gnu.org entry point with a simplified interface
that would only show (say) reproducibility data, lint warnings, and so
forth.  (Then we can also think of other skins, like Guix Weekly News, a
page dedicated to browsing package version history, a security-related
page, etc.  But now I feel like a spoiled kid asking for yet more
presents.  :-))

How does that sound?

Thanks!

Ludo’.

¹ https://guix.gnu.org/en/blog/2020/introduction-to-the-guix-data-service-the-missing-blog-post/


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-02 21:51 ` Ludovic Courtès
@ 2021-05-03 10:30   ` Christopher Baines
  2021-05-04  8:27     ` Ludovic Courtès
  0 siblings, 1 reply; 10+ messages in thread
From: Christopher Baines @ 2021-05-03 10:30 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 6452 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Thanks for your message!  It’s great to see a summary of what’s been
> cooking in the Coordinator and your vision around it.
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> More specifically, while the architecture is similar to daemon
>> offloading, there are some practical advantages. Since the switch to
>> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
>> things across multiple machines in a coordinated manor have also become
>> clear. Additionally, there are things that the Guix Build Coordinator
>> enables, like automated retries, with the option to target specific
>> agents, which can help with avoiding issues with non-reproducible
>> failures, as well as helping to avoid issues when mixing QEMU emulated
>> and native builds.
>
> These are nice examples of current QA blind spots that would be nice to
> address.
>
> [...]
>
>> This is a topic I haven't got directly involved in, until now. I'm just
>> a volunteer, in some ways the most involved I am is that I host an idle
>> ARM build machine, I don't have any particular connection in the default
>> approach to substitutes for users, or the hardware currently
>> involved.
>
> It would be great to have that OverDrive plugged in behind ci.guix
> because we’re still short on CPU power for ARM.  (Not great we had
> forgotten about this one!)
>
> It could be reconfigured with a config similar to
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
> with a different host name and IP.  We can discuss the details in a
> separate thread and/or on guix-sysadmin, let me know!

It was/is being discussed, the relevant thread is here:
  https://lists.gnu.org/mailman/private/guix-sysadmin/2021-March/003487.html

>> However, I think some of the stuff I've been working on could be
>> helpful, as I say, I think the Guix Build Coordinator is a step
>> forward in terms of building things for substitutes, and that shows in
>> the substitute availability percentages.
>>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> My understanding is that, to build substitutes, we need more than the
> Coordinator; we also need something to perform evaluations, similar to
> the “top half” of Cuirass and the Data Service, right?  How would you
> envision setting it up?

The queue builds script bundled in with the guix-build-coordinator seems
to work fine, that's how bayfront is currently set up:

  (service guix-build-coordinator-queue-builds-service-type
         (guix-build-coordinator-queue-builds-configuration
          (systems '("x86_64-linux"))))

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

> To me, the way forward is not necessarily substitutes; it could be QA,
> as you showed that it has helpful features.  You already have access to
> bayfront and it’d be great to experiment there.  Do you have ideas on
> how to do make progress on that front?

Ideas yes, and the two things are interrelated, but my perspective at
the moment is that it'll be harder to try and get some value from the
Guix Build Coordinator in terms of quality assurance, as compared to
substitutes.

> Besides, I feel that the Data Service has a huge role to play with a
> variety of applications, including QA, but also way beyond as you showed
> in the blog post¹.  I think it’s time take advantage of it.  :-)
>
> Specifically, as far as QA is concerned, I think we should place the
> Data Service at the center of the stage.  It already does things that
> Cuirass cannot and will not do, notably: linting, challenging substitute
> servers to estimate reproducibility, and recording the history of all
> this.  Couldn’t it gather these other QA metrics mentioned above,
> possibly from a Coordinator-powered bayfront?

To me, linting and substitute comparison are less important quality
assurance things. I think the key issues are around testing packages and
things like system tests.

There's been some progress, albeit slow progress with the patch testing
setup I've been working on, but I've been viewing that as a way to test
branches as well. Given changes happen through patches and branches, I
think this makes sense, and I think the Guix Data Service has some nice
features for helping with this, like comparing two revisions and working
out what breakages have occurred.

However, with patches and branches, this is very much a continuous
integration tooling issue, much more than substitutes. Cuirass is
already building branches, and it's been gaining features previously
found in the Guix Data Service, like tracking the history of which
derivations relate to which revision, and separating out packages from
system tests so that it's easier to look at them.

As people are going to Cuirass to look at the build status for packages,
system tests and various branches, the problem is similar to that of
substitutes. It doesn't matter if the Guix Build Coordinator seems to do
some things better, when the question comes up about whether something
has built yet, or whether a branch is ready to merge, Cuirass on
ci.guix.gnu.org is where people are going, and that seems unlikely to
change (at least less likely than the setup for providing substitutes).

The Guix Data Service in the center, making it easier to do various
things by providing the data, this was the idea when it started, but the
reality recently is that strategy hasn't been paying off.

> What I’d love to see is a set of “skins” for the Data Service:
> purpose-specific interfaces to the service.  There’s already the
> reproducibility one, the one for package versions, etc.  It would be
> great to have a qa.guix.gnu.org entry point with a simplified interface
> that would only show (say) reproducibility data, lint warnings, and so
> forth.  (Then we can also think of other skins, like Guix Weekly News, a
> page dedicated to browsing package version history, a security-related
> page, etc.  But now I feel like a spoiled kid asking for yet more
> presents.  :-))

Haha, some QA entry point has come up before [1], but I haven't got
around to looking at implementing anything yet.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-03 10:30   ` Christopher Baines
@ 2021-05-04  8:27     ` Ludovic Courtès
  2021-05-04 19:22       ` Christopher Baines
  0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2021-05-04  8:27 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> My understanding is that, to build substitutes, we need more than the
>> Coordinator; we also need something to perform evaluations, similar to
>> the “top half” of Cuirass and the Data Service, right?  How would you
>> envision setting it up?
>
> The queue builds script bundled in with the guix-build-coordinator seems
> to work fine, that's how bayfront is currently set up:
>
>   (service guix-build-coordinator-queue-builds-service-type
>          (guix-build-coordinator-queue-builds-configuration
>           (systems '("x86_64-linux"))))
>
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

Oh, right.

[...]

> There's been some progress, albeit slow progress with the patch testing
> setup I've been working on, but I've been viewing that as a way to test
> branches as well. Given changes happen through patches and branches, I
> think this makes sense, and I think the Guix Data Service has some nice
> features for helping with this, like comparing two revisions and working
> out what breakages have occurred.
>
> However, with patches and branches, this is very much a continuous
> integration tooling issue, much more than substitutes. Cuirass is
> already building branches, and it's been gaining features previously
> found in the Guix Data Service, like tracking the history of which
> derivations relate to which revision, and separating out packages from
> system tests so that it's easier to look at them.

Yes, though to be fair, the system test separation is a separate jobset,
not a “feature” of Cuirass.

From what I can see Cuirass is in the footsteps of Hydra in terms of
design, which was somewhat ad-hoc at times, focusing on answering
developer questions.

> As people are going to Cuirass to look at the build status for packages,
> system tests and various branches, the problem is similar to that of
> substitutes. It doesn't matter if the Guix Build Coordinator seems to do
> some things better, when the question comes up about whether something
> has built yet, or whether a branch is ready to merge, Cuirass on
> ci.guix.gnu.org is where people are going, and that seems unlikely to
> change (at least less likely than the setup for providing substitutes).
>
> The Guix Data Service in the center, making it easier to do various
> things by providing the data, this was the idea when it started, but the
> reality recently is that strategy hasn't been paying off.

The Data Service provides a wealth of information that’s underused!

I think addressing the interface issue (be it web UI or JSON + CLI) is
high priority so we can all start taking advantage of the Data Service.
The current interface is generic enough that it allows you to see
everything, from lint warnings to version changes to reproducitility
rates.

We could have interfaces that answer very specific needs:

  • Which packages are broken on x86_64?

  • How does branch X compare to branch Y in terms of build failures?

  • Which packages are not reproducible?

  • Which packages are “flaky”?

I know all this information is already available from the web UI, but
because it’s generic, it can be hard to find out how to answer very
concrete issues like this.

A QA entry point like you proposed in the thread you mentioned¹ could
certainly help.  A reproducibility entry point would be nice too.  A
package browser for guix.gnu.org like the one Danjela worked on would be
great too, possibly with version browsing facilities.  And Guix Weekly
News!  And the security tracker!  :-)

¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

It seems to me that all that hard work is already done and what I
describe above are rather low-hanging fruits.

Taking Conway’s law into account, we may find it easier to recruit if as
much as possible takes place here, things get deployed behind
*.guix.gnu.org, and relevant bits are made part of Guix proper.  And
also, we must regularly advertise progress; one blog post in all of the
Guix Data Service’s lifetime is not enough.  :-)

Thoughts?

Ludo’.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-01 18:56 Bringing substitutes from the Guix Build Coordinator to users Christopher Baines
  2021-05-02 21:51 ` Ludovic Courtès
@ 2021-05-04 18:38 ` Andreas Enge
  2021-05-04 19:29   ` Christopher Baines
  2021-05-06 16:26   ` Ludovic Courtès
  1 sibling, 2 replies; 10+ messages in thread
From: Andreas Enge @ 2021-05-04 18:38 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hello Chris,

Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
> I think there are some benefits for using the Guix Build Coordinator to
> build things for substitutes, and it would be good to work out how to
> get those benefits to users of Guix generally.

my question is a bit tangential, but how can we get the substitutes that
the build coordinator puts on bayfront? I still have bayfront.guix.info
in my list of substitute servers, but does it use the same signing key
as before? It would be nice to document what a user should do. In my
opinion, a second build farm would be a useful thing, in case one server
goes down, or needs to be updated, for instance. (And then it allows for
reproducibility checks.)

Relatedly, and this may already be documented: How do I know where the
packages are downloaded from?
$ guix package -u
substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
Before the print format overhaul, the full URL would be printed.

Does the guix daemon complain when substitute URLs do not match any
authorised key?

I know this should probably go to help-guix, but my questions fit the thread ;-)

Andreas



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-04  8:27     ` Ludovic Courtès
@ 2021-05-04 19:22       ` Christopher Baines
  2021-05-11 20:18         ` Ludovic Courtès
  0 siblings, 1 reply; 10+ messages in thread
From: Christopher Baines @ 2021-05-04 19:22 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 4023 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
>> As people are going to Cuirass to look at the build status for packages,
>> system tests and various branches, the problem is similar to that of
>> substitutes. It doesn't matter if the Guix Build Coordinator seems to do
>> some things better, when the question comes up about whether something
>> has built yet, or whether a branch is ready to merge, Cuirass on
>> ci.guix.gnu.org is where people are going, and that seems unlikely to
>> change (at least less likely than the setup for providing substitutes).
>>
>> The Guix Data Service in the center, making it easier to do various
>> things by providing the data, this was the idea when it started, but the
>> reality recently is that strategy hasn't been paying off.
>
> The Data Service provides a wealth of information that’s underused!
>
> I think addressing the interface issue (be it web UI or JSON + CLI) is
> high priority so we can all start taking advantage of the Data Service.
> The current interface is generic enough that it allows you to see
> everything, from lint warnings to version changes to reproducitility
> rates.
>
> We could have interfaces that answer very specific needs:
>
>   • Which packages are broken on x86_64?

While the Guix Data Service can sort of do this (if it has builds data),
I think Cuirass has a page for that now, there's not a single URL for
the latest evaulation, but assuming you click on the latest one, this
page list the failed builds for x86_64:

  https://ci.guix.gnu.org/eval/30959/dashboard

>   • How does branch X compare to branch Y in terms of build failures?
>
>   • Which packages are not reproducible?
>
>   • Which packages are “flaky”?
>
> I know all this information is already available from the web UI, but
> because it’s generic, it can be hard to find out how to answer very
> concrete issues like this.
>
> A QA entry point like you proposed in the thread you mentioned¹ could
> certainly help.  A reproducibility entry point would be nice too.  A
> package browser for guix.gnu.org like the one Danjela worked on would be
> great too, possibly with version browsing facilities.  And Guix Weekly
> News!  And the security tracker!  :-)
>
> ¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html
>
> It seems to me that all that hard work is already done and what I
> describe above are rather low-hanging fruits.
>
> Taking Conway’s law into account, we may find it easier to recruit if as
> much as possible takes place here, things get deployed behind
> *.guix.gnu.org, and relevant bits are made part of Guix proper.  And
> also, we must regularly advertise progress; one blog post in all of the
> Guix Data Service’s lifetime is not enough.  :-)
>
> Thoughts?

This doesn't really relate to the subject of substitutes.

Some of the things you mention do relate to work I'm trying to progress
though. I'm still working on automated patch (plus branch) testing, and
I think having a simple overview of patch+branch states is hopefully
something that I'll get to at some point.

On the subject of the patch testing stuff, that isn't under
.guix.gnu.org and I haven't written a blog post yet. I can't see Cuirass
starting to test patches, but then I wouldn't have predicted it would be
managing builds across multiple machines. Maybe there are some risks
with the patch testing work that I haven't done enough/the right stuff
to mitigate.

I've also got too many things in progress at the moment, with the
combination of work on substitutes (I hope to implement things set out
in [1] some point soon), Outreachy, and the security related work that
I'm trying to start, I need to "finish" some things before starting new
ones or going back to unfinished things.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-04 18:38 ` Andreas Enge
@ 2021-05-04 19:29   ` Christopher Baines
  2021-05-12 22:58     ` Christopher Baines
  2021-05-06 16:26   ` Ludovic Courtès
  1 sibling, 1 reply; 10+ messages in thread
From: Christopher Baines @ 2021-05-04 19:29 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> Hello Chris,
>
> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>> I think there are some benefits for using the Guix Build Coordinator to
>> build things for substitutes, and it would be good to work out how to
>> get those benefits to users of Guix generally.
>
> my question is a bit tangential, but how can we get the substitutes that
> the build coordinator puts on bayfront? I still have bayfront.guix.info
> in my list of substitute servers, but does it use the same signing key
> as before? It would be nice to document what a user should do. In my
> opinion, a second build farm would be a useful thing, in case one server
> goes down, or needs to be updated, for instance. (And then it allows for
> reproducibility checks.)

So, bayfront is currently serving substitutes build through the Guix
Build Coordinator, the signing key is the same as it was before.

It's only been building things at pace since yesterday though, so it'll
be a few days at least until it's caught up.

I agree that having multiple independent substitute providers would be
nice, and getting bayfront working well is a step towards that.

> Relatedly, and this may already be documented: How do I know where the
> packages are downloaded from?
> $ guix package -u
> substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>  imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
> Before the print format overhaul, the full URL would be printed.

Maybe you can turn on more detailed output? I'm unsure.

> Does the guix daemon complain when substitute URLs do not match any
> authorised key?

I don't think so, it'll just ignore them.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-04 18:38 ` Andreas Enge
  2021-05-04 19:29   ` Christopher Baines
@ 2021-05-06 16:26   ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2021-05-06 16:26 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Hallo!

Andreas Enge <andreas@enge.fr> skribis:

> Relatedly, and this may already be documented: How do I know where the
> packages are downloaded from?
> $ guix package -u
> substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>  imagemagick-6.9.12-4-doc  4.6MiB                    1.6MiB/s 00:01 [#####             ]  32.5%
> Before the print format overhaul, the full URL would be printed.

You can use ‘-v2’ to see URLs:

  https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html

> Does the guix daemon complain when substitute URLs do not match any
> authorised key?

No, it just ignores it.

Ludo’.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-04 19:22       ` Christopher Baines
@ 2021-05-11 20:18         ` Ludovic Courtès
  0 siblings, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2021-05-11 20:18 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> A QA entry point like you proposed in the thread you mentioned¹ could
>> certainly help.  A reproducibility entry point would be nice too.  A
>> package browser for guix.gnu.org like the one Danjela worked on would be
>> great too, possibly with version browsing facilities.  And Guix Weekly
>> News!  And the security tracker!  :-)
>>
>> ¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html
>>
>> It seems to me that all that hard work is already done and what I
>> describe above are rather low-hanging fruits.
>>
>> Taking Conway’s law into account, we may find it easier to recruit if as
>> much as possible takes place here, things get deployed behind
>> *.guix.gnu.org, and relevant bits are made part of Guix proper.  And
>> also, we must regularly advertise progress; one blog post in all of the
>> Guix Data Service’s lifetime is not enough.  :-)
>>
>> Thoughts?
>
> This doesn't really relate to the subject of substitutes.

Indeed, I slid off-topic, but it’s all there in the big picture.  :-)

> Some of the things you mention do relate to work I'm trying to progress
> though. I'm still working on automated patch (plus branch) testing, and
> I think having a simple overview of patch+branch states is hopefully
> something that I'll get to at some point.
>
> On the subject of the patch testing stuff, that isn't under
> .guix.gnu.org and I haven't written a blog post yet. I can't see Cuirass
> starting to test patches, but then I wouldn't have predicted it would be
> managing builds across multiple machines. Maybe there are some risks
> with the patch testing work that I haven't done enough/the right stuff
> to mitigate.
>
> I've also got too many things in progress at the moment, with the
> combination of work on substitutes (I hope to implement things set out
> in [1] some point soon), Outreachy, and the security related work that
> I'm trying to start, I need to "finish" some things before starting new
> ones or going back to unfinished things.
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Yeah, I do think polishing, deploying, and advertising some of these
things that are already 80% ready would be a great booster for all of
us.  To help recruit people, we can also deploy at guix.gnu.org things
that are not complete yet (the package browser, Guix Weekly News, the QA
entry point, etc.); it’s always easier once the thing is palatable.

HTH!

Ludo’.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Bringing substitutes from the Guix Build Coordinator to users
  2021-05-04 19:29   ` Christopher Baines
@ 2021-05-12 22:58     ` Christopher Baines
  0 siblings, 0 replies; 10+ messages in thread
From: Christopher Baines @ 2021-05-12 22:58 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 3126 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Andreas Enge <andreas@enge.fr> writes:
>
>> Hello Chris,
>>
>> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>>> I think there are some benefits for using the Guix Build Coordinator to
>>> build things for substitutes, and it would be good to work out how to
>>> get those benefits to users of Guix generally.
>>
>> my question is a bit tangential, but how can we get the substitutes that
>> the build coordinator puts on bayfront? I still have bayfront.guix.info
>> in my list of substitute servers, but does it use the same signing key
>> as before? It would be nice to document what a user should do. In my
>> opinion, a second build farm would be a useful thing, in case one server
>> goes down, or needs to be updated, for instance. (And then it allows for
>> reproducibility checks.)
>
> So, bayfront is currently serving substitutes build through the Guix
> Build Coordinator, the signing key is the same as it was before.
>
> It's only been building things at pace since yesterday though, so it'll
> be a few days at least until it's caught up.
>
> I agree that having multiple independent substitute providers would be
> nice, and getting bayfront working well is a step towards that.

Continuing on the subject of bayfront, it seems like the most feasible
way to get substitutes build through the Guix Build Coordinator to users
might be through it. At least there hasn't been any other leads yet from
this thread.

Luckily, from a thread ("Machine things") on guix-sysadmin [1] a number
of months ago, the suggestion of using the Guix Build Coordinator on
bayfront already came up, so it's actually been running for around a
month, and making progress building things for part of that.

1: https://lists.gnu.org/mailman/private/guix-sysadmin/2021-February/003412.html

Getting some benefit from the substitutes will require a few things
though, firstly getting things in order such that a good proportion and
range of substitutes are available, and secondly amending the default
configuration to include bayfront.

On the default configuration point, what are the prerequsites for that?
Beyond having some substitutes to offer, is there any particular
criteria to consider, perhaps about the machines involved?

On the subject of the machines involved, currently there's:

 - x86_64-linux + i686-linux
   - bayfront (coordinator, agent and serving substitutes)
   - harbourfront
   - milano-guix-1
 - aarch64-linux + armhf-linux
   - monokuma

I'd like to get some childhurd VMs running, they can go on the x86_64
machines, and hopefully there will be enough resources to keep up with
x86_64-linux, i686-linux, while doing some i586-gnu builds. If I have
some luck, I might also be able to get some powerpc64 hardware, and I do
have two Raspberry Pi 4's, but they're not running Guix as the OS yet,
so it would be good to work out if they're suitable, or better left to
other tasks.

There's a few intertangled things, but the main question is, if bayfront
can be a source of substitutes, what's the path to actually have that
benefit users?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-05-12 22:59 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-01 18:56 Bringing substitutes from the Guix Build Coordinator to users Christopher Baines
2021-05-02 21:51 ` Ludovic Courtès
2021-05-03 10:30   ` Christopher Baines
2021-05-04  8:27     ` Ludovic Courtès
2021-05-04 19:22       ` Christopher Baines
2021-05-11 20:18         ` Ludovic Courtès
2021-05-04 18:38 ` Andreas Enge
2021-05-04 19:29   ` Christopher Baines
2021-05-12 22:58     ` Christopher Baines
2021-05-06 16:26   ` Ludovic Courtès

unofficial mirror of guix-devel@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git