unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* The Guix Build Coordinator in 2021
@ 2021-02-09 20:30 Christopher Baines
  2021-02-10 10:58 ` Mathieu Othacehe
  0 siblings, 1 reply; 4+ messages in thread
From: Christopher Baines @ 2021-02-09 20:30 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

Hey!

Near the beginning of 2020, things changed such that I suddenly had some
time, and some of that time I spend putting idea's I'd had for a while
around building derivations, including across multiple machines, in to
practice [1].

1: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

Looking back now from 2021, I think it's been pretty successful.

In terms of building things for substitutes, the guix.cbaines.net [2]
site I setup for testing has generally had higher substitute
availability than ci.guix.gnu.org, at least for x86_64-linux, and this
is with far less compute resources.

2: http://guix.cbaines.net/

It's also allowed for massive leaps forward towards being able to
meaningfully test patches. Affected packages and system tests are now
being built, and the results of those builds are being sent to the
relevant Guix Data Service instance.

Up until recently, I've been trying to add more features. I've still got
more features in mind, but I also want to neaten up some things and pay
back some of the technical debt that's been accrued.

I've added a Roadmap section to the README file [3] if you're interested
in the specifics.

3: https://git.cbaines.net/guix/build-coordinator/about/#outline-container-orge5f01b3

Also, given that the Guix Build Coordinator is capably doing what it was
designed to do, there are problems that are now in a good position to
tackle. Things like trying to optimise substitute delivery over HTTP [4]
or otherwise, and making user interface improvements around
patchwork.cbaines.net [5] so that it's better at getting the right
information to the right people.

4: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html
5: https://patchwork.cbaines.net/

If you have any comments, questions, or are interested in getting
involved, please let me know!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: The Guix Build Coordinator in 2021
  2021-02-09 20:30 The Guix Build Coordinator in 2021 Christopher Baines
@ 2021-02-10 10:58 ` Mathieu Othacehe
  2021-02-10 20:07   ` Christopher Baines
  2021-02-10 22:29   ` Ludovic Courtès
  0 siblings, 2 replies; 4+ messages in thread
From: Mathieu Othacehe @ 2021-02-10 10:58 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel


Hello Chris,

> Near the beginning of 2020, things changed such that I suddenly had some
> time, and some of that time I spend putting idea's I'd had for a while
> around building derivations, including across multiple machines, in to
> practice [1].

With the Guix Build Coordinator, you made an impressive work, probably
based on the observation that the "guix-daemon" and its offloading
mechanism was too limited.

We are now in a situation where our continuous integration system,
while performing better and better is getting out of hand. Here are the
different software I'm keeping track of:

* Cuirass, deployed on ci.guix.gnu.org
* The Guix Build Coordinator, deployed on guix.cbaines.net
* The Guix Data Service, deployed on data.guix.gnu.org
* Patchwork, deployed on patchwork.cbaines.net

All those services have databases, using different DBMS on different
servers. Those databases are sometimes overlapping, in the same way as
some of the features of those software.

In particular I feel that what's implemented in the Guix Build
Coordinator can be seen as a subset of Cuirass functionalities. As you
know, I'm reluctant to the idea of connecting Cuirass to the Guix Build
Coordinator, because most of Cuirass PostgreSQL database content would
be duplicated in the SQLite database of the GBC.

On the other hand, maintaining those two software in separate ways seems
like a huge waste of time given the very limited number of people
contributing the maintenance of the CI system.

Furthermore, some of the features we are implementing here, should be
part of the "guix-daemon" itself, which makes me think that we should
not place too much effort in their development.

My proposition would be to make a listing of both Cuirass and the GBC
features, and see how we could merge them. By maintaining a single
software, with a single database, running on the same server, we could
spare some efforts, and quickly converge towards a better CI.

The new Cuirass architecture and the switch to PostgreSQL, make the
software way more modular, and should allow us to add new
functionalities without too much trouble.

What do you think of that proposition?

Thanks,

Mathieu


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: The Guix Build Coordinator in 2021
  2021-02-10 10:58 ` Mathieu Othacehe
@ 2021-02-10 20:07   ` Christopher Baines
  2021-02-10 22:29   ` Ludovic Courtès
  1 sibling, 0 replies; 4+ messages in thread
From: Christopher Baines @ 2021-02-10 20:07 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2756 bytes --]


Mathieu Othacehe <othacehe@gnu.org> writes:

> We are now in a situation where our continuous integration system,
> while performing better and better is getting out of hand. Here are the
> different software I'm keeping track of:
>
> * Cuirass, deployed on ci.guix.gnu.org
> * The Guix Build Coordinator, deployed on guix.cbaines.net
> * The Guix Data Service, deployed on data.guix.gnu.org
> * Patchwork, deployed on patchwork.cbaines.net
>
> All those services have databases, using different DBMS on different
> servers. Those databases are sometimes overlapping, in the same way as
> some of the features of those software.
>
> In particular I feel that what's implemented in the Guix Build
> Coordinator can be seen as a subset of Cuirass functionalities. As you
> know, I'm reluctant to the idea of connecting Cuirass to the Guix Build
> Coordinator, because most of Cuirass PostgreSQL database content would
> be duplicated in the SQLite database of the GBC.
>
> On the other hand, maintaining those two software in separate ways seems
> like a huge waste of time given the very limited number of people
> contributing the maintenance of the CI system.
>
> Furthermore, some of the features we are implementing here, should be
> part of the "guix-daemon" itself, which makes me think that we should
> not place too much effort in their development.
>
> My proposition would be to make a listing of both Cuirass and the GBC
> features, and see how we could merge them. By maintaining a single
> software, with a single database, running on the same server, we could
> spare some efforts, and quickly converge towards a better CI.
>
> The new Cuirass architecture and the switch to PostgreSQL, make the
> software way more modular, and should allow us to add new
> functionalities without too much trouble.
>
> What do you think of that proposition?

I think there's definitely opportunity here, but I'd suggest first
looking at the situation in abstract, ignoring the existing software,
and just focus on what a good situation would be regarding "CI".

With consensus about direction, it should be much easier to make
decisions about the software involved.

Using the "CI" term to describe a system isn't particularly clear to me,
especially in the context of Guix, so I'm going to use different
terms. In my mind at least, I'm currently thinking about two things,
testing patches and non-master branches, and building substitutes for
the master branch and making these available. The direction I've been
looking in also is to treat those two things as quite separate concerns
(which doesn't exclude using the same software for them).

What in your mind would a good "CI system" for Guix do? I'm asking this
in the sense of responsibilties.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: The Guix Build Coordinator in 2021
  2021-02-10 10:58 ` Mathieu Othacehe
  2021-02-10 20:07   ` Christopher Baines
@ 2021-02-10 22:29   ` Ludovic Courtès
  1 sibling, 0 replies; 4+ messages in thread
From: Ludovic Courtès @ 2021-02-10 22:29 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: guix-devel

Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

> We are now in a situation where our continuous integration system,
> while performing better and better is getting out of hand. Here are the
> different software I'm keeping track of:
>
> * Cuirass, deployed on ci.guix.gnu.org
> * The Guix Build Coordinator, deployed on guix.cbaines.net
> * The Guix Data Service, deployed on data.guix.gnu.org
> * Patchwork, deployed on patchwork.cbaines.net

I can see why one would think things are getting out of hands, or at
least that we have prolific developers.  :-)

Still, these four things play different roles: the Coordinator “just”
build things in a distributed fashion, the Data Service evaluates
channels and stores metadata, while Patchwork does patchy things.

Cuirass is in-between because it evaluates things, distributes builds,
and stores metadata.  It made sense to bundle functionalities like this,
because that’s the subset we’re interested in (build stuff, distribute
builds, provide some metadata useful for QA).

The Coordinator/Data Service split is a nice separation of concerns IMO.
Each is good at doing its job; the combination isn’t quite comparable to
Cuirass because it’s more generic, but similar information and actions
are available.

> All those services have databases, using different DBMS on different
> servers. Those databases are sometimes overlapping, in the same way as
> some of the features of those software.

Maybe it’s OK to have multiple databases?  The daemon has one too.  :-)
It can be seen as an implementation detail.

(Plus, I’m told that microservices are hype.)

> In particular I feel that what's implemented in the Guix Build
> Coordinator can be seen as a subset of Cuirass functionalities. As you
> know, I'm reluctant to the idea of connecting Cuirass to the Guix Build
> Coordinator, because most of Cuirass PostgreSQL database content would
> be duplicated in the SQLite database of the GBC.

Does it really need to be duplicated though?  Maybe Cuirass could keep
links to Coordinator builds, just like it doesn’t duplicate the store
database.

> On the other hand, maintaining those two software in separate ways seems
> like a huge waste of time given the very limited number of people
> contributing the maintenance of the CI system.
>
> Furthermore, some of the features we are implementing here, should be
> part of the "guix-daemon" itself, which makes me think that we should
> not place too much effort in their development.

Yeah, but we need to go incremental IMO.  The daemon rewrite is a bit of
a jump, which is why it hasn’t gone anywhere yet.  In the meantime, I
think it’s OK to daemon-like functionality elsewhere; perhaps that’ll be
reused eventually in the daemon.

Ludo’.


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-02-10 23:06 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-09 20:30 The Guix Build Coordinator in 2021 Christopher Baines
2021-02-10 10:58 ` Mathieu Othacehe
2021-02-10 20:07   ` Christopher Baines
2021-02-10 22:29   ` Ludovic Courtès

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).