Tooling for branch workflows

Tooling for branch workflows

[Andreas Enge]

Hello all,

the title says it all, I wish to share some conclusions from working on
the core-updates merge. Clearly our tooling could be improved for the task;
there was some flying by night without instruments, and in the end I
merged the branch without being really able to tell how it compared
to master... (You may also blame it partially on my lack of patience.)
Having feature branches may or may not make things a bit easier, but it
will definitely not solve the problems.
This mail is also of course a bit politically sensitive: It may look like
I am complaining about other people's work, who are volunteers and do what
they can, without offering to work on the code myself. So as a preamble,
let me express my gratitude to the few people who have been working
tirelessly on our tooling and contributing to our infrastructure, without
whom big code changes like we did on core-updates (and now on feature
branches) would simply be impossible; their work is vital to the project
and often not very visible. If I am critical, it is not to diminish their
work, but to discuss about a positive path forward; and I hope more people
will find the motivation to do infrastructure work, which I think will be
decisive for the success of Guix (together with policy and organisational
questions).

We have two build farms, berlin and bordeaux (which is a good thing for
checking reproducibility and for redundancy, but maybe a bit of a problem
concerning hardware requirements for "exotic" architectures), running
two different CI projects, cuirass and the Guix build coordinator (gbc in
the following); both have a very low bus factor (1 to 2?), and it would be
nice to get more people onboard. For this, more documentation would be
helpful. Both have pros and cons, and are architectured quite differently,
so I do not know whether convergence is achievable.

I ended up relying mostly on cuirass for reasons I do not completely
remember any more. The dashboard with its green and red dots is a very
useful tool compared to lists of builds, which become unusable with over
20000 packages. The bigger build power on bordeaux is helpful, and I found
the web interface of gbc a bit slow and down a bit too often. With this
experience, I just filed three wishlist bugs for cuirass:
- Topological sorting in cuirass
  https://issues.guix.gnu.org/63412
  The lack of ordering the builds is a big problem wasting a lot of build
  power; it is solved in gbc and, I think, the reason why the bordeaux
  build farm fares better for aarch64 with fewer machines.
  I would tag this as "important".
- Evaluation comparison on cuirass
  https://issues.guix.gnu.org/63414
  Without being able to compare a branch to master, it is difficult to
  decide whether one should merge. This is sort of solved in gbc, but so
  far the bordeaux build farm has been used more for QA of single patches
  (or a short list of patches featuring in a single issue) than for building
  complete branches.
- Stop and restart builds in cuirass
  https://issues.guix.gnu.org/63413
  Manual intervention is not easy in cuirass (I spent hours clicking on
  "restart" or using the REST API with a shell script through wget, which
  resulted in my IP being banned as a DoS suspect...); and to my knowledge,
  there is no web interface for doing so in gbc. In both systems one can
  probably tinker with the underlying databases, but this also does not
  qualify as "easy".

gdb just got a very nice feature on "blocking builds":
   https://data.guix.gnu.org/revision/8f92dfd9ae7ac491ab7fb4b425799a8c909708a8/blocking-builds?system=aarch64-linux&target=none&limit_results=50
As I understand them, these are the "first failures", derivations all
inputs of which are available, but which fail themselves; so they give
the place where work is needed (and repairs will immediately make
a difference). Once the topological sorting in cuirass is sorted out,
these should be the builds marked as "Failed" (as opposed to "Failed (dependency)"),
so with the first issue above handled, they could easily be shown by
cuirass as well.

This was a long message to say "I filed three bugs", but maybe it can be
the starting point to discuss more items on how to go forward with our
build and CI infrastructure.

Andreas

Re: Tooling for branch workflows

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4532 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> the title says it all, I wish to share some conclusions from working on
> the core-updates merge. Clearly our tooling could be improved for the task;
> there was some flying by night without instruments, and in the end I
> merged the branch without being really able to tell how it compared
> to master... (You may also blame it partially on my lack of patience.)
> Having feature branches may or may not make things a bit easier, but it
> will definitely not solve the problems.
> This mail is also of course a bit politically sensitive: It may look like
> I am complaining about other people's work, who are volunteers and do what
> they can, without offering to work on the code myself. So as a preamble,
> let me express my gratitude to the few people who have been working
> tirelessly on our tooling and contributing to our infrastructure, without
> whom big code changes like we did on core-updates (and now on feature
> branches) would simply be impossible; their work is vital to the project
> and often not very visible. If I am critical, it is not to diminish their
> work, but to discuss about a positive path forward; and I hope more people
> will find the motivation to do infrastructure work, which I think will be
> decisive for the success of Guix (together with policy and organisational
> questions).

I too would like to see improved tooling for managing changes. I've been
working on the qa-frontpage for smaller changes, but I think this can be
extended to large changes too.

> We have two build farms, berlin and bordeaux (which is a good thing for
> checking reproducibility and for redundancy, but maybe a bit of a problem
> concerning hardware requirements for "exotic" architectures), running
> two different CI projects, cuirass and the Guix build coordinator (gbc in
> the following); both have a very low bus factor (1 to 2?), and it would be
> nice to get more people onboard. For this, more documentation would be
> helpful. Both have pros and cons, and are architectured quite differently,
> so I do not know whether convergence is achievable.

This is something I'd like to see too. I've been thinking about this
already and want to spend more time working on at least making it easier
for more people to get involved. Let me know if you have any specific
requests!

> I ended up relying mostly on cuirass for reasons I do not completely
> remember any more. The dashboard with its green and red dots is a very
> useful tool compared to lists of builds, which become unusable with over
> 20000 packages. The bigger build power on bordeaux is helpful, and I found
> the web interface of gbc a bit slow and down a bit too often. With this
> experience, I just filed three wishlist bugs for cuirass:
> - Topological sorting in cuirass
>   https://issues.guix.gnu.org/63412
>   The lack of ordering the builds is a big problem wasting a lot of build
>   power; it is solved in gbc and, I think, the reason why the bordeaux
>   build farm fares better for aarch64 with fewer machines.
>   I would tag this as "important".
> - Evaluation comparison on cuirass
>   https://issues.guix.gnu.org/63414
>   Without being able to compare a branch to master, it is difficult to
>   decide whether one should merge. This is sort of solved in gbc, but so
>   far the bordeaux build farm has been used more for QA of single patches
>   (or a short list of patches featuring in a single issue) than for building
>   complete branches.
> - Stop and restart builds in cuirass
>   https://issues.guix.gnu.org/63413
>   Manual intervention is not easy in cuirass (I spent hours clicking on
>   "restart" or using the REST API with a shell script through wget, which
>   resulted in my IP being banned as a DoS suspect...); and to my knowledge,
>   there is no web interface for doing so in gbc. In both systems one can
>   probably tinker with the underlying databases, but this also does not
>   qualify as "easy".

There's intentionally no concept of restarting builds in the build
coordinator, however unlike Cuirass, it does support having multiple
builds for the same derivation.

So, to try to build something again you just submit another build for
the derivation, which can be as simple as running this on bayfront:

  guix-build-coordinator build /gnu/store/foo.drv

Having to have SSH access to bayfront to do this isn't ideal though, so
we can expose some kind of authenticated interface to do this, maybe
through the new bffe or the qa-frontpage.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Tooling for branch workflows

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

Hi Chris and Andreas,

Christopher Baines <mail@cbaines.net> writes:

> I too would like to see improved tooling for managing changes. I've been
> working on the qa-frontpage for smaller changes, but I think this can be
> extended to large changes too.
>
> [...]
>
> This is something I'd like to see too. I've been thinking about this
> already and want to spend more time working on at least making it easier
> for more people to get involved. Let me know if you have any specific
> requests!

Maybe it would be a good time to create an infrastructure/CI team, and
set some clear goals for it?  I think it might motivate people (me
included) to get more involved in GBC development if there is something
to build towards.

Also, I kind of agree with Andreas about the bus factor of Cuirass and
GBC, and wonder whether it makes sense to duplicate development effort
across both.  It seems to me that Cuirass is working properly as a
generic CI tool, but it's not super well adapted to Guix, something
which the GBC seems to be much better at.  Is replacing Cuirass by GBC
on berlin for Guix CI something that is a future goal, or is it
understood that Cuirass will be running there for the foreseeable
future?

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Tooling for branch workflows

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

On Wed, May 10, 2023 at 01:39:05PM +0200, Andreas Enge wrote:
> Hello all,
> 
> the title says it all, I wish to share some conclusions from working on
> the core-updates merge. Clearly our tooling could be improved for the task;
> there was some flying by night without instruments, and in the end I
> merged the branch without being really able to tell how it compared
> to master... (You may also blame it partially on my lack of patience.)
> Having feature branches may or may not make things a bit easier, but it
> will definitely not solve the problems.

I have to agree with this sentiment. I merged the rust-team branch with
the gut-feeling that nothing new was broken, but I wasn't really sure
how to compare the two.

> This mail is also of course a bit politically sensitive: It may look like
> I am complaining about other people's work, who are volunteers and do what
> they can, without offering to work on the code myself. So as a preamble,
> let me express my gratitude to the few people who have been working
> tirelessly on our tooling and contributing to our infrastructure, without
> whom big code changes like we did on core-updates (and now on feature
> branches) would simply be impossible; their work is vital to the project
> and often not very visible. If I am critical, it is not to diminish their
> work, but to discuss about a positive path forward; and I hope more people
> will find the motivation to do infrastructure work, which I think will be
> decisive for the success of Guix (together with policy and organisational
> questions).

I too wish to thank everyone for all the work they have done. I would
especially like to thank those who work on cuirass and the Guix build
coordinator, neither of which I've worked on but which are fundamental
underpinnings of Guix.


-- 
Efraim Flashner   <efraim@flashner.co.il>   פלשנר אפרים
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Tooling for branch workflows

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]


Josselin Poiret <dev@jpoiret.xyz> writes:

> Hi Chris and Andreas,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> I too would like to see improved tooling for managing changes. I've been
>> working on the qa-frontpage for smaller changes, but I think this can be
>> extended to large changes too.
>>
>> [...]
>>
>> This is something I'd like to see too. I've been thinking about this
>> already and want to spend more time working on at least making it easier
>> for more people to get involved. Let me know if you have any specific
>> requests!
>
> Maybe it would be a good time to create an infrastructure/CI team, and
> set some clear goals for it?  I think it might motivate people (me
> included) to get more involved in GBC development if there is something
> to build towards.

There's the core team, although maybe this doesn't cover this.

However, framing is important and we don't really want
infrastructure/CI, they're means rather than goals.

I'd frame any team stuff around what's wanted like substitutes and up to
date packages.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Tooling for branch workflows

[Ludovic Courtès]

Hello,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> Maybe it would be a good time to create an infrastructure/CI team, and
> set some clear goals for it?  I think it might motivate people (me
> included) to get more involved in GBC development if there is something
> to build towards.

Good idea!

> Also, I kind of agree with Andreas about the bus factor of Cuirass and
> GBC, and wonder whether it makes sense to duplicate development effort
> across both.  It seems to me that Cuirass is working properly as a
> generic CI tool, but it's not super well adapted to Guix, something
> which the GBC seems to be much better at.  Is replacing Cuirass by GBC
> on berlin for Guix CI something that is a future goal, or is it
> understood that Cuirass will be running there for the foreseeable
> future?

Cuirass started as a replacement for Hydra, which is what Nix has been
using, so it’s very similar.  What’s nice is that it’s relatively easy
to set it up to get a bunch of packages built continuously, which is the
major use case for people out there (beyond the project’s own build
farms).

Because the Coordinator itself does nothing but build derivations, one
has to set up some other tool such as the Data Service for that use
case, which may be more difficult (I’m probably biased because I’ve only
set up Cuirass instances).

OTOH, the modularity that the Coordinator/Data Service approach offers
is appealing to me.  It can make it more approachable: one can hack the
Coordinator without worrying about the Data Service.

Anyhow, my suggestion would be to make our infrastructure more robust.
I had to hot-fix Cuirass last week and it was cumbersome because the
test suite only covers basic functionality.  I believe the Coordinator
lacks a test suite, which means it may be harder to modify it (harder to
tell if you broke something) and to get confidence.

So my personal wish list :-) would be improved testing and
documentation.

A longer-term wish: using Spritely Goblins and a modular actor
architecture for those services so we can interact with them through a
rich programming interface and distribute appropriate “capabilities” to
designated developers (a (re)build capability, an “evaluate branch”
capability, etc.).

Ludo’.