Re: 04/09: gnu: mesa: Update to 23.0.3.

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]


guix-commits@gnu.org writes:

> apteryx pushed a commit to branch master
> in repository guix.
>
> commit 0be7838105806819f4586ec9130382a66a22880e
> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
> AuthorDate: Thu May 4 20:12:46 2023 +0000
>
>     gnu: mesa: Update to 23.0.3.
>
>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>     [source]: Remove obsolete patch and update HTTPS url.
>     [arguments]: Enable the crocus gallium driver.
>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>     * gnu/local.mk (dist_patch_DATA): Remove it.
> ---
>  gnu/local.mk                                       |  1 -
>  gnu/packages/gl.scm                                | 14 ++++-------
>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>  3 files changed, 5 insertions(+), 37 deletions(-)

→ guix refresh -l mesa
Building the following 1954 packages would ensure 4257 dependent
packages are rebuilt ...


I know there's been some discussion about changing processes regarding
changes like this that impact lots of packages, but as far as I'm aware,
the documented process hasn't changed yet. So should this have gone to
core-updates, and not been directly pushed to master?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Maxim Cournoyer]

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

> guix-commits@gnu.org writes:
>
>> apteryx pushed a commit to branch master
>> in repository guix.
>>
>> commit 0be7838105806819f4586ec9130382a66a22880e
>> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
>> AuthorDate: Thu May 4 20:12:46 2023 +0000
>>
>>     gnu: mesa: Update to 23.0.3.
>>
>>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>>     [source]: Remove obsolete patch and update HTTPS url.
>>     [arguments]: Enable the crocus gallium driver.
>>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>>     * gnu/local.mk (dist_patch_DATA): Remove it.
>> ---
>>  gnu/local.mk                                       |  1 -
>>  gnu/packages/gl.scm                                | 14 ++++-------
>>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>>  3 files changed, 5 insertions(+), 37 deletions(-)
>
> → guix refresh -l mesa
> Building the following 1954 packages would ensure 4257 dependent
> packages are rebuilt ...
>
>
> I know there's been some discussion about changing processes regarding
> changes like this that impact lots of packages, but as far as I'm aware,
> the documented process hasn't changed yet. So should this have gone to
> core-updates, and not been directly pushed to master?

There isn't currently a core-updates branch, and I need to spend some
time documenting the authorization process for how to create short lived
Cuirass branches.  I think ideally we would have created a
'graphics-team' or similar branch (even the team has yet to be formed)
and let it build.

Seeing the build machines were idling in the European night, I figured I
could get away with it for this time.

But the situation will repeat; I'd like to push some xorg updates that
fix a CVE; we'll nead a 'xorg-team' branch or similar.  Should we create
these branches from the maintenance repository (permanent branches) ?

-- 
Thanks,
Maxim

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3381 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> guix-commits@gnu.org writes:
>>
>>> apteryx pushed a commit to branch master
>>> in repository guix.
>>>
>>> commit 0be7838105806819f4586ec9130382a66a22880e
>>> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
>>> AuthorDate: Thu May 4 20:12:46 2023 +0000
>>>
>>>     gnu: mesa: Update to 23.0.3.
>>>
>>>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>>>     [source]: Remove obsolete patch and update HTTPS url.
>>>     [arguments]: Enable the crocus gallium driver.
>>>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>>>     * gnu/local.mk (dist_patch_DATA): Remove it.
>>> ---
>>>  gnu/local.mk                                       |  1 -
>>>  gnu/packages/gl.scm                                | 14 ++++-------
>>>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>>>  3 files changed, 5 insertions(+), 37 deletions(-)
>>
>> → guix refresh -l mesa
>> Building the following 1954 packages would ensure 4257 dependent
>> packages are rebuilt ...
>>
>>
>> I know there's been some discussion about changing processes regarding
>> changes like this that impact lots of packages, but as far as I'm aware,
>> the documented process hasn't changed yet. So should this have gone to
>> core-updates, and not been directly pushed to master?
>
> There isn't currently a core-updates branch, and I need to spend some
> time documenting the authorization process for how to create short lived
> Cuirass branches.  I think ideally we would have created a
> 'graphics-team' or similar branch (even the team has yet to be formed)
> and let it build.
>
> Seeing the build machines were idling in the European night, I figured I
> could get away with it for this time.

Some build machines may have been idle, but I'm not sure what you mean
by "get away with it"?

While the berlin bulid farm has managed to catch back up for
x86_64-linux and i686-linux within 24 hours, I think these changes
impact other systems as well.

Also, the bordeaux build farm has a lot fewer machines to do these
builds, so while the substitute availability had caught up (and
surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's
going to be several days at least I think before substitute availability
is looking good again.

I was watching the substitute availability recover after the
core-updates merge as I'd like to re-enable testing patches on the
qa-frontpage, but now that'll have to wait some more for all these new
builds to complete.

> But the situation will repeat; I'd like to push some xorg updates that
> fix a CVE; we'll nead a 'xorg-team' branch or similar.  Should we create
> these branches from the maintenance repository (permanent branches) ?

I don't really understand the question, surely the branches would be in
the guix Git repository?

Anyway, package replacements+grafts can be used for security issues so
that shouldn't need to be on a branch as it won't involve lots of
rebuilds.

When it comes to handling changes involving lots of rebuilds though, I
think that this has been and continues to be difficult, but in my mind
that's a reason to slow down and try and work on tooling and processes
to help.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Maxim Cournoyer]

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>>> guix-commits@gnu.org writes:
>>>
>>>> apteryx pushed a commit to branch master
>>>> in repository guix.
>>>>
>>>> commit 0be7838105806819f4586ec9130382a66a22880e
>>>> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
>>>> AuthorDate: Thu May 4 20:12:46 2023 +0000
>>>>
>>>>     gnu: mesa: Update to 23.0.3.
>>>>
>>>>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>>>>     [source]: Remove obsolete patch and update HTTPS url.
>>>>     [arguments]: Enable the crocus gallium driver.
>>>>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>>>>     * gnu/local.mk (dist_patch_DATA): Remove it.
>>>> ---
>>>>  gnu/local.mk                                       |  1 -
>>>>  gnu/packages/gl.scm                                | 14 ++++-------
>>>>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>>>>  3 files changed, 5 insertions(+), 37 deletions(-)
>>>
>>> → guix refresh -l mesa
>>> Building the following 1954 packages would ensure 4257 dependent
>>> packages are rebuilt ...
>>>
>>>
>>> I know there's been some discussion about changing processes regarding
>>> changes like this that impact lots of packages, but as far as I'm aware,
>>> the documented process hasn't changed yet. So should this have gone to
>>> core-updates, and not been directly pushed to master?
>>
>> There isn't currently a core-updates branch, and I need to spend some
>> time documenting the authorization process for how to create short lived
>> Cuirass branches.  I think ideally we would have created a
>> 'graphics-team' or similar branch (even the team has yet to be formed)
>> and let it build.
>>
>> Seeing the build machines were idling in the European night, I figured I
>> could get away with it for this time.
>
> Some build machines may have been idle, but I'm not sure what you mean
> by "get away with it"?

I meant that I believed there was enough capacity to process the 4K
rebuilds (per architecture) in a way that wouldn't negatively affect
users too much.

> While the berlin bulid farm has managed to catch back up for
> x86_64-linux and i686-linux within 24 hours, I think these changes
> impact other systems as well.
>
> Also, the bordeaux build farm has a lot fewer machines to do these
> builds, so while the substitute availability had caught up (and
> surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's
> going to be several days at least I think before substitute availability
> is looking good again.
>
> I was watching the substitute availability recover after the
> core-updates merge as I'd like to re-enable testing patches on the
> qa-frontpage, but now that'll have to wait some more for all these new
> builds to complete.

Hm, sorry about that.  Cuirass seems to have mostly caught up already
(was 64% before, 62% now for the master specification).

>> But the situation will repeat; I'd like to push some xorg updates that
>> fix a CVE; we'll nead a 'xorg-team' branch or similar.  Should we create
>> these branches from the maintenance repository (permanent branches) ?
>
> I don't really understand the question, surely the branches would be in
> the guix Git repository?

Yes, the branch would be in the Guix repository, but I meant regarding
the Cuirass specifications affecting which branches it builds; sorry for
being unclear.

> Anyway, package replacements+grafts can be used for security issues so
> that shouldn't need to be on a branch as it won't involve lots of
> rebuilds.

For this case I think so yes, since it's a patch-level update that
should be safe.

> When it comes to handling changes involving lots of rebuilds though, I
> think that this has been and continues to be difficult, but in my mind
> that's a reason to slow down and try and work on tooling and processes
> to help.

One of the things that has been bothered me has been the lack of
documentation/tooling to recreate TLS user certificates for Cuirass so
that I can configure branches via its web interface again, or retry
failed builds.  I'm currently working on documenting (in Cuirass's
manual) a script Ricardo's made for that task.

But building lots of packages will still require a lot of processing
power, probably more so when it happens in focused team branches
compared to grouped together as it used to be the case for
e.g. core-updates.

-- 
Thanks,
Maxim

Feature branches (was: 04/09: gnu: mesa: Update to 23.0.3)

[Andreas Enge]

Hello,

indeed someone™ should update the documentation to describe the new
process. Probably we should agree on one before doing that as well...
In principle all big updates should go through a feature branch now.

However, this does not solve the problem of limited build power in our
two build farms. Having feature branches that regroup several related
changes should help in not rebuilding too often. In principle it could
also be okay to regroup unrelated changes (mesa and gcc, for instance),
as long as responsibilities are clear. It should be known who is going
to act on what when breakages occur in the branch. And I think there
should be some kind of "branch manager" and a time frame for the merge
so that the branches are short lived. The goal is to avoid the core-updates
experience of random commits being dropped in the same place, while
hoping that someone at some later point will sort it all out.

So how about this suggestion:
A feature branch is created upon request by a team, with a branch manager
designated by the team. It is accompanied by a short description of what
the branch is supposed to achieve, and in which timeframe.
The branch manager has the responsibility to communicate regularly with
guix-devel on the state of the branch and on what remains to be done, and
requests to merge the branch to master once it is ready, and to subsequently
delete the branch.

This does not yet explain how the branches interact with continuous
integration. The branch manager may or may not have commit rights and may
or may not be able to create specifications for cuirass, so the full
process should take this into account.

As written in a different thread, right now I would also suggest to first
build the branch only on x86 and powerpc to not overload our few arm
machines, but this is a technical detail.

Andreas

Re: Feature branches

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> indeed someone™ should update the documentation to describe the new
> process. Probably we should agree on one before doing that as well...
> In principle all big updates should go through a feature branch now.
>
> However, this does not solve the problem of limited build power in our
> two build farms. Having feature branches that regroup several related
> changes should help in not rebuilding too often. In principle it could
> also be okay to regroup unrelated changes (mesa and gcc, for instance),
> as long as responsibilities are clear. It should be known who is going
> to act on what when breakages occur in the branch. And I think there
> should be some kind of "branch manager" and a time frame for the merge
> so that the branches are short lived. The goal is to avoid the core-updates
> experience of random commits being dropped in the same place, while
> hoping that someone at some later point will sort it all out.
>
> So how about this suggestion:
> A feature branch is created upon request by a team, with a branch manager
> designated by the team. It is accompanied by a short description of what
> the branch is supposed to achieve, and in which timeframe.
> The branch manager has the responsibility to communicate regularly with
> guix-devel on the state of the branch and on what remains to be done, and
> requests to merge the branch to master once it is ready, and to subsequently
> delete the branch.
>
> This does not yet explain how the branches interact with continuous
> integration. The branch manager may or may not have commit rights and may
> or may not be able to create specifications for cuirass, so the full
> process should take this into account.
>
> As written in a different thread, right now I would also suggest to first
> build the branch only on x86 and powerpc to not overload our few arm
> machines, but this is a technical detail.

I like your above ideas; I'd like to add:

- I'd make the team branches permanent; e.g. the 'gnome-team' branch
  would always exist, and get synced periodically to master (when enough
  built/deemed stable).  This should reduce the overhead of constantly
  having to adjust the Cuirass specification jobs and serve as an
  integration point for the team (the Cuirass job specs would be defined
  declaratively in the guix-maintenance repository).

- branches judged too experimental to be merged into their team branch
  could be branched from it, with a name such as 'gnome-team/feature-x'
  to make it obvious where it should be merged when deemed ready.
  Cuirass job specifications for such short lived branches would be
  created manually using the Cuirass web interface (users need to be
  authorized as can be done following
  https://issues.guix.gnu.org/63375).

I don't think team branches should be merged together at any point,
ideally, to avoid loosing the benefits of feature branches (limited
scope).

-- 
Thanks,
Maxim

Re: Feature branches

[Andreas Enge]

Hello,

Am Mon, May 08, 2023 at 01:01:05PM -0400 schrieb Maxim Cournoyer:
> - I'd make the team branches permanent; e.g. the 'gnome-team' branch
>   would always exist, and get synced periodically to master (when enough
>   built/deemed stable).  This should reduce the overhead of constantly
>   having to adjust the Cuirass specification jobs and serve as an
>   integration point for the team (the Cuirass job specs would be defined
>   declaratively in the guix-maintenance repository).

I would argument the other way round :)  For instance, I would now remove
the rust-team branch so that it is clear that currently there is no work
on rust. As soon as there is again, the team could spin off a new branch
from master. This would avoid having branches around of which we do not
know any more what they contain, and whether they contain unmerged changes.

$ git branch -a | grep rust
  remotes/origin/rekados-rust-queue
  remotes/origin/rust-team
  remotes/origin/wip-cross-built-rust
  remotes/origin/wip-rekado-rust-team
  remotes/origin/wip-rust

Can any of these be removed? Or brought up to shape?

Notice that this is independent of the cuirass specifications (I think).
I think we could keep the specifications on cuirass, but am not totally
sure what happens when the branch does not exist. Probably nothing. And
then it should be picked up again once it is recreated.

> - branches judged too experimental to be merged into their team branch
>   could be branched from it, with a name such as 'gnome-team/feature-x'
>   to make it obvious where it should be merged when deemed ready.
>   Cuirass job specifications for such short lived branches would be
>   created manually using the Cuirass web interface (users need to be
>   authorized as can be done following
>   https://issues.guix.gnu.org/63375).
> I don't think team branches should be merged together at any point,
> ideally, to avoid loosing the benefits of feature branches (limited
> scope).

Agreed, if we start branching from branches and merging back, we will
probably lose overview.

With my take of not having permanent team branches, it might not even be
necessary to branch from team branches.

Andreas

Re: Feature branches

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> Am Mon, May 08, 2023 at 01:01:05PM -0400 schrieb Maxim Cournoyer:
>> - I'd make the team branches permanent; e.g. the 'gnome-team' branch
>>   would always exist, and get synced periodically to master (when enough
>>   built/deemed stable).  This should reduce the overhead of constantly
>>   having to adjust the Cuirass specification jobs and serve as an
>>   integration point for the team (the Cuirass job specs would be defined
>>   declaratively in the guix-maintenance repository).
>
> I would argument the other way round :)  For instance, I would now remove
> the rust-team branch so that it is clear that currently there is no work
> on rust. As soon as there is again, the team could spin off a new branch
> from master. This would avoid having branches around of which we do not
> know any more what they contain, and whether they contain unmerged changes.

My worry here would be to introduce the overhead of having to
continually adjust the Cuirass job specs either via the web interface or
declaratively via git.  But perhaps it's already possible to have it
build all the branches matching a regexp, such as '-team$' ?  That would
solve the issue.

> $ git branch -a | grep rust
>   remotes/origin/rekados-rust-queue
>   remotes/origin/rust-team
>   remotes/origin/wip-cross-built-rust
>   remotes/origin/wip-rekado-rust-team
>   remotes/origin/wip-rust
>
> Can any of these be removed? Or brought up to shape?

Feel free to remove 'wip-cross-built-rust'; I don't intend to work on
this anymore (I hit a wall where rust could not be statically built
using cargo, how ironic, it had something to do with the way macros are
implemented in Rust).

> Notice that this is independent of the cuirass specifications (I think).
> I think we could keep the specifications on cuirass, but am not totally
> sure what happens when the branch does not exist. Probably nothing. And
> then it should be picked up again once it is recreated.

We'd have to try, I would assume it may cause errors in Cuirass (it'd
make sense that it let you know: hey, you've defined a job spec that
won't build anything!)

-- 
Thanks,
Maxim

Re: Feature branches

[Andreas Enge]

Am Wed, May 10, 2023 at 09:23:11AM -0400 schrieb Maxim Cournoyer:
> Feel free to remove 'wip-cross-built-rust'

Done!

> We'd have to try, I would assume it may cause errors in Cuirass (it'd
> make sense that it let you know: hey, you've defined a job spec that
> won't build anything!)

I just deleted the rust-team branch, we will see what happens.

Andreas

Re: Feature branches

[Andreas Enge]

Am Wed, May 10, 2023 at 03:40:31PM +0200 schrieb Andreas Enge:
> I just deleted the rust-team branch, we will see what happens.

Apparently nothing. Here is an excerpt of /var/log/cuirass.log:
2023-05-10 15:44:10 Fetching channels for spec 'gnuzilla-updates'.
2023-05-10 15:44:19 Fetching channels for spec 'guile'.
2023-05-10 15:44:28 Fetching channels for spec 'guix'.
2023-05-10 15:44:28 evaluating spec 'guile'
2023-05-10 15:44:36 Fetching channels for spec 'kernel-updates'.
2023-05-10 15:44:45 Fetching channels for spec 'master'.
2023-05-10 15:44:54 Fetching channels for spec 'rust-team'.
2023-05-10 15:45:02 Fetching channels for spec 'shepherd'.
2023-05-10 15:45:03 Fetching channels for spec 'source'.
2023-05-10 15:45:12 Fetching channels for spec 'tex-team'.
2023-05-10 15:45:20 next evaluation in 300 seconds
2023-05-10 15:47:36 evaluation 459246 for 'guile' completed
2023-05-10 15:47:36 evaluation 459246 registered 0 new derivations
2023-05-10 15:47:40 evaluation 459201 for 'master' completed
2023-05-10 15:47:40 evaluation 459201 registered 136 new derivations
2023-05-10 15:50:20 Fetching channels for spec 'gnuzilla-updates'.
...

Maybe this is a cuirass bug, maybe it is a feature, but it seems to
survive without problem to an unavailable branch :)

Andreas

Re: Feature branches

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Am Wed, May 10, 2023 at 03:40:31PM +0200 schrieb Andreas Enge:
>> I just deleted the rust-team branch, we will see what happens.
>
> Apparently nothing. Here is an excerpt of /var/log/cuirass.log:
> 2023-05-10 15:44:10 Fetching channels for spec 'gnuzilla-updates'.
> 2023-05-10 15:44:19 Fetching channels for spec 'guile'.
> 2023-05-10 15:44:28 Fetching channels for spec 'guix'.
> 2023-05-10 15:44:28 evaluating spec 'guile'
> 2023-05-10 15:44:36 Fetching channels for spec 'kernel-updates'.
> 2023-05-10 15:44:45 Fetching channels for spec 'master'.
> 2023-05-10 15:44:54 Fetching channels for spec 'rust-team'.
> 2023-05-10 15:45:02 Fetching channels for spec 'shepherd'.
> 2023-05-10 15:45:03 Fetching channels for spec 'source'.
> 2023-05-10 15:45:12 Fetching channels for spec 'tex-team'.
> 2023-05-10 15:45:20 next evaluation in 300 seconds
> 2023-05-10 15:47:36 evaluation 459246 for 'guile' completed
> 2023-05-10 15:47:36 evaluation 459246 registered 0 new derivations
> 2023-05-10 15:47:40 evaluation 459201 for 'master' completed
> 2023-05-10 15:47:40 evaluation 459201 registered 136 new derivations
> 2023-05-10 15:50:20 Fetching channels for spec 'gnuzilla-updates'.
> ...
>
> Maybe this is a cuirass bug, maybe it is a feature, but it seems to
> survive without problem to an unavailable branch :)

Interesting.  Thanks for following up.

-- 
Thanks,
Maxim

Re: Feature branches

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Mon, May 8, 2023 at 9:34 AM Andreas Enge <andreas@enge.fr> wrote:
>
> This does not yet explain how the branches interact with continuous
> integration.

How about requiring prior to merging a feature branch that substitutes
exist for all changed derivations? It would prevent build failures and
preempt local builds, and thereby improve the experience for average
users.

Kind regards
Felix

Re: Feature branches

[Andreas Enge]

Am Mon, May 08, 2023 at 10:15:56AM -0700 schrieb Felix Lechner:
> How about requiring prior to merging a feature branch that substitutes
> exist for all changed derivations? It would prevent build failures and
> preempt local builds, and thereby improve the experience for average
> users.

Taken absolutely, that sounds a bit excessive; but the idea is of course
there, the branch should be built out as much as possible.

It is probably unavoidable that some things stop working going forward,
and I wonder how realistic it is to iron out all problems before a merge.

Andreas

Re: Feature branches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> Am Mon, May 08, 2023 at 10:15:56AM -0700 schrieb Felix Lechner:
>> How about requiring prior to merging a feature branch that substitutes
>> exist for all changed derivations? It would prevent build failures and
>> preempt local builds, and thereby improve the experience for average
>> users.
>
> Taken absolutely, that sounds a bit excessive; but the idea is of course
> there, the branch should be built out as much as possible.
>
> It is probably unavoidable that some things stop working going forward,
> and I wonder how realistic it is to iron out all problems before a merge.

I think the thing next to aim for is at least to be informed about the
effects of any changes.

Knowning the x package breaks on y architecture and deciding to merge
anyway is a lot better (and I think the end goal) than the current
situation of not really knowing what the effects of non-trivial changes
are.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4945 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>>> Seeing the build machines were idling in the European night, I figured I
>>> could get away with it for this time.
>>
>> Some build machines may have been idle, but I'm not sure what you mean
>> by "get away with it"?
>
> I meant that I believed there was enough capacity to process the 4K
> rebuilds (per architecture) in a way that wouldn't negatively affect
> users too much.

That may well be the case, but I see problems using this as
justification for similar actions in the future.

Even if it's unlikely that people use mesa or it's dependencies on
systems other than x86_64-linux and i686-linux, this still adds to the
backlog of builds for other architectures.

Also, while the berlin build farm may be able to build things very
quickly for these systems, I think it's good to try and reduce the churn
where possible by batching together changes (aka
staging/core-updates). Not doing so and just pushing when the build farm
can cope will generate more data to store, more for users to download
and more to build for people who don't use substitutes.

>> While the berlin bulid farm has managed to catch back up for
>> x86_64-linux and i686-linux within 24 hours, I think these changes
>> impact other systems as well.
>>
>> Also, the bordeaux build farm has a lot fewer machines to do these
>> builds, so while the substitute availability had caught up (and
>> surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's
>> going to be several days at least I think before substitute availability
>> is looking good again.
>>
>> I was watching the substitute availability recover after the
>> core-updates merge as I'd like to re-enable testing patches on the
>> qa-frontpage, but now that'll have to wait some more for all these new
>> builds to complete.
>
> Hm, sorry about that.  Cuirass seems to have mostly caught up already
> (was 64% before, 62% now for the master specification).

I think this is a problematic metric to use.

Cuirass should be building for aarch64-linux, but substitute
availability sits below 20% for that system. Even though the bordeaux
build farm has less machines, it has 70%+ substitute availability. So
for aarch64-linux for the berlin build farm, I think these builds have
just been added to the long backlog. This metric doesn't articulate how
the situation has got worse in this way.

(also, I don't know what this number means, but if it's something like
substitute availability, ~60% seems pretty bad)

>>> But the situation will repeat; I'd like to push some xorg updates that
>>> fix a CVE; we'll nead a 'xorg-team' branch or similar.  Should we create
>>> these branches from the maintenance repository (permanent branches) ?
>>
>> I don't really understand the question, surely the branches would be in
>> the guix Git repository?
>
> Yes, the branch would be in the Guix repository, but I meant regarding
> the Cuirass specifications affecting which branches it builds; sorry for
> being unclear.

No problem, this is something that needs looking at. On the berlin build
farm side, yes, configuring Cuirass to look at the branches is one
thing, but there's also bigger issues, e.g. the lack of substitutes for
aarch64-linux and armhf-linux (and thus delays in testing/merging
branches due to this).

On the bordeaux build farm side, there's also a "how to start and stop
building a branch issue". Currently it's a code change in the
qa-frontpage (plus reconfiguring bayfront and restarting the service),
but it would be good to make this easier too. Plus, like the berlin
build farm, there's also issues getting things built fast enough.

>> Anyway, package replacements+grafts can be used for security issues so
>> that shouldn't need to be on a branch as it won't involve lots of
>> rebuilds.
>
> For this case I think so yes, since it's a patch-level update that
> should be safe.
>
>> When it comes to handling changes involving lots of rebuilds though, I
>> think that this has been and continues to be difficult, but in my mind
>> that's a reason to slow down and try and work on tooling and processes
>> to help.
>
> One of the things that has been bothered me has been the lack of
> documentation/tooling to recreate TLS user certificates for Cuirass so
> that I can configure branches via its web interface again, or retry
> failed builds.  I'm currently working on documenting (in Cuirass's
> manual) a script Ricardo's made for that task.
>
> But building lots of packages will still require a lot of processing
> power, probably more so when it happens in focused team branches
> compared to grouped together as it used to be the case for
> e.g. core-updates.

I agree. I'm still not really sold on the idea of team specific branches
for this reason. Anyway, I think there's still tooling to build
(e.g. analyzing the overlap for builds between two changes) and more
thinking to do about processes for this.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 04/09: gnu: mesa: Update to 23.0.3.

[John Kehayias]

Hi Christopher and Maxim,

On Mon, May 08, 2023 at 02:41 AM, Christopher Baines wrote:

> guix-commits@gnu.org writes:
>
>> apteryx pushed a commit to branch master
>> in repository guix.
>>
>> commit 0be7838105806819f4586ec9130382a66a22880e
>> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
>> AuthorDate: Thu May 4 20:12:46 2023 +0000
>>
>>     gnu: mesa: Update to 23.0.3.
>>
>>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>>     [source]: Remove obsolete patch and update HTTPS url.
>>     [arguments]: Enable the crocus gallium driver.
>>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>>     * gnu/local.mk (dist_patch_DATA): Remove it.
>> ---
>>  gnu/local.mk                                       |  1 -
>>  gnu/packages/gl.scm                                | 14 ++++-------
>>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>>  3 files changed, 5 insertions(+), 37 deletions(-)
>
> → guix refresh -l mesa
> Building the following 1954 packages would ensure 4257 dependent
> packages are rebuilt ...
>
>
> I know there's been some discussion about changing processes regarding
> changes like this that impact lots of packages, but as far as I'm aware,
> the documented process hasn't changed yet. So should this have gone to
> core-updates, and not been directly pushed to master?
>

I should take some responsibility over what happened here as I had
volunteered as a sort of "branch manager" (to use the terminology
later in this thread) but I was a bit slower than I wanted. My plan
was roughly in line with what we are discussing, reviewing the
patches, pushing to a new "mesa-updates" branch, and then asking for
someone to start a build job. We could then have people more easily
test or just to check build for build failures and ensure good
substitute coverage upon merging.

Anyway, I appreciate this coming up for discussion here, so thanks
Christopher. And thanks Maxim for pushing through these changes
anyway, as there were some important fixes for people (some couldn't
use wayland I think due to missing Mesa drivers, video decoding
acceleration didn't work, etc.) and selfishly I'm happy to have the
latest Mesa sooner.

My apologies for not helping lead more on this front after my public
volunteering to do so!

John

Re: 04/09: gnu: mesa: Update to 23.0.3.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]


John Kehayias <john.kehayias@protonmail.com> writes:

> Hi Christopher and Maxim,
>
> On Mon, May 08, 2023 at 02:41 AM, Christopher Baines wrote:
>
>> guix-commits@gnu.org writes:
>>
>>> apteryx pushed a commit to branch master
>>> in repository guix.
>>>
>>> commit 0be7838105806819f4586ec9130382a66a22880e
>>> Author: Kaelyn Takata <kaelyn.alexi@protonmail.com>
>>> AuthorDate: Thu May 4 20:12:46 2023 +0000
>>>
>>>     gnu: mesa: Update to 23.0.3.
>>>
>>>     * gnu/packages/gl.scm (mesa): Update to 23.0.3.
>>>     [source]: Remove obsolete patch and update HTTPS url.
>>>     [arguments]: Enable the crocus gallium driver.
>>>     * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file.
>>>     * gnu/local.mk (dist_patch_DATA): Remove it.
>>> ---
>>>  gnu/local.mk                                       |  1 -
>>>  gnu/packages/gl.scm                                | 14 ++++-------
>>>  .../patches/mesa-fix-sporadic-test-failures.patch  | 27 ----------------------
>>>  3 files changed, 5 insertions(+), 37 deletions(-)
>>
>> → guix refresh -l mesa
>> Building the following 1954 packages would ensure 4257 dependent
>> packages are rebuilt ...
>>
>>
>> I know there's been some discussion about changing processes regarding
>> changes like this that impact lots of packages, but as far as I'm aware,
>> the documented process hasn't changed yet. So should this have gone to
>> core-updates, and not been directly pushed to master?
>>
>
> I should take some responsibility over what happened here as I had
> volunteered as a sort of "branch manager" (to use the terminology
> later in this thread) but I was a bit slower than I wanted. My plan
> was roughly in line with what we are discussing, reviewing the
> patches, pushing to a new "mesa-updates" branch, and then asking for
> someone to start a build job. We could then have people more easily
> test or just to check build for build failures and ensure good
> substitute coverage upon merging.

Don't feel responsible for not doing things John, I'm advocating here
for less haste and more caution.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]