Branch (and team?) for mesa updates

Branch (and team?) for mesa updates

[John Kehayias]

Hi everyone,

With our move to a branching strategy for patches that require many rebuilds, I would like to propose a branch for Mesa updates. Based on how Mesa has been developed the last few years, there should be frequent (roughly a couple of months or quicker) releases that shouldn't be breaking anything or requiring lots of packaging work.

Famous last words, I know, but at least in the last few years the only big change I know of was the one we hit on the last core-updates cycle with old hardware being dropped. The previous cycle had some build changes, perhaps due to missing many intermediate version changes. Both were rather self-contained and resolved quickly.

So, I have submitted a patch for the latest stable release at <https://issues.guix.gnu.org/64175> I'm aware of one other patch that should also go here, <https://issues.guix.gnu.org/64044> are there any others?

I've tentatively labeled with "mesa-updates" as a proposed branch name. With Mesa's release cycle I propose keeping this as a branch for Mesa updates, and I suppose related required changes (say libdrm). If everything goes smoothly, we can give the build farm some time to build everything, check for any breakages, and then push to master with substitutes available. Master can be merged into this branch just prior to a patches going to this branch with the expectation merging back to master will be soon after and changes are only affecting packages that won't be touched on master anyway. I think this should be relatively clean and straightforward, a good use of our new branching/building strategy.

Thoughts? Can someone set up a build job for this branch and/or let me know how to do that? (I would also require access to Cuirass.)

Do we want a "Mesa team" or something a bit larger? Not sure what exactly, since "graphics" is perhaps too broad. Happy to help spearhead the Mesa front for Guix (the very package that got me first involved in the patching process).

Thanks!
John

Re: Branch (and team?) for mesa updates

[Andreas Enge]

Hello,

Am Mon, Jun 19, 2023 at 06:25:04PM +0000 schrieb John Kehayias:
> Master can be merged into this branch just prior to a patches going to this branch with the expectation merging back to master will be soon after and changes are only affecting packages that won't be touched on master anyway. I think this should be relatively clean and straightforward, a good use of our new branching/building strategy.

I am a bit wary of branches just sitting around; after a while we will
forget whether a branch still needs merging, or is just a placeholder.
So I would suggest to delete every branch right after merging, and then
branch off of master later when a new patch is going to be applied.
This will also create a cleaner history by avoiding a merge.

This is compatible with how cuirass works. We can keep the mesa-updates job,
and when the branch does not exist, it will just do nothing. You just need
to be sure to use the same branch name again next time, and it should be
picked up by the cuirass job.

Andreas

Re: Branch (and team?) for mesa updates

[Katherine Cox-Buday]

On 7/2/23 4:30 AM, Andreas Enge wrote:

> So I would suggest to delete every branch right after merging, and then
> branch off of master later when a new patch is going to be applied.
> This will also create a cleaner history by avoiding a merge.

I disagree with this because it seems like Mesa moves along at a pretty 
brisk pace and I feel like we'd be constantly recreating the same branch:

23.1.3, 2023-06-22 (14 days)
23.1.2, 2023-06-08 ( 9 days)
23.0.4, 2023-05-30 ( 5 days)
23.1.1, 2023-05-25 (15 days)
23.1.0, 2023-05-10

And so on.

 >> Am Mon, Jun 19, 2023 at 06:25:04PM +0000 schrieb John Kehayias:
 >> Master can be merged into this branch just prior to a patches going 
to this branch with the expectation merging back to master will be soon 
after and changes are only affecting packages that won't be touched on 
master anyway. I think this should be relatively clean and 
straightforward, a good use of our new branching/building strategy.

This seems more reasonable to me and like it's standard Git practice.

Re: Branch (and team?) for mesa updates

[Andreas Enge]

Am Wed, Jul 05, 2023 at 09:47:06AM -0600 schrieb Katherine Cox-Buday:
> I disagree with this because it seems like Mesa moves along at a pretty
> brisk pace and I feel like we'd be constantly recreating the same branch:
> 23.1.3, 2023-06-22 (14 days)
> 23.1.2, 2023-06-08 ( 9 days)
> 23.0.4, 2023-05-30 ( 5 days)
> 23.1.1, 2023-05-25 (15 days)
> 23.1.0, 2023-05-10

I doubt we would be able to move at such a brisk pace and update mesa
every other week! It is a package with more than 4000 dependents, so in
any case it would be good to regroup with somewhat related changes.

Andreas

Re: Branch (and team?) for mesa updates

[John Kehayias]

Hello,

On Wed, Jul 05, 2023 at 08:08 PM, Andreas Enge wrote:

> Am Wed, Jul 05, 2023 at 09:47:06AM -0600 schrieb Katherine Cox-Buday:
>> I disagree with this because it seems like Mesa moves along at a pretty
>> brisk pace and I feel like we'd be constantly recreating the same branch:
>> 23.1.3, 2023-06-22 (14 days)
>> 23.1.2, 2023-06-08 ( 9 days)
>> 23.0.4, 2023-05-30 ( 5 days)
>> 23.1.1, 2023-05-25 (15 days)
>> 23.1.0, 2023-05-10
>
> I doubt we would be able to move at such a brisk pace and update mesa
> every other week! It is a package with more than 4000 dependents, so in
> any case it would be good to regroup with somewhat related changes.
>

Since the Cuirass job remains, I don't think it is really any extra
work or not to recreate the branch as needed, if that is preferred.
I'll defer to whatever people want, we can always change our mind if
anything comes up.

As for the pace, on the main 23.1.x releases we see about 2 weeks
currently. If that is a bit fast I think about once a month is a
reasonable pace, maybe depending if it is a small bugfix release or
what other changes we may need causing other rebuilds to be grouped on
the branch. We can slow down if major changes are introduced, but
seems Mesa is moving along without breaking changes or major changes
in building/support these days.

As you can follow along on <https://issues.guix.gnu.org/64369> within a
couple of days x86_64 and i686 had good weather (though I can't see
specifically what may have broken yet) while as we'd expect things
like aarch64 are still churning away. The speed of this build job
might determine how fast we can go anyway, but waiting too long will
mean a new minor release to build :)

John

Re: Branch (and team?) for mesa updates

[John Kehayias]

Hello again,

On Mon, Jun 19, 2023 at 02:10 PM, John Kehayias wrote:

> Hi everyone,
>
> With our move to a branching strategy for patches that require many
> rebuilds, I would like to propose a branch for Mesa updates. Based on
> how Mesa has been developed the last few years, there should be
> frequent (roughly a couple of months or quicker) releases that
> shouldn't be breaking anything or requiring lots of packaging work.
>
> Famous last words, I know, but at least in the last few years the only
> big change I know of was the one we hit on the last core-updates cycle
> with old hardware being dropped. The previous cycle had some build
> changes, perhaps due to missing many intermediate version changes.
> Both were rather self-contained and resolved quickly.
>
> So, I have submitted a patch for the latest stable release at
> <https://issues.guix.gnu.org/64175> I'm aware of one other patch that
> should also go here, <https://issues.guix.gnu.org/64044> are there any
> others?
>

I've created a "mesa-updates" branch with these two commits (the one
updating mesa has been updated to the latest version from last week).

> I've tentatively labeled with "mesa-updates" as a proposed branch
> name. With Mesa's release cycle I propose keeping this as a branch for
> Mesa updates, and I suppose related required changes (say libdrm). If
> everything goes smoothly, we can give the build farm some time to
> build everything, check for any breakages, and then push to master
> with substitutes available. Master can be merged into this branch just
> prior to a patches going to this branch with the expectation merging
> back to master will be soon after and changes are only affecting
> packages that won't be touched on master anyway. I think this should
> be relatively clean and straightforward, a good use of our new
> branching/building strategy.
>
> Thoughts? Can someone set up a build job for this branch and/or let me
> know how to do that? (I would also require access to Cuirass.)
>

I'll open a branch merge request issue later today as per new
procedure for QA. Though I believe that only builds 2 branches, which
is occupied at the moment. Or can someone set a separate build job
specifically for mesa-updates, especially if we think it is a good
idea to have this going forward?

> Do we want a "Mesa team" or something a bit larger? Not sure what
> exactly, since "graphics" is perhaps too broad. Happy to help
> spearhead the Mesa front for Guix (the very package that got me first
> involved in the patching process).
>

This is still a good question I think, of how we want to have a
team(s) to handle things like xorg, wayland, mesa, and related
packages. They are a bit all over the place in terms of scope and what
they touch. For now I'd like to go ahead with a regular mesa-updates
branch since that sees regular releases and is pretty self-contained
currently.

Thanks!
John

Re: Branch (and team?) for mesa updates

[Maxim Cournoyer]

Hi John,

John Kehayias <john.kehayias@protonmail.com> writes:

[...]

> I'll open a branch merge request issue later today as per new
> procedure for QA. Though I believe that only builds 2 branches, which
> is occupied at the moment. Or can someone set a separate build job
> specifically for mesa-updates, especially if we think it is a good
> idea to have this going forward?

Do you already have admin access to Cuirass?  We can issue client certs
for team members needing to create branches on it or restart builds
there, etc.

>> Do we want a "Mesa team" or something a bit larger? Not sure what
>> exactly, since "graphics" is perhaps too broad. Happy to help
>> spearhead the Mesa front for Guix (the very package that got me first
>> involved in the patching process).
>>
>
> This is still a good question I think, of how we want to have a
> team(s) to handle things like xorg, wayland, mesa, and related
> packages. They are a bit all over the place in terms of scope and what
> they touch. For now I'd like to go ahead with a regular mesa-updates
> branch since that sees regular releases and is pretty self-contained
> currently.

It seems a 'desktop' team could make sense, covering some of the things
listed here that makes sense / are already well separated in modules in
Guix to avoid being added to two teams:
https://www.freedesktop.org/wiki/Software/.

-- 
Thanks,
Maxim

Re: Branch (and team?) for mesa updates

[John Kehayias]

Hi Maxim,

On Sun, Jul 30, 2023 at 09:50 PM, Maxim Cournoyer wrote:

> Hi John,
>
> John Kehayias <john.kehayias@protonmail.com> writes:
>
> [...]
>
>> I'll open a branch merge request issue later today as per new
>> procedure for QA. Though I believe that only builds 2 branches, which
>> is occupied at the moment. Or can someone set a separate build job
>> specifically for mesa-updates, especially if we think it is a good
>> idea to have this going forward?
>
> Do you already have admin access to Cuirass?  We can issue client certs
> for team members needing to create branches on it or restart builds
> there, etc.
>

I do not have access. The mesa-updates branch remains (and still an
active job I think, just nothing pushed since the merge). I plan on
making use of it as soon as 23.2 is out, along with a handful of
pending patches I've seen that will make sense here.

I haven't used Cuirass before but if a hand would be helpful I'm happy
to lend it (let me know if there is someone I should contact directly
or message me off list).

>>> Do we want a "Mesa team" or something a bit larger? Not sure what
>>> exactly, since "graphics" is perhaps too broad. Happy to help
>>> spearhead the Mesa front for Guix (the very package that got me first
>>> involved in the patching process).
>>>
>>
>> This is still a good question I think, of how we want to have a
>> team(s) to handle things like xorg, wayland, mesa, and related
>> packages. They are a bit all over the place in terms of scope and what
>> they touch. For now I'd like to go ahead with a regular mesa-updates
>> branch since that sees regular releases and is pretty self-contained
>> currently.
>
> It seems a 'desktop' team could make sense, covering some of the things
> listed here that makes sense / are already well separated in modules in
> Guix to avoid being added to two teams:
> <https://www.freedesktop.org/wiki/Software/>.

The problem I'm thinking of for a "desktop" team is setting the
correct scope of package files to make use of e.g. auto cc-ing on
patch submissions. Though at least (gnu packages gl) looks pretty
reasonable to start for maybe a graphics team? Maybe with vulkan?

I'm still not sure but I should probably propose something concrete
with at least myself for gl since those patches generally will go to
the mesa-updates branch for convenient building.

Anyone else want in?

John

Re: Branch (and team?) for mesa updates

[Maxim Cournoyer]

Hi John,

John Kehayias <john.kehayias@protonmail.com> writes:

> Hi Maxim,
>
> On Sun, Jul 30, 2023 at 09:50 PM, Maxim Cournoyer wrote:
>
>> Hi John,
>>
>> John Kehayias <john.kehayias@protonmail.com> writes:
>>
>> [...]
>>
>>> I'll open a branch merge request issue later today as per new
>>> procedure for QA. Though I believe that only builds 2 branches, which
>>> is occupied at the moment. Or can someone set a separate build job
>>> specifically for mesa-updates, especially if we think it is a good
>>> idea to have this going forward?
>>
>> Do you already have admin access to Cuirass?  We can issue client certs
>> for team members needing to create branches on it or restart builds
>> there, etc.
>>
>
> I do not have access. The mesa-updates branch remains (and still an
> active job I think, just nothing pushed since the merge). I plan on
> making use of it as soon as 23.2 is out, along with a handful of
> pending patches I've seen that will make sense here.

I'll email you your TLS client cert that you can install into your
favorite browser.

> I haven't used Cuirass before but if a hand would be helpful I'm happy
> to lend it (let me know if there is someone I should contact directly
> or message me off list).

It's unfortunately kind of necessary to baby sit the builds and restart
those that failed due to bugs in our CI infrastructure such as #54447
("cuirass: missing derivation error").  The more hands, the better.

>>>> Do we want a "Mesa team" or something a bit larger? Not sure what
>>>> exactly, since "graphics" is perhaps too broad. Happy to help
>>>> spearhead the Mesa front for Guix (the very package that got me first
>>>> involved in the patching process).

>>>
>>> This is still a good question I think, of how we want to have a
>>> team(s) to handle things like xorg, wayland, mesa, and related
>>> packages. They are a bit all over the place in terms of scope and what
>>> they touch. For now I'd like to go ahead with a regular mesa-updates
>>> branch since that sees regular releases and is pretty self-contained
>>> currently.
>>
>> It seems a 'desktop' team could make sense, covering some of the things
>> listed here that makes sense / are already well separated in modules in
>> Guix to avoid being added to two teams:
>> <https://www.freedesktop.org/wiki/Software/>.
>
> The problem I'm thinking of for a "desktop" team is setting the
> correct scope of package files to make use of e.g. auto cc-ing on
> patch submissions. Though at least (gnu packages gl) looks pretty
> reasonable to start for maybe a graphics team? Maybe with vulkan?

Yeah, it's not super focused, but I think it may be best to start to
broad and refine later to have more teams coverage.

> I'm still not sure but I should probably propose something concrete
> with at least myself for gl since those patches generally will go to
> the mesa-updates branch for convenient building.

A "graphics" or "desktop" team which would include freedesktop and the
gl modules sounds good to me.

-- 
Thanks,
Maxim

Re: Branch (and team?) for mesa updates

[dan]

Hi John,

Aug 26, 2023 08:51:49 John Kehayias <john.kehayias@protonmail.com>:
>
>
> Though at least (gnu packages gl) looks pretty
> reasonable to start for maybe a graphics team? Maybe with vulkan?
>
> I'm still not sure but I should probably propose something concrete
> with at least myself for gl since those patches generally will go to
> the mesa-updates branch for convenient building.
>
> Anyone else want in?

I use vulkan and sdl2 for my day to day development, and I once submitted 
a patch series updating various vulkan packages.  I'm interested in 
maintaining these vulkan packages and I like the idea of having a 
graphics team.  Meanwhile, I'm not so sure if the team should be 
responsible for sdl2, but I would like to discuss the possibility first.

Re: Branch (and team?) for mesa updates

[Tobias Platen]

On Sun, 2023-08-27 at 12:45 +0800, dan wrote:
> Hi John,
> 
> Aug 26, 2023 08:51:49 John Kehayias <john.kehayias@protonmail.com>:
> > 
> > 
> > Though at least (gnu packages gl) looks pretty
> > reasonable to start for maybe a graphics team? Maybe with vulkan?
> > 
> > I'm still not sure but I should probably propose something concrete
> > with at least myself for gl since those patches generally will go
> > to
> > the mesa-updates branch for convenient building.
> > 
> > Anyone else want in?
> 
> I use vulkan and sdl2 for my day to day development, and I once
> submitted 
> a patch series updating various vulkan packages.  I'm interested in 
> maintaining these vulkan packages and I like the idea of having a 
> graphics team.  Meanwhile, I'm not so sure if the team should be 
> responsible for sdl2, but I would like to discuss the possibility
> first.
> 
I would also want to submit the libre-soc vulkan driver to a branch,
once it exists. Currently all vulkan compatible GPUs need non-free
firmware blobs.

Re: Branch (and team?) for mesa updates

[John Kehayias]

On Thu, Jun 29, 2023 at 12:09 PM, John Kehayias wrote:

[snip]
>
> I'll open a branch merge request issue later today as per new
> procedure for QA. Though I believe that only builds 2 branches, which
> is occupied at the moment. Or can someone set a separate build job
> specifically for mesa-updates, especially if we think it is a good
> idea to have this going forward?
>

Merging request for "mesa-updates" branch created:
<https://issues.guix.gnu.org/64369>

>> Do we want a "Mesa team" or something a bit larger? Not sure what
>> exactly, since "graphics" is perhaps too broad. Happy to help
>> spearhead the Mesa front for Guix (the very package that got me first
>> involved in the patching process).
>>
>
> This is still a good question I think, of how we want to have a
> team(s) to handle things like xorg, wayland, mesa, and related
> packages. They are a bit all over the place in terms of scope and what
> they touch. For now I'd like to go ahead with a regular mesa-updates
> branch since that sees regular releases and is pretty self-contained
> currently.
>
> Thanks!
> John