Re: mesa-updates: call for patches

Re: mesa-updates: call for patches

[John Kehayias]

Hi everyone,

Update below:

On Sun, Nov 05, 2023 at 11:47 PM, John Kehayias wrote:
[snippy snip snip]
>>
>> Happy to! Substitutes will eventually become available, but there's
>> quite a few builds to be done. This takes care of some ungrafts and
>> updates with I hope minimal disruption. I'll be keeping an eye out and
>> using locally as well. Please test and report, thanks everyone!
>>
>> John
>
> An issue was created to track merging the mesa-updates branch here:
> <https://issues.guix.gnu.org/66964>. Please use that bug number as
> needed (and cc me or use wide-reply in emacs debbugs).

At this point I feel we are just about ready to go, unless there are
objections?

Substitute coverage, according to
<https://qa.guix.gnu.org/branch/mesa-updates> is good on x86_64 and
i686 (about 95% and 83%, respectively) while, as usual, other
architectures are behind. The next best is aarch64 at 54% on bordeaux,
and then falling to 24% for armhf, with others we build in the teens.
I think this is to be expected? In any event builds continue very
slowly and in the past I think this is about where we merge.

I should note: please check for any breakages. I didn't expect too
much, but did get more than I thought. It seems the ungrafting version
changes caused some things to fail. Also, the libx11 ungraft mean
python and rust were rebuilt, with the many packages that entails.

I fixed big ones I saw, like QT (unrelated: it was libxkbcommon
upgrade), but other leaf packages I saw had tests failing for reasons
I didn't see. For instance, php fails tests. The current ones are due
to the curl update, but updating php and removing an obsolete patch
had a different test fail. It would be great if someone more familiar
will take a look. With few dependents I figure this can just be done
on master after the merge.

So, shall I merge this to master in the next couple of days? I've been
merging master into mesa-updates smoothly so far. Please do check and
feel free to object if this needs more time.

Thanks everyone,
John


PS: I forgot to email the various patches/issues that are done on
mesa-updates, as listed in a previous message. I will do that too.

[bug#66964] mesa-updates: call for patches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]


John Kehayias <john.kehayias@protonmail.com> writes:

> Hi everyone,
>
> Update below:
>
> On Sun, Nov 05, 2023 at 11:47 PM, John Kehayias wrote:
> [snippy snip snip]
>>>
>>> Happy to! Substitutes will eventually become available, but there's
>>> quite a few builds to be done. This takes care of some ungrafts and
>>> updates with I hope minimal disruption. I'll be keeping an eye out and
>>> using locally as well. Please test and report, thanks everyone!
>>>
>>> John
>>
>> An issue was created to track merging the mesa-updates branch here:
>> <https://issues.guix.gnu.org/66964>. Please use that bug number as
>> needed (and cc me or use wide-reply in emacs debbugs).
>
> At this point I feel we are just about ready to go, unless there are
> objections?
>
> Substitute coverage, according to
> <https://qa.guix.gnu.org/branch/mesa-updates> is good on x86_64 and
> i686 (about 95% and 83%, respectively) while, as usual, other
> architectures are behind. The next best is aarch64 at 54% on bordeaux,
> and then falling to 24% for armhf, with others we build in the teens.
> I think this is to be expected? In any event builds continue very
> slowly and in the past I think this is about where we merge.

I think some changes have been pushed since this email, since the
aarch64 substitute availability has dropped from 54% to 25%.

> So, shall I merge this to master in the next couple of days? I've been
> merging master into mesa-updates smoothly so far. Please do check and
> feel free to object if this needs more time.

I guess this has been held up by the changes on the 15th, but still, I
think we need to wait for substitute availability to improve more prior
to merging, unless there's a specific and significant reason why we
don't want to wait.

Targets are arbitrary, but guix weather defines ☀ as 80%+, so I think
that's what we should aim for at least for x86_64-linux, i686-linux,
aarch64-linux and armhf-linux. This isn't just about substitute
availability though as this is key for discovering what things fail to
build.

Obviously delays in merging aren't ideal, but we should tackle the
problems around this, maybe by deciding that testing and providing
substitutes for ARM isn't a priority and thus isn't something we should
wait for, or look at getting more ARM hardware to speed up the process
(we also have a lack of x86_64 hardware on the bordeaux build farm).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#66964] mesa-updates: call for patches

[John Kehayias via Guix-patches via]

Hi,

On Sat, Nov 18, 2023 at 11:07 AM, Christopher Baines wrote:

>
> John Kehayias <john.kehayias@protonmail.com> writes:
>
>> Hi everyone,
>>
>> Update below:
>>
>> On Sun, Nov 05, 2023 at 11:47 PM, John Kehayias wrote:
>> [snippy snip snip]
>>
>> At this point I feel we are just about ready to go, unless there are
>> objections?
>>
>> Substitute coverage, according to
>> <https://qa.guix.gnu.org/branch/mesa-updates> is good on x86_64 and
>> i686 (about 95% and 83%, respectively) while, as usual, other
>> architectures are behind. The next best is aarch64 at 54% on bordeaux,
>> and then falling to 24% for armhf, with others we build in the teens.
>> I think this is to be expected? In any event builds continue very
>> slowly and in the past I think this is about where we merge.
>
> I think some changes have been pushed since this email, since the
> aarch64 substitute availability has dropped from 54% to 25%.
>

Yes, Efraim chimed in to help on some other architectures and some big
rebuilds were/are happening for those. I see them slowly ticking up
but it will still need some time.

>> So, shall I merge this to master in the next couple of days? I've been
>> merging master into mesa-updates smoothly so far. Please do check and
>> feel free to object if this needs more time.
>
> I guess this has been held up by the changes on the 15th, but still, I
> think we need to wait for substitute availability to improve more prior
> to merging, unless there's a specific and significant reason why we
> don't want to wait.
>

Yes, agreed. I'm not as clear on how well we do typically on non-x86
but getting a sense of it now, which is why I wanted to ask.

> Targets are arbitrary, but guix weather defines ☀ as 80%+, so I think
> that's what we should aim for at least for x86_64-linux, i686-linux,
> aarch64-linux and armhf-linux. This isn't just about substitute
> availability though as this is key for discovering what things fail to
> build.
>

I think this is something we could better clarify and quantify as many
of us probably only pay attention to x86_64, where for others we can
be strapped for both hardware and people. So I didn't want to wait for
some substitutes that would never come but also don't want to
inconvenience people on other architectures, especially if builds
there take much much longer in the first place.

Perhaps we can look at some historical data on what we've hit in
substitute coverage and try to at least keep up with that if not set
some goals for better coverage? While we might also expect further
difficulties as some get left behind by upstream (as we've had to work
around rust on i686 so far, I believe).

All that is to say, yes, let's make sure we have good substitute
coverage and are clear on what architectures we want to make sure
users get substitutes for.

> Obviously delays in merging aren't ideal, but we should tackle the
> problems around this, maybe by deciding that testing and providing
> substitutes for ARM isn't a priority and thus isn't something we should
> wait for, or look at getting more ARM hardware to speed up the process
> (we also have a lack of x86_64 hardware on the bordeaux build farm).
>

Agreed. We should have some clear Guix-wide standards and goals. I'm
sure we can get some hardware from Guix-ers and/or funding too,
especially if people know exactly what it will go towards improving.

Thanks for chiming in here and all your work on this front!

In the meantime, I'll go back to refreshing the CI and QA pages every
so often to make sure we keep getting closer...

Re: mesa-updates: call for patches

[John Kehayias]

> Details of changes below, which included some more ungrafting:
>
>>> Here is a list of what has been on my radar and I'll be looking to
>>> apply on the mesa-updates branch. If there's something I missed and
>>> you think makes sense here (e.g. lower level graphical/X related
>>> libraries) please let me know. And yes, I need to make a Mesa team. It
>>> is on my list!
>>>
>>> 1. ungraft libx11 and libxpm
>>>
>
> These are done.
>
> Also, since I didn't realize libx11 means building python, and rust,
> and so on, I figured I would ungraft more that was being rebuilt
> anyway. So nghttp2 and curl have been ungrafted.
>
> The only more involved change was in ungrafting curl (to keep another
> test skip for Hurd), in
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00442f15d46cd9d5d02499827946d23426aad0ba&h=mesa-updates>
> and then running parallel tests (very nice speed up for building curl
> I came across in the documentation!) in
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d70f2b788e56545f93dc24238d2617e20fde7460&h=mesa-updates>.
> Please do take a look in case I missed something here, but all built
> fine locally on x86_64 at least.
>
>>> 2. update mesa (23.2.1)
>>>
>>> 3. <https://issues.guix.gnu.org/65375> (libepoxy fix for GTK, see
>>> <https://issues.guix.gnu.org/64981>)
>>>
>
> These are done.
>
>>> 4. <https://issues.guix.gnu.org/65155> (mesa vulkan search-paths)
>>>
>
> I don't think this is a correct change as written (search-path should
> be in vulkan-loader if I'm understanding what is supposed to happen
> here). Anyway, will follow up on that issue and left it out for now.
>
>>> 5. <https://issues.guix.gnu.org/65153> (sdl2 vulkan-loader; and update sdl2?)
>>>
>>> 6. <https://issues.guix.gnu.org/64637> (libdrm update; to even newer version now)
>>>
>>> 7. <https://issues.guix.gnu.org/66727> (libxkbcommon update)
>>>
>>> 8. <https://issues.guix.gnu.org/64639> (pixman update)
>>>
>
> All these done (sdl2 was also updated to latest version).
>
>>> 9. submit patch for mesa team
>>>
>
> But not this one, the easy one :) It will be sent.
>
>>> Thanks everyone! And when you see the updates pushed on mesa-updates
>>> and builds become available, please do test and let me know. Or if
>>> you'd like to join this team of one, happy to have you on board.
>>
>> Sounds like a fine plan!  Thank you for tackling this!
>
> Happy to! Substitutes will eventually become available, but there's
> quite a few builds to be done. This takes care of some ungrafts and
> updates with I hope minimal disruption. I'll be keeping an eye out and
> using locally as well. Please test and report, thanks everyone!
>
> John

An issue was created to track merging the mesa-updates branch here:
<https://issues.guix.gnu.org/66964>. Please use that bug number as
needed (and cc me or use wide-reply in emacs debbugs).

[bug#66964] mesa-updates: call for patches

[Kaelyn via Guix-patches via]

Hi,

I've just submitted a pair of patches for the mesa-updates branch: <https://issues.guix.gnu.org/67136> updating xorgproto and xorg-server-xwayland. The xorgproto is a high-impact update (guix refresh reports rebuilding 8710 packages would ensure 22871 dependent packages are rebuilt), but required to update to the latest xwayland as xwayland requires a newer version of presentproto than in the current guix xorgproto package. The updating and ungrafting of mesa and a number of X.org related libraries seemed like a good time (and place) to update xorgproto as well.

Cheers,
Kaelyn