Mesa vulkan layer path fix for core-updates

Mesa vulkan layer path fix for core-updates

[Kaelyn]

Hi,

Some time back I mailed in https://issues.guix.gnu.org/59453 to fix an issue with the manifests for the vulkan layers that ship with mesa. Basically the error is that the manifests don't include the store path to the referenced libraries, only the file name. An example the error can be seen by running "vulkaninfo &| less", where this message is printed a few times:

ERROR: [Loader Message] Code 0 : libVkLayer_MESA_device_select.so: cannot open shared object file: No such file or directory


While I know it is late in the process of preparing core-updates for merging into master, I wanted to ask if it would be possible to have the patch addressed prior to the merge? I just encountered the error message again after having a need to check vulkaninfo and it reminded me of the patch. (The vulkan-validationlayers package that was recently merged to master from staging includes a similar phase for fixing the layer object path in its layer manifest.)

Thanks,
Kaelyn

Re: Mesa vulkan layer path fix for core-updates

[Andreas Enge]

Hello,

thanks for bringing this back to our attention!

Am Wed, Apr 19, 2023 at 02:41:57PM +0000 schrieb Kaelyn:
> While I know it is late in the process of preparing core-updates for merging into master, I wanted to ask if it would be possible to have the patch addressed prior to the merge?

Given how many packages depend on mesa and their importance, I think it
would be safer to postpone the fix until after the merge; be it in a
dedicated mesa feature branch that could quickly be merged afterwards,
or better yet regroup other changes in this area. (Searching for open mesa
packages on issues.guix.gnu.org returns quite a few matches, this could be
a good occasion to sort them out.)

Andreas

Re: Mesa vulkan layer path fix for core-updates

[Kaelyn]

------- Original Message -------
On Wednesday, April 19th, 2023 at 3:26 PM, Andreas Enge <andreas@enge.fr> wrote:


> 
> 
> Hello,
> 
> thanks for bringing this back to our attention!

You're welcome! :)

> 
> Am Wed, Apr 19, 2023 at 02:41:57PM +0000 schrieb Kaelyn:
> 
> > While I know it is late in the process of preparing core-updates for merging into master, I wanted to ask if it would be possible to have the patch addressed prior to the merge?
> 
> 
> Given how many packages depend on mesa and their importance, I think it
> would be safer to postpone the fix until after the merge; be it in a
> dedicated mesa feature branch that could quickly be merged afterwards,
> or better yet regroup other changes in this area. (Searching for open mesa
> packages on issues.guix.gnu.org returns quite a few matches, this could be
> a good occasion to sort them out.)

I'm okay with it waiting until after the core-updates merge and going into a feature branch, especially if that branch includes updating mesa to the latest release (currently 23.0.0, or 22.3.7 if going with the latest patch of the previous feature release instead of the x.y.0 release of the newest series).

Inspired by your comment, I also just did a quick scan of https://issues.guix.gnu.org/search?query=mesa+is%3Aopen and took a peek at some tickets that looked like they may be related to the mesa package. From that perspective, six other tickets caught my eye in the results:

* https://issues.guix.gnu.org/62176 can be closed when core-updates is merged, since core-updates contains mesa 22.2.4

* https://issues.guix.gnu.org/58887 looks like an alternate way of solving the layer path issues that https://issues.guix.gnu.org/59453 also addresses. Additionally, it adds two new nonstandard VK_*_PATH variables to vulkan-loader, with at least VK_ILAYER_PATH seeming very similar in functionality to VK_LAYER_PATH and VK_ADD_LAYER_PATH described at https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md

* https://issues.guix.gnu.org/58251 would be fixed by https://issues.guix.gnu.org/59453

* https://issues.guix.gnu.org/62313 might need a modification to mesa e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one possible solution; in my home profile I made VDPAU work by setting "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").

* https://issues.guix.gnu.org/48868 appears to be the same VDPAU_DRIVER_PATH issue as https://issues.guix.gnu.org/62313.

* Though not exactly mesa-related, https://issues.guix.gnu.org/61364 can possibly be closed now, and almost certainly once the core-updates merge is completed. (The ticket is a number of workarounds the user applied in early Feb to be able to build their system profile using core-updates, to pick up Mesa 22 for newer hardware support. I'm not sure if any of the patches are still relevant.)

Cheers,
Kaelyn

Re: Mesa vulkan layer path fix for core-updates

[John Kehayias]

Hi Kaelyn, Andreas, and Guix!

On Wed, Apr 19, 2023 at 04:07 PM, Kaelyn wrote:

> ------- Original Message -------
> On Wednesday, April 19th, 2023 at 3:26 PM, Andreas Enge <andreas@enge.fr> wrote:
>
>>
>> Hello,
>>
>> thanks for bringing this back to our attention!
>
> You're welcome! :)

Yes, thanks Kaelyn, especially since I believe I said I wanted to help
these patches through for core-updates, before I had to step away for
a bit (all good now!).

>>
>> Am Wed, Apr 19, 2023 at 02:41:57PM +0000 schrieb Kaelyn:
>>
>> Given how many packages depend on mesa and their importance, I think it
>> would be safer to postpone the fix until after the merge; be it in a
>> dedicated mesa feature branch that could quickly be merged afterwards,
>> or better yet regroup other changes in this area. (Searching for open mesa
>> packages on issues.guix.gnu.org returns quite a few matches, this could be
>> a good occasion to sort them out.)
>
> I'm okay with it waiting until after the core-updates merge and going
> into a feature branch, especially if that branch includes updating
> mesa to the latest release (currently 23.0.0, or 22.3.7 if going with
> the latest patch of the previous feature release instead of the x.y.0
> release of the newest series).
>

I was going to suggest the same, a feature branch just after
core-updates is merged, including getting Mesa to the latest version
(I suspect a stable v23 will be here soon). I've used a newer Mesa
locally for some packages and I didn't hit any issues in the packaging
for v23.

I would be happy to team up with you to prepare this branch once it is
time. I see you did a good search below collecting the various related
issues/patches. I can take a look as well.

> Inspired by your comment, I also just did a quick scan of
> <https://issues.guix.gnu.org/search?query=mesa+is%3Aopen> and took a
> peek at some tickets that looked like they may be related to the mesa
> package. From that perspective, six other tickets caught my eye in the
> results:
>
> * <https://issues.guix.gnu.org/62176> can be closed when core-updates is
> merged, since core-updates contains mesa 22.2.4
>

Great, we'll be able to close it Real Soon Now.

> * <https://issues.guix.gnu.org/58887> looks like an alternate way of
> solving the layer path issues that <https://issues.guix.gnu.org/59453>
> also addresses. Additionally, it adds two new nonstandard VK_*_PATH
> variables to vulkan-loader, with at least VK_ILAYER_PATH seeming very
> similar in functionality to VK_LAYER_PATH and VK_ADD_LAYER_PATH
> described at
> <https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md>
>

Thanks, will have a look at the differences.

> * <https://issues.guix.gnu.org/58251> would be fixed by <https://issues.guix.gnu.org/59453>
>

Great, always good to close some issues :)

> * <https://issues.guix.gnu.org/62313> might need a modification to mesa
> e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one possible
> solution; in my home profile I made VDPAU work by setting
> "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").
>

Right, this one is my report. I haven't had a chance to return to it
to figure out what is best here. Things do work with manual
intervention. It is a little tricky since my guess is we want VDPAU to
work without needing to install e.g. Mesa explicitly. But I don't
remember the details right now. Happy to discuss!

> * <https://issues.guix.gnu.org/48868> appears to be the same
> VDPAU_DRIVER_PATH issue as <https://issues.guix.gnu.org/62313>.
>

Yes, I think so too, though VLC does have Mesa as an input so the fix
might be easier directly for VLC at least.

> * Though not exactly mesa-related, <https://issues.guix.gnu.org/61364>
> can possibly be closed now, and almost certainly once the core-updates
> merge is completed. (The ticket is a number of workarounds the user
> applied in early Feb to be able to build their system profile using
> core-updates, to pick up Mesa 22 for newer hardware support. I'm not
> sure if any of the patches are still relevant.)
>

Also my quick reading of it; will double check none of those are still
out of date and we can close with core-updates.

> Cheers,
> Kaelyn

Thanks again Kaelyn! I think this will make for a small (but
non-trivial) set of patches for a feature branch and trying out that
workflow. We can then have the CI build the branch and people can try
their system with the newer Mesa plus fixes for testing. I don't think
there is any hardware support being dropped (unlike with v22) and it
will help for newer hardware, so I don't anticipate issues. But let me
not say that too loudly.

Looking forward to it!

John

Re: Mesa vulkan layer path fix for core-updates

[Andreas Enge]

Hello Kaelyn,

thanks for your research!

Am Wed, Apr 19, 2023 at 04:07:51PM +0000 schrieb Kaelyn:
> * https://issues.guix.gnu.org/62176 can be closed when core-updates is merged, since core-updates contains mesa 22.2.4
> * Though not exactly mesa-related, https://issues.guix.gnu.org/61364 can possibly be closed now, and almost certainly once the core-updates merge is completed. (The ticket is a number of workarounds the user applied in early Feb to be able to build their system profile using core-updates, to pick up Mesa 22 for newer hardware support. I'm not sure if any of the patches are still relevant.)

I just closed these two.

> * https://issues.guix.gnu.org/58887 looks like an alternate way of solving the layer path issues that https://issues.guix.gnu.org/59453 also addresses. Additionally, it adds two new nonstandard VK_*_PATH variables to vulkan-loader, with at least VK_ILAYER_PATH seeming very similar in functionality to VK_LAYER_PATH and VK_ADD_LAYER_PATH described at https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md
> * https://issues.guix.gnu.org/58251 would be fixed by https://issues.guix.gnu.org/59453
> * https://issues.guix.gnu.org/62313 might need a modification to mesa e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one possible solution; in my home profile I made VDPAU work by setting "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").
> * https://issues.guix.gnu.org/48868 appears to be the same VDPAU_DRIVER_PATH issue as https://issues.guix.gnu.org/62313.

And leave these to you and anybody who wants to work on them!

Andreas

Re: Mesa vulkan layer path fix for core-updates

[Kaelyn]

Hi,

------- Original Message -------
On Tuesday, April 25th, 2023 at 2:15 PM, Andreas Enge <andreas@enge.fr> wrote:


> 
> 
> Hello Kaelyn,
> 
> thanks for your research!

You're welcome! :)

> Am Wed, Apr 19, 2023 at 04:07:51PM +0000 schrieb Kaelyn:
> 
> > * https://issues.guix.gnu.org/62176 can be closed when core-updates is merged, since core-updates contains mesa 22.2.4
> > * Though not exactly mesa-related, https://issues.guix.gnu.org/61364 can possibly be closed now, and almost certainly once the core-updates merge is completed. (The ticket is a number of workarounds the user applied in early Feb to be able to build their system profile using core-updates, to pick up Mesa 22 for newer hardware support. I'm not sure if any of the patches are still relevant.)
> 
> 
> I just closed these two.
> 
> > * https://issues.guix.gnu.org/58887 looks like an alternate way of solving the layer path issues that https://issues.guix.gnu.org/59453 also addresses. Additionally, it adds two new nonstandard VK_*_PATH variables to vulkan-loader, with at least VK_ILAYER_PATH seeming very similar in functionality to VK_LAYER_PATH and VK_ADD_LAYER_PATH described at https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md
> > * https://issues.guix.gnu.org/58251 would be fixed by https://issues.guix.gnu.org/59453

I feel these two can be closed once 59453 lands, as then the manifests will have the store path to their corresponding shared objects. I also think having the full paths in the manifests will lead to fewer cross-version/cross-package mixing of objects, compared to setting and using environment variables of directories to search.

> > * https://issues.guix.gnu.org/62313 might need a modification to mesa e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one possible solution; in my home profile I made VDPAU work by setting "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").
> > * https://issues.guix.gnu.org/48868 appears to be the same VDPAU_DRIVER_PATH issue as https://issues.guix.gnu.org/62313.

For the VDPAU drivers, I plan to do a little more digging and some experimenting but I suspect defining VDPAU_DRIVER_PATH as a native-search-path is the best way forward. I'll send a patch once I've tested a change locally without having my profile set VDPAU_DRIVER_PATH to /run/current-system/profile/lib/vdpau.

Cheers,
Kaelyn
> 
> 
> And leave these to you and anybody who wants to work on them!
> 
> Andreas

Re: Mesa vulkan layer path fix for core-updates

[John Kehayias]

Hello,

On Tue, Apr 25, 2023 at 04:40 PM, Kaelyn wrote:

> Hi,
>
> ------- Original Message -------
> On Tuesday, April 25th, 2023 at 2:15 PM, Andreas Enge <andreas@enge.fr> wrote:
>
>> Hello Kaelyn,
>>
>> thanks for your research!
>
> You're welcome! :)
>
>> Am Wed, Apr 19, 2023 at 04:07:51PM +0000 schrieb Kaelyn:
>>
>> > * <https://issues.guix.gnu.org/62176> can be closed when
>> > core-updates is merged, since core-updates contains mesa 22.2.4
>> > * Though not exactly mesa-related,
>> > <https://issues.guix.gnu.org/61364> can possibly be closed now, and
>> > almost certainly once the core-updates merge is completed. (The
>> > ticket is a number of workarounds the user applied in early Feb to
>> > be able to build their system profile using core-updates, to pick
>> > up Mesa 22 for newer hardware support. I'm not sure if any of the
>> > patches are still relevant.)
>>
>> I just closed these two.
>>
>> > * <https://issues.guix.gnu.org/58887> looks like an alternate way of
>> > solving the layer path issues that
>> > <https://issues.guix.gnu.org/59453> also addresses. Additionally, it
>> > adds two new nonstandard VK_*_PATH variables to vulkan-loader,
>> > with at least VK_ILAYER_PATH seeming very similar in functionality
>> > to VK_LAYER_PATH and VK_ADD_LAYER_PATH described at
>> > <https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md>
>> > * <https://issues.guix.gnu.org/58251> would be fixed by
>> > <https://issues.guix.gnu.org/59453>
>
> I feel these two can be closed once 59453 lands, as then the manifests
> will have the store path to their corresponding shared objects. I also
> think having the full paths in the manifests will lead to fewer
> cross-version/cross-package mixing of objects, compared to setting and
> using environment variables of directories to search.
>

I haven't looked at the details, but I'm guessing these can all be
closed now?

>> > * <https://issues.guix.gnu.org/62313> might need a modification to
>> > mesa e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one
>> > possible solution; in my home profile I made VDPAU work by setting
>> > "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").
>> > * <https://issues.guix.gnu.org/48868> appears to be the same
>> > VDPAU_DRIVER_PATH issue as <https://issues.guix.gnu.org/62313>.
>
> For the VDPAU drivers, I plan to do a little more digging and some
> experimenting but I suspect defining VDPAU_DRIVER_PATH as a
> native-search-path is the best way forward. I'll send a patch once
> I've tested a change locally without having my profile set
> VDPAU_DRIVER_PATH to /run/current-system/profile/lib/vdpau.
>

I checked that both 48868 and 62313 were fixed in the recent updates
and closed both.

Thanks for the patches!
John

Re: Mesa vulkan layer path fix for core-updates

[Kaelyn]

Hi,

------- Original Message -------
On Tuesday, May 9th, 2023 at 4:51 AM, John Kehayias <john.kehayias@protonmail.com> wrote:

> 
> Hello,
> 
> On Tue, Apr 25, 2023 at 04:40 PM, Kaelyn wrote:
> 
> > Hi,
> > 
> > ------- Original Message -------
> > On Tuesday, April 25th, 2023 at 2:15 PM, Andreas Enge andreas@enge.fr wrote:
> > 
> > > Hello Kaelyn,
> > > 
> > > thanks for your research!
> > 
> > You're welcome! :)
> > 
> > > Am Wed, Apr 19, 2023 at 04:07:51PM +0000 schrieb Kaelyn:
> > > 
> > > > * https://issues.guix.gnu.org/62176 can be closed when
> > > > core-updates is merged, since core-updates contains mesa 22.2.4
> > > > * Though not exactly mesa-related,
> > > > https://issues.guix.gnu.org/61364 can possibly be closed now, and
> > > > almost certainly once the core-updates merge is completed. (The
> > > > ticket is a number of workarounds the user applied in early Feb to
> > > > be able to build their system profile using core-updates, to pick
> > > > up Mesa 22 for newer hardware support. I'm not sure if any of the
> > > > patches are still relevant.)
> > > 
> > > I just closed these two.
> > > 
> > > > * https://issues.guix.gnu.org/58887 looks like an alternate way of
> > > > solving the layer path issues that
> > > > https://issues.guix.gnu.org/59453 also addresses. Additionally, it
> > > > adds two new nonstandard VK_*_PATH variables to vulkan-loader,
> > > > with at least VK_ILAYER_PATH seeming very similar in functionality
> > > > to VK_LAYER_PATH and VK_ADD_LAYER_PATH described at
> > > > https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderInterfaceArchitecture.md
> > > > * https://issues.guix.gnu.org/58251 would be fixed by
> > > > https://issues.guix.gnu.org/59453
> > 
> > I feel these two can be closed once 59453 lands, as then the manifests
> > will have the store path to their corresponding shared objects. I also
> > think having the full paths in the manifests will lead to fewer
> > cross-version/cross-package mixing of objects, compared to setting and
> > using environment variables of directories to search.
> 
> 
> I haven't looked at the details, but I'm guessing these can all be
> closed now?

Yes they can be. As a quick smoke test, prior to these patches landing, "vulkaninfo > /dev/null" would consistently print out vulkan loader errors about not being able to open libVkLayer_MESA_device_select.so. (I call it a smoke test because that layer not being found isn't fatal; I first encountered the errors in a breaking way when packaging vulkan-validationlayers for use with the tutorial at https://vulkan-tutorial.com/.)

Slightly off-topic speaking of mesa bugs: another one that can be closed is https://issues.guix.gnu.org/43849 as the issue with mesa's reproducibility was fixed with a patch that landed in meson 0.59.0 (which Guix picked up in commit b15c3dd9b0 from July 2021).

> 
> > > > * https://issues.guix.gnu.org/62313 might need a modification to
> > > > mesa e.g. to add VDPAU_DRIVER_PATH as a native-search-path (one
> > > > possible solution; in my home profile I made VDPAU work by setting
> > > > "VDPAU_DRIVER_PATH=/run/current-system/profile/lib/vdpau").
> > > > * https://issues.guix.gnu.org/48868 appears to be the same
> > > > VDPAU_DRIVER_PATH issue as https://issues.guix.gnu.org/62313.
> > 
> > For the VDPAU drivers, I plan to do a little more digging and some
> > experimenting but I suspect defining VDPAU_DRIVER_PATH as a
> > native-search-path is the best way forward. I'll send a patch once
> > I've tested a change locally without having my profile set
> > VDPAU_DRIVER_PATH to /run/current-system/profile/lib/vdpau.
> 
> 
> I checked that both 48868 and 62313 were fixed in the recent updates
> and closed both.
> 
> Thanks for the patches!
> John

You're welcome! And thank you for checking and resolving the bug reports!

Cheers,
Kaelyn

Re: Mesa vulkan layer path fix for core-updates

[Maxim Cournoyer]

Hi Kaelyn,

[...]

> Yes they can be. As a quick smoke test, prior to these patches
> landing, "vulkaninfo > /dev/null" would consistently print out vulkan
> loader errors about not being able to open
> libVkLayer_MESA_device_select.so. (I call it a smoke test because that
> layer not being found isn't fatal; I first encountered the errors in a
> breaking way when packaging vulkan-validationlayers for use with the
> tutorial at https://vulkan-tutorial.com/.)
>
> Slightly off-topic speaking of mesa bugs: another one that can be
> closed is https://issues.guix.gnu.org/43849 as the issue with mesa's
> reproducibility was fixed with a patch that landed in meson 0.59.0
> (which Guix picked up in commit b15c3dd9b0 from July 2021).

I've verified that mesa builds reproducibly now and closed 43849.

Thanks for the heads-up!

-- 
Thanks,
Maxim