This issue seems to be gone for me with kernel 5.10.x.
I guess it was a kernel bug then.

-- 
Pierre Neidhardt
https://ambrevar.xyz/