I can confirm the problem is exclusively related to automatic point moves which happen during scrolling, landing on an invisible character at the start of a line.  This seems to happen due to automatic redisplay point repositioning, and not line-move, for example.

I tested Po Lu’s new updates in master to pixel-scroll-precision (of Aug, 2023).  With the new version, the behavior w.r.t. invisible chars has changed: when point moves to an invisible character as you scroll point towards the window's bottom, that line jumps back up to the middle of the window, but is not “trapped”: scroll can continue, and you can get past the line.  As before, scrolling the other direction, from top of buffer to bottom does not place point on invisible characters and so is unaffected.

So some improvement, but still not correct.