I believe the issue happens when the line you are moving the cursor from a line that is hscrolled to a line that is shorter than (frame_width - hscroll_margin), the issue occurs. However, if the line you are moving the cursor to is greater than or equal to (frame_width - hscroll_margin), the problem does not occur. This is more evident when you set the hscroll-margin to something smaller, like 1 or 2.
The patch I attached has modified the if statement that has this condition in it (point >= frame_width - hscroll_margin) or something like that, I don't have the code in front of me.