Ergonomics: mouse clicking with scroll-margin

[Yuri D'Elia <wavexx@thregr.org>]

The combined behavior of the mouse and "scroll-margin" is a bit odd
currently and I wonder if there's something that can be done to improve
it.

If you have a buffer which has "scroll-margin" enabled, and you click
anywhere in the scroll-margin region, the point is moved and the buffer
is immediately scrolled.

This in turn often results in the click becoming a "drag" instead,
selecting parts of the buffer.

I rarely use the mouse, but when I do (usually in customize groups, or
when using eww), it trips me all the time since I use a rather large
scroll-margin.

Can this be improved _without_ disabling scroll-margin?

If feels like that moving the point when using the mouse shouldn't
trigger scroll-margin ever, since any action that causes a scroll would
immediately move the current target away and result in unintended
behavior. I can see the same effect when performing a selection by
dragging: once you reach the scroll-margin area every single motion
events selects half the visible buffer, which is also broken.

Is there a better approach?