Are you using the development version of Emacs (aka Emacs 24)?
If so, I have no idea whether this will help or help you find the problem, but
you might try doing this to see if it makes a difference:
(setq-default bidi-display-reordering nil)
This helped me with my slow scroll issues. Holding down arrow on a fully maximized frame on a 1080p monitor was spiking my duo-core CPU to 20-25%. Scrolling text. 50% of one CPU. Just to scroll text. And it would start jumping around almost right away. With this setting, it goes down to 8-14% while scrolling. Still kind of silly, but that's better than emacs23.