bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii <eliz@gnu.org>]

> Date: Fri, 19 Jun 2015 22:07:47 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20847@debbugs.gnu.org
> 
> > Problem 2:
> > 
> > 7. Type "l", see it wrapped to the next line.
> > 
> > 8. Type backspace. See the cursor move to the second paragraph.
> > 
> > 9. Continue backspacing. See the completion popup disappear, and the text
> > being deleted in the second paragraph.
> 
> I didn't yet finish debugging this part, but I clearly see that some
> code actually _moves_ point to that place in the second paragraph, I'm
> not yet sure why.  For starters, if you turn off font-lock in the
> buffer, this second problem doesn't happen at all.  I see some weird
> interaction between JIT Font Lock and the post-command-hook installed
> by Company, they seem to somehow conspire to force point to move to
> that place.  I'll try to debug more to see why this happens.  (Any
> idea why Company's post-command-hook calls sit-for, thus forcing
> redisplay?)

I think I understand why this is happening, and I find nothing wrong
with what the display engine does under these circumstances.

In a nutshell, when a screen line ends in a newline that comes from an
overlay string, we don't want to display the cursor on that line.  The
reasons are heuristic, but they give good results, and we used this
heuristic for a very long time, so changing it now is out of question.

Due to this, when you type Backspace to delete "l" in "hell", and the
Company post-command-hook runs and puts an overlay string on the same
line that begins with a newline, the display engine decides, after
redrawing the window, that it doesn't want to put the cursor there,
and looks for an alternative place.  The first such place is after the
overlay string, so point is moved there.

The font-lock part of this riddle is that when font-lock-mode is
active in the buffer, making any changes to buffer text cause JIT Lock
to spring to action, which doesn't really do anything, but disables a
certain redisplay optimization, which bypasses the above test.

My suggestion would be to use the 'cursor' property on the overlay
string in some place where it could be picked up by the display engine
(i.e. not on a newline), to countermand this problem.  E.g., perhaps
begin the overlay string a few characters earlier, so that it replaces
part of buffer text in "hel", and have the 'cursor' property on that
part of the string.

I still think the overlay string is constructed incorrectly in this
case, something that should be fixed in Company.  The above special
setting of 'cursor' could be part of that fix.