[Adding Eric to CC]

David Engster writes:
> Stefan Monnier writes:
>>> The short of it is that I occasionally witness point jump to elsewhere
>>> in the buffer while I'm editing and then back again.
>>
>> So it's "displayed at A" then "displayed at B" then "displayed at
>> A again"?  what happens between each one of those 3 displays?
>>
>> If "nothing", then I suspect there's something like a `sit-for'
>> somewhere that causes a redisplay in the middle of the command (i.e. in
>> the middle of a save-excursion).
>
> I sometimes see this as well, and yes, "nothing" happens between those
> three displays. I also think there's a redisplay triggered by another
> background task while Semantic does its idle parsing stuff.

I think I tracked this down. I first instrumented `sit-for' to see which
background tasks trigger it. On my machine, this happens by two things:
`jit-lock-deferred-fontify', and `display-time-event-handler'.

The display-time handler is called twice per minute: I have
(display-time) in my .emacs, and every full minute the time gets updated
in the mode-line. Also, it is called in the
gnus-after-get-new-news-hook, and since I check for new mails in the
background, it gets called through this, too.

It's kinda hard to trigger this problem through jit-lock, since its idle
time is much smaller than the one from Semantic. So I disabled it and
tried to trigger the jump by stopping typing at roughly XX:XX:59. The
semantic idle function kicks in after 1 second, and lo and behold, I saw
a jump. It's still difficult to reproduce, but I managed to get two
backtraces in the past hour, which are attached.

As you can see, the display-time-event-handler does indeed interrupt the
lexing phase. It does a `sit-for', the display jumps. Not sure what
happens after that. Does the semantic-idle function resume? Anyway,
somehow point gets back to its original position, and through
`trace-redisplay', I saw the following on stderr:

0x7f35178 (test.cpp): same window start
0x7f35178 (test.cpp): 1
redisplay_preserve_echo_area (2)
redisplay_internal 0
0x7f35178 (test.cpp): try_scrolling
redisplay_preserve_echo_area (8)
redisplay_internal 0
0x7f35178 (test.cpp): same window start
0x7f35178 (test.cpp): 1
0x7f35178 (test.cpp): try_scrolling
redisplay_preserve_echo_area (8)

I guess this just says that redisplay made point visible by scrolling,
right?

You might wonder how the display-time-event-handler can interrupt the
Semantic lexer. In the two backtraces, you see that it calls
`accept-process-output' and `input-pending-p'. This is hidden inside the
macro `semantic-throw-on-input', which can be called in code wrapped
inside `semantic-exit-on-input'; it's our poor-man's 'yield'. It's used
extensively in the idle function code, and it's just there to do a
non-local exit in case the user does something. However, now I know that
it also allows other timers to run.

If you look in the `define-lex' macro, you see that it calls
`semantic-throw-on-input' after each identified token. The problem is
that it does not restore the cursor position before that, so I guess the
fix is simply to change this call to

	   (save-excursion
	     (goto-char starting-position)
	     (semantic-throw-on-input 'lex))

Obviously, we will have to check all other calls to
`semantic-throw-on-input' for this as well. However, I also wonder if
display-time-event-handler couldn't just call `force-mode-line-update';
or does this repaint the whole display as well?

-David