bug#15045: Point jumps inappropriately around time of Semantic lexing

["Eric M. Ludlam" <eric@siege-engine.com>]

On 08/08/2013 01:21 PM, David Engster wrote:
> [Adding Eric to CC]
>
> David Engster writes:
>
> It's kinda hard to trigger this problem through jit-lock, since its idle
> time is much smaller than the one from Semantic. So I disabled it and
> tried to trigger the jump by stopping typing at roughly XX:XX:59. The
> semantic idle function kicks in after 1 second, and lo and behold, I saw
> a jump. It's still difficult to reproduce, but I managed to get two
> backtraces in the past hour, which are attached.
>
> As you can see, the display-time-event-handler does indeed interrupt the
> lexing phase. It does a `sit-for', the display jumps. Not sure what
> happens after that. Does the semantic-idle function resume? Anyway,
> somehow point gets back to its original position, and through
> `trace-redisplay', I saw the following on stderr:

David, this is some impressing debugging.  Thanks for investigating so 
thoroughly.

> You might wonder how the display-time-event-handler can interrupt the
> Semantic lexer. In the two backtraces, you see that it calls
> `accept-process-output' and `input-pending-p'. This is hidden inside the
> macro `semantic-throw-on-input', which can be called in code wrapped
> inside `semantic-exit-on-input'; it's our poor-man's 'yield'. It's used
> extensively in the idle function code, and it's just there to do a
> non-local exit in case the user does something. However, now I know that
> it also allows other timers to run.
>
> If you look in the `define-lex' macro, you see that it calls
> `semantic-throw-on-input' after each identified token. The problem is
> that it does not restore the cursor position before that, so I guess the
> fix is simply to change this call to
>
> 	   (save-excursion
> 	     (goto-char starting-position)
> 	     (semantic-throw-on-input 'lex))

I pulled up some of the bigger C files in Emacs, ran the lexer, and they 
all took less than .3 second to lex.  It may also be safe to remove this 
interactive optimization and just depend on input checking that happens 
during parsing.

The detriment is that if someone had a really big file, they might 
notice that Emacs stops responding if lexing takes a long time.

Eric