How can I predict which regions xdisp will present me for font-locking?

How can I predict which regions xdisp will present me for font-locking?

[Alan Mackenzie]

Hi, all.

I hope this isn't one of these "send complete details" questions, but:

In a file.hpp, inside a 3355 line macro (I kid you not), I type a single
character.  xdisp responds by presenting jit-lock, successively, with
four areas to fontify.  (This was determined by
trace-function-background on `font-lock-fontify-region').

None of these areas starts/stops at the top/bottom of the window.  It is the
third area which encloses the changed line.  The fourth area finishes
below the window's last line.

I can't see how this wierd sequence of fontification areas can be
being calculated by CC Mode.

This seemingly superfluous fontification seems to be causing a
performance degradation.

How can this series of fontification regions be explained?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How can I predict which regions xdisp will present me for font-locking?

[Eli Zaretskii]

> Date: Sun, 11 Mar 2012 20:59:03 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> In a file.hpp, inside a 3355 line macro (I kid you not), I type a single
> character.  xdisp responds by presenting jit-lock, successively, with
> four areas to fontify.  (This was determined by
> trace-function-background on `font-lock-fontify-region').
> 
> None of these areas starts/stops at the top/bottom of the window.  It is the
> third area which encloses the changed line.  The fourth area finishes
> below the window's last line.
> 
> I can't see how this wierd sequence of fontification areas can be
> being calculated by CC Mode.
> 
> This seemingly superfluous fontification seems to be causing a
> performance degradation.
> 
> How can this series of fontification regions be explained?

I think what happens is this:

  . every change in the buffer triggers a call to
    font-lock-extend-jit-lock-region-after-change (via
    jit-lock-after-change-extend-region-functions)

  . font-lock-extend-jit-lock-region-after-change extends the region
    of the change according to the font-lock properties of the major
    mode, then marks that region unfontified by putting on it a
    `fontified' text property with the value nil

  . the next redisplay cycle calls jit-lock-function (via
    fontification-functions) at the beginning of the region whose
    `fontified' property is nil, and refontifies the region from there
    in chunks of 500 characters (but every chunk is extended so that
    its start is at the beginning of a line and its end is at the end
    of a line)

Does this explain what you see?

Re: How can I predict which regions xdisp will present me for font-locking?

[John Yates]

On Sun, Mar 11, 2012 at 5:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 11 Mar 2012 20:59:03 +0000
>> From: Alan Mackenzie <acm@muc.de>
>>
>> In a file.hpp, inside a 3355 line macro (I kid you not), I type a single
>> character.  xdisp responds by presenting jit-lock, successively, with
>> four areas to fontify.  (This was determined by
>> trace-function-background on `font-lock-fontify-region').
>>
>> None of these areas starts/stops at the top/bottom of the window.  It is the
>> third area which encloses the changed line.  The fourth area finishes
>> below the window's last line.
>>
>> I can't see how this wierd sequence of fontification areas can be
>> being calculated by CC Mode.
>>
>> This seemingly superfluous fontification seems to be causing a
>> performance degradation.
>>
>> How can this series of fontification regions be explained?
>
> I think what happens is this:
>
>  . every change in the buffer triggers a call to
>    font-lock-extend-jit-lock-region-after-change (via
>    jit-lock-after-change-extend-region-functions)
>
>  . font-lock-extend-jit-lock-region-after-change extends the region
>    of the change according to the font-lock properties of the major
>    mode, then marks that region unfontified by putting on it a
>    `fontified' text property with the value nil
>
>  . the next redisplay cycle calls jit-lock-function (via
>    fontification-functions) at the beginning of the region whose
>    `fontified' property is nil, and refontifies the region from there
>    in chunks of 500 characters (but every chunk is extended so that
>    its start is at the beginning of a line and its end is at the end
>    of a line)
>
> Does this explain what you see?

How does this strategy differ from 23.x?

I am seeing a horrendous slowdown when editing atypical macros (though
no more than 50 lines long).  Worse still I see the same slowdown when
editing a comment if even a fragment of my atypical macro is visible.

The pattern of the macro is

#define LIST_NAME \
  _ROW_( <literal columns> ) \
, _ROW_( <literal columns> ) \
... \
, _ROW_( <literal columns> ) \

/john

Re: How can I predict which regions xdisp will present me for font-locking?

[Eli Zaretskii]

> Date: Mon, 12 Mar 2012 13:47:00 -0400
> From: John Yates <john@yates-sheets.org>
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> 
> >  . every change in the buffer triggers a call to
> >    font-lock-extend-jit-lock-region-after-change (via
> >    jit-lock-after-change-extend-region-functions)
> >
> >  . font-lock-extend-jit-lock-region-after-change extends the region
> >    of the change according to the font-lock properties of the major
> >    mode, then marks that region unfontified by putting on it a
> >    `fontified' text property with the value nil
> >
> >  . the next redisplay cycle calls jit-lock-function (via
> >    fontification-functions) at the beginning of the region whose
> >    `fontified' property is nil, and refontifies the region from there
> >    in chunks of 500 characters (but every chunk is extended so that
> >    its start is at the beginning of a line and its end is at the end
> >    of a line)
> >
> > Does this explain what you see?
> 
> How does this strategy differ from 23.x?

It doesn't, not in the display engine.

> I am seeing a horrendous slowdown when editing atypical macros (though
> no more than 50 lines long).

Can you give more quantitative description of the slowdown?  E.g., how
much time does it take to move the cursor by one character or one
line, or to insert or delete a single character?

If you replace cc-mode Lisp files in v24 with those in v23, does the
problem go away?

Re: How can I predict which regions xdisp will present me for font-locking?

[John Yates]

On Mon, Mar 12, 2012 at 2:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Can you give more quantitative description of the slowdown?  E.g., how
> much time does it take to move the cursor by one character or one
> line, or to insert or delete a single character?

My work machine (4 year old technology):
o Intel Core2 Duo E8400 (Wolfdale)
  - dual core, 3GHz, 6MB shared L2, no L3
  - 8GB DDr2 via separate Northbridge chip
Barely tolerable, perceptible delay inserting characters

Home machine (3 year old technology):
o AMD Phenom II x4 945
  - quad core, 3GHz, 512KB private L2, 6 MB shared L3
  - 8GB DDR2 via low latency on-chip memory controller
Really painful, at least a second to insert a single character.

Both machines run Ubuntu, the same emacs-snapshot and the same set of
init files.  My guess is that the much larger low latency Intel L2
cache is what makes the difference.

The other thing I am noticing on both machine is that when I use
keyboard repeat to "C-n" or arrow down I get scrolling behaves akin to
an iPhone: after I release my key a large number of buffered input
events continue the scrolling.

> If you replace cc-mode Lisp files in v24 with those in v23, does the
> problem go away?

I am really not able to do so.  I do not build emacs from scratch.  I
simply run the current state of Julien Danjou's emacs-snapshot on
Ubuntu (11.04 at home, 11.10 at work).

/john

Re: How can I predict which regions xdisp will present me for font-locking?

[Alan Mackenzie]

Hello again, John.

On Tue, Mar 13, 2012 at 03:50:31PM -0400, John Yates wrote:
> On Mon, Mar 12, 2012 at 2:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Can you give more quantitative description of the slowdown?  E.g., how
> > much time does it take to move the cursor by one character or one
> > line, or to insert or delete a single character?

> My work machine (4 year old technology):
> o Intel Core2 Duo E8400 (Wolfdale)
>   - dual core, 3GHz, 6MB shared L2, no L3
>   - 8GB DDr2 via separate Northbridge chip
> Barely tolerable, perceptible delay inserting characters

> Home machine (3 year old technology):
> o AMD Phenom II x4 945
>   - quad core, 3GHz, 512KB private L2, 6 MB shared L3
>   - 8GB DDR2 via low latency on-chip memory controller
> Really painful, at least a second to insert a single character.

This is in CC Mode, right?  It would be _really_ helpful to have a
description of which characters (e.g., comma, semicolon) are slow, and
what the buffer looks like in the vicinity of the slowness.  Is there a
dearth of semicolons and/or braces there?

For example, I'm struggling with getting the speed up on a 3355 line C++
macro.  Typing a comma towards the end of it is taking ~0.89 seconds on
my hardware (similarly specced to your home machine).

> The other thing I am noticing on both machine is that when I use
> keyboard repeat to "C-n" or arrow down I get scrolling behaves akin to
> an iPhone: after I release my key a large number of buffered input
> events continue the scrolling.

Is this in general, or in CC Mode?  In my mega-macro, I can't reproduce
this.

> > If you replace cc-mode Lisp files in v24 with those in v23, does the
> > problem go away?

> I am really not able to do so.  I do not build emacs from scratch.  I
> simply run the current state of Julien Danjou's emacs-snapshot on
> Ubuntu (11.04 at home, 11.10 at work).

I put in a considerable speed up to (long) macros on 1st March, and
another speed up on 2nd March.  Could it perhaps be that the snapshot you
use hasn't yet been updated for these?

> /john

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How can I predict which regions xdisp will present me for font-locking?

[Eli Zaretskii]

> Date: Tue, 13 Mar 2012 15:50:31 -0400
> From: John Yates <john@yates-sheets.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> > If you replace cc-mode Lisp files in v24 with those in v23, does the
> > problem go away?
> 
> I am really not able to do so.  I do not build emacs from scratch.  I
> simply run the current state of Julien Danjou's emacs-snapshot on
> Ubuntu (11.04 at home, 11.10 at work).

Since CC Mode is not preloaded, all you need to do is copy the the
cc-*.elc files from v23 over their v24 namesakes, and then fire up
Emacs.

Re: How can I predict which regions xdisp will present me for font-locking?

[Alan Mackenzie]

Hello again, John.

On Mon, Mar 12, 2012 at 01:47:00PM -0400, John Yates wrote:
> On Sun, Mar 11, 2012 at 5:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Sun, 11 Mar 2012 20:59:03 +0000

[ .... ]

> I am seeing a horrendous slowdown when editing atypical macros (though
> no more than 50 lines long).  Worse still I see the same slowdown when
> editing a comment if even a fragment of my atypical macro is visible.

What does horrendous mean?  It could mean 10s per key (i.e. totally
broken) or ¼s per key (very sluggish).  Is the horrenditity any worse 
typing the comma?

> The pattern of the macro is

> #define LIST_NAME \
>   _ROW_( <literal columns> ) \
> , _ROW_( <literal columns> ) \
> ... \
> , _ROW_( <literal columns> ) \

Here's an update to the latest current state of my optimisations.  Try it
out, please, and see if it make things any better.


=== modified file 'lisp/progmodes/cc-engine.el'
*** lisp/progmodes/cc-engine.el	2012-03-02 22:16:21 +0000
--- lisp/progmodes/cc-engine.el	2012-03-12 18:32:00 +0000
***************
*** 1246,1252 ****
  		       (c-at-vsemi-p))))
  	      (throw 'done vsemi-pos))
  	     ;; In a string/comment?
! 	     ((setq lit-range (c-literal-limits))
  	      (goto-char (cdr lit-range)))
  	     ((eq (char-after) ?:)
  	      (forward-char)
--- 1246,1252 ----
  		       (c-at-vsemi-p))))
  	      (throw 'done vsemi-pos))
  	     ;; In a string/comment?
! 	     ((setq lit-range (c-literal-limits from))
  	      (goto-char (cdr lit-range)))
  	     ((eq (char-after) ?:)
  	      (forward-char)
***************
*** 3250,3257 ****
  	      (if scan-forward-p
  		  (progn (narrow-to-region (point-min) here)
  			 (c-append-to-state-cache good-pos))
! 
! 		(c-get-cache-scan-pos good-pos))))
  
         (t ; (eq strategy 'IN-LIT)
  	(setq c-state-cache nil
--- 3250,3256 ----
  	      (if scan-forward-p
  		  (progn (narrow-to-region (point-min) here)
  			 (c-append-to-state-cache good-pos))
! 		good-pos)))
  
         (t ; (eq strategy 'IN-LIT)
  	(setq c-state-cache nil
***************
*** 4563,4568 ****
--- 4562,4599 ----
  	(point-min))
         (t
  	(c-determine-limit (- how-far-back count) base try-size))))))
+ 
+ (defun c-determine-+ve-limit (how-far &optional start-pos)
+   ;; Return a buffer position about HOW-FAR non-literal characters forward
+   ;; from START-POS (default point), which must not be inside a literal.
+   (save-excursion
+     (let ((pos (or start-pos (point)))
+ 	  (count how-far)
+ 	  (s (parse-partial-sexp (point) (point)))) ; null state
+       (while (and (not (eobp))
+ 		  (> count 0))
+ 	;; Scan over counted characters.
+ 	(setq s (parse-partial-sexp
+ 		 pos
+ 		 (min (+ pos count) (point-max))
+ 		 nil			; target-depth
+ 		 nil			; stop-before
+ 		 s			; state
+ 		 'syntax-table))	; stop-comment
+ 	(setq count (- count (- (point) pos) 1)
+ 	      pos (point))
+ 	;; Scan over literal characters.
+ 	(if (nth 8 s)
+ 	    (setq s (parse-partial-sexp
+ 		     pos
+ 		     (point-max)
+ 		     nil		; target-depth
+ 		     nil		; stop-before
+ 		     s			; state
+ 		     'syntax-table)	; stop-comment
+ 		  pos (point))))
+       (point))))
+ 
  \f
  ;; `c-find-decl-spots' and accompanying stuff.
  
***************
*** 8635,8641 ****
  	(setq pos (point)))
        (and
         c-macro-with-semi-re
-        (not (c-in-literal))
         (eq (skip-chars-backward " \t") 0)
  
         ;; Check we've got nothing after this except comments and empty lines
--- 8666,8671 ----
***************
*** 8666,8672 ****
  	     (c-backward-syntactic-ws)
  	     t))
         (c-simple-skip-symbol-backward)
!        (looking-at c-macro-with-semi-re)))))
  
  (defun c-macro-vsemi-status-unknown-p () t) ; See cc-defs.el.
  
--- 8696,8704 ----
  	     (c-backward-syntactic-ws)
  	     t))
         (c-simple-skip-symbol-backward)
!        (looking-at c-macro-with-semi-re)
!        (goto-char pos)
!        (not (c-in-literal))))))		; The most expensive check last. 
  
  (defun c-macro-vsemi-status-unknown-p () t) ; See cc-defs.el.
  
***************
*** 9571,9577 ****
  	 ;; CASE 5B: After a function header but before the body (or
  	 ;; the ending semicolon if there's no body).
  	 ((save-excursion
! 	    (when (setq placeholder (c-just-after-func-arglist-p lim))
  	      (setq tmp-pos (point))))
  	  (cond
  
--- 9603,9610 ----
  	 ;; CASE 5B: After a function header but before the body (or
  	 ;; the ending semicolon if there's no body).
  	 ((save-excursion
! 	    (when (setq placeholder (c-just-after-func-arglist-p
! 				     (max lim (c-determine-limit 500))))
  	      (setq tmp-pos (point))))
  	  (cond
  

=== modified file 'lisp/progmodes/cc-mode.el'
*** lisp/progmodes/cc-mode.el	2012-02-22 19:34:32 +0000
--- lisp/progmodes/cc-mode.el	2012-03-12 18:31:49 +0000
***************
*** 925,932 ****
      ;; inside a string, comment, or macro.
      (setq new-bounds (c-extend-font-lock-region-for-macros
  		      c-new-BEG c-new-END old-len))
!     (setq c-new-BEG (car new-bounds)
! 	  c-new-END (cdr new-bounds))
      ;; Clear all old relevant properties.
      (c-clear-char-property-with-value c-new-BEG c-new-END 'syntax-table '(1))
      (c-clear-char-property-with-value c-new-BEG c-new-END 'category 'c-cpp-delimiter)
--- 925,932 ----
      ;; inside a string, comment, or macro.
      (setq new-bounds (c-extend-font-lock-region-for-macros
  		      c-new-BEG c-new-END old-len))
!     (setq c-new-BEG (max (car new-bounds) (c-determine-limit 500 begg))
! 	  c-new-END (min (cdr new-bounds) (c-determine-+ve-limit 500 endd)))
      ;; Clear all old relevant properties.
      (c-clear-char-property-with-value c-new-BEG c-new-END 'syntax-table '(1))
      (c-clear-char-property-with-value c-new-BEG c-new-END 'category 'c-cpp-delimiter)


> /john

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How can I predict which regions xdisp will present me for font-locking?

[Stefan Monnier]

> I hope this isn't one of these "send complete details" questions, but:

> In a file.hpp, inside a 3355 line macro (I kid you not), I type a single
> character.  xdisp responds by presenting jit-lock, successively, with
> four areas to fontify.  (This was determined by
> trace-function-background on `font-lock-fontify-region').

While "complete details" would be nice, just showing the actual four
areas, plus the corresponding position of point (and maybe also the
position of beginning and end of the macro) might be a good starting
point,


        Stefan

Re: How can I predict which regions xdisp will present me for font-locking?

[Alan Mackenzie]

Hello, Stefan.

On Mon, Mar 12, 2012 at 03:25:26PM -0400, Stefan Monnier wrote:
> > I hope this isn't one of these "send complete details" questions, but:

> > In a file.hpp, inside a 3355 line macro (I kid you not), I type a single
> > character.  xdisp responds by presenting jit-lock, successively, with
> > four areas to fontify.  (This was determined by
> > trace-function-background on `font-lock-fontify-region').

> While "complete details" would be nice, just showing the actual four
> areas, plus the corresponding position of point (and maybe also the
> position of beginning and end of the macro) might be a good starting
> point,

The font locking boundaries were indeed set by CC Mode. 

Sorry for wasting people's time.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).