font-lock-extend-region-function: Final refinements.

font-lock-extend-region-function: Final refinements.

[Alan Mackenzie]

Hi, Emacs!

It has been apparent for some time that the current mechanism for calling
font-lock-extend-region-function[*] from the two font-lock after-change
functions is incomplete - it needs to be supplemented by callng the
function from jit-lock-fontify-now and font-lock-default-fontify-region.

Here is a patch which does precisely this.  I will install it if nobody
sees any problems with it.

The Elisp manual still needs some updating - I plan to do this soon.

[*] The purpose of a (major mode specific) function in this hook is to
extend the region to be fontified to "safe" boundaries, such that Emacs
won't try to start or stop fontification in the middle of a syntactic
construct.


2006-04-30  Alan Mackenzie  <acm@muc.de>

        * font-lock.el (font-lock-default-fontify-region): call
        font-lock-extend-region-function, if set, to determine the region
        to fontify.

        * jit-lock.el (jit-lock-fontify-now): call
        font-lock-extend-region-function, if set, to determine the region
        to fontify.



Index: font-lock.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/font-lock.el,v
retrieving revision 1.296
diff -c -r1.296 font-lock.el
*** font-lock.el	15 Mar 2006 22:26:08 -0000	1.296
--- font-lock.el	30 Apr 2006 10:35:03 -0000
***************
*** 1025,1055 ****
    (save-buffer-state
        ((parse-sexp-lookup-properties
          (or parse-sexp-lookup-properties font-lock-syntactic-keywords))
!        (old-syntax-table (syntax-table)))
      (unwind-protect
  	(save-restriction
  	  (unless font-lock-dont-widen (widen))
  	  ;; Use the fontification syntax table, if any.
  	  (when font-lock-syntax-table
  	    (set-syntax-table font-lock-syntax-table))
!           (goto-char beg)
! 	  (setq beg (line-beginning-position))
! 	  ;; check to see if we should expand the beg/end area for
! 	  ;; proper multiline matches
! 	  (when (and (> beg (point-min))
! 		     (get-text-property (1- beg) 'font-lock-multiline))
! 	    ;; We are just after or in a multiline match.
! 	    (setq beg (or (previous-single-property-change
! 			   beg 'font-lock-multiline)
! 			  (point-min)))
  	    (goto-char beg)
! 	    (setq beg (line-beginning-position)))
!           (setq end (or (text-property-any end (point-max)
!                                            'font-lock-multiline nil)
!                         (point-max)))
! 	  (goto-char end)
! 	  ;; Round up to a whole line.
!           (unless (bolp) (setq end (line-beginning-position 2)))
  	  ;; Now do the fontification.
  	  (font-lock-unfontify-region beg end)
  	  (when font-lock-syntactic-keywords
--- 1025,1060 ----
    (save-buffer-state
        ((parse-sexp-lookup-properties
          (or parse-sexp-lookup-properties font-lock-syntactic-keywords))
!        (old-syntax-table (syntax-table))
!        (extended-region (font-lock-extend-region beg end nil)))
      (unwind-protect
  	(save-restriction
  	  (unless font-lock-dont-widen (widen))
  	  ;; Use the fontification syntax table, if any.
  	  (when font-lock-syntax-table
  	    (set-syntax-table font-lock-syntax-table))
! 	  (if extended-region
! 	      (setq beg (car extended-region)
! 		    end (cdr extended-region))
  	    (goto-char beg)
! 	    (setq beg (line-beginning-position))
! 	    ;; check to see if we should expand the beg/end area for
! 	    ;; proper multiline matches
! 	    (when (and (> beg (point-min))
! 		       (get-text-property (1- beg) 'font-lock-multiline))
! 	      ;; We are just after or in a multiline match.
! 	      (setq beg (or (previous-single-property-change
! 			     beg 'font-lock-multiline)
! 			    (point-min)))
! 	      (goto-char beg)
! 	      (setq beg (line-beginning-position)))
! 	    (setq end (or (text-property-any end (point-max)
! 					     'font-lock-multiline nil)
! 			  (point-max)))
! 	    (goto-char end)
! 	    ;; Round up to a whole line.
! 	    (unless (bolp) (setq end (line-beginning-position 2))))
! 
  	  ;; Now do the fontification.
  	  (font-lock-unfontify-region beg end)
  	  (when font-lock-syntactic-keywords
***************
*** 1085,1096 ****
  (defun font-lock-after-change-function (beg end old-len)
    (let ((inhibit-point-motion-hooks t)
  	(inhibit-quit t)
! 	(region (font-lock-extend-region beg end old-len)))
      (save-excursion
        (save-match-data
! 	(if region
  	    ;; Fontify the region the major mode has specified.
! 	    (setq beg (car region) end (cdr region))
  	  ;; Fontify the whole lines which enclose the region.
  	  (setq beg (progn (goto-char beg) (line-beginning-position))
  		end (progn (goto-char end) (line-beginning-position 2))))
--- 1090,1101 ----
  (defun font-lock-after-change-function (beg end old-len)
    (let ((inhibit-point-motion-hooks t)
  	(inhibit-quit t)
! 	(extended-region (font-lock-extend-region beg end old-len)))
      (save-excursion
        (save-match-data
! 	(if extended-region
  	    ;; Fontify the region the major mode has specified.
! 	    (setq beg (car extended-region) end (cdr extended-region))
  	  ;; Fontify the whole lines which enclose the region.
  	  (setq beg (progn (goto-char beg) (line-beginning-position))
  		end (progn (goto-char end) (line-beginning-position 2))))
Index: jit-lock.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/jit-lock.el,v
retrieving revision 1.51
diff -c -r1.51 jit-lock.el
*** jit-lock.el	20 Mar 2006 07:52:03 -0000	1.51
--- jit-lock.el	30 Apr 2006 10:35:05 -0000
***************
*** 331,337 ****
       ;; from the end of a buffer to its start, can do repeated
       ;; `parse-partial-sexp' starting from `point-min', which can
       ;; take a long time in a large buffer.
!      (let (next)
         (save-match-data
  	 ;; Fontify chunks beginning at START.  The end of a
  	 ;; chunk is either `end', or the start of a region
--- 331,337 ----
       ;; from the end of a buffer to its start, can do repeated
       ;; `parse-partial-sexp' starting from `point-min', which can
       ;; take a long time in a large buffer.
!      (let (next extended-region)
         (save-match-data
  	 ;; Fontify chunks beginning at START.  The end of a
  	 ;; chunk is either `end', or the start of a region
***************
*** 340,354 ****
  	   ;; Determine the end of this chunk.
  	   (setq next (or (text-property-any start end 'fontified t)
  			  end))
! 
! 	   ;; Decide which range of text should be fontified.
! 	   ;; The problem is that START and NEXT may be in the
! 	   ;; middle of something matched by a font-lock regexp.
! 	   ;; Until someone has a better idea, let's start
! 	   ;; at the start of the line containing START and
! 	   ;; stop at the start of the line following NEXT.
! 	   (goto-char next)  (setq next (line-beginning-position 2))
! 	   (goto-char start) (setq start (line-beginning-position))
  
             ;; Make sure the contextual refontification doesn't re-refontify
             ;; what's already been refontified.
--- 340,357 ----
  	   ;; Determine the end of this chunk.
  	   (setq next (or (text-property-any start end 'fontified t)
  			  end))
! 	   ;; Decide which range of text should be fontified.  The problem is
! 	   ;; that START and NEXT may be in the middle of something matched by
! 	   ;; a font-lock regexp.
! 	   ;; If the major mode has supplied an extend-region-function, use
! 	   ;; this to ensure clean boundaries.  Otherwise, until someone has a
! 	   ;; better idea, let's start at the start of the line containing
! 	   ;; START and stop at the start of the line following NEXT.
! 	   (if (setq extended-region (font-lock-extend-region start next nil))
! 	       (setq start (car extended-region)
! 		     next (cdr extended-region))
! 	     (goto-char next)  (setq next (line-beginning-position 2))
! 	     (goto-char start) (setq start (line-beginning-position)))
  
             ;; Make sure the contextual refontification doesn't re-refontify
             ;; what's already been refontified.
***************
*** 557,600 ****
  in case the syntax of those lines has changed.  Refontification
  will take place when text is fontified stealthily."
    (when (and jit-lock-mode (not memory-full))
!     (let ((region (font-lock-extend-region start end old-len)))
        (save-excursion
  	(with-buffer-prepared-for-jit-lock
  	 ;; It's important that the `fontified' property be set from the
  	 ;; beginning of the line, else font-lock will properly change the
  	 ;; text's face, but the display will have been done already and will
  	 ;; be inconsistent with the buffer's content.
! 	 ;; 
! 	 ;; FIXME!!! (Alan Mackenzie, 2006-03-14): If start isn't at a BOL,
! 	 ;; expanding the region to BOL might mis-fontify, should the BOL not
! 	 ;; be at a "safe" position.
! 	 (setq start (if region
! 			 (car region)
! 		       (goto-char start)
! 		       (line-beginning-position)))
! 
! 	 ;; If we're in text that matches a multi-line font-lock pattern,
! 	 ;; make sure the whole text will be redisplayed.
! 	 ;; I'm not sure this is ever necessary and/or sufficient.  -stef
! 	 (when (get-text-property start 'font-lock-multiline)
! 	   (setq start (or (previous-single-property-change
! 			    start 'font-lock-multiline)
! 			   (point-min))))
! 
! 	 (if region (setq end (cdr region)))
! 	 ;; Make sure we change at least one char (in case of deletions).
! 	 (setq end (min (max end (1+ start)) (point-max)))
  	 ;; Request refontification.
! 	 (put-text-property start end 'fontified nil))
  	;; Mark the change for deferred contextual refontification.
! 	(when jit-lock-context-unfontify-pos
! 	  (setq jit-lock-context-unfontify-pos
! 		;; Here we use `start' because nothing guarantees that the
! 		;; text between start and end will be otherwise refontified:
! 		;; usually it will be refontified by virtue of being
! 		;; displayed, but if it's outside of any displayed area in the
! 		;; buffer, only jit-lock-context-* will re-fontify it.
! 		(min jit-lock-context-unfontify-pos start)))))))
  
  (provide 'jit-lock)
  
--- 560,600 ----
  in case the syntax of those lines has changed.  Refontification
  will take place when text is fontified stealthily."
    (when (and jit-lock-mode (not memory-full))
!     (let ((extended-region (font-lock-extend-region start end old-len)))
        (save-excursion
  	(with-buffer-prepared-for-jit-lock
  	 ;; It's important that the `fontified' property be set from the
  	 ;; beginning of the line, else font-lock will properly change the
  	 ;; text's face, but the display will have been done already and will
  	 ;; be inconsistent with the buffer's content.
! 	 (if extended-region
! 	     (setq start (car extended-region)
! 		   end (cdr extended-region))
! 	   (goto-char start)
! 	   (setq start (line-beginning-position))
! 	   ;; If we're in text that matches a multi-line font-lock pattern,
! 	   ;; make sure the whole text will be redisplayed.
! 	   ;; I'm not sure this is ever necessary and/or sufficient.  -stef
! 	   (when (get-text-property start 'font-lock-multiline)
! 	     (setq start (or (previous-single-property-change
! 			      start 'font-lock-multiline)
! 			     (point-min))))
! 
! 	   ;; Make sure we change at least one char (in case of deletions).
! 	   (setq end (min (max end (1+ start)) (point-max))))
! 
  	 ;; Request refontification.
! 	 (put-text-property start end 'fontified nil)
! 
  	;; Mark the change for deferred contextual refontification.
! 	 (when jit-lock-context-unfontify-pos
! 	   (setq jit-lock-context-unfontify-pos
! 		 ;; Here we use `start' because nothing guarantees that the
! 		 ;; text between start and end will be otherwise refontified:
! 		 ;; usually it will be refontified by virtue of being
! 		 ;; displayed, but if it's outside of any displayed area in the
! 		 ;; buffer, only jit-lock-context-* will re-fontify it.
! 		 (min jit-lock-context-unfontify-pos start))))))))
  
  (provide 'jit-lock)
  


-- 
Alan.

Re: font-lock-extend-region-function: Final refinements.

[Stefan Monnier]

> It has been apparent for some time that the current mechanism for calling
> font-lock-extend-region-function[*] from the two font-lock after-change
> functions is incomplete - it needs to be supplemented by callng the
> function from jit-lock-fontify-now and font-lock-default-fontify-region.

> Here is a patch which does precisely this.  I will install it if nobody
> sees any problems with it.

Since it depends on Richard's response to our query, I'd recomend you wait
(just like I'm waiting to remove/move the after-change-function part).

Also please don't add the jit-lock-fontify-now part, since I think it's not
the right approach to solve this problem and it's not needed for correctness.


        Stefan

Re: font-lock-extend-region-function: Final refinements.

[Richard Stallman]

    Here is a patch which does precisely this.  I will install it if nobody
    sees any problems with it.

Please don't install it unless I say ok.

Re: font-lock-extend-region-function: Final refinements.

[Alan Mackenzie]

Morning, Stefan!

[ cc: emacs-devel added.]

On Thu, Aug 03, 2006 at 02:43:23PM -0400, Stefan Monnier wrote:
> > One small point - there are slightly different mechanisms in
> > jit-lock-after-change and font-lock-after-change-function for
> > extending the region.  Is there a reason for this?

> Yes.

> Basically here is what happens in f-l-after-change:
> A1 - obey f-l-extend-after-change-function.

> Here is what happens in j-l-after-change (via
> f-l-extend-jit-lock-region-after-change):
> B1 - obey f-l-extend-after-change-function.
> B2 - obey font-lock-multiline (just once: no looping).
> B3 - round up to whole lines.

> - A1 == B1.

That's not quite true.  B1 uses j-l-after-change-extend-region-functions,
which has the same purpose as f-l-extend-after-change-function, but a
different calling convention.  I can't see any reason why these two need
to be different - it seems to be an unnecessary complication.

font-lock-extend-region-functions uses yet a third calling convention,
although what it is doing is almost (but not quite) the same as A1 and
B1.

Could we not decide on the one or the other calling convention and use it
in both places?  At the very least, could we not use just one pair of
variables rather than both `font-lock-beg/end' and `jit-lock-start/end'?
With both pairs of variables, I can foresee lots of major mode code
ugliness like this:

   (set (if from-jit-a-c 'jit-lock-start 'font-lock-beg) construct-start)

> - B3 is only present so as to avoid double redisplay.  I.e. it's not needed
>   for correctness.  It could/should be replaced by the same code as what is
>   done in f-l-default-fontify-region, except that would be potentially
>   too costly.
> - B2 OTOH is important: it's the main way font-lock-multiline works: it
>   marks the part of the text that will need to be refontified before being
>   redisplayed.  It does not care about it being refontified atomically,
>   which is why it does not loop.

> Why is there no corresponding A2?  Because f-l-after-change ends up
> calling f-l-fontify-region where f-l-multiline is obeyed anyway.

I've thought this over for some time, and I'm convinced that's all
right.

> Note that the use of f-l-multiline in f-l-fontify-region is not
> reliable: it's basically using it to do /identification/ which is
> known not be reliable, but experience shows that it can be useful to
> do it, so that if a piece of text does get identified somehow at some
> point, its multilineness will usually be preserved.

I can't see that it's unreliable, but I'll take your word for it.  The
lack of f-l-multiline (or like-minded hook function) is what fouled up
the highlighting in the example I gave a day or two ago, after 500
bytes:

[This one:
#ifndef VMS   /* VMS hates multi-line '#if's */
# if !defined(ibm032)                    && \
     !defined(__convex__)                &&          \
     !(defined(vax) && !defined(ultrix)) &&           \
...
...
]

I agree with you that f-l-multiline needs to be checked in
f-l-fontify-region.

>         Stefan

-- 
Alan.

Re: font-lock-extend-region-function: Final refinements.

[Stefan Monnier]

>> > One small point - there are slightly different mechanisms in
>> > jit-lock-after-change and font-lock-after-change-function for
>> > extending the region.  Is there a reason for this?

>> Yes.

>> Basically here is what happens in f-l-after-change:
>> A1 - obey f-l-extend-after-change-function.

>> Here is what happens in j-l-after-change (via
>> f-l-extend-jit-lock-region-after-change):
>> B1 - obey f-l-extend-after-change-function.
>> B2 - obey font-lock-multiline (just once: no looping).
>> B3 - round up to whole lines.

>> - A1 == B1.

> That's not quite true.  B1 uses j-l-after-change-extend-region-functions,
> which has the same purpose as f-l-extend-after-change-function, but a
> different calling convention.  I can't see any reason why these two need
> to be different - it seems to be an unnecessary complication.

jit-lock.el is a package that provides a service used by font-lock.el and
glasses.el and potentially many more packages.  Do a grep for `font-lock'
in jit-lock.el and you'll see that it's only mentioned in docstrings
and comments.
j-l-after-change-extend-region-functions is a hook provided by jit-lock
for its client packages.  font-lock uses it for various purposes, including
to run its own f-l-extend-after-change-function.

I.e. j-l-after-change-extend-region-functions should *not* be used by major
modes for their font-lock needs.  There's f-l-extend-after-change-function
for that (with the added advantage that it also works when font-lock doesn't
use jit-lock).


        Stefan

Re: font-lock-extend-region-function: Final refinements.

[Alan Mackenzie]

Hi, Stefan!

On Fri, Aug 04, 2006 at 10:59:05AM -0400, Stefan Monnier wrote:
> >> > One small point - there are slightly different mechanisms in
> >> > jit-lock-after-change and font-lock-after-change-function for
> >> > extending the region.  Is there a reason for this?

> >> Yes.

> >> Basically here is what happens in f-l-after-change:
> >> A1 - obey f-l-extend-after-change-function.

> >> Here is what happens in j-l-after-change (via
> >> f-l-extend-jit-lock-region-after-change):
> >> B1 - obey f-l-extend-after-change-function.
> >> B2 - obey font-lock-multiline (just once: no looping).
> >> B3 - round up to whole lines.

> >> - A1 == B1.

> > That's not quite true.  B1 uses
> > j-l-after-change-extend-region-functions, which has the same purpose
> > as f-l-extend-after-change-function, but a different calling
> > convention.  I can't see any reason why these two need to be
> > different - it seems to be an unnecessary complication.

> jit-lock.el is a package that provides a service used by font-lock.el
> and glasses.el and potentially many more packages.  Do a grep for
> `font-lock' in jit-lock.el and you'll see that it's only mentioned in
> docstrings and comments.

I don't understand what you're saying here - what is the relevance of
that fact to the way the hook functions get used?

> j-l-after-change-extend-region-functions is a hook provided by jit-lock
> for its client packages.

What do you mean by "client packages" here.  Is this things like
glasses.el?  Does it include font-lock.el, does it include foo-mode.el?
Are you saying that j-l-a-c-extend-region-f is a hook purely for the use
of font-lock itself (i.e. only in the three files font-lock.el,
font-core.el and jit-lock.el)?

> font-lock uses it [j-l-after-change-extend-region-functions] for
> various purposes, including to run its own
> f-l-extend-after-change-function.

I'm fairly confused at the moment, particularly by some of the function
names.  For example, what does the "jit-lock" part of the name
`font-lock-extend-jit-lock-region-after-change' mean?  Does it mean that
this function only gets called when jit-lock is active?  Could you
possibly give these new functions doc strings (or even comments) which
explain the context in which they're called?

> I.e. j-l-after-change-extend-region-functions should *not* be used by
> major modes for their font-lock needs.  There's
> f-l-extend-after-change-function for that (with the added advantage
> that it also works when font-lock doesn't use jit-lock).

OK.  I missed the fact that f-l-extend-a-c-function gets called from
jit-lock. 

But I'm not sure you've answered my question; if you have, I've not
understood the answer.  So let me ask it again: why are there two
separate hooks for extending the region after a buffer change [that's
jit-lock-after-change-region-functions and
font-lock-extend-after-change-region-function]?  What would become more
difficult or impossible if these two were unified into a single hook?
It seems to me that the two (or three) incompatible calling conventions
for these hooks might well cause unnecessary confusion.

Again, would it be possible to simplify the whole mechanism by using
only one of the pairs `font-lock-beg/end' and `jit-lock-start/end'?

>         Stefan

-- 
Alan.

Re: font-lock-extend-region-function: Final refinements.

[Stefan Monnier]

> What do you mean by "client packages" here.  Is this things like
> glasses.el?

Yes, and font-lock.el

> Does it include font-lock.el,

Yes.  Any package which uses jit-lock-register, basically.

> does it include foo-mode.el?

If you mean a major mode "Foo", then no: major modes typically don't use
jit-lock: they use font-lock (which may or may not internally use jit-lock,
but that's mostly none of their business, unless they fiddle with
font-lock-support-mode).

> Are you saying that j-l-a-c-extend-region-f is a hook purely for the use
> of font-lock itself (i.e. only in the three files font-lock.el,
> font-core.el and jit-lock.el)?

And glasses.el, yes.

> I'm fairly confused at the moment, particularly by some of the function
> names.  For example, what does the "jit-lock" part of the name
> `font-lock-extend-jit-lock-region-after-change' mean?  Does it mean that
> this function only gets called when jit-lock is active?

Yes.

> Could you possibly give these new functions doc strings (or even comments)
> which explain the context in which they're called?

It's on the way.

> But I'm not sure you've answered my question; if you have, I've not
> understood the answer.  So let me ask it again: why are there two
> separate hooks for extending the region after a buffer change [that's
> jit-lock-after-change-region-functions and
> font-lock-extend-after-change-region-function]?

Because you can use jit-lock without font-lock and vice-versa.  One is for
jit-lock clients (e.g. font-lock), and the other for font-lock clients
(e.g. c-mode).

> Again, would it be possible to simplify the whole mechanism by using
> only one of the pairs `font-lock-beg/end' and `jit-lock-start/end'?

Why would you care?
c-mode should never need to know about jit-lock-(beg|end).


        Stefan

Re: font-lock-extend-region-function: Final refinements.

[Alan Mackenzie]

Evening, Stefan!

On Mon, Aug 07, 2006 at 01:45:02PM -0400, Stefan Monnier wrote:

[ .... ]

> If you mean a major mode "Foo", then no: major modes typically don't
> use jit-lock: they use font-lock (which may or may not internally use
> jit-lock, but that's mostly none of their business, unless they fiddle
> with font-lock-support-mode).

OK.

> > Are you saying that j-l-a-c-extend-region-f is a hook purely for the
> > use of font-lock itself (i.e. only in the three files font-lock.el,
> > font-core.el and jit-lock.el)?

> And glasses.el, yes.

glasses.el is a really neat mode!

> > I'm fairly confused at the moment, particularly by some of the
> > function names.  For example, what does the "jit-lock" part of the
> > name `font-lock-extend-jit-lock-region-after-change' mean?  Does it
> > mean that this function only gets called when jit-lock is active?

> Yes.

> > Could you possibly give these new functions doc strings (or even
> > comments) which explain the context in which they're called?

> It's on the way.

Thanks.

> > But I'm not sure you've answered my question; if you have, I've not
> > understood the answer.  So let me ask it again: why are there two
> > separate hooks for extending the region after a buffer change [that's
> > jit-lock-after-change-region-functions and
> > font-lock-extend-after-change-region-function]?

> Because you can use jit-lock without font-lock and vice-versa.  One is
> for jit-lock clients (e.g. font-lock), and the other for font-lock
> clients (e.g. c-mode).

EUREKA!!!  I hadn't realised that jit-lock could be used for things other
than fontification.  You just have to stuff a different function, such as
the glasses one, into `jit-lock-functions'.  Thanks for the
clarification!

Actually, there are places, quite a lot of them, in jit-lock.el (e.g. the
doc string for j-l-after-change-extend-r-f) which explicitly refer to
fontification.  Maybe these could be reformulated with words more
accurate and more abstract than "fontify".

> > Again, would it be possible to simplify the whole mechanism by using
> > only one of the pairs `font-lock-beg/end' and `jit-lock-start/end'?

> Why would you care?
> c-mode should never need to know about jit-lock-(beg|end).

I'm happy about that!

>         Stefan

-- 
Alan.