Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

[emacs-devel@gnu.org has been added]

[The discussion is about a font-locking problem reported in CC Mode:  The
user types the start of a C function, but puts a NL in the middle:

void function
                        (int x);

The second line gets incorrectly fontified, because in its
after-change-functions, font-lock considers only that line in and of
itself.]

Hi, Richard!

On Wed, 9 Mar 2005, Richard Stallman wrote:

>    Why not use the mechanism from AWK Mode, which was designed to
>    refont the opening quote of a (multi-line) string when the closing
>    quote gets typed?  This device advises the four critical font-lock
>    functions (e.g.  `font-lock-after-change-function').

>The technique may be a good one, but advising other parts of Emacs is a
>bug!

Hmmm.  The advice facility exists within Emacs, and it is difficult to
see what it could be used on, aside from another part of Emacs.  After
all, one does not need to advise one's own code, having full control over
the source.

But I take the point.  Advising font-lock is suboptimal, since it ties
AWK Mode to the precise internal implementation of font-lock.  Should
font-lock's internals change, AWK Mode will break.  Also, this use of
advice is ugly, and that's reason enough in itself to change it.  But
what we change it to must be _less_ ugly.

However:  I did complain about this shortcoming of font-lock on
gnu.emacs.bug way back, on Friday 10th May 2002 (Subject: Font-lock:
Major mode should be able to specify region to fontify.  Message-Id:
<r77hba.r5.ln@acm.acm>), and that post of mine contained a patch to add
the facility to font-lock.  Nobody apart from Kim Storm (who objected
(quite reasonably) to the prolix function names I'd used) responded to
the post at the time, and nothing happened.  That's when I decided that
advising font-lock was the least bad way of solving the problem. 

>Let's look for a way to fix this to avoid using advice.  One idea is to
>have a hook that these functions run which can extend the region to be
>fontified.

By "these functions", I take it you mean
"font-lock-after-change-function" etc., rather than the AWK Mode
functions?  I believe it is a rather common need for Major modes to
calculate the region to fontify.  If so, let me start the ball rolling by
reposting my patch from 2002.  I can't guarantee it'll still work (or
even that it worked properly in 2002), but if you agree with the approach
in principle, I am prepared to rework the patch for a modern CVS version,
and amend the appropriate elisp.texi file.


"The context-diff is based on the source files from Emacs 21.1."

;;;;;; Changelog entry
2002-05-10  Alan Mackenzie  <acm@muc.de>

        The following set of changes allows a major mode dynamically to
        specify the region to fontify after a buffer change. (fast-lock.el
        doesn't need changing.)
        * font-lock.el
        New abnormal hook before-font-lock-after-change-function,
        New function font-lock-run-before-after-change-hook.
        Changed font-lock-after-change-function.
        * jit-lock.el
        Changed jit-lock-after-change
        * lazy-lock.el
        Changed lazy-lock-defer-line-after-change and
        lazy-lock-defer-rest-after-change.
;;;;;; End of Changelog entry

;;;;;; Patch
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/font-lock.el NEW/font-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/font-lock.el	Wed Sep  5 13:36:29 2001
--- NEW/font-lock.el	Fri May 10 19:24:25 2002
***************
*** 1260,1270 ****
  				'(face nil syntax-table nil font-lock-multiline nil)
  			      '(face nil font-lock-multiline nil)))))
  
  ;; Called when any modification is made to buffer text.
  (defun font-lock-after-change-function (beg end old-len)
!   (let ((inhibit-point-motion-hooks t))
      (save-excursion
        (save-match-data
  	;; Rescan between start of lines enclosing the region.
  	(font-lock-fontify-region
  	 (progn (goto-char beg) (beginning-of-line) (point))
--- 1260,1309 ----
  				'(face nil syntax-table nil font-lock-multiline nil)
  			      '(face nil font-lock-multiline nil)))))
  
+ (defvar before-font-lock-after-change-function nil
+   "If set to a function, this can specify the region to fontify after a change.
+ 
+ This variable is a buffer-local abnormal hook whose value, if set, should be a
+ single function.  The function gets called from the active font-lock
+ after-change function, and is intended to allow a major mode to calculate for
+ itself the region to be fontified after a buffer change.
+ 
+ The function is given three parameters, the standard BEG, END and OLD-LEN from
+ after-change-functions.  It should return either a cons of the beginning and end
+ buffer-positions \(in that order\) of the region to fontify, or nil (in which
+ case the default region will be used).")
+ (make-variable-buffer-local 'before-font-lock-after-change-function)
+ 
+ (defun font-lock-run-before-after-change-hook (beg end old-len)
+   "Run the hook function, if any, in before-font-lock-after-change-function,
+ returning its value (a cons of beg and end), if it's valid, else nil.
+ 
+ BEG END and OLD-LEN are the three parameters supplied by
+ after-change-functions."
+   (cond ((null before-font-lock-after-change-function) nil)
+         ((not (functionp before-font-lock-after-change-function))
+          (message "before-font-lock-after-change-function's value is not a \
+ function in buffer %S"
+                   (buffer-name))
+          nil)
+         (t (let ((region (funcall before-font-lock-after-change-function beg end old-len)))
+              (cond ((null region) nil)
+                    ((or (not (consp region))
+                         (not (wholenump (car region))) (not (wholenump (cdr region)))
+                         (not (<= (car region) (cdr region))))
+                     (message "before-font-lock-after-change-function returned %S.  \
+ This isn't a cons \(beg.end\), with beg and end numbers, and beg <= end")
+                     nil)
+                    (t region))))))
+ 
  ;; Called when any modification is made to buffer text.
  (defun font-lock-after-change-function (beg end old-len)
!   (let ((inhibit-point-motion-hooks t) region)
      (save-excursion
        (save-match-data
+         ;; Does the major mode have its own ideas about the region to fontify?
+         (setq region (font-lock-run-before-after-change-hook beg end old-len))
+         (if region (setq beg (car region)  end (cdr region)))
  	;; Rescan between start of lines enclosing the region.
  	(font-lock-fontify-region
  	 (progn (goto-char beg) (beginning-of-line) (point))
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/jit-lock.el NEW/jit-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/jit-lock.el	Mon Jul 16 12:22:58 2001
--- NEW/jit-lock.el	Fri May 10 18:59:36 2002
***************
*** 426,431 ****
--- 426,434 ----
    (when jit-lock-mode
      (save-excursion
        (with-buffer-prepared-for-jit-lock
+        ;; Does the major mode have its own ideas about the region to fontify?
+        (let ((region (font-lock-run-before-after-change-hook start end old-len)))
+          (if region (setq start (car region)  end (cdr region))))
         ;; It's important that the `fontified' property be set from the
         ;; beginning of the line, else font-lock will properly change the
         ;; text's face, but the display will have been done already and will
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/lazy-lock.el NEW/lazy-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/lazy-lock.el	Thu Aug 16 14:25:15 2001
--- NEW/lazy-lock.el	Fri May 10 19:00:47 2002
***************
*** 771,776 ****
--- 771,779 ----
    (save-buffer-state nil
      (unless (memq (current-buffer) lazy-lock-buffers)
        (push (current-buffer) lazy-lock-buffers))
+     ;; Does the major mode have its own ideas about what to fontify?
+     (let ((region (font-lock-run-before-after-change-hook beg end old-len)))
+       (if region (setq beg (car region)  end (cdr region))))
      (remove-text-properties (max (1- beg) (point-min))
  			    (min (1+ end) (point-max))
  			    '(lazy-lock nil))))
***************
*** 784,789 ****
--- 787,795 ----
        (push (current-buffer) lazy-lock-buffers))
      (save-restriction
        (widen)
+       ;; Does the major mode have its own ideas about what to fontify?
+       (let ((region (font-lock-run-before-after-change-hook beg end old-len)))
+         (if region (setq beg (car region)  end (cdr region))))
        (remove-text-properties (max (1- beg) (point-min))
  			      (point-max)
  			      '(lazy-lock nil)))))
;;;;;; end of patch.

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> [The discussion is about a font-locking problem reported in CC Mode:  The
> user types the start of a C function, but puts a NL in the middle:

> void function
>                         (int x);

> The second line gets incorrectly fontified, because in its
> after-change-functions, font-lock considers only that line in and of
> itself.]

[...]

> However:  I did complain about this shortcoming of font-lock on
> gnu.emacs.bug way back, on Friday 10th May 2002 (Subject: Font-lock:
> Major mode should be able to specify region to fontify.  Message-Id:
> <r77hba.r5.ln@acm.acm>), and that post of mine contained a patch to add
> the facility to font-lock.  Nobody apart from Kim Storm (who objected
> (quite reasonably) to the prolix function names I'd used) responded to
> the post at the time, and nothing happened.  That's when I decided that
> advising font-lock was the least bad way of solving the problem. 

Hmm... I can't remember such a post and I have a hard time imagining why
I wouldn't have jumped on it.

Could you explain smoe more about your two cases (the one in awk and the
one in the subject of this article):
- how do they use your new before-font-lock-after-change-function hook?
- why can't they use the font-lock-multiline property?
- why can't they use the jit-lock-defer-multiline property?

This will allow us the better judge which is the better course of action.
Using advice on font-lock functions from awk-mode is pretty ugly, so it's
important we come up with a better way to solve this issue.


        Stefan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Stefan.

On Wed, 9 Mar 2005, Stefan Monnier wrote:

>> [The discussion is about a font-locking problem reported in CC Mode:  The
>> user types the start of a C function, but puts a NL in the middle:

>> void function
>>                         (int x);

>> The second line gets incorrectly fontified, because in its
>> after-change-functions, font-lock considers only that line in and of
>> itself.]

>[...]

>> However:  I did complain about this shortcoming of font-lock on
>> gnu.emacs.bug way back, on Friday 10th May 2002 (Subject: Font-lock:
>> Major mode should be able to specify region to fontify.  Message-Id:
>> <r77hba.r5.ln@acm.acm>), and that post of mine contained a patch to add
>> the facility to font-lock.  Nobody apart from Kim Storm (who objected
>> (quite reasonably) to the prolix function names I'd used) responded to
>> the post at the time, and nothing happened.  That's when I decided that
>> advising font-lock was the least bad way of solving the problem. 

>Hmm... I can't remember such a post and I have a hard time imagining why
>I wouldn't have jumped on it.

We talked about it in bug-cc-mode just after CC Mode 5.30 had been
released.  (Subject: Advising in cc-awk.el and namespace   Date: Wed, 09
Jul 2003 13:20:20 -0400).  I drew attention then (Jul 2003) to the patch
I had submitted in 2002, which you acknowledged.  However, the topic of
conversation drifted onto why jit-lock consumed so much processor power.

>Could you explain some more about your two cases (the one in awk and the
>one in the subject of this article):
>- how do they use your new before-font-lock-after-change-function hook?

The one in AWK is with a string like this:

print "multi-l\
ine\
 string
       ^
       |
     point

The opening quote has been given warning font, since it is an unmated
quote.  When the closing quote is typed, the opening quote's face gets
changed to string face.  This is done by the advice on
font-lock-after-change-function (or the jit-lock thingy ...) amending the
BEGIN parameter to point to the start of the "print" line, before passing
this into font-lock.

The C example at the top is essentially the same; the single line
containing "(int x);" currently gets fontified by after-change in
isolation; correct fontification requires that it be fontified with the
previous line as a unit.

The patch I proposed would achieve this result via a user supplied
function.

>- why can't they use the font-lock-multiline property?

I couldn't find any documentation for the facility back in 2002, and I
couldn't figure out from the source code exactly what it did.  I've now
got it working for me in Texinfo mode, though.

Also, font-lock-multiline doesn't (yet?) exist in XEmacs, so I'd need to
implement a second solution for XEmacs if f-l-multiline was used.  CC
Mode still supports Emacs 20.x, and I think I found that f-l-multiline
was introduced later than 20.1 (though I can't remember exactly).

>- why can't they use the jit-lock-defer-multiline property?

I'll have to look at that one.  But presumably that doesn't exist in both
of XEmacs and Emacs 20.x either.

>This will allow us the better judge which is the better course of
>action.  Using advice on font-lock functions from awk-mode is pretty
>ugly, so it's important we come up with a better way to solve this
>issue.

Agreed on all points.  I think, as I have done since 2002, that the
major-mode should be able to supply functions to determine the start and
end of the region to be fontified.   It seems a natural and idiomatic
Emacsy way of doing things.

Even so, it would be several years before major modes could start using
it in earnest, due to the need to continue supporting older Emacs
versions.  (CC Mode 5.30 gave up support for 19.34 by starting to use
text properties.)  How long is it going to be before 22.1 becomes the
oldest Emacs version supported by a major mode?

>        Stefan

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> We talked about it in bug-cc-mode just after CC Mode 5.30 had been
> released.  (Subject: Advising in cc-awk.el and namespace   Date: Wed, 09
> Jul 2003 13:20:20 -0400).  I drew attention then (Jul 2003) to the patch
> I had submitted in 2002, which you acknowledged.  However, the topic of
> conversation drifted onto why jit-lock consumed so much processor power.

Ah...topic drift.  Sorry about not being more careful.

>> Could you explain some more about your two cases (the one in awk and the
>> one in the subject of this article):
>> - how do they use your new before-font-lock-after-change-function hook?

> The one in AWK is with a string like this:

> print "multi-l\
> ine\
>  string
>        ^
>        |
>      point

> The opening quote has been given warning font, since it is an unmated
> quote.  When the closing quote is typed, the opening quote's face gets
> changed to string face.  This is done by the advice on
> font-lock-after-change-function (or the jit-lock thingy ...) amending the
> BEGIN parameter to point to the start of the "print" line, before passing
> this into font-lock.

> The C example at the top is essentially the same; the single line
> containing "(int x);" currently gets fontified by after-change in
> isolation; correct fontification requires that it be fontified with the
> previous line as a unit.

> The patch I proposed would achieve this result via a user supplied
> function.

Well my question was really about "how" rather than "why": what does the
elisp hook code look like for each of the two cases.

>> - why can't they use the font-lock-multiline property?

> I couldn't find any documentation for the facility back in 2002, and I
> couldn't figure out from the source code exactly what it did.  I've now
> got it working for me in Texinfo mode, though.

> Also, font-lock-multiline doesn't (yet?) exist in XEmacs, so I'd need to
> implement a second solution for XEmacs if f-l-multiline was used.  CC
> Mode still supports Emacs 20.x, and I think I found that f-l-multiline
> was introduced later than 20.1 (though I can't remember exactly).

Your new hook won't exist in Emacs-20 either, so this seems irrelevant to
the discussion at hand.  It just risks drifting the topic.

>> - why can't they use the jit-lock-defer-multiline property?

> I'll have to look at that one.  But presumably that doesn't exist in both
> of XEmacs and Emacs 20.x either.

Again, irrelevant.

> I think, as I have done since 2002, that the major-mode should be able to
> supply functions to determine the start and end of the region to be
> fontified.   It seems a natural and idiomatic Emacsy way of doing things.

You may be right, but I'd like to see your code first, to see how easy and
robust it is to use your hook.

On a side note, I'm wondering why you go through so much trouble with the
"unmatched opening quote" in AWK.  No other mode does that AFAIK, even
though pretty much every major mode in lisp/progmodes/ has pretty much the
same need AFAICT.  Does it have to do with something specific to AWK, or is
it non-specific and maybe it should be done for all modes?


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[D. R. E. Moonfire]

> On a side note, I'm wondering why you go through so much trouble with the
> "unmatched opening quote" in AWK.  No other mode does that AFAIK, even
> though pretty much every major mode in lisp/progmodes/ has pretty much the
> same need AFAICT.  Does it have to do with something specific to AWK, or is
> it non-specific and maybe it should be done for all modes?

I know that C# allows that type of string:

   string a = @"I
                   like
              cheese";

... is perfectly valid (at least to my complier) and a test executable. 
So, it might be something useful for the work I'm doing with the baby 
csharp-mode stuff. :)

Cheers,
Dylan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> On a side note, I'm wondering why you go through so much trouble with the
>> "unmatched opening quote" in AWK.  No other mode does that AFAIK, even
>> though pretty much every major mode in lisp/progmodes/ has pretty much the
>> same need AFAICT.  Does it have to do with something specific to AWK, or is
>> it non-specific and maybe it should be done for all modes?

> I know that C# allows that type of string:

>    string a = @"I
>                    like
>               cheese";

> ... is perfectly valid

If it is, then current normal Emacs support is already perfect.  Alan's code
seems to be trying to take care of other situations.


        Stefan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Stephan.

On Thu, 10 Mar 2005, Stefan Monnier wrote:

>> We talked about it in bug-cc-mode just after CC Mode 5.30 had been
>> released.  (Subject: Advising in cc-awk.el and namespace   Date: Wed, 09
>> Jul 2003 13:20:20 -0400).  I drew attention then (Jul 2003) to the patch
>> I had submitted in 2002, which you acknowledged.  However, the topic of
>> conversation drifted onto why jit-lock consumed so much processor power.

>Ah...topic drift.  Sorry about not being more careful.

I think it was more me that derailed the topic, but it was a long time
ago.

>>> Could you explain some more about your two cases (the one in awk and
>>> the one in the subject of this article): - how do they use your new
>>> before-font-lock-after-change-function hook?

[ .... ]

>> The patch I proposed would achieve this result via a user supplied
>> function.

>Well my question was really about "how" rather than "why": what does the
>elisp hook code look like for each of the two cases.

Sorry, I think I was a bit confused in my last post.  We might be talking
at cross purposes.  The "before-font-lock-after-change-function-hook"
never made it into any released code.  The advice around
font-lock-after-change-function, etc., got into cc-awk.el.

>>> - why can't they use the font-lock-multiline property?

[ .... ]

>> Also, font-lock-multiline doesn't (yet?) exist in XEmacs, so I'd need to
>> implement a second solution for XEmacs if f-l-multiline was used.  CC
>> Mode still supports Emacs 20.x, and I think I found that f-l-multiline
>> was introduced later than 20.1 (though I can't remember exactly).

>Your new hook won't exist in Emacs-20 either, so this seems irrelevant to
>the discussion at hand.  It just risks drifting the topic.

Hmm.  Yes, but-  If it got into Emacs now, in 6 or 7 years time
major-moders could with good conscience say "we're now dropping support
for all versions previous to 22.1", just like we recently said "we're
dropping support for Emacs 19.xx, so as to use text properties".  It's
just a fact of life that major modes can't really use "new" elisp
features until they've penetrated fully into the installed user base -
after several years and several releases.  But that's no reason for not
making these enhancements.

[ .... ]

>> I think, as I have done since 2002, that the major-mode should be able
>> to supply functions to determine the start and end of the region to be
>> fontified.   It seems a natural and idiomatic Emacsy way of doing
>> things.

>You may be right, but I'd like to see your code first, to see how easy and
>robust it is to use your hook.

I re-posted my tentative 2002 patch here last night.  Or did you mean
something else?  I think it's robust enough, but it could certainly do
with some tidying up.  To use it, one must merely set a buffer local
variable to the major-mode's function, this function being supplied the
after-change's BEGIN END and OLD-LEN, and returning (NEW-BEGIN .
NEW-END).  I think that variable would be better called
`font-lock-get-region-function' than the
`before-font-lock-after-change-function' I proposed back then.

>On a side note, I'm wondering why you go through so much trouble with
>the "unmatched opening quote" in AWK.  No other mode does that AFAIK,
>even though pretty much every major mode in lisp/progmodes/ has pretty
>much the same need AFAICT.  Does it have to do with something specific
>to AWK, or is it non-specific and maybe it should be done for all modes?

No, it was a pure act of unadulterated joyous hacking.  It wasn't
difficult in AWK, a line oriented language, because the "safe place" from
which to begin font-locking is merely the most recent BOL which isn't
preceded by a \.  In C, it would be somewhat trickier, but not too
difficult.

I think this putting of warning-face on unmatched quotes would be a good
idea generally.  Not stunningly important in itself, but one of these
minor details, the sum total of which makes Emacs such a joy to use.

>        Stefan

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> You may be right, but I'd like to see your code first, to see how easy and
>> robust it is to use your hook.
> I re-posted my tentative 2002 patch here last night.

Read again: I said "use" not "define".
I.e. I don't want to see the font-lock side of the code, but the awk-mode or
c++-mode side of it.  Most importantly, I'd like to see some code that would
solve the problem at hand (see subject) using your new hook.

> I think this putting of warning-face on unmatched quotes would be a good
> idea generally.

It would help if you actually explained what you're talking about.
Not everyone uses awk-mode.


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Martin Stjernholm]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> - why can't they use the font-lock-multiline property?
> - why can't they use the jit-lock-defer-multiline property?

One argument for a hook alternative to these things is that it might
not be efficient/convenient to spread text properties all over the
place:

o  Text properties stay behind when the buffer changes, and so they
   might become invalid. Adding code to correctly remove them before
   that can be tricky.

o  Putting text properties in place and handling stickiness etc
   properly is decidedly more complex than just returning a region
   from a hook.

o  Text properties take space and make various buffer operations
   (slightly) slower. If large quantities of them would be needed they
   might have noticeable effect on performance. (There are already
   plenty enough in fontified buffers.)

Providing a hook where the font lock pattern writer can extended the
region to refontify is imho a very straightforward way to give control
over that aspect. I don't contend that the text properties probably
are more suitable in many cases, though.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

On Thu, 10 Mar 2005, Stefan Monnier wrote:

>>> You may be right, but I'd like to see your code first, to see how easy and
>>> robust it is to use your hook.
>> I re-posted my tentative 2002 patch here last night.

>Read again: I said "use" not "define".
>I.e. I don't want to see the font-lock side of the code, but the awk-mode or
>c++-mode side of it.  Most importantly, I'd like to see some code that would
>solve the problem at hand (see subject) using your new hook.

Sorry, I didn't quite understand what you were asking for.  I hope the
following will do.  I can't produce any working c-mode code for this in a
hurry.  But here is the stuff from cc-awk.el.  It has the functionality
to fit the hook I'd proposed, but not the same interface.

What it does is to expand the region in an AWK buffer to be
after-change-font-locked from a physical line to the containing
@dfn{logical line}, the maximal sequence of physical lines joined by
escaped newlines.  It does the Right Thing when a backslash at an EOL
gets inserted or deleted.

(defun c-awk-beginning-of-logical-line (&optional pos)
;; Go back to the start of the (apparent) current line (or the start of the
;; line containing POS), returning the buffer position of that point.  I.e.,
;; go back to the last line which doesn't have an escaped EOL before it.
;;
;; This is guaranteed to be "safe" for syntactic analysis, i.e. outwith any
;; comment, string or regexp.  IT MAY WELL BE that this function should not be
;; executed on a narrowed buffer.
;;
;; This function might do hidden buffer changes.
  (if pos (goto-char pos))
  (forward-line 0)
  (while (and (> (point) (point-min))
              (eq (char-before (1- (point))) ?\\))
    (forward-line -1))
  (point))

(defun c-awk-end-of-logical-line (&optional pos)
;; Go forward to the end of the (apparent) current logical line (or the end of
;; the line containing POS), returning the buffer position of that point.  I.e.,
;; go to the end of the next line which doesn't have an escaped EOL.
;;
;; This is guaranteed to be "safe" for syntactic analysis, i.e. outwith any
;; comment, string or regexp.  IT MAY WELL BE that this function should not be
;; executed on a narrowed buffer.
;;
;; This function might do hidden buffer changes.
  (if pos (goto-char pos))
  (end-of-line)
  (while (and (< (point) (point-max))
              (eq (char-before) ?\\))
    (end-of-line 2))
  (point))

(defvar c-awk-old-EOLL 0)
(make-variable-buffer-local 'c-awk-old-EOLL)
;; End of logical line following the region which is about to be changed.  Set
;; in c-awk-before-change and used in c-awk-after-change.

(defun c-awk-before-change (beg end)
;; This function is called exclusively from the before-change-functions hook.
;; It does two things: Finds the end of the (logical) line on which END lies,
;; and clears c-awk-NL-prop text properties from this point onwards.
;;
;; This function might do hidden buffer changes.
  (save-restriction
    (save-excursion
      (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
      (c-save-buffer-state nil
       (c-awk-clear-NL-props end (point-max))))))

(defun c-awk-end-of-change-region (beg end old-len)
  ;; Find the end of the region which needs to be font-locked after a change.
  ;; This is the end of the logical line on which the change happened, either
  ;; as it was before the change, or as it is now, which ever is later.
  ;; N.B. point is left undefined.
  ;;
  ;; This function might do hidden buffer changes.
  (max (+ (- c-awk-old-EOLL old-len) (- end beg))
       (c-awk-end-of-logical-line end)))

;; ACM 2002/5/25.  When font-locking is invoked by a buffer change, the region
;; specified by the font-lock after-change function must be expanded to
;; include ALL of any string or regexp within the region.  The simplest way to
;; do this in practice is to use the beginning/end-of-logical-line functions.
;; Don't overlook the possibility of the buffer change being the "recapturing"
;; of a previously escaped newline.
(defmacro c-awk-advise-fl-for-awk-region (function)
  `(defadvice ,function (before get-awk-region activate)
;; When font-locking an AWK Mode buffer, make sure that any string/regexp is
;; completely font-locked.
  (when (eq major-mode 'awk-mode)
    (save-excursion
      (ad-set-arg 1 (c-awk-end-of-change-region
                     (ad-get-arg 0)     ; beg
                     (ad-get-arg 1)     ; end
                     (ad-get-arg 2)))   ; old-len
      (ad-set-arg 0 (c-awk-beginning-of-logical-line (ad-get-arg 0)))))))

(c-awk-advise-fl-for-awk-region font-lock-after-change-function)
(c-awk-advise-fl-for-awk-region jit-lock-after-change)
(c-awk-advise-fl-for-awk-region lazy-lock-defer-rest-after-change)
(c-awk-advise-fl-for-awk-region lazy-lock-defer-line-after-change)



>> I think this putting of warning-face on unmatched quotes would be a good
>> idea generally.

>It would help if you actually explained what you're talking about.
>Not everyone uses awk-mode.

I'll try; I'm not doing very well at explaining at the moment.  Open a
new AWK mode buffer with font locking enabled:  Start by entering a
string:

"string

At this point, the opening quote has font-lock-warning-face, giving the
user a constant niggling reminder to close the string at some point; the
remainder of the text has font-lock-string-face.  Use escaped NLs to
continue the string thus:

"string\
on several\
lines.

The entire string still has string-face, and the opening quote still has
warning-face.  Finally, type the closing quote:

"string\
on several\
lines."

This last action now causes the opening quote to be refonted with
string-face.

>        Stefan

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> - why can't they use the font-lock-multiline property?
>> - why can't they use the jit-lock-defer-multiline property?

> One argument for a hook alternative to these things is that it might
> not be efficient/convenient to spread text properties all over the
> place:

You misunderstood: I'm not particularly happy about either of the
*-multiline text-properties.  I wrote both of them, but neither of them is
satisfactory, really.

But I want to better understand the limitations and strength of each
approach before adding a new hook.

When I added the jit-lock-defer-multiline property (mostly for perl-mode),
I first considered adding a hook somewhat like Alan's but it turns out that
it's pretty damn difficult to write this hook:

The problem is: how to get font-lock to refontify

    s[foo]{
      bar
    }x

when it's changed to

    s[foo]{
      bar
    }xe

since the first `bar' should be treated like a string whereas the second
should be treated like code.

Now, when I'm in the middle of font-locking, I've done the work of figuring
out that I'm in an s/foo/bar/ expression and I can easily add
a text-property on the whole thing.  OTOH using something like Alan's hook,
I'm stuck because if I see "}xe" I don't even know if I'm within code, or
within a string, let along if the } happens to be closing the second arg of
an `s' operation.  It can be done, of course, but it's a fair bit of
redundant work especially since you have to make sure it's not too slow
in the "usual" case (where the hook shouldn't do anything).

I suspect that in the case mentioned in the subject the problem might
be similar.

> o  Text properties stay behind when the buffer changes, and so they
>    might become invalid. Adding code to correctly remove them before
>    that can be tricky.

font-lock (and jit-lock) should take care of that for you.

> o  Putting text properties in place and handling stickiness etc
>    properly is decidedly more complex than just returning a region
>    from a hook.

I haven't seen code that bothers with stickiness when handling *-multiline,
so it doesn't seem complex.

I do agree that for the awk-mode case, Alan's hook is probably one of the
most straightforward solution.  But I must also say I'm not convinced by the
resulting behavior in awk-mode (or c-mode for that matter).  After all, it
catches the problem of forgetting a \, so the warning face should be applied
to the \n char where the \ is missing, not to the opening/closing quote.

This would tie in with another "todo" feature: provide font-lock-keywords
specific to a particular syntatic context: you could thus have separate
keywords highlighted in comments and in strings.  Quite handy for
Javadoc thingies.


        Stefan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    Hmmm.  The advice facility exists within Emacs, and it is difficult to
    see what it could be used on, aside from another part of Emacs.  After
    all, one does not need to advise one's own code, having full control over
    the source.

It is meant for users to use in their Lisp code.  We can't anticipate
all of what they need.  But when something is part of Emacs, it should
not use advice.  When we install code in Emacs, if it advises some
other part of Emacs, then we should add a hook for it to use instead
of the advice.

The reason the hook is better is that it is visible in the source.  If
you are debugging a call to the function foo, there is nothing to show
you it has advice.  That can be a real pain in debugging.  How much
time would you waste before it occurred to you to wonder if the
code that's running really matches the source?

If foo has a hook, at least you'll see `run-hooks' in the code for
foo, so you will ask yourself "What's in that hook?"


I just had an idea that could perhaps eliminate this problem.  Emacs
Lisp mode could have a feature to put an overlay on the defun for a
function that has advice.  The overlay could have `after-text' like "
[Function has advice]".  If you see

(defun foo [Function has advice] (x y) 

you would not be confused by the advice.  The overlay could also perhaps
be a button that you could use to view the advice.

I will add this to TODO.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

This patch to font-lock is exactly the sort of change I was thinking
of.  Could someone please install it, then rename
before-font-lock-after-change-function to
font-lock-extend-region-function, and rename
font-lock-run-before-after-change-hook to font-lock-extend-region?


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    - why can't they use the font-lock-multiline property?

People are looking for a clearer explanation of what this does
and how to use it.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    Even so, it would be several years before major modes could start using
    it in earnest, due to the need to continue supporting older Emacs
    versions.

Supporting other Emacs versions is a secondary issue.  We will start
using this feature right away, in all the modes where it helps.




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>     - why can't they use the font-lock-multiline property?
> People are looking for a clearer explanation of what this does
> and how to use it.

All the consecutive chars with a non-nil font-lock-multiline property will
always be re-fontified together.  E.g. if it is put on three consecutive
lines, a modification on any of the three lines will cause all three to be
re-fontified.


        Stefan



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Richard!

On Thu, 10 Mar 2005, Richard Stallman wrote:

>    Even so, it would be several years before major modes could start
>    using it in earnest, due to the need to continue supporting older
>    Emacs versions.

>Supporting other Emacs versions is a secondary issue.  We will start
>using this feature right away, in all the modes where it helps.

Indeed, that is a "luxury" you enjoy in the core team.  Since people will
be downloading Emacs as a Mega-tarball, everything works with everything
else, modulo the occasional bug.

Down at the CC Mode, though, we put a lot of effort into keeping our
product compatible with as many recent (and not-all-that-recent) Emacs
and XEmacs releases as we can.  We know that people will be downloading
and installing CC Mode directly rather than getting it with new Emacs
releases, if perhaps not all that many people.  We don't like to mess
them around.

I suspect other mode teams feel the same.  Just a small point.

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    You misunderstood: I'm not particularly happy about either of the
    *-multiline text-properties.  I wrote both of them, but neither of them is
    satisfactory, really.

Could you write clearer documentation for them?

    Providing a hook where the font lock pattern writer can extended the
    region to refontify is imho a very straightforward way to give control
    over that aspect. I don't contend that the text properties probably
    are more suitable in many cases, though.

I think we should add that hook now.
It is simple and easy to understand.
Even if in some cases it is not a good solution,
it is a good solution part of the time.
That is enough reason.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

      It's
    just a fact of life that major modes can't really use "new" elisp
    features until they've penetrated fully into the installed user base -
    after several years and several releases.

We can use this feature right away.  There are a few major modes that
are distributed separately, and try to support old Emacs versions.  It
is ok to do that, but that must not be an obstacle to making the next
version of Emacs work right.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    All the consecutive chars with a non-nil font-lock-multiline property will
    always be re-fontified together.  E.g. if it is put on three consecutive
    lines, a modification on any of the three lines will cause all three to be
    re-fontified.

Is there a way they can use font-lock to install these properties
when necessary, automatically?

Could you write documentation for this feature to put into lispref/modes.texi?


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>     All the consecutive chars with a non-nil font-lock-multiline property
>     will always be re-fontified together.  E.g. if it is put on three
>     consecutive lines, a modification on any of the three lines will cause
>     all three to be re-fontified.

> Is there a way they can use font-lock to install these properties
> when necessary, automatically?

You can add those properties "manually" from the font-lock-keywords.

You can also set the font-lock-multiline property so as to tell font-lock to
try and add this property automatically: if a font-lock keyword's match
extends over more than one line, font-lock will then add
a font-lock-multiline property over the whole match.

> Could you write documentation for this feature to put into
> lispref/modes.texi?

I don't have much time for it right now, but I'll keep it in mind,


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

    > Is there a way they can use font-lock to install these properties
    > when necessary, automatically?

    You can add those properties "manually" from the font-lock-keywords.

I think an example would help people see how to use font-lock-multiline
to solve a real problem.

    > Could you write documentation for this feature to put into
    > lispref/modes.texi?

    I don't have much time for it right now, but I'll keep it in mind,

Could you do it before the release, please?


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Martin Stjernholm]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> When I added the jit-lock-defer-multiline property (mostly for perl-mode),
> I first considered adding a hook somewhat like Alan's but it turns out that
> it's pretty damn difficult to write this hook:

Indeed it is.

> The problem is: how to get font-lock to refontify
>
>     s[foo]{
>       bar
>     }x
/.../
> Now, when I'm in the middle of font-locking, I've done the work of figuring
> out that I'm in an s/foo/bar/ expression and I can easily add
> a text-property on the whole thing.

That doesn't work very well while the thing is entered, does it? First
you see "s[foo]{" while it's being entered, then you see the next line
" bar", and lastly "}x". Your patterns will never see the whole
construct at once. (They will however see the buffer end, or even
worse some completely unrelated code that happen to be on the
following lines and which might confuse them.)

A text property can help if you can recognize the construct from the
first line only, but then you'll have to make another pattern that
matches the "}" and contains logic to extend the property put in place
by the pattern matching the first line.

Another problem is that the "bar" part can get arbitrarily large if
it's code, in which case it becomes a performance problem to put a
property on it to refontify the whole thing. Another case to consider
is that the intervening code can contain nested constructs of the same
sort. (I'm not saying it's so in Perl - I'm extending the discussion
to the general case.)

If the construct can't be recognized from the first line alone, which
might very well happen for a C-like declaration, then I see no choice
but to have a function that can find the start when the last line is
entered. That function can of course use various properties put in
place by the major mode to speed up its work. (CC Mode, for instance,
already has some internal properties to cache interesting syntactic
positions, and I'm thinking about extending that approach a lot.)

> I haven't seen code that bothers with stickiness when handling
> *-multiline, so it doesn't seem complex.

It's still more complex to understand and use text properties at all
than to just return a region, especially if one wants the code to work
in the two major emacs flavors which have completely different
interfaces in this area.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> The problem is: how to get font-lock to refontify
>> 
>> s[foo]{
>> bar
>> }x
> /.../
>> Now, when I'm in the middle of font-locking, I've done the work of figuring
>> out that I'm in an s/foo/bar/ expression and I can easily add
>> a text-property on the whole thing.

> That doesn't work very well while the thing is entered, does it? First
> you see "s[foo]{" while it's being entered, then you see the next line
> " bar", and lastly "}x". Your patterns will never see the whole
> construct at once. (They will however see the buffer end, or even
> worse some completely unrelated code that happen to be on the
> following lines and which might confuse them.)

I use font-lock-syntactic-face-function, so I don't need to "see the whole
pattern" (I basically keep track of the intermediate state in the
parse-partial-sexp state).  It works just fine (well, admittedly, I just
tried it and there's a bug in the code that in this case ends up placing
a string-fence char on the `s' instead of the `[').

> Another problem is that the "bar" part can get arbitrarily large if
> it's code, in which case it becomes a performance problem to put a
> property on it to refontify the whole thing.

That's why I used jit-lock-defer-multiline: I only refontify the whole thing
in the "deferred" refontification, not the on-the-fly one.


        Stefan


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Martin Stjernholm]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> That doesn't work very well while the thing is entered, does it? First
>> you see "s[foo]{" while it's being entered, then you see the next line
>> " bar", and lastly "}x". Your patterns will never see the whole
>> construct at once. (They will however see the buffer end, or even
>> worse some completely unrelated code that happen to be on the
>> following lines and which might confuse them.)
>
> I use font-lock-syntactic-face-function,

(I had to investigate what that really does, in case it worked out
some fantastic magic. But afaics it doesn't. ;)

> so I don't need to "see the whole pattern" (I basically keep track
> of the intermediate state in the parse-partial-sexp state).

If I understand this correctly, this means that your example falls
under the special case when the construct can be recognized from the
first line alone. Unfortunately things aren't always that easy.

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>>> That doesn't work very well while the thing is entered, does it? First
>>> you see "s[foo]{" while it's being entered, then you see the next line
>>> " bar", and lastly "}x". Your patterns will never see the whole
>>> construct at once. (They will however see the buffer end, or even
>>> worse some completely unrelated code that happen to be on the
>>> following lines and which might confuse them.)
>> 
>> I use font-lock-syntactic-face-function,

> (I had to investigate what that really does, in case it worked out
> some fantastic magic. But afaics it doesn't. ;)

No, no fantastic magic.  But in some cases, together with
font-lock-syntactic-keywords, it's a good way to get "multi-line keywords".
Other examples: verbatim envs in LaTeX, heredocs in sh-script, things
like that.  I also had a hack that marked strings with a missing \ at the
end of the line using this trick, but the behavior had some rough edges.

>> so I don't need to "see the whole pattern" (I basically keep track
>> of the intermediate state in the parse-partial-sexp state).
> If I understand this correctly, this means that your example falls
> under the special case when the construct can be recognized from the
> first line alone.  Unfortunately things aren't always that easy.

This trick is of restricted applicability, of course,


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

The quoted message is from a discussion back in last March regarding
the addition of a hook for extending backwards the region to be
fontified with a hook to be called before
`font-lock-after-change-function'.

* Richard Stallman (2005-03-11) writes:

> This patch to font-lock is exactly the sort of change I was thinking
> of.  Could someone please install it, then rename
> before-font-lock-after-change-function to
> font-lock-extend-region-function, and rename
> font-lock-run-before-after-change-hook to font-lock-extend-region?

Apparently the hook hasn't been added yet.  Is it still planned to add
it?

The reason I am asking this is that I am currently wrecking my brain
about how to make font locking for LaTeX buffers controlled by AUCTeX
more robust.  On a regular basis we are getting reports about font
locking of multiline constructs failing, thereby spilling color all
over the buffer and/or slowing down editing quite severly.  (We are
using `font-lock-multiline' for fontification of multiline
constructs.)  This mainly happens with text which is erroneously
identified as the start of a multiline construct but has no matching
closing tag.  The last report we got was about "<<" which can be an
opening quotation mark but which may also be used unmatched in some
math constructs.

An idea for making font locking more robust in this respect might be
to desist from coloring the rest of the buffer in case of an unmatched
opening tag, i.e. to leave it alone if a matching closing tag cannot
be found in an arbitrarily large region after the opening tag.  That
would mean fontification will only be applied as soon as the closing
tag is typed.  For that to work, however, I'd have to look backwards
for an opening tag which could be done with a hook like the one
proposed above.

-- 
Ralf

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> This patch to font-lock is exactly the sort of change I was thinking of.
>> Could someone please install it, then rename
>> before-font-lock-after-change-function to
>> font-lock-extend-region-function, and rename
>> font-lock-run-before-after-change-hook to font-lock-extend-region?

I can't find this patch any more.  Based on the name, I suppose it's some
kind of hook in font-lock-after-change-function, in which case I'd be
tempted to suggest to move it to font-lock-fontify-region instead, to reduce
the performance impact and make it easier to deal with lazy-lock&jit-lock
since these tend to use their own after-change-function.

> The reason I am asking this is that I am currently wrecking my brain
> about how to make font locking for LaTeX buffers controlled by AUCTeX
> more robust.  On a regular basis we are getting reports about font
> locking of multiline constructs failing, thereby spilling color all
> over the buffer and/or slowing down editing quite severly.  (We are
> using `font-lock-multiline' for fontification of multiline
> constructs.)  This mainly happens with text which is erroneously
> identified as the start of a multiline construct but has no matching
> closing tag.  The last report we got was about "<<" which can be an
> opening quotation mark but which may also be used unmatched in some
> math constructs.

> An idea for making font locking more robust in this respect might be
> to desist from coloring the rest of the buffer in case of an unmatched
> opening tag, i.e. to leave it alone if a matching closing tag cannot
> be found in an arbitrarily large region after the opening tag.  That
> would mean fontification will only be applied as soon as the closing
> tag is typed.  For that to work, however, I'd have to look backwards
> for an opening tag which could be done with a hook like the one
> proposed above.

I'm not sure how that relates to before-font-lock-after-change-function.
Could you describe how you currently font-lock those <<...>> multiline
elements and how you'd use before-font-lock-after-change-function to solve
the problem?


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Stefan Monnier (2006-02-12) writes:

>>> This patch to font-lock is exactly the sort of change I was thinking of.
>>> Could someone please install it, then rename
>>> before-font-lock-after-change-function to
>>> font-lock-extend-region-function, and rename
>>> font-lock-run-before-after-change-hook to font-lock-extend-region?
>
> I can't find this patch any more.  Based on the name, I suppose it's some
> kind of hook in font-lock-after-change-function, in which case I'd be
> tempted to suggest to move it to font-lock-fontify-region instead, to reduce
> the performance impact and make it easier to deal with lazy-lock&jit-lock
> since these tend to use their own after-change-function.

You can find the patch at
<URL:http://mid.gmane.org/Pine.LNX.3.96.1050309193630.414A-100000@acm.acm>.

>> An idea for making font locking more robust in this respect might be
>> to desist from coloring the rest of the buffer in case of an unmatched
>> opening tag, i.e. to leave it alone if a matching closing tag cannot
>> be found in an arbitrarily large region after the opening tag.  That
>> would mean fontification will only be applied as soon as the closing
>> tag is typed.  For that to work, however, I'd have to look backwards
>> for an opening tag which could be done with a hook like the one
>> proposed above.
>
> I'm not sure how that relates to before-font-lock-after-change-function.
> Could you describe how you currently font-lock those <<...>> multiline
> elements and how you'd use before-font-lock-after-change-function to solve
> the problem?

Currently a function is used as matcher in `font-lock-keywords' for
this functionality.  It basically operates like this:

  (catch 'match
    (while (re-search-forward "<<" limit t)
      (let ((beg (match-beginning 0)))
      (search-forward ">>" limit 'move)
      (store-match-data (list beg (point)))
      (throw 'match t))))

That means, search for an opening tag and if you find one search for
the closing tag till `limit'.  If an opening tag was found, set the
match to the region from the opening tag to the closing tag, or
`limit' if none was found.

What I'd like to use instead is this:

  (catch 'match
    (while (re-search-forward "<<" limit t)
      (let* ((beg (match-beginning 0))
           (after-beg (match-end 0))
                (point-of-surrender (+ beg 1000)))
                (search-forward ">>" point-of-surrender 'move)
                (if (= (point) point-of-surrender)
                    (progn
                          (goto-char after-beg)
                                (store-match-data (list after-beg after-beg)))
                                  (store-match-data (list beg (point))))
                                  (throw 'match t))))))

That means, search for an opening tag and if one is found search for
the closing tag till an arbitrarily large region after the opening tag
(1000 characters large in the code above).  If no closing tag is
found, set an empty match; if one is found, set the match from the
opening to the closing tag.

With the matcher function above text in quotation marks won't be
fontified when I start typing stuff like "<<foo" as long as there is
no closing quotation mark.  Now if the closing quotation mark is
entered a few lines below the line containing the opening quotation
mark, font locking won't see the opening quotation mark and the
multiline quotation won't be fontified.

I'd use something like `before-font-lock-after-change-function' to
look backwards a few characters from the point where text was entered
and detect closing tags.  For example if ">>" was detected after
typing it, I'd search backwards in an arbitrarily large region for a
starting "<<" quotation mark and extend the region to be fontified
backwards till that point.  Now font-lock will find both the opening
and closing quotation marks and fontify the enclosed text.

-- 
Ralf

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard M. Stallman]

Here's the change I was talking about.  Would someone please adapt
this _now_ to the current sources, and install it?  Then please rename
before-font-lock-after-change-function to
font-lock-extend-region-function, and rename
font-lock-run-before-after-change-hook to font-lock-extend-region?

Then please ack this message.


;;;;;; Changelog entry
2002-05-10  Alan Mackenzie  <acm@muc.de>

        The following set of changes allows a major mode dynamically to
        specify the region to fontify after a buffer change. (fast-lock.el
        doesn't need changing.)
        * font-lock.el
        New abnormal hook before-font-lock-after-change-function,
        New function font-lock-run-before-after-change-hook.
        Changed font-lock-after-change-function.
        * jit-lock.el
        Changed jit-lock-after-change
        * lazy-lock.el
        Changed lazy-lock-defer-line-after-change and
        lazy-lock-defer-rest-after-change.
;;;;;; End of Changelog entry

;;;;;; Patch
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/font-lock.el NEW/font-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/font-lock.el	Wed Sep  5 13:36:29 2001
--- NEW/font-lock.el	Fri May 10 19:24:25 2002
***************
*** 1260,1270 ****
  				'(face nil syntax-table nil font-lock-multiline nil)
  			      '(face nil font-lock-multiline nil)))))
  
  ;; Called when any modification is made to buffer text.
  (defun font-lock-after-change-function (beg end old-len)
!   (let ((inhibit-point-motion-hooks t))
      (save-excursion
        (save-match-data
  	;; Rescan between start of lines enclosing the region.
  	(font-lock-fontify-region
  	 (progn (goto-char beg) (beginning-of-line) (point))
--- 1260,1309 ----
  				'(face nil syntax-table nil font-lock-multiline nil)
  			      '(face nil font-lock-multiline nil)))))
  
+ (defvar before-font-lock-after-change-function nil
+   "If set to a function, this can specify the region to fontify after a change.
+ 
+ This variable is a buffer-local abnormal hook whose value, if set, should be a
+ single function.  The function gets called from the active font-lock
+ after-change function, and is intended to allow a major mode to calculate for
+ itself the region to be fontified after a buffer change.
+ 
+ The function is given three parameters, the standard BEG, END and OLD-LEN from
+ after-change-functions.  It should return either a cons of the beginning and end
+ buffer-positions \(in that order\) of the region to fontify, or nil (in which
+ case the default region will be used).")
+ (make-variable-buffer-local 'before-font-lock-after-change-function)
+ 
+ (defun font-lock-run-before-after-change-hook (beg end old-len)
+   "Run the hook function, if any, in before-font-lock-after-change-function,
+ returning its value (a cons of beg and end), if it's valid, else nil.
+ 
+ BEG END and OLD-LEN are the three parameters supplied by
+ after-change-functions."
+   (cond ((null before-font-lock-after-change-function) nil)
+         ((not (functionp before-font-lock-after-change-function))
+          (message "before-font-lock-after-change-function's value is not a \
+ function in buffer %S"
+                   (buffer-name))
+          nil)
+         (t (let ((region (funcall before-font-lock-after-change-function beg end old-len)))
+              (cond ((null region) nil)
+                    ((or (not (consp region))
+                         (not (wholenump (car region))) (not (wholenump (cdr region)))
+                         (not (<= (car region) (cdr region))))
+                     (message "before-font-lock-after-change-function returned %S.  \
+ This isn't a cons \(beg.end\), with beg and end numbers, and beg <= end")
+                     nil)
+                    (t region))))))
+ 
  ;; Called when any modification is made to buffer text.
  (defun font-lock-after-change-function (beg end old-len)
!   (let ((inhibit-point-motion-hooks t) region)
      (save-excursion
        (save-match-data
+         ;; Does the major mode have its own ideas about the region to fontify?
+         (setq region (font-lock-run-before-after-change-hook beg end old-len))
+         (if region (setq beg (car region)  end (cdr region)))
  	;; Rescan between start of lines enclosing the region.
  	(font-lock-fontify-region
  	 (progn (goto-char beg) (beginning-of-line) (point))
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/jit-lock.el NEW/jit-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/jit-lock.el	Mon Jul 16 12:22:58 2001
--- NEW/jit-lock.el	Fri May 10 18:59:36 2002
***************
*** 426,431 ****
--- 426,434 ----
    (when jit-lock-mode
      (save-excursion
        (with-buffer-prepared-for-jit-lock
+        ;; Does the major mode have its own ideas about the region to fontify?
+        (let ((region (font-lock-run-before-after-change-hook start end old-len)))
+          (if region (setq start (car region)  end (cdr region))))
         ;; It's important that the `fontified' property be set from the
         ;; beginning of the line, else font-lock will properly change the
         ;; text's face, but the display will have been done already and will
diff -c -x ChangeLog -x Makefile -x README /usr/src/packages/BUILD/emacs-21.1/lisp/lazy-lock.el NEW/lazy-lock.el
*** /usr/src/packages/BUILD/emacs-21.1/lisp/lazy-lock.el	Thu Aug 16 14:25:15 2001
--- NEW/lazy-lock.el	Fri May 10 19:00:47 2002
***************
*** 771,776 ****
--- 771,779 ----
    (save-buffer-state nil
      (unless (memq (current-buffer) lazy-lock-buffers)
        (push (current-buffer) lazy-lock-buffers))
+     ;; Does the major mode have its own ideas about what to fontify?
+     (let ((region (font-lock-run-before-after-change-hook beg end old-len)))
+       (if region (setq beg (car region)  end (cdr region))))
      (remove-text-properties (max (1- beg) (point-min))
  			    (min (1+ end) (point-max))
  			    '(lazy-lock nil))))
***************
*** 784,789 ****
--- 787,795 ----
        (push (current-buffer) lazy-lock-buffers))
      (save-restriction
        (widen)
+       ;; Does the major mode have its own ideas about what to fontify?
+       (let ((region (font-lock-run-before-after-change-hook beg end old-len)))
+         (if region (setq beg (car region)  end (cdr region))))
        (remove-text-properties (max (1- beg) (point-min))
  			      (point-max)
  			      '(lazy-lock nil)))))
;;;;;; end of patch.

-- 
Alan Mackenzie (Munich, Germany)


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> Here's the change I was talking about.  Would someone please adapt
> this _now_ to the current sources, and install it?  Then please rename
> before-font-lock-after-change-function to
> font-lock-extend-region-function, and rename
> font-lock-run-before-after-change-hook to font-lock-extend-region?

Could we also drop the font-lock-before-lines, since the above feature
subsumes it (and it wasn't part of Emacs-21.4 so it's never been released
anyway).


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> Currently a function is used as matcher in `font-lock-keywords' for
> this functionality.  It basically operates like this:

>   (catch 'match
>     (while (re-search-forward "<<" limit t)
>       (let ((beg (match-beginning 0)))
>       (search-forward ">>" limit 'move)
>       (store-match-data (list beg (point)))
>       (throw 'match t))))

I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

> That means, search for an opening tag and if you find one search for
> the closing tag till `limit'.  If an opening tag was found, set the
> match to the region from the opening tag to the closing tag, or
> `limit' if none was found.

> What I'd like to use instead is this:

>   (catch 'match
>     (while (re-search-forward "<<" limit t)
>       (let* ((beg (match-beginning 0))
>            (after-beg (match-end 0))
>                 (point-of-surrender (+ beg 1000)))
>                 (search-forward ">>" point-of-surrender 'move)
>                 (if (= (point) point-of-surrender)
>                     (progn
>                           (goto-char after-beg)
>                                 (store-match-data (list after-beg after-beg)))
>                                   (store-match-data (list beg (point))))
>                                   (throw 'match t))))))

> That means, search for an opening tag and if one is found search for
> the closing tag till an arbitrarily large region after the opening tag
> (1000 characters large in the code above).  If no closing tag is
> found, set an empty match; if one is found, set the match from the
> opening to the closing tag.

> With the matcher function above text in quotation marks won't be
> fontified when I start typing stuff like "<<foo" as long as there is
> no closing quotation mark.  Now if the closing quotation mark is
> entered a few lines below the line containing the opening quotation
> mark, font locking won't see the opening quotation mark and the
> multiline quotation won't be fontified.

Indeed.  You can use contextual refontification, tho:

   (if jit-lock-context-unfontify-pos
       (setq jit-lock-context-unfontify-pos
             (min jit-lock-context-unfontify-pos
                  (re-search-backward "<<" limit t))))

it's specific to jit-lock, tho.

Also it will not refontify immediately but only after jit-lock-contex-time.
I'd consider it a feature, but others may disagree.

Finally, the above code is naive: you'd have to make sure it's only used
when refontification will indeed make a difference, otherwise the code could
be triggered again and the context refontification could repeat over and over
again.

I like the idea behind your before-font-lock-after-change-function, except
that I'd want it to be in font-lock-default-fontify-region.  I.e. it should
then be possible to remove mention of font-lock-multiline from
font-lock-default-fontify-region by moving it to this new hook (which we
could call font-lock-fontify-extend-region-functions).


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard M. Stallman]

    Could we also drop the font-lock-before-lines, since the above feature
    subsumes it (and it wasn't part of Emacs-21.4 so it's never been released
    anyway).

Once this change is installed, if you want to drop font-lock-before-lines
and replace whatever uses it, I don't mind.



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 >>With the matcher function above text in quotation marks won't be
 >>fontified when I start typing stuff like "<<foo" as long as there is
 >>no closing quotation mark.  Now if the closing quotation mark is
 >>entered a few lines below the line containing the opening quotation
 >>mark, font locking won't see the opening quotation mark and the
 >>multiline quotation won't be fontified.
 >
 >
 > Indeed.  You can use contextual refontification, tho:
 >
 >    (if jit-lock-context-unfontify-pos
 >        (setq jit-lock-context-unfontify-pos
 >              (min jit-lock-context-unfontify-pos
 >                   (re-search-backward "<<" limit t))))
 >
 > it's specific to jit-lock, tho.

Provided the "<<" is still after window-start, though.

In any case, it's completely inappropriate to search tags in the hook.
The hook should trigger an idle timed function that would do the search.
And you could calmly replace the 1000 characters limit by something more
useful.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Werner LEMBERG]

> > Currently a function is used as matcher in `font-lock-keywords' for
> > this functionality.  It basically operates like this:
>
> I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

It might be a good idea to add the above regexp to the emacs info file
as an example how to write a regexp which spans more than a single
line.


    Werner

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Stefan Monnier (2006-02-13) writes:

>> Currently a function is used as matcher in `font-lock-keywords' for
>> this functionality.  It basically operates like this:
>
>>   (catch 'match
>>     (while (re-search-forward "<<" limit t)
>>       (let ((beg (match-beginning 0)))
>>       (search-forward ">>" limit 'move)
>>       (store-match-data (list beg (point)))
>>       (throw 'match t))))

Ouch.  Pasting the code into a wterm messed up indentation pretty
badly. )c:

> I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

The function actually used in AUCTeX does a bit more.  For example it
checks if a face for verbatim content is present and does not set a
match in such a case.

>> With the matcher function above text in quotation marks won't be
>> fontified when I start typing stuff like "<<foo" as long as there is
>> no closing quotation mark.  Now if the closing quotation mark is
>> entered a few lines below the line containing the opening quotation
>> mark, font locking won't see the opening quotation mark and the
>> multiline quotation won't be fontified.
>
> Indeed.  You can use contextual refontification, tho:
>
>    (if jit-lock-context-unfontify-pos
>        (setq jit-lock-context-unfontify-pos
>              (min jit-lock-context-unfontify-pos
>                   (re-search-backward "<<" limit t))))
>
> it's specific to jit-lock, tho.

Hm, I would not like leaving people who disabled jit-lock out in the
cold.  So the hook is probably a better alternative.  But thanks for
that hint anyway.

> Also it will not refontify immediately but only after jit-lock-contex-time.
> I'd consider it a feature, but others may disagree.

I am tempted to disagree but maybe this is a case of getting used to it.

> I like the idea behind your before-font-lock-after-change-function, except

Uh, Alan Mackenzie is the originator of this proposal, not me. (c:

> that I'd want it to be in font-lock-default-fontify-region.  I.e. it should
> then be possible to remove mention of font-lock-multiline from
> font-lock-default-fontify-region by moving it to this new hook (which we
> could call font-lock-fontify-extend-region-functions).

Hm, how would it be possible to detect closing tags in this case?
Maybe with an initial search for these tags across the region to be
fontified.  Or on a case by case basis for every closing tag which is
encountered during fontication of the region?  This would be rather
inefficient compared to using a hook to be called before
`font-lock-after-change-function'.  But maybe you are thinking about
something completely different.

-- 
Ralf

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Indeed.  You can use contextual refontification, tho:
>> 
>> (if jit-lock-context-unfontify-pos
>> (setq jit-lock-context-unfontify-pos
>> (min jit-lock-context-unfontify-pos
>> (re-search-backward "<<" limit t))))
>> 
>> it's specific to jit-lock, tho.

> Provided the "<<" is still after window-start, though.

Huh?  Why would window-start matter?

> In any case, it's completely inappropriate to search tags in the hook.
> The hook should trigger an idle timed function that would do the search.
> And you could calmly replace the 1000 characters limit by something more
> useful.

I'm not sure which hook you're talking about.  The code I wrote above is
meant to be used on font-lock-keywords when finding a ">>".


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

> The function actually used in AUCTeX does a bit more.  For example it
> checks if a face for verbatim content is present and does not set a
> match in such a case.

Sure.  Also it's more efficient than the above regexp.

>> it's specific to jit-lock, tho.
> Hm, I would not like leaving people who disabled jit-lock out in the
> cold.  So the hook is probably a better alternative.  But thanks for
> that hint anyway.

jit-lock is already needed for correct refontification of multi-line
comments and strings, so I wouldn't worry too much about the effect it would
have of foncitifcation of other multiline constructs.

>> that I'd want it to be in font-lock-default-fontify-region.  I.e. it should
>> then be possible to remove mention of font-lock-multiline from
>> font-lock-default-fontify-region by moving it to this new hook (which we
>> could call font-lock-fontify-extend-region-functions).

> Hm, how would it be possible to detect closing tags in this case?
> Maybe with an initial search for these tags across the region to be
> fontified.  Or on a case by case basis for every closing tag which is
> encountered during fontication of the region?  This would be rather
> inefficient compared to using a hook to be called before
> `font-lock-after-change-function'.  But maybe you are thinking about
> something completely different.

Huh?  I must be missing something: I don't see what's different between
font-lock-after-change-function and font-lock-default-fontify-region, other
than the fact that they're called at different moments.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > Huh?  Why would window-start matter?

Because fontification triggered by redisplay will fontify the displayed
area first.  `jit-lock-context-fontify' would reset the fontified
property for everything after the "<<".  Redisplay now would trigger
refontification below window-start.  It won't care about the "<<".

 > I'm not sure which hook you're talking about.  The code I wrote above is
 > meant to be used on font-lock-keywords when finding a ">>".

The hook that would trigger searching for the "<<" after a buffer
change, `before-font-lock-after-change-function' or whatever it will be
called.  I simply believe that searching some 1000 characters every time
you type one single character may slow down editing noticeably.  But
maybe I didn't understand the scenario correctly.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Huh?  Why would window-start matter?

> Because fontification triggered by redisplay will fontify the displayed
> area first.  `jit-lock-context-fontify' would reset the fontified
> property for everything after the "<<".  Redisplay now would trigger
> refontification below window-start.  It won't care about the "<<".

Good point.  Later fontification (e.g. stealth) may pay attention to the <<
at some point, tho it won't refontify the parts that were already fontified.

>> I'm not sure which hook you're talking about.  The code I wrote above is
>> meant to be used on font-lock-keywords when finding a ">>".

> The hook that would trigger searching for the "<<" after a buffer
> change, `before-font-lock-after-change-function' or whatever it will be
> called.  I simply believe that searching some 1000 characters every time
> you type one single character may slow down editing noticeably.  But
> maybe I didn't understand the scenario correctly.

Yes, there's a misunderstanding: my 4 lines of code were meant to be used
*instead of* before-font-lock-after-change-function (i.e. placed on
font-lock-keywords).

And yes, I agree about the speed penalty which is why I want to run such
a hook in font-lock-default-fontify-region rather than in
font-lock-after-change-function.

Note, tho, that the 1000-byte search would probably only only happen when
the line has a ">>" on it.  And it's all very much fast enough for
self-insert-command.  The problem is much more severe when you're running
a command that does many buffer modification and thus runs the hook
many times without intervening user interaction.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Stefan Monnier (2006-02-14) writes:

>> Hm, how would it be possible to detect closing tags in this case?
>> Maybe with an initial search for these tags across the region to be
>> fontified.  Or on a case by case basis for every closing tag which is
>> encountered during fontication of the region?  This would be rather
>> inefficient compared to using a hook to be called before
>> `font-lock-after-change-function'.  But maybe you are thinking about
>> something completely different.
>
> Huh?  I must be missing something: I don't see what's different between
> font-lock-after-change-function and font-lock-default-fontify-region, other
> than the fact that they're called at different moments.

You are right, it is not /that/ different.  However, the proposed
hook, like the other functions in `after-change-functions', will get
the begin and end of the changed region which will mostly be smaller
than the region passed to `font-lock-default-fontify-region' and the
end of the region will be nearer to the point I am interested in.  So
detecting an ending tag with something like (search-backward ">>" (1-
beg) t) will be faster.  I am not sure, however, if the solution with
`font-lock-default-fontify-region' is more efficient overall because
the function might not be called as often as a hook in
`after-change-functions'.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > Note, tho, that the 1000-byte search would probably only only happen when
 > the line has a ">>" on it.

... has or _had_ a ">>" on it.

 > And it's all very much fast enough for
 > self-insert-command.

It may show up when using auto-repeat for typing characters.

 > The problem is much more severe when you're running
 > a command that does many buffer modification and thus runs the hook
 > many times without intervening user interaction.

In particular the hook might trigger fontification for regions that are
not displayed thus defeating the purpose of jit-lock.  When you indent,
fill, delete larger regions you have to search all beginnings and ends
of those regions within the 1000 char limit for "<<"s and ">>"s that
might have changed.

I still believe that an idle timer would be more appropriate here.  Have
the hook accumulate the bounds of the most recent changes to a buffer.
A timed function would scan from some suitable position before the
displayed area - this can be some 1000 chars before window-start but I'd
prefer something more Archimedean here like the start of a syntactic
entity that cannot appear within "<<" ... ">>" - and search for matching
"<<" ">>" pairs between that position and the maximum of the last
changed position within the displayed area and the next syntactic entity
that can't be within "<<" ... ">>".  Obviously, this should be done for
every window showing that buffer.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* martin rudalics (2006-02-15) writes:

> In particular the hook might trigger fontification for regions that are
> not displayed thus defeating the purpose of jit-lock.

What alternative is there if fontification of a visible portion of the
buffer depends on text not being displayed?

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> You are right, it is not /that/ different.  However, the proposed
> hook, like the other functions in `after-change-functions', will get
> the begin and end of the changed region which will mostly be smaller
> than the region passed to `font-lock-default-fontify-region' and the
> end of the region will be nearer to the point I am interested in.  So
> detecting an ending tag with something like (search-backward ">>" (1-
> beg) t) will be faster.

Typically, the region processed by `font-lock-default-fontify-region' will
just have one more line than the one in after-change-functions, so unless
you have really long lines, the difference will be lost in the noise.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Stefan Monnier (2006-02-15) writes:

> Typically, the region processed by `font-lock-default-fontify-region' will
> just have one more line than the one in after-change-functions, so unless
> you have really long lines, the difference will be lost in the noise.

Another typical region size for `font-lock-default-fontify-region'
will be `jit-lock-chunk-size' which admittedly is not much larger
either.  But in any case one will have to deal with extremes like
`font-lock-fontify-buffer' as well.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 >>In particular the hook might trigger fontification for regions that are
 >>not displayed thus defeating the purpose of jit-lock.
 >
 >
 > What alternative is there if fontification of a visible portion of the
 > buffer depends on text not being displayed?

Fontifying text that does not reside near the beginning of a buffer
usually always depends on analyzing the syntax of text that is not
displayed.  In fact, fontification must work independently of whether
text is displayed or not.  That's, for example, the driving principle of
stealth fontification.  However, fontification of some region A of a
buffer should not depend on whether or how some region B of a buffer has
been fontified before (multiline keywords are a notorious exception to
this rule).

Your proposal - please correct me if I'm wrong - may fontify modified
text disregarding whether the text appears in some window or not.  That,
in principle, defeats the aim of jit-lock to fontify as few text as
necessary in order to display it correctly.  I don't see anything wrong
with jit-fontifiying portions of text around displayed regions if such
portions would have to be scanned / searched anyway in order to
determine how displayed text shall be fontified.  Often a user might
want to scroll into such regions sooner or later anyway.  If, however,
filling or indenting larger regions will have to refontify text
regardless of whether the text is displayed anywhere, the overhead might
get intrusive.



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Ralf Angeli (2006-02-15) writes:

> * Stefan Monnier (2006-02-15) writes:
>
>> Typically, the region processed by `font-lock-default-fontify-region' will
>> just have one more line than the one in after-change-functions, so unless
>> you have really long lines, the difference will be lost in the noise.
>
> Another typical region size for `font-lock-default-fontify-region'
> will be `jit-lock-chunk-size' which admittedly is not much larger
> either.  But in any case one will have to deal with extremes like
> `font-lock-fontify-buffer' as well.

P.S.: Admittedly there is not much content left in this cornercase for
extending the region.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* martin rudalics (2006-02-15) writes:

> Your proposal

It's not my proposal.

> - please correct me if I'm wrong - may fontify modified
> text disregarding whether the text appears in some window or not.

That's how I understand it.  The region to be fontified will get
extended backwards to the starting tag regardless of this point being
visible in the window or not.  If it will be fontified right away or
marked to be fontified by jit-lock depends upon either
`jit-lock-after-change' or `font-lock-after-change-function' being
present in `after-change-functions'.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > That's how I understand it.  The region to be fontified will get
 > extended backwards to the starting tag regardless of this point being
 > visible in the window or not.  If it will be fontified right away or
 > marked to be fontified by jit-lock depends upon either
 > `jit-lock-after-change' or `font-lock-after-change-function' being
 > present in `after-change-functions'.

Presumably the "region to be fontified gets extended" by resetting its
fontified text-property to nil.  `jit-lock-after-change' should be able
to do this if you provide the correct start and end values.  But keep in
mind that with jit-lock redisplay immediately triggers refontification
of displayed text whose fontified text-property has been reset to nil.
This refontification won't recognize any "starting tag" preceding
window-start unless you explicitly tell it to go there, for example, by
using a multiline property.  If you decide on the `font-lock-multiline'
property, every time you insert or delete a character nearby, redisplay
has to refontify the entire contiguous area covered by the property.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* martin rudalics (2006-02-15) writes:

> But keep in
> mind that with jit-lock redisplay immediately triggers refontification
> of displayed text whose fontified text-property has been reset to nil.
> This refontification won't recognize any "starting tag" preceding
> window-start unless you explicitly tell it to go there, for example, by
> using a multiline property.

Ugh, that is not good.  The main reason I want to use the proposed
hook is to get rid of `font-lock-multiline'.  With
`font-lock-multiline' I am forced to set the match to the end of the
region to be fontified in a matcher function as long as no ending tag
was found.  Otherwise the multiline text property will not be picked
up in the next chunk to be fontified.  Unfortunately in LaTeX (and
especially docTeX) buffers can be so much junk not being used the
"normal" way that this practice often leads to color being spilled all
over the buffer.  The idea with the hook was to apply fontification
only as soon as an ending tag is found.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

On Sun, 12 Feb 2006, Stefan Monnier wrote:

>>> This patch to font-lock is exactly the sort of change I was thinking of.
>>> Could someone please install it, then rename
>>> before-font-lock-after-change-function to
>>> font-lock-extend-region-function, and rename
>>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>.... Based on the name, I suppose it's some kind of hook in
>font-lock-after-change-function, in which case I'd be tempted to suggest
>to move it to font-lock-fontify-region instead, to reduce the
>performance impact and make it easier to deal with lazy-lock&jit-lock
>since these tend to use their own after-change-function.

I strongly oppose such a change.  With that change:

(i) font-lock-fontify-region would no longer be fontifying the region
specified by its paramters, but a different (possibly larger) one. 
(ii) the hook function (which recalculates BEG and END) might well refer
to variables set by a before-change-functions hook.  (This is done in AWK
mode, for example).  f-l-f-region is regularly called when there is no
buffer change in progress.

Both of these things would make debugging a hook function much more
difficult than it already is.  Determining the region to fontify and
actually fontifying it are two logically distinct operations.  They
shouldn't be intermingled with eachother.

[ .... ]

>        Stefan

-- 
Alan Mackenzie (Munich, Germany)





-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Ralf!

On Sun, 12 Feb 2006, Ralf Angeli wrote:

>The quoted message is from a discussion back in last March regarding
>the addition of a hook for extending backwards the region to be
>fontified with a hook to be called before
>`font-lock-after-change-function'.

>* Richard Stallman (2005-03-11) writes:

>> This patch to font-lock is exactly the sort of change I was thinking
>> of.  Could someone please install it, then rename
>> before-font-lock-after-change-function to
>> font-lock-extend-region-function, and rename
>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>Apparently the hook hasn't been added yet.  Is it still planned to add
>it?

>The reason I am asking this is that I am currently wrecking my brain
>about how to make font locking for LaTeX buffers controlled by AUCTeX
>more robust.  On a regular basis we are getting reports about font
>locking of multiline constructs failing, thereby spilling color all
>over the buffer and/or slowing down editing quite severly.  (We are
>using `font-lock-multiline' for fontification of multiline
>constructs.)  This mainly happens with text which is erroneously
>identified as the start of a multiline construct but has no matching
>closing tag.  The last report we got was about "<<" which can be an
>opening quotation mark but which may also be used unmatched in some
>math constructs.

Why don't you construct a regular expression which would only find a "<<"
which isn't in a maths construct?  I don't know LaTex, but assuming these
"<<" ">>" pairs can't be nested (is this the case?), you could make this
regexp stop (with "not found") at a second "<<" (not within a maths
thingy).

In your font-lock matcher function, you would first go back to a "safe
place", then scan forward for this regexp.  Even though the regexp would
be quite intricate, it would still be reasonably fast, even if you end up
scanning to the end of the buffer.

Another idea would be to maintain a list of "<<" ">>" pairs, which
you would update in an after-change-function.

>An idea for making font locking more robust in this respect might be
>to desist from coloring the rest of the buffer in case of an unmatched
>opening tag, i.e. to leave it alone if a matching closing tag cannot
>be found in an arbitrarily large region after the opening tag.

I suggest that you font lock only the complete construct "<< ..... >>".
You might want to give an unmated "<<" or ">>" font-lock-warning-face
until the user completes the construct.

>That would mean fontification will only be applied as soon as the
>closing tag is typed.  For that to work, however, I'd have to look
>backwards for an opening tag which could be done with a hook like the
>one proposed above.

>Ralf

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Martin!

On Tue, 14 Feb 2006, martin rudalics wrote:

> >>With the matcher function above text in quotation marks won't be
> >>fontified when I start typing stuff like "<<foo" as long as there is
> >>no closing quotation mark.  Now if the closing quotation mark is
> >>entered a few lines below the line containing the opening quotation
> >>mark, font locking won't see the opening quotation mark and the
> >>multiline quotation won't be fontified.


> > Indeed.  You can use contextual refontification, tho:

> >    (if jit-lock-context-unfontify-pos
> >        (setq jit-lock-context-unfontify-pos
> >              (min jit-lock-context-unfontify-pos
> >                   (re-search-backward "<<" limit t))))

> > it's specific to jit-lock, tho.

>Provided the "<<" is still after window-start, though.

>In any case, it's completely inappropriate to search tags in the hook.

[ Small linguistic point: "search" = "durchsuchen"; "search for" =
"suchen".]

>The hook should trigger an idle timed function that would do the search.
>And you could calmly replace the 1000 characters limit by something more
>useful.

I disagree profoundly.  This would interact with other idly timed
functions, some of which might well already be fontifying up to the bits
being searched.  The font locking code is already very complicated, and
this extra complication would make it very difficult indeed to debug.

I think the function should be run from the after-change hook, brute
force fashion.  Should it prove too slow[*], then would be the time to
implement a more sophisticated strategy.

[*] I have a 166 MHz PC, available for testing for "too slow".

-- 
Alan Mackenzie (Munich, Germany)




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

On Mon, 13 Feb 2006, Stefan Monnier wrote:

>> Currently a function is used as matcher in `font-lock-keywords' for
>> this functionality.  It basically operates like this:

>>   (catch 'match
>>     (while (re-search-forward "<<" limit t)
>>       (let ((beg (match-beginning 0)))
>>       (search-forward ">>" limit 'move)
>>       (store-match-data (list beg (point)))
>>       (throw 'match t))))

>I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".
                              ^^

Small point:  It might be a good idea here to write [\n\r] rather than
just \n, in case something has changed \n's to \r's for selective display
(Node of that name in the Elisp manual).

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Stefan!

On Tue, 14 Feb 2006, Stefan Monnier wrote:

[ .... ]

>Note, tho, that the 1000-byte search would probably only only happen when
>the line has a ">>" on it.  And it's all very much fast enough for
>self-insert-command.  The problem is much more severe when you're running
>a command that does many buffer modification and thus runs the hook
>many times without intervening user interaction.

There's the macro combine-after-change-calls to mitigate this.

>        Stefan


>_______________________________________________
>Emacs-devel mailing list
>Emacs-devel@gnu.org
>http://lists.gnu.org/mailman/listinfo/emacs-devel




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Martin!

On Tue, 14 Feb 2006, martin rudalics wrote:

> > Huh?  Why would window-start matter?

>Because fontification triggered by redisplay will fontify the displayed
>area first.  `jit-lock-context-fontify' would reset the fontified
>property for everything after the "<<".  Redisplay now would trigger
>refontification below window-start.  It won't care about the "<<".

> > I'm not sure which hook you're talking about.  The code I wrote above is
> > meant to be used on font-lock-keywords when finding a ">>".

>The hook that would trigger searching for the "<<" after a buffer
>change, `before-font-lock-after-change-function' or whatever it will be
>called.  I simply believe that searching some 1000 characters every time
>you type one single character may slow down editing noticeably.  But
>maybe I didn't understand the scenario correctly.

Surely it would be possible to arrange for this searching to be done only
when one of these delimiters is inserted or deleted?

In a before-change function, you check whether or not a change is
happening around a ">>", ">", "<" or "<<", and record this in some
variables.  In the after-change you see if a delimiter has actually been
inserted or deleted.  Only then need you search.  Since this searching
will be done only rarely, you probably don't need to worry about
searching even the entire buffer.

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>>> Currently a function is used as matcher in `font-lock-keywords' for
>>> this functionality.  It basically operates like this:

>>> (catch 'match
>>> (while (re-search-forward "<<" limit t)
>>> (let ((beg (match-beginning 0)))
>>> (search-forward ">>" limit 'move)
>>> (store-match-data (list beg (point)))
>>> (throw 'match t))))

>> I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

> Small point:  It might be a good idea here to write [\n\r] rather than
> just \n, in case something has changed \n's to \r's for selective display
> (Node of that name in the Elisp manual).

Last time I checked a "." in a regexp does match \r even if
selective-display is activated, so always \\(.\\|\n\\) matches *any* char.


        Stefan


PS: Furthermore, selective-display is on its way out.
The only code left that uses it (in Emacs-CVS) is in dired-aux.el,
gnus-sum.el, and locate.el, none of which interact with other major modes.


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Alan Mackenzie (2006-02-15) writes:

> On Sun, 12 Feb 2006, Stefan Monnier wrote:
>
>>.... Based on the name, I suppose it's some kind of hook in
>>font-lock-after-change-function, in which case I'd be tempted to suggest
>>to move it to font-lock-fontify-region instead, to reduce the
>>performance impact and make it easier to deal with lazy-lock&jit-lock
>>since these tend to use their own after-change-function.
>
> I strongly oppose such a change.  With that change:
>
> (i) font-lock-fontify-region would no longer be fontifying the region
> specified by its paramters, but a different (possibly larger) one. 
> (ii) the hook function (which recalculates BEG and END) might well refer
> to variables set by a before-change-functions hook.  (This is done in AWK
> mode, for example).  f-l-f-region is regularly called when there is no
> buffer change in progress.
>
> Both of these things would make debugging a hook function much more
> difficult than it already is.  Determining the region to fontify and
> actually fontifying it are two logically distinct operations.  They
> shouldn't be intermingled with eachother.

This is all well and good, but in contrast to the after-change hook, a
hook in `font-lock-default-fontify-region' could not only adjust the
region after a change but also during fontification by chunks as done
by jit-lock.  (That's an advantage I haven't noticed before.)
Following your reasoning of separating the determination of the region
to be fontified from the fontification itself would require that
jit-lock determines the chunks to be fontified more intelligently.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

On Wed, 15 Feb 2006, Stefan Monnier wrote:

>>> I.e. equivalent to "<<\\(.\\|\n\\)*?\\(>>\\)?".

>> Small point:  It might be a good idea here to write [\n\r] rather than
>> just \n, in case something has changed \n's to \r's for selective
>> display (Node of that name in the Elisp manual).

>Last time I checked a "." in a regexp does match \r even if
>selective-display is activated, so always \\(.\\|\n\\) matches *any* char.

Ah!  OK, thanks!

>        Stefan

>PS: Furthermore, selective-display is on its way out.  The only code
>left that uses it (in Emacs-CVS) is in dired-aux.el, gnus-sum.el, and
>locate.el, none of which interact with other major modes.

But for people writing modes which have to work with older (X)Emacsen,
it's worth bearing in mind.

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>>>> This patch to font-lock is exactly the sort of change I was thinking of.
>>>> Could someone please install it, then rename
>>>> before-font-lock-after-change-function to
>>>> font-lock-extend-region-function, and rename
>>>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>> .... Based on the name, I suppose it's some kind of hook in
>> font-lock-after-change-function, in which case I'd be tempted to suggest
>> to move it to font-lock-fontify-region instead, to reduce the
>> performance impact and make it easier to deal with lazy-lock&jit-lock
>> since these tend to use their own after-change-function.

> I strongly oppose such a change.  With that change:

> (i) font-lock-fontify-region would no longer be fontifying the region
> specified by its paramters, but a different (possibly larger) one.

That's already the case since font-lock-fontify-region will typically not
fontify from BEG to END from the beginning of line before BEG to the end of
line after END.  Then that can be extended yet further because of
font-lock-multiline.  Nobody has ever complained about this.

So why is it a problem?

> (ii) the hook function (which recalculates BEG and END) might well refer
> to variables set by a before-change-functions hook.  (This is done in AWK
> mode, for example).

Which variables would that be?

> f-l-f-region is regularly called when there is no buffer change in progress.

Indeed.  Again, why is it a problem?

> Both of these things would make debugging a hook function much more
> difficult than it already is.  Determining the region to fontify and
> actually fontifying it are two logically distinct operations.

With jit-lock, the region to fontify is not determined by the
after-change-functions either.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Note, tho, that the 1000-byte search would probably only only happen when
>> the line has a ">>" on it.  And it's all very much fast enough for
>> self-insert-command.  The problem is much more severe when you're running
>> a command that does many buffer modification and thus runs the hook
>> many times without intervening user interaction.

> There's the macro combine-after-change-calls to mitigate this.

Yes, for code that knows about it.  A grep shows that it is *very*
rarely used.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > There's the macro combine-after-change-calls to mitigate this.

...

 > Surely it would be possible to arrange for this searching to be done only
 > when one of these delimiters is inserted or deleted?
 >
 > In a before-change function, you check whether or not a change is

`combine-after-change-calls' won't combine anything when
`before-change-functions' is non-nil.

 > happening around a ">>", ">", "<" or "<<", and record this in some
 > variables.  In the after-change you see if a delimiter has actually been
 > inserted or deleted.  Only then need you search.  Since this searching
 > will be done only rarely, you probably don't need to worry about
 > searching even the entire buffer.

When you fill text you would have to (1) scan every line for occurrences
of these strings in `before-change-functions' and record these
occurrences somehow (maybe using markers), (2) apply the change for that
line, and (3) scan the line again for matching occurrences of these
strings in `after-change-functions' (and maybe release the markers).
All this in the presence of some LaTeX syntax specific math context
whose bounds you have to recalculate for every single change too.  And,
what's worse, all this is completely useless since filling shouldn't
affect fontification anyway.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 >>The hook should trigger an idle timed function that would do the search.
 >>And you could calmly replace the 1000 characters limit by something more
 >>useful.
 >
 >
 > I disagree profoundly.  This would interact with other idly timed
 > functions, some of which might well already be fontifying up to the bits
 > being searched.  The font locking code is already very complicated, and
 > this extra complication would make it very difficult indeed to debug.

You don't even have to introduce another timer.  You could run this from
within `jit-lock-context-fontify'.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Alan Mackenzie (2006-02-15) writes:

> Why don't you construct a regular expression which would only find a "<<"
> which isn't in a maths construct?  I don't know LaTex, but assuming these
> "<<" ">>" pairs can't be nested (is this the case?), you could make this
> regexp stop (with "not found") at a second "<<" (not within a maths
> thingy).

The quotation marks can be nested, there isn't even a requirement that
they have to come in pairs but they are usually used like this.  With
csquotes there is even a LaTeX package which produces sensible output
from nested quotation marks of the same kind.  For example an input
like «foo «bar» baz» could result in a printed output of «foo ‹bar›
baz» or „foo ‚bar‘ baz“.

Anyway, quotation marks are just an example.  There are many other
places where font locking can go wrong and where I'd like to minimize
the impact of such a failure.

> You might want to give an unmated "<<" or ">>" font-lock-warning-face
> until the user completes the construct.

Yes, that's what I had in mind as well.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

> Why don't you construct a regular expression which would only find a "<<"
> which isn't in a maths construct?

A regular expressions that knows about syntactic context?

> I don't know LaTex, but assuming these
> "<<" ">>" pairs can't be nested (is this the case?), you could make this
> regexp stop (with "not found") at a second "<<" (not within a maths
> thingy).

This means when you change a "maths thingy" you have to check for "<<"s
and ">>"s that are now in- / outside the thingy as well.

> In your font-lock matcher function, you would first go back to a "safe
> place", then scan forward for this regexp.  Even though the regexp would
> be quite intricate, it would still be reasonably fast, even if you end up
> scanning to the end of the buffer.

That wouldn't be just-in-time any more.

> Another idea would be to maintain a list of "<<" ">>" pairs, which
> you would update in an after-change-function.

Using markers?

> I suggest that you font lock only the complete construct "<< ..... >>".
> You might want to give an unmated "<<" or ">>" font-lock-warning-face
> until the user completes the construct.

That works efficiently for syntactically fontified text like comments or
strings.  In the present case it's yet another complication - typing
">>" might cause searching from bob till point.





-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

> I strongly oppose such a change.  With that change:
>
> (i) font-lock-fontify-region would no longer be fontifying the region
> specified by its paramters, but a different (possibly larger) one.

Note that jit-lock doesn't care about any "parameters".  It fontifies
whatever redisplay or a timer triggered function ask it to fontify.

> Both of these things would make debugging a hook function much more
> difficult than it already is.  Determining the region to fontify and
> actually fontifying it are two logically distinct operations.  They
> shouldn't be intermingled with eachother.

With jit-lock you don't determine "the region to fontify".  You only
determine regions that might need (re-)fontification when they are
redisplayed or - with stealth fontification - emacs is idle.





-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Good Morning!

On Wed, 15 Feb 2006, Stefan Monnier wrote:

>>>>> This patch to font-lock is exactly the sort of change I was thinking of.
>>>>> Could someone please install it, then rename
>>>>> before-font-lock-after-change-function to
>>>>> font-lock-extend-region-function, and rename
>>>>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>>> .... Based on the name, I suppose it's some kind of hook in
>>> font-lock-after-change-function, in which case I'd be tempted to suggest
>>> to move it to font-lock-fontify-region instead, to reduce the
>>> performance impact and make it easier to deal with lazy-lock&jit-lock
>>> since these tend to use their own after-change-function.

My original patch patched the \(lazy\|jit\)-lock after-change-functions,
too.  Maybe it would be cleaner to have a new function,
(font-lock-get-after-change-region BEG END OLD-LEN), to return that
region.  This function would also deal with extending the region to whole
lines.

After all, the region to fontify must be the same, regardless of the
support mode (if any) in use.

>> I strongly oppose such a change.  With that change:

>> (i) font-lock-fontify-region would no longer be fontifying the region
>> specified by its paramters, but a different (possibly larger) one.

>That's already the case since font-lock-fontify-region will typically not
>fontify from BEG to END from the beginning of line before BEG to the end of
>line after END.  Then that can be extended yet further because of
>font-lock-multiline.  Nobody has ever complained about this.

>So why is it a problem?

"Nobody complaining" has never been good evidence for something being OK.
Intimidation by authorities (though that doesn't happen here, hi
Richard!) will produce the same effect.  In fact, if Emacs 22 introduced
some wonderful new user facility, and there were NO complaints (or
feature requests), most likely nobody would be using it.

I think, in general, functions MUST do what they say they do, on the
exact parameters given by the caller.  For example, if SQRT (9) is
returned rather than SQRT (9.00322), because it's "easier to calculate",
this would be a bad thing.  In fact, Intel did more or less this with
some of their chips ~10 years ago.  Nobody complained, at least, not for
a while.  That didn't make it OK.

The burden of proof is surely the other way round.  If somebody believes
that, when a caller calls (font-lock-fontify-region 173 258), it's OK to
font lock the region (1 250) instead, the onus is on him to show that
this really is OK.

>> (ii) the hook function (which recalculates BEG and END) might well refer
>> to variables set by a before-change-functions hook.  (This is done in AWK
>> mode, for example).

>Which variables would that be?

Just one, actually - c-awk-old-EOLL, in cc-awk.el ~L787 - L811.  This
records the end-of-logical-line position before the change, in case the
change deletes the \ of an escaped newline.

>> f-l-f-region is regularly called when there is no buffer change in progress.

>Indeed.  Again, why is it a problem?

It would be a problem if f-l-f-region called the hook function, because
it would use c-awk-old-EOLL, which wouldn't now have a meaningful value.
Or, the hook would have to code round this, behaving differently
depending on where f-l-f-region was called from, which would be a Very
Bad Thing.

Or have I totally lost the thread of the argument, here?

>> Both of these things would make debugging a hook function much more
>> difficult than it already is.  Determining the region to fontify and
>> actually fontifying it are two logically distinct operations.

>With jit-lock, the region to fontify is not determined by the
>after-change-functions either.

(jit-lock-after-change BEG END OLD-LEN) determines that region in a
well defined fashion, surely?

The after-change (BEG END OLD-LEN) has a critically different meaning
from the (BEG END) in (font-lock-fontify-region BEG END).  The first of
these says how the buffer has been changed, the second defines a region
to fontify.  Deriving the second of these from the first is precisely the
problem we're trying to solve.

>        Stefan

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Martin!

On Wed, 15 Feb 2006, martin rudalics wrote:

> > That's how I understand it.  The region to be fontified will get
> > extended backwards to the starting tag regardless of this point being
> > visible in the window or not.  If it will be fontified right away or
> > marked to be fontified by jit-lock depends upon either
> > `jit-lock-after-change' or `font-lock-after-change-function' being
> > present in `after-change-functions'.

>Presumably the "region to be fontified gets extended" by resetting its
>fontified text-property to nil.  `jit-lock-after-change' should be able
>to do this if you provide the correct start and end values.  But keep in
>mind that with jit-lock redisplay immediately triggers refontification
>of displayed text whose fontified text-property has been reset to nil.
>This refontification won't recognize any "starting tag" preceding
>window-start unless you explicitly tell it to go there, for example, by
>using a multiline property.  If you decide on the `font-lock-multiline'
>property, every time you insert or delete a character nearby, redisplay
>has to refontify the entire contiguous area covered by the property.

There's a fundamental mismatch between Font Lock's implementation and
reality: Font Lock assumes that, with the exception of comments and
strings, the fontification of a region is dependent only on text near the
region.  This works pretty well for programming languages, but badly for
mark-up languages.

Conceptually, Emacs partitions the problem into a low-intensity global
task (locating comments and strings), for which it has the fast syntax
routines, and a high-intensity local task (hairy syntax analysis near
point.)  It is worth noting that the syntax routines support 2-character
delimiters (like "/*"), if only by kludge.

I am convinced that, in the long term, we need an analogous, fast, global
mechanism for for locating and characterizing regions bounded by
arbitrary delimiters - in this case "<<" and ">>", but could just as well
be Texinfo's "{" and "}" or Lex's and Yacc's "%{", "%}" and "%%" or
"literate programing"'s boundaries between narrative text and executable
code, or "here documents" within a shell script.

Until we have this, I think we'll be entangling ourselves in an ever
stickier web of ad-hoc workarounds.

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Ralf!

On Wed, 5 Feb 2006, Ralf Angeli wrote:

>* Alan Mackenzie (2006-02-15) writes:

>> On Sun, 12 Feb 2006, Stefan Monnier wrote:

>>>.... Based on the name, I suppose it's some kind of hook in
>>>font-lock-after-change-function, in which case I'd be tempted to suggest
>>>to move it to font-lock-fontify-region instead, to reduce the
>>>performance impact and make it easier to deal with lazy-lock&jit-lock
>>>since these tend to use their own after-change-function.

>> I strongly oppose such a change.  With that change:

>> (i) font-lock-fontify-region would no longer be fontifying the region
>> specified by its paramters, but a different (possibly larger) one. 
>> (ii) the hook function (which recalculates BEG and END) might well refer
>> to variables set by a before-change-functions hook.  (This is done in AWK
>> mode, for example).  f-l-f-region is regularly called when there is no
>> buffer change in progress.

>> Both of these things would make debugging a hook function much more
>> difficult than it already is.  Determining the region to fontify and
>> actually fontifying it are two logically distinct operations.  They
>> shouldn't be intermingled with eachother.

>This is all well and good, but in contrast to the after-change hook, a
>hook in `font-lock-default-fontify-region' could not only adjust the
>region after a change but also during fontification by chunks as done
>by jit-lock.  (That's an advantage I haven't noticed before.)

That is horrible.  f-l-d-f-r currently fontifies the region requested by
the caller, not a different region that Font Lock finds more convenient.
If we change its functionality, something somewhere will surely break.

>Following your reasoning of separating the determination of the region
>to be fontified from the fontification itself would require that
>jit-lock determines the chunks to be fontified more intelligently.

I don't follow that.  There are several different ways that jit-lock
determines its chunks.  There's after-change, after-scrolling, stealth,
at least.  I don't see why determining a jit-lock after-change region
need have anything to do with determining an after-scrolling or stealth
region.

>Ralf

-- 
Alan.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Vivek Dasmohapatra]

On Thu, 16 Feb 2006, Alan Mackenzie wrote:

> I am convinced that, in the long term, we need an analogous, fast, global
> mechanism for for locating and characterizing regions bounded by
> arbitrary delimiters - in this case "<<" and ">>", but could just as well
> be Texinfo's "{" and "}" or Lex's and Yacc's "%{", "%}" and "%%" or
> "literate programing"'s boundaries between narrative text and executable
> code, or "here documents" within a shell script.
>
> Until we have this, I think we'll be entangling ourselves in an ever
> stickier web of ad-hoc workarounds.

For what it's worth (sticking my oar in) I agree - and it would be even 
better if such regions could be arbitrarily put into other major modes...
(although that's more of a blue-sky wishlist thing...)

I believe mmm-mode tries do do something along these lines but from what I 
hear it's quite hard to set up and get working.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Alan Mackenzie (2006-02-16) writes:

> Hi, Ralf!
>
> On Wed, 5 Feb 2006, Ralf Angeli wrote:
>
>>This is all well and good, but in contrast to the after-change hook, a
>>hook in `font-lock-default-fontify-region' could not only adjust the
>>region after a change but also during fontification by chunks as done
>>by jit-lock.  (That's an advantage I haven't noticed before.)
>
> That is horrible.  f-l-d-f-r currently fontifies the region requested by
> the caller, not a different region that Font Lock finds more convenient.

As Stefan already pointed out this is not the case because the region
may be extended if `font-lock-multiline' is activated.

> If we change its functionality, something somewhere will surely break.
>
>>Following your reasoning of separating the determination of the region
>>to be fontified from the fontification itself would require that
>>jit-lock determines the chunks to be fontified more intelligently.
>
> I don't follow that.  There are several different ways that jit-lock
> determines its chunks.  There's after-change, after-scrolling, stealth,
> at least.  I don't see why determining a jit-lock after-change region
> need have anything to do with determining an after-scrolling or stealth
> region.

I was referring to stealth fontification during which jit-lock
fontifies chunks of the size determined by `jit-lock-chunk-size' one
after another.  If the start of one of those chunks is located inside
of a "<<...>>" construct an after-change hook will not let me extend
the region backwards to the starting tag in that case because there
hasn't happened a change.

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> PS: Furthermore, selective-display is on its way out.

.. in terms of usage, but I haven't heard anyone declare it obsolete.

> The only code left that uses it (in Emacs-CVS) is in dired-aux.el,
> gnus-sum.el, and locate.el, none of which interact with other major modes.

I have three other packages locally which uses it (I haven't looked
whether any of these have been updated in recent years to not
use selective-display): 

tags-menu.el, crypt++.el, dig-browser.el

--
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Ralf!

On Thu, 16 Feb 2006, Ralf Angeli wrote:

>* Alan Mackenzie (2006-02-16) writes:

>> Hi, Ralf!

>> On Wed, 5 Feb 2006, Ralf Angeli wrote:

>>>This is all well and good, but in contrast to the after-change hook, a
>>>hook in `font-lock-default-fontify-region' could not only adjust the
>>>region after a change but also during fontification by chunks as done
>>>by jit-lock.  (That's an advantage I haven't noticed before.)

>> That is horrible.  f-l-d-f-r currently fontifies the region requested
>> by the caller, not a different region that Font Lock finds more
>> convenient.

>As Stefan already pointed out this is not the case because the region
>may be extended if `font-lock-multiline' is activated.

>> If we change its functionality, something somewhere will surely break.

>>>Following your reasoning of separating the determination of the region
>>>to be fontified from the fontification itself would require that
>>>jit-lock determines the chunks to be fontified more intelligently.

>> I don't follow that.  There are several different ways that jit-lock
>> determines its chunks.  There's after-change, after-scrolling, stealth,
>> at least.  I don't see why determining a jit-lock after-change region
>> need have anything to do with determining an after-scrolling or stealth
>> region.

>I was referring to stealth fontification during which jit-lock
>fontifies chunks of the size determined by `jit-lock-chunk-size' one
>after another.  If the start of one of those chunks is located inside
>of a "<<...>>" construct an after-change hook will not let me extend
>the region backwards to the starting tag in that case because there
>hasn't happened a change.

OK, I've got you now!  It sounds like jit-lock perhaps isn't really the
right strategy for mark-up modes.  But it seems like a hook is required
in the jit-lock stealth bit.  This is surely needed as _well_ as the
after-change hook that expands the region, not instead of it.

For example, if a change were the insertion of a ">> .... <<", the region
to fontify would include some text before the new insertion.  The
after-change-expand-region hook would detect this by using information
recorded by a before-change-function.  AWK Mode uses this stragem.  A
stealth-expand-region hook would scan back to the ">>", and stop.

>Ralf

-- 
Alan.



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> I am convinced that, in the long term, we need an analogous, fast, global
> mechanism for for locating and characterizing regions bounded by arbitrary
> delimiters - in this case "<<" and ">>", but could just as well be
> Texinfo's "{" and "}" or Lex's and Yacc's "%{", "%}" and "%%" or "literate
> programing"'s boundaries between narrative text and executable code, or
> "here documents" within a shell script.

You're preaching to a converted.  I have a student here which I'm trying to
get to work on exactly this.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>>> (i) font-lock-fontify-region would no longer be fontifying the region
>>> specified by its paramters, but a different (possibly larger) one.

>> That's already the case since font-lock-fontify-region will typically not
>> fontify from BEG to END from the beginning of line before BEG to the end of
>> line after END.  Then that can be extended yet further because of
>> font-lock-multiline.  Nobody has ever complained about this.

>> So why is it a problem?

> "Nobody complaining" has never been good evidence for something being OK.
[ Follows some exaggeration which I find tends to weaken your point, so
  I prefer to cut it. ]

But the fact is:
- font-lock-fontify-region doesn't have a docstring a doesn't seem
  documented in the Emacs manual either.  So when you say "functions MUST do
  what they say they do", it's not clear to me that font-lock-fontify-region
  says anything about what it does.
- font-lock-fontify-region AFAICT has *never* limited its scope to
  BEG...END.  So anybody expecting font-lock-fontify-region to only fontify
  from BEG to END is deluding himself.

Basically, the way it has been defined and used until now is the following:
fontify the smallest region that includes BEG..END and which doesn't
start or end in the middle of a font-lock-keyword.

For font-lock-keywords which can span several lines, this implies extending
the region by the corresponding number of lines.  Now the region matched by
a given font-lock-keyword can't be automatically determined by
font-lock-fontify-region, so the heuristic used is to extend the region to
a whole number of lines and if that's not enough some other piece of code
(e.g. user-provided code) has to tell it via the font-lock-multiline
property (or as I suggest, via a font-lock-extend-region-functions hook).

Now let's say you have a multiline element which your font-lock-keyword is
somehow able to (re)highlight line-by-line, then you indeed don't need to
extend the region in font-lock-fontify-region.

Of course it gets trickier when your font-lock-keyword can (re)highlight
your multiline element line-by-line but only if the first line of the
element has already been fontified (where it recognizes the element and
somehow marks the text so that fontification of further lines is done
properly).  And even worse if the fontification of the first line only
recognizes the whole element depending on some text further down
(e.g. a terminator), since then the first (and succeeding) line(s) need(s)
to be somehow rehighlighted when the terminator is added/removed.
For such a situation, I currently don't know of a "canonical" way to get
font/jit-lock to do the right thing, other than to simply force the whole
multiline element to always be rehighlighted as a whole (i.e. defeating your
efforts to make your font-lock-keyword able to (re)highlight on
a line-by-line basis).

If that's the case you're trying to solve, then an after-change-function
which marks the whole multiline entity for refontification whenever the
terminator is added/removed won't cut it: if the first line is outside of
the window, jit-lock won't refontify it.

Maybe what will cut it is: using some kind of after/before-change-function
(which doesn't have to be in font-lock-after-change-function) detect when
a terminator is added/removed and in that case add a font-lock-multiline
property on the whole element (which will cause the whole element to be
(re)fontified not matter what jit-lock thinks, but the whole-element
fontification will only happen this one time since the font-lock-multiline
property is removed afterwards).

Another way: move the special treatment of the first line from
font-lock-keywords to font-lock-syntactic-keywords.  Then the
after/before-change-function only needs to move
jit-lock-context-unfontify-pos.

Yet another way: again, move the special treatment of the first line from
font-lock-keywords to font-lock-syntactic-keywords, and have it place
a jit-lock-defer-multiline property on the whole element.  But if you need
the terminator to determine whether and where is the whole element, you
clearly can't use that since you wouldn't know where/when to put that
property such that adding a terminator causes the
appropriate refontification.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> I was referring to stealth fontification during which jit-lock fontifies
> chunks of the size determined by `jit-lock-chunk-size' one after another.

jit-lock-chunk-size is not only used for stealth fontification.  It's also
used for the main normal case of display-triggered fontification.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> The only code left that uses it (in Emacs-CVS) is in dired-aux.el,
>> gnus-sum.el, and locate.el, none of which interact with other major modes.

> I have three other packages locally which uses it (I haven't looked
> whether any of these have been updated in recent years to not
> use selective-display): 

> tags-menu.el,
> dig-browser.el

These are probably similar to locate.el: they don't interact with other
major modes.

> crypt++.el,

This one doesn't use selective-display, AFAICT (if you just grepped for
selective-display, you've been fooled: many packages refer to it without
actually using it: they only use the variable because they need to do
something with it in case some other package is using it).


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > There's a fundamental mismatch between Font Lock's implementation and
 > reality: Font Lock assumes that, with the exception of comments and
 > strings, the fontification of a region is dependent only on text near the
 > region.  This works pretty well for programming languages, but badly for
 > mark-up languages.

Font Lock is not to blame here, you must improve `parse-partial-sexp'.
The following comment in font-lock has been here ever since ...

   "Perhaps one day someone will write
    some syntactic parsers for common languages and a son-of-font-lock.el could
    use them rather then relying so heavily on the keyword (regexp) pass."

 > I am convinced that, in the long term, we need an analogous, fast, global
 > mechanism for for locating and characterizing regions bounded by
 > arbitrary delimiters - in this case "<<" and ">>", but could just as well
 > be Texinfo's "{" and "}" or Lex's and Yacc's "%{", "%}" and "%%" or
 > "literate programing"'s boundaries between narrative text and executable
 > code, or "here documents" within a shell script.
 >
 > Until we have this, I think we'll be entangling ourselves in an ever
 > stickier web of ad-hoc workarounds.

In the short term, such mechanisms needn't even be fast.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 >>With jit-lock, the region to fontify is not determined by the
 >>after-change-functions either.
 >
 >
 > (jit-lock-after-change BEG END OLD-LEN) determines that region in a
 > well defined fashion, surely?
 >

No.  If BEG < window-start or END > window-end the non-displayed regions
are not necessarily fontified.  If you don't display them and don't
fontifiy them stealthily they will be never fontified.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I am convinced that, in the long term, we need an analogous, fast,
>> global mechanism for for locating and characterizing regions
>> bounded by arbitrary delimiters - in this case "<<" and ">>", but
>> could just as well be Texinfo's "{" and "}" or Lex's and Yacc's
>> "%{", "%}" and "%%" or "literate programing"'s boundaries between
>> narrative text and executable code, or "here documents" within a
>> shell script.
>
> You're preaching to a converted.  I have a student here which I'm
> trying to get to work on exactly this.

We really need to get 22.1 out the door, because fundamental changes
like that should only be done once Emacs 23.1 has been released (we
really need to get emacs-unicode-2 and possibly the multi-tty branch
as well as current package updates out before getting caught in
another multi-year delay cycle because of new features).

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > Maybe what will cut it is: using some kind of after/before-change-function
 > (which doesn't have to be in font-lock-after-change-function) detect when
 > a terminator is added/removed and in that case add a font-lock-multiline
 > property on the whole element (which will cause the whole element to be
 > (re)fontified not matter what jit-lock thinks, but the whole-element
 > fontification will only happen this one time since the font-lock-multiline
 > property is removed afterwards).

Text within "<<" ... ">>" would have to keep its multiline property
forever.  Take:

<<
...
bar
...
 >

"bar" wouldn't be highlighted specially.  Insert the closing ">":

<<
...
bar
...
 >>

"bar" would be highlighted specially. Now overwrite "r" by "z":

<<
...
baz
...
 >>

Who'd be responsible for highlighting "baz" specially?

Also, checking for the presence of "<<" ... ">>" round window-start just
when the buffer changes is insufficient.  You have to check also
whenever you jump to an arbitrary buffer position preceded by
unfontified text, scroll the window back into unfontified text, or
change your window configuration - lazy-lock.el knows the according
hooks.

 > Another way: move the special treatment of the first line from
 > font-lock-keywords to font-lock-syntactic-keywords.  Then the
 > after/before-change-function only needs to move
 > jit-lock-context-unfontify-pos.

With jit-lock and the "first line" before window-start that wouldn't
change a thing.  `font-lock-fontify-syntactically-region' and
`font-lock-fontify-keywords-region' have identic arguments - at least
when called by `font-lock-default-fontify-region'.  Calling
`font-lock-fontify-syntactically-region' from

(min jit-lock-context-unfontify-pos beg)

instead of `beg' would fix this but you won't get rid of a multiline
property here either.  And, when you insert the contents of an entire
file into the current buffer you would have to fontify them all before
you were able to continue working.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > OK, I've got you now!  It sounds like jit-lock perhaps isn't really the
 > right strategy for mark-up modes.  But it seems like a hook is required
 > in the jit-lock stealth bit.  This is surely needed as _well_ as the
 > after-change hook that expands the region, not instead of it.

Modes shouldn't rely on stealth fontification doing something important.
Stealth fontification has one single purpose: Prefontify something the
user might visit eventually - useful for slow terminals.  Using stealth
fontification for other purposes means asking for troubles.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* martin rudalics (2006-02-17) writes:

>  > OK, I've got you now!  It sounds like jit-lock perhaps isn't really the
>  > right strategy for mark-up modes.  But it seems like a hook is required
>  > in the jit-lock stealth bit.  This is surely needed as _well_ as the
>  > after-change hook that expands the region, not instead of it.
>
> Modes shouldn't rely on stealth fontification doing something important.
> Stealth fontification has one single purpose: Prefontify something the
> user might visit eventually - useful for slow terminals.  Using stealth
> fontification for other purposes means asking for troubles.

Sorry, but I don't understand what you mean.  Nobody proposed to use
stealth fontification for other purposes than fontification.  There
just has to be a way for it to adjust the region for fontification in
case of multiline constructs.  (Maybe this is already possible with
contextual fontification as proposed by Stefan, but I haven't looked
deeper into this yet.)

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 > Sorry, but I don't understand what you mean.  Nobody proposed to use
 > stealth fontification for other purposes than fontification.  There
 > just has to be a way for it to adjust the region for fontification in
 > case of multiline constructs.  (Maybe this is already possible with
 > contextual fontification as proposed by Stefan, but I haven't looked
 > deeper into this yet.)

I meant that you should not rely on stealth fontification to correct any
misfontifications.  Stealth fontification does not try to refontify
anything that is already fontified.  Hence, it makes no sense to "adjust
the region for fontification".  When `font-lock-default-fontify-region'
detects that the region it should fontify stealthily is near a multiline
property, it will extend the region appropriately.




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* martin rudalics (2006-02-17) writes:

> I meant that you should not rely on stealth fontification to correct any
> misfontifications.  Stealth fontification does not try to refontify
> anything that is already fontified.  Hence, it makes no sense to "adjust
> the region for fontification".  When `font-lock-default-fontify-region'
> detects that the region it should fontify stealthily is near a multiline
> property, it will extend the region appropriately.

If fontification of the multiline construct in question is only done
when its closing tag is found, there won't be any multiline property
nearby.  That's the proplem to be solved by "the hook".

-- 
Ralf


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> You're preaching to a converted.  I have a student here which I'm
>> trying to get to work on exactly this.

> We really need to get 22.1 out the door, because fundamental changes
> like that should only be done once Emacs 23.1 has been released

Don't you worry.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Maybe what will cut it is: using some kind of after/before-change-function
>> (which doesn't have to be in font-lock-after-change-function) detect when
>> a terminator is added/removed and in that case add a font-lock-multiline
>> property on the whole element (which will cause the whole element to be
>> (re)fontified not matter what jit-lock thinks, but the whole-element
>> fontification will only happen this one time since the font-lock-multiline
>> property is removed afterwards).

> Text within "<<" ... ">>" would have to keep its multiline property
> forever.

The above way said in the context of:

> Now let's say you have a multiline element which your font-lock-keyword is
> somehow able to (re)highlight line-by-line, then you indeed don't need to
> extend the region in font-lock-fontify-region.

So the answer to:

> Who'd be responsible for highlighting "baz" specially?

is font-lock-keywords.  See the "recent" change to smerge-mode's
font-lock-keywords for an example.


        Stefan


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Hi, Richard!

On Sun, 12 Feb 2006, Richard M. Stallman wrote:

>Here's the change I was talking about.  Would someone please adapt
>this _now_ to the current sources, and install it?  Then please rename
>before-font-lock-after-change-function to
>font-lock-extend-region-function, and rename
>font-lock-run-before-after-change-hook to font-lock-extend-region?

>Then please ack this message.

DONE.  In the process, I moved the processing of font-lock-lines-before
from font-lock-default-fontify-region into
font-lock-after-change-function and jit-lock-after-change, so as to be
able to stop them interfering with eachother.

More controversially, I've explicitly documented that the region returned
by the f-l-extend-region may start or end in the middle of a line.  I'm
not sure whether that will work properly at the moment (I suspect it
won't), but I think it can be and should be fixed.  I'm thinking about a
piece of badly formatted C code something like:

1. int foo (int
2.          bar) {printf "Hello world"} ; int baz
3.   (int omg) {

The "baz" on L3 hasn't a snowball's chance in Hades of getting fontified
with f-l-function-name-face unless the font lock region is allowed to
start at the "int" on L2.  There are surely much better formatted buffers
which also need the region to start in the middle of a line.

I've also created a new @subsection in modes.texi, "Region to Fontify",
which documents font-lock-extend-region-function, and I've moved
font-lock-lines-before into it.

-- 
Alan.






-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Here's the change I was talking about.  Would someone please adapt
>> this _now_ to the current sources, and install it?  Then please rename
>> before-font-lock-after-change-function to
>> font-lock-extend-region-function, and rename
>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>> Then please ack this message.

> DONE.  In the process, I moved the processing of font-lock-lines-before
> from font-lock-default-fontify-region into
> font-lock-after-change-function and jit-lock-after-change, so as to be
> able to stop them interfering with eachother.

As discussed and agreed in this list, font-lock-lines-before should simply
be removed since it is subsumed by the new feature.

Also, I thought we had agreed that this new feature should be in
f-l-default-fontify-region rather than f-l-after-change-function.  I believe
it is *wrong* for it to be in f-l-after-change-function where it is mostly
useless: you can already get the same result by using an
(non-f-l)after-change-functions hook that sets the
font-lock-multiline property.

> More controversially, I've explicitly documented that the region returned
> by the f-l-extend-region may start or end in the middle of a line.

I don't care much either way.  If someone does that, it's his problem: it
won't hurt font-lock.el.  But it may screw up the other font-lock-keywords.

> I'm not sure whether that will work properly at the moment (I suspect it
> won't), but I think it can be and should be fixed.  I'm thinking about
> a piece of badly formatted C code something like:

> 1. int foo (int
> 2.          bar) {printf "Hello world"} ; int baz
> 3.   (int omg) {

> The "baz" on L3 hasn't a snowball's chance in Hades of getting fontified
> with f-l-function-name-face unless the font lock region is allowed to
> start at the "int" on L2.

Why?  After all, when you open the file, the region will start at BOB.


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[martin rudalics]

 >>DONE.  In the process, I moved the processing of font-lock-lines-before
 >>from font-lock-default-fontify-region into
 >>font-lock-after-change-function and jit-lock-after-change, so as to be
 >>able to stop them interfering with eachother.
 >
 >
 > As discussed and agreed in this list, font-lock-lines-before should simply
 > be removed since it is subsumed by the new feature.
 >
 > Also, I thought we had agreed that this new feature should be in
 > f-l-default-fontify-region rather than f-l-after-change-function.  I believe
 > it is *wrong* for it to be in f-l-after-change-function where it is mostly
 > useless: you can already get the same result by using an
 > (non-f-l)after-change-functions hook that sets the
 > font-lock-multiline property.

Any functionality replacing `font-lock-lines-before' should be able to
handle the following `jit-lock' related scenario: "Display an arbitrary
range of a buffer such that the area preceding that range has not been
(syntactically- or keyword-) fontified yet."  With other words, any such
functionality should handle changes of `window-buffer', `window-start',
or `window-height' as well.  `after-change-functions' is of no use here.

An aside: About 20 major modes set `font-lock-syntactic-keywords'
currently.  None of them sets `font-lock-syntactically-fontified'.



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Ralf Angeli]

* Stefan Monnier (2006-03-14) writes:

> Also, I thought we had agreed that this new feature should be in
> f-l-default-fontify-region rather than f-l-after-change-function.  I believe
> it is *wrong* for it to be in f-l-after-change-function where it is mostly
> useless:

I think so, too.

> you can already get the same result by using an
> (non-f-l)after-change-functions hook that sets the
> font-lock-multiline property.

BTW, I am currently extending the region without a hook or an advice
by calculating the start of the (possibly extended) region to be
fontified in a custom function plugged into font lock via
`font-lock-fontify-region-function' which calls
`font-lock-default-fontify-region' with the adapted region start.

For that to work correctly I also had to adapt
`jit-lock-context-unfontify-pos' in the custom function as suggested
by you earlier.  Otherwise the part of the region before the line
where a change happened won't get refontified correctly.  That means
it will only be refontified upon redisplay (I guess), e.g. if one
enters the minibuffer with `M-x'.  So I wonder if a hook in
`font-lock-default-fontify-region' might have to cater for that as
well.

-- 
Ralf

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

> For that to work correctly I also had to adapt
> `jit-lock-context-unfontify-pos' in the custom function as suggested by
> you earlier.  Otherwise the part of the region before the line where
> a change happened won't get refontified correctly.  That means it will
> only be refontified upon redisplay (I guess), e.g. if one enters the
> minibuffer with `M-x'.  So I wonder if a hook in
> `font-lock-default-fontify-region' might have to cater for that as well.

Yes, indeed.  But we can make font-lock-default-fontify-region take care of
it transparently without the hook having to know about it.


        Stefan

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Good morning, Stefan!

On Tue, 14 Mar 2006, Stefan Monnier wrote:

>>> Here's the change I was talking about.  Would someone please adapt
>>> this _now_ to the current sources, and install it?  Then please rename
>>> before-font-lock-after-change-function to
>>> font-lock-extend-region-function, and rename
>>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>>> Then please ack this message.

>> DONE.  In the process, I moved the processing of font-lock-lines-before
>> from font-lock-default-fontify-region into
>> font-lock-after-change-function and jit-lock-after-change, so as to be
>> able to stop them interfering with eachother.

>As discussed and agreed in this list, font-lock-lines-before should simply
>be removed since it is subsumed by the new feature.

I wasn't sure about that, but I'm certainly agreeable.

>Also, I thought we had agreed that this new feature should be in
>f-l-default-fontify-region rather than f-l-after-change-function.  I believe
>it is *wrong* for it to be in f-l-after-change-function where it is mostly
>useless:

It actually works quite well, I've tested it.  :-)  In particular, it can
extend the fontification region backwards, to some "safe" position to
start analysis, or even when refontification is wanted before the changed
line.

I'm convinced this feature needs to be in the font-lock/jit-lock
after-change-functions.  The fontification region is determined by the
_change_, not merely by the buffer contents after the change.  For
example:

1. "string \
2. over \
3. several        <========= point is at EOL 3, having just deleted a \
4. #lines."

You've just deleted a backslash on L3.  The fontification region thus
extends from L1 (where the " gets changed from string-face to
warning-face) up to L4 (where the entire line gets changed from
string-face to comment-face).

With point at L3, a static analysis in f-l-default-fontify-region could
not know that it must refontify L4.

However, I think there should be analogous features for stealth/display
fontification, something like this:

    (defvar font-lock-safe-floor-position-function nil
      "A function that determines the latest \"safe\" position for Font
    Locking analysis \(or nil).  It is given one paramter, POS.")

    (defvar font-lock-safe-ceiling-position-function nil
      " ...")

The stealth/display fontification would extend their regions using these
functions, thus fixing the problem where after-change fontification is
done right, then damaged 3 seconds later when contextual fontification
kicks in.  With these two functions, contextual fontification would be
redundant.

There is already font-lock-beginning-of-syntax-function, but it is
over-constrained.  To start syntactic analysis, you don't need to go to
the top level (i.e. outside of braces).  You merely need to go somewhere
which is "top-level" to font lock (e.g. start of a statement), or even
some point where the font-lock state is known (e.g. it's been cached). 

>you can already get the same result by using an
>(non-f-l)after-change-functions hook that sets the font-lock-multiline
>property.

OK, but for that, font-lock-multiline must first be documented.  I've
tried to figure out from the source how f-l-multiline works, but haven't
succeeded yet.  Can f-l-multiline be used to extend the fontification
region backwards?

With the f-l-multiline approach, you need somehow to ensure that the
non-f-l-after-change function gets called before f-l-after-change.  It
can be done, but is hassle.  Also, f-l-multiline is an internal Font Lock
mechanism, and major modes shouldn't have to manipulate it directly,
since it would increase the coupling between modes and Font Lock
enormously, making it expensive to change approach in the future.

#########################################################################

>> More controversially, I've explicitly documented that the region returned
>> by the f-l-extend-region may start or end in the middle of a line.

>I don't care much either way.  If someone does that, it's his problem:
>it won't hurt font-lock.el.  But it may screw up the other
>font-lock-keywords.

Why would it be a problem?  Given that f-l-extend-region-function and
font-lock-keywords both belong to the major mode, they can be made to
match eachother.

A comment in jit-lock says that if the f-l region starts in the middle of
a line, that line won't be re-displayed.  I admit I haven't tried this
out yet, but assuming it's true, would it not be easy to remove the
restriction?

Are there any other problems in starting a font-lock region in the middle
of a line?

>> I'm not sure whether that will work properly at the moment (I suspect it
>> won't), but I think it can be and should be fixed.  I'm thinking about
>> a piece of badly formatted C code something like:

>> 1. int foo (int
>> 2.          bar) {printf "Hello world"} ; int baz
>> 3.   (int omg) {

>> The "baz" on L3 hasn't a snowball's chance in Hades of getting fontified
>> with f-l-function-name-face unless the font lock region is allowed to
>> start at the "int" on L2.

>Why?  After all, when you open the file, the region will start at BOB.

When you open the file, yes.

(i) You type a space on L2, "baz" changes to variable-name-face.

(ii) You then type a space on L3, it flips back to function-name-face,
but "bar" goes to default face.  (font-lock-lines-before == 1)

(iii) Type a space on L1, "bar" is restored to variable-name-face after
~0.5 second (contextual fontification?).

(iv) reload the file, do (i), then put point over "baz" (currently
variable-name-face).  Do C-u C-x =, to display its properties.  It
changes instantly to function-name-face, which it reports in the other
window.  Hello Heisenberg!

I honestly don't think it's worthwhile to investigate each of these
happenings individually - it's amusing to watch for a few minutes, but
rapidly becomes tedious.  They all stem from a single cause, namely the
misapprehension that font lock can start analyzing code at an arbitrary BOL.
It isn't just candidates for the Obfuscated C contest that are affected.
There's an open bug report at CC Mode, thus:

  Fontification is incorrect for enums, structs, classes, etc.  of the
  following form:

  enum Enum { A, B, C }
      e;

and another about the fontification in:

    #ifndef VMS   /* VMS hates multi-line '#if's */
    # if !defined(ibm032)                    && \
         !defined(__convex__)                &&          \
         !(defined(vax) && !defined(ultrix)) &&           \
         !defined(mips)                      &&          \
         !defined(apollo)                    &&          \
         !defined(pyr)                       &&          \
         !defined(__UMAXV__)                 &&          \
         !defined(bsd43)                     &&          \
         !defined(__bsd43)                   &&          \
         !(defined(BSD) && (BSD >= 199103))  &&          \
         !defined(aux)                       &&          \
         !defined(__bsdi__)                  &&          \
         !defined(sequent)



OK, these are likely to be bugs in CC Mode's font lock patterns, but the
uncertainty caused by font-lock's instability makes them more difficult
to debug.

#########################################################################

Summary of how I think about Font Lock should be fixed:

(i) f-l-extend-region-function should be supplemented by
f-l-safe-\(floor\|ceiling\)-position-function; if non-nil, these
functions give guaranteed good boundaries for fontification.  They would
render contextual fontification redundant.

(ii) Code expanding the font lock region to whole lines should be
removed (except as in (iii)).  font-lock-multiline would become
redundant.

(iii) When any of these three variables is unset, the pertinent region
boundary should be chosen as it currently is done.

(iv) The interface between Font Lock and the display code should be
amended so that a font lock region need not start at column 0.

(v) Any remaining restrictions on multi-line patterns should be removed.

>        Stefan

-- 
Alan.

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>> Also, I thought we had agreed that this new feature should be in
>> f-l-default-fontify-region rather than f-l-after-change-function.  I believe
>> it is *wrong* for it to be in f-l-after-change-function where it is mostly
>> useless:

> It actually works quite well, I've tested it.  :-)

Please re-read what I wrote.  I don't say it won't work.  It has various
problems, tho, which you most likely wouldn't notice in random testing,
especially if you turn off jit-lock.

> I'm convinced this feature needs to be in the font-lock/jit-lock
> after-change-functions.  The fontification region is determined by the
> _change_, not merely by the buffer contents after the change.  For
> example:

> The stealth/display fontification would extend their regions using these
> functions,

Thinking of stealth fontification as a special case will not lead you to
a good solution.  Been there, done that.  All the problems I've ever
encountered that were supposedly "due to jit-lock" could also be reproduced
without jit-lock.

>> you can already get the same result by using an
>> (non-f-l)after-change-functions hook that sets the
>> font-lock-multiline property.

> OK, but for that, font-lock-multiline must first be documented.

Go for it.  Note that I'm not talking about the variable, but about the
text-property.

> I've tried to figure out from the source how f-l-multiline works, but
> haven't succeeded yet.  Can f-l-multiline be used to extend the
> fontification region backwards?

Of course.  Put font-lock-multiline property on all the chars of a region
that should be refontified as a whole because the font-lock-keywords don't
know how to refontify it correctly unless they get to see the whole thing
at once.  f-l-default-fontify-region then extends the region
forward&backward so the region it refontifies doesn't start or end in the
middle of a font-lock-multiline property.

> With the f-l-multiline approach, you need somehow to ensure that the
> non-f-l-after-change function gets called before f-l-after-change.

Not if you use jit-lock ;-)

> It can be done, but is hassle.  Also, f-l-multiline is an internal Font
> Lock mechanism, and major modes shouldn't have to manipulate it directly,
> since it would increase the coupling between modes and Font Lock
> enormously, making it expensive to change approach in the future.

The feature you added suffers from the exact same problem.

> #########################################################################

>>> More controversially, I've explicitly documented that the region returned
>>> by the f-l-extend-region may start or end in the middle of a line.

>> I don't care much either way.  If someone does that, it's his problem:
>> it won't hurt font-lock.el.  But it may screw up the other
>> font-lock-keywords.

> Why would it be a problem?  Given that f-l-extend-region-function and
> font-lock-keywords both belong to the major mode, they can be made to
> match eachother.

Exactly: the author can only shoot himself, so "it's his problem".
Except of course for minor modes which add their own keywords (via
font-lock-add-keywords, for example), but these already suffer from several
other comparable minor problems, so it's probably not making things
noticeably worse.

> A comment in jit-lock says that if the f-l region starts in the middle of
> a line, that line won't be re-displayed.

Which comment?

>>> The "baz" on L3 hasn't a snowball's chance in Hades of getting fontified
>>> with f-l-function-name-face unless the font lock region is allowed to
>>> start at the "int" on L2.

>> Why?  After all, when you open the file, the region will start at BOB.

> When you open the file, yes.

> (i) You type a space on L2, "baz" changes to variable-name-face.

> (ii) You then type a space on L3, it flips back to function-name-face,
> but "bar" goes to default face.  (font-lock-lines-before == 1)

> (iii) Type a space on L1, "bar" is restored to variable-name-face after
> ~0.5 second (contextual fontification?).

> (iv) reload the file, do (i), then put point over "baz" (currently
> variable-name-face).  Do C-u C-x =, to display its properties.  It
> changes instantly to function-name-face, which it reports in the other
> window.  Hello Heisenberg!

I dno't see where the above example requires the region to start "at the int
on L2" (or more generally in the middle of a line).

> I honestly don't think it's worthwhile to investigate each of these
> happenings individually - it's amusing to watch for a few minutes, but
> rapidly becomes tedious.  They all stem from a single cause, namely the
> misapprehension that font lock can start analyzing code at an
> arbitrary BOL.

It's not a misapprehension: it's a constraint.  If your font-lock-keywords
don't obey this constraint, then yes, you'll see funny things because your
font-lock-keywords are broken.  Lifting this constraint is what "adding
multiline keywords support" is all about.  But I still haven't seen any
example where it needs to be possible to start in the middle of a line.
What is often neded is to be able to prevent starting/ending at some
particular BOLs.

> OK, these are likely to be bugs in CC Mode's font lock patterns, but the
> uncertainty caused by font-lock's instability makes them more difficult
> to debug.

Easy: turn off jit-lock and it becomes much more deterministic (and those
bugs can still be easily reproduced).  Sometimes you just need to cut&paste
a block of text at the same spot (i.e. mark and then C-w C-y).

> (i) f-l-extend-region-function should be supplemented by
> f-l-safe-\(floor\|ceiling\)-position-function; if non-nil, these
> functions give guaranteed good boundaries for fontification.  They would
> render contextual fontification redundant.

No, it's still not redundant at all.  Contextual fontification is solving
other cases of multiline patterns where the approach of extending the region
is unworkable (because the region may end up extended much too far, bringing
Emacs to a crawl): e.g. removing/adding a " in a C file typically causes the
whole rest of the buffer to be refontified.

> (ii) Code expanding the font lock region to whole lines should be
> removed (except as in (iii)).  font-lock-multiline would become
> redundant.

I fail to see the conection between font-lock-multiline and whole lines.

> (iv) The interface between Font Lock and the display code should be
> amended so that a font lock region need not start at column 0.

I don't think there is any such requirement.


        Stefan


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Richard Stallman]

Thanks for these changes.


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

font-lock-extend-region (was: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows])

[Stefan Monnier]

> DONE.  In the process, I moved the processing of font-lock-lines-before
> from font-lock-default-fontify-region into
> font-lock-after-change-function and jit-lock-after-change, so as to be
> able to stop them interfering with eachother.

OK, having thought some more about it, I'm really convinced doing it in
after-change-function is the wrong way: your jit-lock code won't always do
the right thing, because even though you mark the whole extended region for
refontification, jit-lock may refontify it in chunks (and maybe not even in
the intended order).

So I will move the font-lock-extend-region code to
font-lock-default-fontify-region where it belongs (which is why that's also
the place where font-lock-extra-lines was handled and where
font-lock-multiline is handled).

Now IIUC that means it'll break some/all of your uses of that variable.
Clearly you won't be pleased, but think about it this way: it'll save you
bug reports from users seeing odd behavior in conjunction with jit-lock.

Anyway, as I said, for your use case what you should be using is an
after-change-functions hook that puts a font-lock-multiline property.
But as you noted, this will only work if your hook happens to be placed in
after-change-functions before font-lock's own (or jit-lock's, though that
one is much less serious).

Also as I mentioned elsewhere, another solution is to change your
requirement such that some of the responsibility of the refontification is
passed on to contextual refontification: I would tend to prefer this
solution myself (it moves work away from the time-critical path).
But admittedly, setting jit-lock-context-time to 0 is believed to be a bit
too costly right now (not enough optimizations), so if you really want the
refontification to be immediate (rather than delayed by 0.5 idle seconds),
it's not a good solution.

So what I offer you is to introduce a new
`font-lock-before-after-change-functions' which is just like
after-change-functions except it's run by font-lock's (or jit-lock's)
after-change-function and before it does anything else.
You can then use this hook to place a function that computes the extended
region and places a font-lock-multiline property on it.

Do we have a deal?


        Stefan

Re: font-lock-extend-region (was: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows])

[Alan Mackenzie]

Hi, Stefan!

Before I get going, I'd like to say I've spent some time getting to grips
with jit-lock, and I think I now understand some of the things you were
telling me.  I also apologize for getting a bit grumpy about it last
week.

Here is a diagram of the sequence of functions that a piece of text is
processed by:

#########################################################################
"*" means that the routine adjusts the region to whole lines.

DEMAND (called by display code)  <----------------------------|
      [fontification-functions] (hook)                        |
      jit-lock-function ->-|                                  |
|-------------<------------+ <is defer-fontif. enabled?>      |
|                          |                                  |
|                          v                                  ^
| |-----------<------------|                                  |
| |                                                           |
| |->* jit-lock-fontify-now  <----------------------------|   |
|      [jit-lock-functions] (hook)                        |   |
v      font-lock-fontify-region  <------------------------+---+----|
|      [font-lock-fontify-region-function] (hook)         |   |    |
|    * font-lock-default-fontify-region                   |   |    |
|                                                         |   |    |
|DEFERRED (invoked by jit-lock-defer-timer)               |   |    |
|-->   jit-lock-deferred-fontify  ------------------>-----+---|    |
         This sets 'fontified to nil, and calls sit-for,  |   |    |
         causing immediate display (by DEMAND).           |   |    |
                                                          |   |    |
STEALTH (invoked by jit-lock-stealth-timer)               |   |    |
      jit-lock-stealth-fontify                            |   ^    |
      jit-lock-fontify-now ----------------->-------------|   |    |
                                                              |    |
CONTEXT (invoked by jit-lock-context-timer)                   |    |
      jit-lock-context-fontify (from timer) -------->---------+    |
        This sets 'fontified to nil on the (extended)         |    |     
        region.                                               |    |
                                                              |    |
JIT AFTER CHANGE (called from the after-change hook)          |    ^
    * jit-lock-after-change --------------->------------------|    |
      This sets 'fontified to nil, and relies on DEMAND            |
      to refontify the changed bit during display.                 |
                                                                   |
ORDINARY AFTER CHANGE (without jit, called from after-change)      |
    * font-lock-after-change-function                              |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |
COMMANDS                                                           |
      font-lock-fontify-block                                      |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |   
      font-lock-fontify-buffer                                     |
      [font-lock-fontify-buffer-function] (hook)                   |
      font-lock-default-fontify-buffer                             |
      font-lock-fontify-region ---------------->-------------------|
#########################################################################

On Mon, 20 Mar 2006, Stefan Monnier wrote:

>OK, having thought some more about it, I'm really convinced doing it
>[extending the fontification region with
>font-lock-extend-region-function] in after-change-function is the wrong
>way: your jit-lock code won't always do the right thing, because even
>though you mark the whole extended region for refontification, jit-lock
>may refontify it in chunks (and maybe not even in the intended order).

OK, I see what you're saying, now, I think - if you insert a large chunk
of text with C-y, font-lock-extend-region will calculate a starting
position off the top of the screen.  The display engine will then,
however, call jit-lock-function with the screen beginning as the place to
start, and this won't work properly.  I agree with you now.

>So I will move the font-lock-extend-region code to
>font-lock-default-fontify-region where it belongs (which is why that's
>also the place where font-lock-extra-lines was handled and where
>font-lock-multiline is handled).

I think there are two distinct issues here that we're confusing, and this
is why we've found it so hard to agree:

    (i) calculating the region which needs refontifying.
    (ii) finding a safe place to start fontifying a single chunk.

font-lock-extend-region-function is intended to do (i).  The
functionality you're suggesting for f-l-default-fontify-region is for
doing (ii). 

>Now IIUC that means it'll break some/all of your uses of that variable.
>Clearly you won't be pleased, but think about it this way: it'll save
>you bug reports from users seeing odd behavior in conjunction with
>jit-lock.

OK, I understand this now.

>Anyway, as I said, for your use case what you should be using is an
>after-change-functions hook that puts a font-lock-multiline property.
>But as you noted, this will only work if your hook happens to be placed
>in after-change-functions before font-lock's own (or jit-lock's, though
>that one is much less serious).

I think the essence of the font-lock-multiline property is that it marks
a chunk of text to be fontified atomically.  Please confirm this
impression or correct it for me.

Here's why I think the font-lock-muliline way is wrong.  Taking my AWK
example again:

1. "string \
2. over \
3. several \       <=========
4. #lines."

Suppose the user replaces the backslash on L3 with 20k of code from the
kill ring with M-y.  The region to fontify now extends from L1 to EOL4
(actually, it's now L1073).  The display engine is going to request
fontification from L1034.  If I mark this entire region with
font-lock-multiline, these 1073 lines will be (unnecessarily) fontified
atomically, defeating the aims of jit-lock in this case.  What I think we
need is a function called from f-l-default-f-region which will get a safe
starting position at or before L1034.

>Also as I mentioned elsewhere, another solution is to change your
>requirement such that some of the responsibility of the refontification
>is passed on to contextual refontification: I would tend to prefer this
>solution myself (it moves work away from the time-critical path).  But
>admittedly, setting jit-lock-context-time to 0 is believed to be a bit
>too costly right now (not enough optimizations), so if you really want
>the refontification to be immediate (rather than delayed by 0.5 idle
>seconds), it's not a good solution.

I do, and it's not.  :-)

>So what I offer you is to introduce a new
>`font-lock-before-after-change-functions' which is just like
>after-change-functions except it's run by font-lock's (or jit-lock's)
>after-change-function and before it does anything else.  You can then
>use this hook to place a function that computes the extended region and
>places a font-lock-multiline property on it.

For the reasons I've given above, I don't think this is the right thing
to do.  What I think we should do is to put a hook into
f-l-default-f-region to calculate a safe starting position (and probably
also a safe stopping position).

>Do we have a deal?

Not quite.  But I'm sure we'll soon be there.  :-)

Incidentally, referring to my diagram above, the region gets extended to
whole lines more than once.  For demand fontification, it is done first
in jit-lock-fontify-now then in font-lock-default-fontify-region.  For
after-change fontification, it is done yet a third time in
jit-lock-after-change.

How about doing this only in f-l-default-f-r?  This would make it easier
for a mode maintainer to switch off this action, since he would then just
have to put a modified function into the hook
font-lock-fontify-region-function.

>        Stefan

-- 
Alan.

Re: font-lock-extend-region

[Stefan Monnier]

[ I hope some of this exchange can get into some documentation somewhere.
  I find it very useful to have to explain it in such details. ]

> Before I get going, I'd like to say I've spent some time getting to grips
> with jit-lock, and I think I now understand some of the things you were
> telling me.  I also apologize for getting a bit grumpy about it last
> week.

Don't worry about it.  I've been so edgy these days that I wouldn't notice
anyone else getting grumpy.

> I think there are two distinct issues here that we're confusing, and this
> is why we've found it so hard to agree:

>     (i) calculating the region which needs refontifying.
>     (ii) finding a safe place to start fontifying a single chunk.

Right.  The first is generally handled by jit-lock-context-* (including the
jit-lock-defer-multiline property).  The second is generally handled by the
font-lock-multiline property and by rounding up to a whole number of lines.

> font-lock-extend-region-function is intended to do (i).  The functionality
> you're suggesting for f-l-default-fontify-region is for doing (ii).

In other words your use of font-lock-extend-region-function is specifically
to deal with issues that jit-lock-context-* tries to handle as well.

> I think the essence of the font-lock-multiline property is that it marks
> a chunk of text to be fontified atomically.  Please confirm this
> impression or correct it for me.

Correct.

> Here's why I think the font-lock-multiline way is wrong.  Taking my AWK
> example again:

> 1. "string \
> 2. over \
> 3. several \       <=========
> 4. #lines."

> Suppose the user replaces the backslash on L3 with 20k of code from the
> kill ring with M-y.  The region to fontify now extends from L1 to EOL4
> (actually, it's now L1073).  The display engine is going to request
> fontification from L1034.  If I mark this entire region with
> font-lock-multiline, these 1073 lines will be (unnecessarily) fontified
> atomically, defeating the aims of jit-lock in this case.

The region is not automatically marked with font-lock-multiline, so you
can't really fault font-lock-multiline for it: it's your code that marks it
that is at fault.
[ Now don't get me wrong: the font-lock-multiline property is not perfect. ]

> What I think we need is a function called from f-l-default-f-region which
> will get a safe starting position at or before L1034.

Agreed.  And I suggested we name it font-lock-extend-region-function.

You seem to be saying that you'd also want such a thing in
after-change-functions.

My belief is that you don't need it for the following reason: if you need to
refontify more than the 20K of code you just yanked, it can only be because
of elements at the beginning/end that need to be refontified atomically, so
you can just either place a font-lock-multiline property on them or extend
the region from f-l-default-fontify-region.

But maybe this is only true in theory, and reasons of performance (or
presence/absence of various info in different contexts) make it that you do
need an "extend-region" in font-lock-after-change-function.  Is that what
you are saying?

Or is there some other reason to extend the region from
after-change-functions, other than atomic elements at the boundaries?

> What I think we should do is to put a hook into f-l-default-f-region to
> calculate a safe starting position (and probably also a safe stopping
> position).

Yes, we agree on that.

> Incidentally, referring to my diagram above, the region gets extended to
> whole lines more than once.  For demand fontification, it is done first
> in jit-lock-fontify-now then in font-lock-default-fontify-region.  For
> after-change fontification, it is done yet a third time in
> jit-lock-after-change.

Yes, it's a bit messy, partly for historical reasons.

> How about doing this only in f-l-default-f-r?  This would make it easier
> for a mode maintainer to switch off this action, since he would then just
> have to put a modified function into the hook
> font-lock-fontify-region-function.

I believe the one in jit-lock-fontify-now could be removed (but this
function is also sadly called from external packages, so there may be some
minor compatibility issues).

But the one in jit-lock-after-change is needed because of what the comment
there says.  Basically here is the scenario:

  start with C code like

    void foo 

  and add an open parenthesis at the end.  The modified chunk is just the
  open paren, so if you don't reset the `fontified' property on the whole
  line, the redisplay engine will not redisplay `foo' and even if jit-lock
  changes the `face' property on `foo' it does it after the display engine
  decided what `foo' would look like.  So if jit-lock-after-change doesn't
  round up to whole lines, `foo' in the above scenario would only be
  refontified at the next screen refresh :-(

I'd like to be able to solve this problem elsewhere than in
jit-lock-after-change (e.g. some way for jit-lock to say "hey, font-lock
modified this text before BEGIN, please make sure you redisplay it
immediately"), but even if I knew how to do that, I'd probably not use it
for whole-line-round-up because it would simply cause (almost) all redisplay
to be done twice.


        Stefan


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: font-lock-extend-region

[Alan Mackenzie]

Hi, Stefan!

On Mon, 20 Mar 2006, Stefan Monnier wrote:

>[ I hope some of this exchange can get into some documentation somewhere.
>  I find it very useful to have to explain it in such detail. ]

Agreed on both points!  :-)

>> I think there are two distinct issues here that we're confusing, and this
>> is why we've found it so hard to agree:

>>     (i) calculating the region which needs refontifying.
>>     (ii) finding a safe place to start fontifying a single chunk.

>Right.  The first is generally handled by jit-lock-context-* (including the
>jit-lock-defer-multiline property).  The second is generally handled by the
>font-lock-multiline property and by rounding up to a whole number of lines.

>> font-lock-extend-region-function is intended to do (i).  The
>> functionality you're suggesting for f-l-default-fontify-region is for
>> doing (ii).

>In other words your use of font-lock-extend-region-function is specifically
>to deal with issues that jit-lock-context-* tries to handle as well.

Er, maybe.  ;-)

>> I think the essence of the font-lock-multiline property is that it
>> marks a chunk of text to be fontified atomically.  Please confirm this
>> impression or correct it for me.

>Correct.

OK, time for me to document this in modes.texi, I think.

>> Here's why I think the font-lock-multiline way is wrong.  Taking my AWK
>> example again:

>> 1. "string \
>> 2. over \
>> 3. several \       <=========
>> 4. #lines."

>> Suppose the user replaces the backslash on L3 with 20k of code from the
>> kill ring with M-y.  The region to fontify now extends from L1 to EOL4
>> (actually, it's now L1073).  The display engine is going to request
>> fontification from L1034.  If I mark this entire region with
>> font-lock-multiline, these 1073 lines will be (unnecessarily) fontified
>> atomically, defeating the aims of jit-lock in this case.

>The region is not automatically marked with font-lock-multiline, so you
>can't really fault font-lock-multiline for it: it's your code that marks it
>that is at fault.
>[ Now don't get me wrong: the font-lock-multiline property is not perfect. ]

Hang on a moment!  The CC Mode code _doesn't_ currently mark text with
font-lock-multiline.  I understood your suggestion was that it should do
this.

>> What I think we need is a function called from f-l-default-f-region which
>> will get a safe starting position at or before L1034.

>Agreed.  And I suggested we name it font-lock-extend-region-function.

font-lock-extend-region-function is already in use for purpose (i),
determining the region to be fontified.  I'm saying we also need a
distinct function for purpose (ii), determining a safe place to start
fontifying a chunk which could easily be in the middle of the region
returned by (i).

>You seem to be saying that you'd also want such a thing in
>after-change-functions.

Yes, that's exactly what I'm saying.

>My belief is that you don't need it for the following reason: if you need to
>refontify more than the 20K of code you just yanked, it can only be because
>of elements at the beginning/end that need to be refontified atomically, so
>you can just either place a font-lock-multiline property on them or extend
>the region from f-l-default-fontify-region.

   1. "string \
   2. over \
   3. several    <========= backslash/NL here replaced by ....
<....20k of text from kill ring.>
1073. #lines."

>But maybe this is only true in theory, and reasons of performance (or
>presence/absence of various info in different contexts) make it that you
>do need an "extend-region" in font-lock-after-change-function.  Is that
>what you are saying?

Yes.

After a buffer change, the region can't be extended within
f-l-default-fontify-region, since that function doesn't have access to to
BEG, END and OLD-LEN of after-change-functions, nor to OLD-BEG and
OLD-END of before-change-functions.  This analysis _must_ be done within
an after-change function.

On the other hand, determining the boundaries of a chunk of text that
needs fontifying atomically _must_ be done in f-l-default-f-region.  I
can't see any advantages in doing it in the after-change function too.

It is also conceivable that f-l-extend-region-f returns a start-position
well before the change, yet the interval from there to the change
position could be fontified in several distinct chunks rather than
atomically.  I admit I can't think of an realistic example right now,
though.

>Or is there some other reason to extend the region from
>after-change-functions, other than atomic elements at the boundaries?

The region sometimes has to be extended to the end of what was an atomic
element before the change, but is no longer so.

>> What I think we should do is to put a hook into f-l-default-f-region
>> to calculate a safe starting position (and probably also a safe
>> stopping position).

>Yes, we agree on that.

:-)

>> Incidentally, referring to my diagram above, the region gets extended
>> to whole lines more than once.  For demand fontification, it is done
>> first in jit-lock-fontify-now then in
>> font-lock-default-fontify-region.  For after-change fontification, it
>> is done yet a third time in jit-lock-after-change.

>Yes, it's a bit messy, partly for historical reasons.

>> How about doing this only in f-l-default-f-r?  This would make it easier
>> for a mode maintainer to switch off this action, since he would then just
>> have to put a modified function into the hook
>> font-lock-fontify-region-function.

>I believe the one in jit-lock-fontify-now could be removed (but this
>function is also sadly called from external packages, so there may be some
>minor compatibility issues).

I think any text region handled by jit-l-f-now cannot avoid being
processed by f-l-fontify-region.  This means f-l-default-f-region, unless
the hook f-l-f-region-function has been changed.  Somebody who does this
surely knows what she is doing.

>But the one in jit-lock-after-change is needed because of what the
>comment there says.  Basically here is the scenario:

>  start with C code like

>    void foo 

>  and add an open parenthesis at the end.  The modified chunk is just the
>  open paren, so if you don't reset the `fontified' property on the whole
>  line, the redisplay engine will not redisplay `foo' and even if jit-lock
>  changes the `face' property on `foo' it does it after the display engine
>  decided what `foo' would look like.  So if jit-lock-after-change doesn't
>  round up to whole lines, `foo' in the above scenario would only be
>  refontified at the next screen refresh :-(

I don't see this, yet.  If 'fontified is cleared only on the new open
paren at position 1055, the display engine calls successively
  (jit-lock-function 1055),
  (jit-lock-fontify-now 1055 1555),
    and (assuming j-l-f-now DOESN'T extend to whole lines)
  (run-hook jit-lock-functions 1055 1555),
  (font-lock-fontify-region 1055 1555),
  (funcall font-lock-fontify-region-function 1055 1555),
  (font-lock-default-fontify-region 1055 1555).
Here, in f-l-default-f-r, 1055 is set back to 1044 (BOL), and this whole
line, 1044 - 1056, gets its face properties rearranged.  These then get
redisplayed.  What am I missing here?

>I'd like to be able to solve this problem elsewhere than in
>jit-lock-after-change (e.g. some way for jit-lock to say "hey, font-lock
>modified this text before BEGIN, please make sure you redisplay it
>immediately"), but even if I knew how to do that, I'd probably not use it
>for whole-line-round-up because it would simply cause (almost) all redisplay
>to be done twice.


>        Stefan

-- 
Alan.



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: font-lock-extend-region

[Stefan Monnier]

>> The region is not automatically marked with font-lock-multiline, so you
>> can't really fault font-lock-multiline for it: it's your code that marks it
>> that is at fault.
>> [ Now don't get me wrong: the font-lock-multiline property is not perfect. ]

> Hang on a moment!  The CC Mode code _doesn't_ currently mark text with
> font-lock-multiline.  I understood your suggestion was that it should do
> this.

My response was to your complaint about font-lock-multiline because of what
would happen if the C mode placed a font-lock-multiline on the whole 20K of
yanked text.  I understand that C mode doesn't do that right now.

>>> What I think we need is a function called from f-l-default-f-region which
>>> will get a safe starting position at or before L1034.

>> Agreed.  And I suggested we name it font-lock-extend-region-function.

> font-lock-extend-region-function is already in use for purpose (i),
> determining the region to be fontified.

Yes, I understand that, but I'm trying to understand why you need it and
whether font-lock-extend-region-function is then the right answer.

> I'm saying we also need a distinct function for purpose (ii), determining
> a safe place to start fontifying a chunk which could easily be in the
> middle of the region returned by (i).

I think I need to start over.  IIUC you're trying to handle the "\\\n"
line-continuation feature, right?

For that, you need to always font-lock the whole multiline-line at
a time, right?  So my natural answer (given the hammer I have in my hand) is
to tell you to put a font-lock-multiline property on the whole
multiline-line.  My naive understanding says that it should be sufficient:
no need for font-lock-extend-region anywhere.

>> Or is there some other reason to extend the region from
>> after-change-functions, other than atomic elements at the boundaries?
> The region sometimes has to be extended to the end of what was an atomic
> element before the change, but is no longer so.

If this previously-atomic element had a font-lock-multiline property, then
it will be refontified atomically (at which point it will be discovered
that it's not atomic anymore so the font-lock-multiline property won't be
preserved).

>> I believe the one in jit-lock-fontify-now could be removed (but this
>> function is also sadly called from external packages, so there may be some
>> minor compatibility issues).
> I think any text region handled by jit-l-f-now cannot avoid being
> processed by f-l-fontify-region.

Indeed, as I said, I think it can be removed, although of course we'll then
have to update glasses-mode first (which registers itself with jit-lock).
I think glasses-mode and font-lock-mode are the only two to use jit-lock.

> I don't see this, yet.  If 'fontified is cleared only on the new open
> paren at position 1055, the display engine calls successively
>   (jit-lock-function 1055),
>   (jit-lock-fontify-now 1055 1555),
>     and (assuming j-l-f-now DOESN'T extend to whole lines)
>   (run-hook jit-lock-functions 1055 1555),
>   (font-lock-fontify-region 1055 1555),
>   (funcall font-lock-fontify-region-function 1055 1555),
>   (font-lock-default-fontify-region 1055 1555).
> Here, in f-l-default-f-r, 1055 is set back to 1044 (BOL), and this whole
> line, 1044 - 1056, gets its face properties rearranged.

You're right until here.

> These then get redisplayed.

But this is what doesn't happen.

> What am I missing here?

When the redisplay engine calls jit-lock-function with arg 1055, it has
already redisplayed all the text until 1054 and changes in the buffer that
affect text before 1055 will not cause redisplay to restart when
it finishes.


        Stefan


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: font-lock-extend-region

[Alan Mackenzie]

Hi, Stefan!

On Tue, 21 Mar 2006, Stefan Monnier wrote:

>>>> What I think we need is a function called from f-l-default-f-region
>>>> which will get a safe starting position at or before L1034.

>>> Agreed.  And I suggested we name it font-lock-extend-region-function.

>> font-lock-extend-region-function is already in use for purpose (i),
>> determining the region to be fontified.

>Yes, I understand that, but I'm trying to understand why you need it and
>whether font-lock-extend-region-function is then the right answer.

I think it's the right answer.

>> I'm saying we also need a distinct function for purpose (ii),
>> determining a safe place to start fontifying a chunk which could
>> easily be in the middle of the region returned by (i).

>I think I need to start over.  IIUC you're trying to handle the "\\\n"
>line-continuation feature, right?

Yes, in essence.

>For that, you need to always font-lock the whole multiline-line at a
>time, right?  So my natural answer (given the hammer I have in my hand)
>is to tell you to put a font-lock-multiline property on the whole
>multiline-line.  My naive understanding says that it should be
>sufficient: no need for font-lock-extend-region anywhere.

OK.  Yes, I think I agree with you now - the font-lock-multiline property
could indeed do this job successfully (that's assuming it can be set on
an arbitrary sequence of characters, not just on entire lines).  It is
marginally less good than f-l-e-r-function in one extreme case:  when the
extended after-change region is large, and consists of two contiguous
"atomic" ranges.  f-l-multiline would force jit-lock to fontify this
atomically, whilst f-l-e-r-function would allow
jit-lock-extend-chunk-function to fontify it as two chunks.

I still think that f-l-extend-region-f is a more natural approach - IMAO,
it is easier for a major mode to use.

Whatever we do, I think we are agreed that we need a major-mode supplied
function that will extend a jit-lock chunk (as opposed to an after-change
region).  Maybe jit-lock-extend-chunk-function taking two arguments, BEG
and END?

[ .... ]

[ talking about the number of times a fontification region is extended to
entire lines:]
>> I think any text region handled by jit-l-f-now cannot avoid being
>> processed by f-l-fontify-region.

>Indeed, as I said, I think it can be removed, although of course we'll then
>have to update glasses-mode first (which registers itself with jit-lock).
>I think glasses-mode and font-lock-mode are the only two to use jit-lock.

>> I don't see this, yet.  If 'fontified is cleared only on the new open
>> paren at position 1055, the display engine calls successively
>>   (jit-lock-function 1055),
>>   (jit-lock-fontify-now 1055 1555),
>>     and (assuming j-l-f-now DOESN'T extend to whole lines)
>>   (run-hook jit-lock-functions 1055 1555),
>>   (font-lock-fontify-region 1055 1555),
>>   (funcall font-lock-fontify-region-function 1055 1555),
>>   (font-lock-default-fontify-region 1055 1555).
>> Here, in f-l-default-f-r, 1055 is set back to 1044 (BOL), and this whole
>> line, 1044 - 1056, gets its face properties rearranged.

>You're right until here.

>> These then get redisplayed.

>But this is what doesn't happen.

>> What am I missing here?

>When the redisplay engine calls jit-lock-function with arg 1055, it has
>already redisplayed all the text until 1054 and changes in the buffer that
>affect text before 1055 will not cause redisplay to restart when
>it finishes.

Ah, got it!  Thanks!

Maybe the extension-to-whole-lines needs to stay in jit-lock-fontify-now
(since _something_ has to clear the 'fontified text property), and it has
to stay in f-l-default-f-region (since jit-lock isn't always enabled).
That only leaves the after-change one to take out.  Maybe.

>        Stefan

-- 
Alan.

Re: font-lock-extend-region

[Stefan Monnier]

> OK.  Yes, I think I agree with you now - the font-lock-multiline property
> could indeed do this job successfully (that's assuming it can be set on
> an arbitrary sequence of characters, not just on entire lines).

This assumption is valid.  Actually,the exact chars on which the
font-lock-multiline property is added is something important (e.g. you'd
have to include the \n after the \\).

> It is marginally less good than f-l-e-r-function in one extreme case:
> when the extended after-change region is large, and consists of two
> contiguous "atomic" ranges.  f-l-multiline would force jit-lock to fontify
> this atomically, whilst f-l-e-r-function would allow
> jit-lock-extend-chunk-function to fontify it as two chunks.

No, jit-lock would not be forced to refontify both chunks atomically at the
same time.  Remember jit-lock only deals with what needs to be refontified
(i.e. point (i) in your earlier email) then chunkifies it.
Only font-lock-default-fontify-region cares about atomicity (point (ii) in
your earlier email).

> I still think that f-l-extend-region-f is a more natural approach - IMAO,
> it is easier for a major mode to use.

I guess it's a matter of taste rather.  Often, you can put the
font-lock-multiline property very cheaply at the time you're highlighting
the multiline element, so you don't need to do any extra work at all to
discover its boundaries.  In contrast when using f-l-extend-region-f you may
have to re-discover the boundaries that you had found last time you
highlighted the element.

Furthermore, as you've discovered, if you use f-l-extend-region-f you get
into trouble when the time comes to refresh an element that used to be
multiline but isn't any more: you have to somehow figure out that it was
multiline earlier, then find the boundaries it had, then tell jit-lock that
this also needs to be refontified.  So you end up needing
a before-change-function, and an after-change-function, ...
Whereas with font-lock-multiline the problem doesn't even appear because it
uses text properties to remember the earlier multilineness of the element.

> Whatever we do, I think we are agreed that we need a major-mode supplied
> function that will extend a jit-lock chunk (as opposed to an after-change
> region).

The more I think about it, the less I'm convinced it's worth the trouble.
I didn't think hard about the design of font-lock-multiline when
I implemented it, but it turns out it works really well for what it's
designed to do.

>> When the redisplay engine calls jit-lock-function with arg 1055, it has
>> already redisplayed all the text until 1054 and changes in the buffer that
>> affect text before 1055 will not cause redisplay to restart when
>> it finishes.

> Ah, got it!  Thanks!

> Maybe the extension-to-whole-lines needs to stay in jit-lock-fontify-now
> (since _something_ has to clear the 'fontified text property), and it has
> to stay in f-l-default-f-region (since jit-lock isn't always enabled).
> That only leaves the after-change one to take out.  Maybe.

The above problem to which you replied "Ah, I got it" is the one that's
solved in the after-change function, so no: we can't take out the one in
after-change-function.

Clearing the `fontified' property is not a good justification for the
code in jit-lock-fontify-now.  I really think this one can go.
OTOH I think it'd be good if font-lock could return to jit-lock the extent
of the area that was actually refreshed (including extension due to
font-lock-multiline), so that jit-lock can clear more of the `fontified'
property and avoid some redundant font-locking.


        Stefan