Re: font-lock-extend-region (was: [sigra@home.se: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows])

[Alan Mackenzie <acm@muc.de>]

Hi, Stefan!

Before I get going, I'd like to say I've spent some time getting to grips
with jit-lock, and I think I now understand some of the things you were
telling me.  I also apologize for getting a bit grumpy about it last
week.

Here is a diagram of the sequence of functions that a piece of text is
processed by:

#########################################################################
"*" means that the routine adjusts the region to whole lines.

DEMAND (called by display code)  <----------------------------|
      [fontification-functions] (hook)                        |
      jit-lock-function ->-|                                  |
|-------------<------------+ <is defer-fontif. enabled?>      |
|                          |                                  |
|                          v                                  ^
| |-----------<------------|                                  |
| |                                                           |
| |->* jit-lock-fontify-now  <----------------------------|   |
|      [jit-lock-functions] (hook)                        |   |
v      font-lock-fontify-region  <------------------------+---+----|
|      [font-lock-fontify-region-function] (hook)         |   |    |
|    * font-lock-default-fontify-region                   |   |    |
|                                                         |   |    |
|DEFERRED (invoked by jit-lock-defer-timer)               |   |    |
|-->   jit-lock-deferred-fontify  ------------------>-----+---|    |
         This sets 'fontified to nil, and calls sit-for,  |   |    |
         causing immediate display (by DEMAND).           |   |    |
                                                          |   |    |
STEALTH (invoked by jit-lock-stealth-timer)               |   |    |
      jit-lock-stealth-fontify                            |   ^    |
      jit-lock-fontify-now ----------------->-------------|   |    |
                                                              |    |
CONTEXT (invoked by jit-lock-context-timer)                   |    |
      jit-lock-context-fontify (from timer) -------->---------+    |
        This sets 'fontified to nil on the (extended)         |    |     
        region.                                               |    |
                                                              |    |
JIT AFTER CHANGE (called from the after-change hook)          |    ^
    * jit-lock-after-change --------------->------------------|    |
      This sets 'fontified to nil, and relies on DEMAND            |
      to refontify the changed bit during display.                 |
                                                                   |
ORDINARY AFTER CHANGE (without jit, called from after-change)      |
    * font-lock-after-change-function                              |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |
COMMANDS                                                           |
      font-lock-fontify-block                                      |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |   
      font-lock-fontify-buffer                                     |
      [font-lock-fontify-buffer-function] (hook)                   |
      font-lock-default-fontify-buffer                             |
      font-lock-fontify-region ---------------->-------------------|
#########################################################################

On Mon, 20 Mar 2006, Stefan Monnier wrote:

>OK, having thought some more about it, I'm really convinced doing it
>[extending the fontification region with
>font-lock-extend-region-function] in after-change-function is the wrong
>way: your jit-lock code won't always do the right thing, because even
>though you mark the whole extended region for refontification, jit-lock
>may refontify it in chunks (and maybe not even in the intended order).

OK, I see what you're saying, now, I think - if you insert a large chunk
of text with C-y, font-lock-extend-region will calculate a starting
position off the top of the screen.  The display engine will then,
however, call jit-lock-function with the screen beginning as the place to
start, and this won't work properly.  I agree with you now.

>So I will move the font-lock-extend-region code to
>font-lock-default-fontify-region where it belongs (which is why that's
>also the place where font-lock-extra-lines was handled and where
>font-lock-multiline is handled).

I think there are two distinct issues here that we're confusing, and this
is why we've found it so hard to agree:

    (i) calculating the region which needs refontifying.
    (ii) finding a safe place to start fontifying a single chunk.

font-lock-extend-region-function is intended to do (i).  The
functionality you're suggesting for f-l-default-fontify-region is for
doing (ii). 

>Now IIUC that means it'll break some/all of your uses of that variable.
>Clearly you won't be pleased, but think about it this way: it'll save
>you bug reports from users seeing odd behavior in conjunction with
>jit-lock.

OK, I understand this now.

>Anyway, as I said, for your use case what you should be using is an
>after-change-functions hook that puts a font-lock-multiline property.
>But as you noted, this will only work if your hook happens to be placed
>in after-change-functions before font-lock's own (or jit-lock's, though
>that one is much less serious).

I think the essence of the font-lock-multiline property is that it marks
a chunk of text to be fontified atomically.  Please confirm this
impression or correct it for me.

Here's why I think the font-lock-muliline way is wrong.  Taking my AWK
example again:

1. "string \
2. over \
3. several \       <=========
4. #lines."

Suppose the user replaces the backslash on L3 with 20k of code from the
kill ring with M-y.  The region to fontify now extends from L1 to EOL4
(actually, it's now L1073).  The display engine is going to request
fontification from L1034.  If I mark this entire region with
font-lock-multiline, these 1073 lines will be (unnecessarily) fontified
atomically, defeating the aims of jit-lock in this case.  What I think we
need is a function called from f-l-default-f-region which will get a safe
starting position at or before L1034.

>Also as I mentioned elsewhere, another solution is to change your
>requirement such that some of the responsibility of the refontification
>is passed on to contextual refontification: I would tend to prefer this
>solution myself (it moves work away from the time-critical path).  But
>admittedly, setting jit-lock-context-time to 0 is believed to be a bit
>too costly right now (not enough optimizations), so if you really want
>the refontification to be immediate (rather than delayed by 0.5 idle
>seconds), it's not a good solution.

I do, and it's not.  :-)

>So what I offer you is to introduce a new
>`font-lock-before-after-change-functions' which is just like
>after-change-functions except it's run by font-lock's (or jit-lock's)
>after-change-function and before it does anything else.  You can then
>use this hook to place a function that computes the extended region and
>places a font-lock-multiline property on it.

For the reasons I've given above, I don't think this is the right thing
to do.  What I think we should do is to put a hook into
f-l-default-f-region to calculate a safe starting position (and probably
also a safe stopping position).

>Do we have a deal?

Not quite.  But I'm sure we'll soon be there.  :-)

Incidentally, referring to my diagram above, the region gets extended to
whole lines more than once.  For demand fontification, it is done first
in jit-lock-fontify-now then in font-lock-default-fontify-region.  For
after-change fontification, it is done yet a third time in
jit-lock-after-change.

How about doing this only in f-l-default-f-r?  This would make it easier
for a mode maintainer to switch off this action, since he would then just
have to put a modified function into the hook
font-lock-fontify-region-function.

>        Stefan

-- 
Alan.