Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> I've generally been assuming that your code would explicitly do
>> a put-text-property, but indeed font-lock.el includes some code which tries
>> to do that for you.

> It would do this, presumably, in an after-change (or before-change)
> function, either directly on one of these hooks or called from
> font-lock-after-change.

No, it's done in font-lock-default-fontify-region.

> OK.  So the font-lock-multiline property is put on exactly a region of
> text which needs fontifying atomically, yet straddles a line break.  An
> after-change function (and/or a before-change function) is what will do
> this putting and erasing of f-l-multiline.

No, it's done in font-lock-default-fontify-region.

> I think it's more accurate just to say you want to remove the
> extend-region stuff from f-l-after-change.  I want it to stay.  :-)
> Why don't we call the thing we need in f-l-d-f-r
> "font-lock-extend-CHUNK-function", since it's more likely to be a
> jit-lock chunk than a region supplied by the user or major mode?

Since it's a hook run from font-lock-default-fontify-region, it makes sense
to call it "extend-region", don't you think?  Naming (and documenting)
something based on how it is used rather than what it does is usually
a bad idea.

> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
> might gobble too much processor time?

That and the fact that it's not necessary, and that to use it (as seen in
your example) you need a good bit more work/code than doing it "the other
way" (with code solely run from font-lock-default-fontify-region).

> It needs to be in j-l-fontify-now, so that that function knows what bytes
> to apply the 'fontified property to.

This is not needed for correctness.  It's only a matter of avoiding
redundant work.  This need wouldn't be new (it has existed since Emacs-21
becaue of font-lock-multiline) and doesn't need to be addressed right away.

>> Yes it is important in general.  But this discussion is about the need to
>> have *any* a-c-f hook.  If we need one, its place is in f-l-a-c-f, indeed,
>> but I argue that we don't need one, so the precise place is not really
>> relevant to the discussion.

> I can't see how a major mode can manipulate f-l-multiline properties
> anywhere BUT an a-c-f (or a before-c-f).

What about the sample solution I provided: it's all manipulated from
font-lock-keywords (i.e. from font-lock-default-fontify-region)?

>> Look ma!  No old-len!  Tadaaa!
> .... but some code, somewhere, sometime needs to adjust the f-l-m
> property, removing it from L4.  In general, it will also need to _split_
> the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m region
> could gradually engulf an entire buffer, making a nonsense of jit-lock.
> (OK, this would be an extreme situation.  :-)

Yes, of course.  It's done this way: font-lock-unfontify-region (called from
font-lock-default-fontify-region) removes the font-lock-mutliline property
on the whole fontified region jsut before proceeding with the actual
fontification, so the font-lock-keywords need to re-add it at those places
where it's still needed.

>> If you insert text (instead of removing it), it gets a bit more
>> interesting (because the inserted text doesn't have a
>> font-lock-multiline property) but it basically works along the same
>> lines.

> Something, somewhere, sometime has to analyse the newly inserted code,
> putting f-l-m properties throughout it, I think, and adjusting those at
> the boundaries of the new region.  Is this to be done at the time of the
> change or at the time it gets font locked?  I think it has to be done at
> the change, otherwise the f-l-m settings will be wrong when it comes time
> to fontify a chunk.

Let me extend my example.  In the last episode, we stopped just before
refontification was to take place and had seen that the f-l-m property was
going to make the refontification apply to all 4 lines, as needed.
The fontification will work as follows:

5. font-lock-default-fontify-region gets asked to refontify line 3.

6. it extends the refontified region at beg and at end because of the
   presence of f-l-m property.  So the refontified region now spans all
   4 lines.

7. Prior to applying the new fontification, font-lock-unfontify-region is
   called which removes all `face' properties and `font-lock-multiline'
   properties on all 4 lines.

8. the actual refontification takes place, using font-lock-keywords.
   During this step, the rule

     (".*\\\\\n.*"
      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                   'font-lock-multiline t)
                nil)))

   is used to re-add a font-lock-multiline property to the first 3 lines
   (the 4th is not part of the multiline-line any more).

   If we had remoed the \ on line 2 instead, the refontification would have
   added the font-lock-multiline to lines 1&2 and then 2&3 (the two
   separated by a \n which is does not have this property).

See, it does analyse the new text to adjust the font-lock-multiline property.

> 1. I think font-lock-extend-region-function, called as it is from
> j-l-after-change, is a good thing.  It is a correct way of giving a major
> mode an opportunity to massage a fontification region.  I think it should
> stay.  You think it should be removed.

Yup.

> 2. f-l-d-f-r / j-l-f-now absolutely need a
> "font-lock-extend-chunk-function" functionality (regardless of its
> precise name).  We agree on this. 

Yup.

> 3. You have convinced me that it is possible to use the f-l-m properties.

Good.

> 4. I think that f-l-extend-region-f, called from f-l-after-change, is
> likely to be more convenient and less error prone than the direct
> manipulation of the f-l-multiline property by major mode code.

You mean adding a single put-text-property call in your font-lock-keywords
rule is more error prone than cooking up a b-c-f plus an a-c-f plus some way
to store info between the two so that the a-c-f can figure out what
happened, ...?

> 5. I suspect you think that f-l-extend-region-f is likely to consume more
> processor time than direct manipulation of the f-l-m property.

The problemis not what is done but when.  after-change-fucntions get run
more often than font-lock-fontify-region.

> Is there any reason why we can't leave both methods side by side in
> the code?

Unnecessary complexity.  And it will mislead people into thinking that
using a a-c-f is a good way to solve their problem.


        Stefan