Multiline font lock questions

Multiline font lock questions

[David Abrahams]

A simplified view of what I'm trying to handle is to fontify
[...]-enclosed text that doesn't cross a paragraph boundary.

Quoting from the info documentation:

   There are three ways to ensure correct identification of multiline
   constructs:

   * Place a `font-lock-multiline' property on the construct when it is
     added to the buffer.

I don't understand how that can help.  If I start with

+------------+
|This [is    |
|paragraph 1 |
|            |
|This is     |
|paragraph] 2|
+------------+

then presumably nothing is marked with the `font-lock-multiline'
property, because there are no multiline constructs.  If I then kill
the blank line, how is it going to know to start identification at the
beginning of the newly-formed paragraph?

   * Use `font-lock-fontify-region-function' hook to extend the scan so
     that the scanned text never starts or ends in the middle of a
     multiline construct.

That's understandable; I suppose I can just change the region
boundaries to encompass any paragraphs that intersect the region and
then forward on to font-lock-default-fontify-region (?)

   * Add a function to `font-lock-extend-region-functions' that does
     the _identification_ and extends the scan so that the scanned text
     never starts or ends in the middle of a multiline construct.

This is presumably just a way to do the same thing as the above, only
more cooperatively (?)

   There are three ways to do rehighlighting of multiline constructs:

   * Place a `font-lock-multiline' property on the construct.  This
     will rehighlight the whole construct if any part of it is changed.
     In some cases you can do this automatically by setting the
     `font-lock-multiline' variable.

In which cases?  Oh, I see that if I read the next info page it will
tell me.  I suggest a link or something here.

   * Use `jit-lock-contextually'....  

What does "Use" mean?  I suppose "set it to non-nil," but I had to
guess that jit-lock-contextually was a variable; I suggest a doc tweak.

     ...This will only rehighlight the part
     of the construct that follows the actual change, and will do it
     after a short delay.  This only works if the highlighting of the
     various parts of your multiline construct never depends on text in
     subsequent lines.  Since `jit-lock-contextually' is activated by
     default, this can be an attractive solution.

That clearly wouldn't work for the example I showed above.

   * Place a `jit-lock-defer-multiline' property on the construct.
     This works only if `jit-lock-contextually' is used, but it can
     handle the case where highlighting depends on subsequent lines.

That seems like it could make it work.  What's the advantage of using
this method?  Doesn't slow down typing as much?

This very page links to another page describing
font-lock-extend-after-change-region-function, which seems like it
constitutes a fourth way to "do rehighlighting of multiline
constructs."  Am I misunderstanding that text?

Thanks in advance for your explanations,

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Multiline font lock questions

[rgb]

>    * Place a `font-lock-multiline' property on the construct when it is
>      added to the buffer.
>
> I don't understand how that can help.  If I start with
>
> +------------+
> |This [is    |
> |paragraph 1 |
> |            |
> |This is     |
> |paragraph] 2|
> +------------+
>
> then presumably nothing is marked with the `font-lock-multiline'
> property, because there are no multiline constructs.  If I then kill
> the blank line, how is it going to know to start identification at the
> beginning of the newly-formed paragraph?

Look at the SYNTAX-BEGIN argument of font-lock-defaults.

Re: Multiline font lock questions

[David Abrahams]

"rgb" <rbielaws@i1.net> writes:

>>    * Place a `font-lock-multiline' property on the construct when it is
>>      added to the buffer.
>>
>> I don't understand how that can help.  If I start with
>>
>> +------------+
>> |This [is    |
>> |paragraph 1 |
>> |            |
>> |This is     |
>> |paragraph] 2|
>> +------------+
>>
>> then presumably nothing is marked with the `font-lock-multiline'
>> property, because there are no multiline constructs.  If I then kill
>> the blank line, how is it going to know to start identification at the
>> beginning of the newly-formed paragraph?
>
> Look at the SYNTAX-BEGIN argument of font-lock-defaults.

IIUC that is only for `syntactic fontification,' which apparently
refers just to fontifying comments and strings.  Did I misinterpret
the doc?
 
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Multiline font lock questions

[rgb]

David Abrahams wrote:
> "rgb" <rbielaws@i1.net> writes:
>
> >>    * Place a `font-lock-multiline' property on the construct when it is
> >>      added to the buffer.
> >>
> >> I don't understand how that can help.  If I start with
> >>
> >> +------------+
> >> |This [is    |
> >> |paragraph 1 |
> >> |            |
> >> |This is     |
> >> |paragraph] 2|
> >> +------------+
> >>
> >> then presumably nothing is marked with the `font-lock-multiline'
> >> property, because there are no multiline constructs.  If I then kill
> >> the blank line, how is it going to know to start identification at the
> >> beginning of the newly-formed paragraph?
> >
> > Look at the SYNTAX-BEGIN argument of font-lock-defaults.
>
> IIUC that is only for `syntactic fontification,' which apparently
> refers just to fontifying comments and strings.  Did I misinterpret
> the doc?

AFAIK font-lock does ALL fontification (outside of any
you do yourself of course).  Not just comments and
strings.  Moreover it takes several passes across the
text to accomplish fontification.

font-lock-syntactic-keywords allows you to pick and choose
which characters have begin string fence syntax, comment end
syntax etc. before a pass marks strings and comments.
Only then can keywords be fontified so as to avoid words
in comments or strings.

So it depends on how you plan to apply font-lock-multiline.
If it were me, I'd be using the fontification routines to apply
the font-lock-multiline property to the text.  But to do that
I would need to insure that, as in your example, when something
like a blank line is deleted, fontification starts at the beginning
of the paragraph; which now is 2 lines prior to the deleted line.
I believe syntax-begin defaults to beginning-of-line.  So by
default the multi-line construct would not get recognized.

I've never actually had to change syntax-begin for my own
modes but I've written font-lock support for some fairly unusual
syntax.  Leading me to diluded myself into thinking I know
how it works.

Re: Multiline font lock questions

[David Abrahams]

"rgb" <rbielaws@i1.net> writes:

> David Abrahams wrote:
>> "rgb" <rbielaws@i1.net> writes:
>>
>> >>    * Place a `font-lock-multiline' property on the construct when it is
>> >>      added to the buffer.
>> >>
>> >> I don't understand how that can help.  If I start with
>> >>
>> >> +------------+
>> >> |This [is    |
>> >> |paragraph 1 |
>> >> |            |
>> >> |This is     |
>> >> |paragraph] 2|
>> >> +------------+
>> >>
>> >> then presumably nothing is marked with the `font-lock-multiline'
>> >> property, because there are no multiline constructs.  If I then kill
>> >> the blank line, how is it going to know to start identification at the
>> >> beginning of the newly-formed paragraph?
>> >
>> > Look at the SYNTAX-BEGIN argument of font-lock-defaults.
>>
>> IIUC that is only for `syntactic fontification,' which apparently
>> refers just to fontifying comments and strings.  Did I misinterpret
>> the doc?
>
> AFAIK font-lock does ALL fontification 

I know that.

> (outside of any you do yourself of course).

"Do yourself?"  That might be just what I need, actually.  How do I do
that?

> Not just comments and strings.  Moreover it takes several passes
> across the text to accomplish fontification.
>
> font-lock-syntactic-keywords allows you to pick and choose
> which characters have begin string fence syntax, comment end
> syntax etc. before a pass marks strings and comments.
> Only then can keywords be fontified so as to avoid words
> in comments or strings.

Right.

However, the docs for font-lock-defaults say:

  If SYNTAX-BEGIN is non-nil, it should be a function with no args
  used to move backwards outside any enclosing syntactic block, 
  for syntactic fontification.  
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^

Maybe "syntactic fontification" means something different from the
"just comments and strings" meaning of "syntactic"...?

  Typical values are `beginning-of-line'
  (i.e., the start of the line is known to be outside a syntactic
  block), or `beginning-of-defun' for programming modes or
  `backward-paragraph' for textual modes (i.e., the mode-dependent
  function is known to move outside a syntactic block).  If nil, the
  beginning of the buffer is used as a position outside of a syntactic
  block, in the worst case.

> So it depends on how you plan to apply font-lock-multiline.
> If it were me, I'd be using the fontification routines 

Sorry, which are those?

> to apply the font-lock-multiline property to the text.  But to do
> that I would need to insure that, as in your example, when something
> like a blank line is deleted, fontification starts at the beginning
> of the paragraph; which now is 2 lines prior to the deleted line.  I
> believe syntax-begin defaults to beginning-of-line.  

In my case, it's already nil, which according to the above should mean
that it goes all the way back to the beginning of the buffer ("in the
worst case," whatever that means) so it doesn't seem like this is the
right lever to pull.

> So by default the multi-line construct would not get recognized.
>
> I've never actually had to change syntax-begin for my own
> modes but I've written font-lock support for some fairly unusual
> syntax.  Leading me to diluded myself into thinking I know
> how it works.

Well, I am really stumped.  I am also, it seems, getting some
combinatoric regexp evaluation.  Whoopee!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Multiline font lock questions

[Stefan Monnier]

[ Since this refers to doc and vars that are specific to Emacs-22, it might
  be better to discuss it on emacs-devel. ]

>    There are three ways to ensure correct identification of multiline
>    constructs:

>    * Place a `font-lock-multiline' property on the construct when it is
>      added to the buffer.

> I don't understand how that can help.

Indeed, it's a bit awkwardly placed and written.  This is basically saying
"you can do it yourself manually somehow".

>    * Use `font-lock-fontify-region-function' hook to extend the scan so
>      that the scanned text never starts or ends in the middle of a
>      multiline construct.

> That's understandable; I suppose I can just change the region
> boundaries to encompass any paragraphs that intersect the region and
> then forward on to font-lock-default-fontify-region (?)

Right.

>    * Add a function to `font-lock-extend-region-functions' that does
>      the _identification_ and extends the scan so that the scanned text
>      never starts or ends in the middle of a multiline construct.

> This is presumably just a way to do the same thing as the above, only
> more cooperatively (?)

Indeed.  But this new var only exists in Emacs-22, so if you want your code
to work with Emacs-21, you should use
font-lock-fontify-region-function instead.

>    There are three ways to do rehighlighting of multiline constructs:

>    * Place a `font-lock-multiline' property on the construct.  This
>      will rehighlight the whole construct if any part of it is changed.
>      In some cases you can do this automatically by setting the
>      `font-lock-multiline' variable.

> In which cases?  Oh, I see that if I read the next info page it will
> tell me.  I suggest a link or something here.

Thanks.

>    * Use `jit-lock-contextually'....  

> What does "Use" mean?  I suppose "set it to non-nil," but I had to
> guess that jit-lock-contextually was a variable; I suggest a doc tweak.

Problem is: in 99% of the cases, it's already set, so there's nothing to do
really, other than rely on it doing its job.  That's why I set "use".

>      ...This will only rehighlight the part
>      of the construct that follows the actual change, and will do it
>      after a short delay.  This only works if the highlighting of the
>      various parts of your multiline construct never depends on text in
>      subsequent lines.  Since `jit-lock-contextually' is activated by
>      default, this can be an attractive solution.

> That clearly wouldn't work for the example I showed above.

Indeed.

>    * Place a `jit-lock-defer-multiline' property on the construct.
>      This works only if `jit-lock-contextually' is used, but it can
>      handle the case where highlighting depends on subsequent lines.

> That seems like it could make it work.  What's the advantage of using
> this method?

Over what?  `font-lock-multiline'?

> Doesn't slow down typing as much?

Yes: the re-highlighting is delayed a little bit (see
jit-lock-context-time), so it doesn't happen after every keystroke.
For small multiline elements, it's not a big change, but for larger ones, it
can be significant.

Another difference is that it only works if font-lock uses jit-lock.

> This very page links to another page describing
> font-lock-extend-after-change-region-function, which seems like it
> constitutes a fourth way to "do rehighlighting of multiline constructs."

Indeed.  Alan McKenzie likes this, but I would rather discourage
its use.


        Stefan

Re: Multiline font lock questions

[David Abrahams]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan, thanks for your reply.

>> What does "Use" mean?  I suppose "set it to non-nil," but I had to
>> guess that jit-lock-contextually was a variable; I suggest a doc tweak.
>
> Problem is: in 99% of the cases, it's already set, so there's nothing to do
> really, other than rely on it doing its job.  That's why I set "use".

Saying something explicit like

       Ensure that jit-lock-contextually is set (as it is by default)
       and rely on it doing its job.

would be more useful

>>      ...This will only rehighlight the part
>>      of the construct that follows the actual change, and will do it
>>      after a short delay.  This only works if the highlighting of the
>>      various parts of your multiline construct never depends on text in
>>      subsequent lines.  Since `jit-lock-contextually' is activated by
>>      default, this can be an attractive solution.
>
>> That clearly wouldn't work for the example I showed above.
>
> Indeed.
>
>>    * Place a `jit-lock-defer-multiline' property on the construct.
>>      This works only if `jit-lock-contextually' is used, but it can
>>      handle the case where highlighting depends on subsequent lines.
>
>> That seems like it could make it work.  What's the advantage of using
>> this method?
>
> Over what?  `font-lock-multiline'?

Any other methods you mention in this piece of doc.

>> Doesn't slow down typing as much?
>
> Yes: the re-highlighting is delayed a little bit (see
> jit-lock-context-time), so it doesn't happen after every keystroke.
> For small multiline elements, it's not a big change, but for larger ones, it
> can be significant.
>
> Another difference is that it only works if font-lock uses jit-lock.

It would be great if the doc would give me the information I need in
order to make an informed choice about which method to use.

>> This very page links to another page describing
>> font-lock-extend-after-change-region-function, which seems like it
>> constitutes a fourth way to "do rehighlighting of multiline
>> constructs."
>
> Indeed.  Alan McKenzie likes this, but I would rather discourage its
> use.

Why does he like it, and why do you dislike it?

Can we make the doc consistent, please?  If there are really four ways
documented, there shouldn't be a page saying there are three ways.
You can always add disparaging words about the fourth way if you
like.  As it stands it's hard to tell whether the information even
makes sense.

Thanks,

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Multiline font lock questions

[Stefan Monnier]

>>> This very page links to another page describing
>>> font-lock-extend-after-change-region-function, which seems like it
>>> constitutes a fourth way to "do rehighlighting of multiline
>>> constructs."
>> 
>> Indeed.  Alan McKenzie likes this, but I would rather discourage its
>> use.

> Why does he like it, and why do you dislike it?

That was the subject of a neverending thread :-(

> Can we make the doc consistent, please?  If there are really four ways
> documented, there shouldn't be a page saying there are three ways.
> You can always add disparaging words about the fourth way if you
> like.  As it stands it's hard to tell whether the information even
> makes sense.

Which way to fix the doc depends on the end of the neverending thread, hence
the current state.


        Stefan