bug#192: regexp does not work as documented

[David Koppelman <koppel@ece.lsu.edu>]

I've decided against having hi-lock turn on font-lock-multiline or
even apply font-lock-multiline text properties, too much potential to
slow things down to a crawl when an unsuspecting user enters a regexp.

If I understand things correctly, the font-lock-multiline property is
used to extend a region to be fontified, a region to be used for *all*
keywords. This would have disastrous effects when multi-line patterns
span, say, 100's of lines for modes with hundreds of keywords. I had
been toying with the idea of limiting extended regions to something
like 100 lines, but that still seems wasteful when most keywords are
single line (I haven't benchmarked anything yet).

A better solution would be to have font-lock use multi-line extended
regions selectively. Perhaps a hint in the current keyword syntax
(say, explicitly applying the font-lock-multiline property), or a
separate method for providing multi-line keywords to font-lock.
Such keywords would get the multi-line extended regions, the other
just the whole-line extensions (or whatever the hooks do).

Is this something the font-lock maintainers would consider?

What I'll do now is just document the limitations for hi-lock and
perhaps provide a warning when a multiline pattern is used.


> That may not be enough.  You'll probably want to do something like what
> smerge does:
>
>   (while (re-search-forward <RE> nil t)
>     (font-lock-fontify-region (match-beginning 0) (match-end 0)))

I wouldn't do that without suppressing other keywords. 


> If someone wants that, I have a parser that takes a regexp and turns it
> into something like `rx' syntax.  It uses my lex.el library (which
> takes an `rx'-like input syntax).

That sounds useful, either E-mail it to me or let me know
where to find it. 


Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> My latest plan is to do what Chong Yidong suggests, setting up text
>> properties so that font-lock DTRT, though it doesn't seem as hard as he
>> suggests (I'm still in the naive enthusiasm stage).
>
> Indeed, it shouldn't be that hard.
>
>> I tried adding the font-lock-multiline property to the face property
>> list passed to font lock and that did the trick, even with the
>> font-lock-multiline variable nil.
>
> That may not be enough.  You'll probably want to do something like what
> smerge does:
>
>   (while (re-search-forward <RE> nil t)
>     (font-lock-fontify-region (match-beginning 0) (match-end 0)))
>
> this will find all the multiline elements.  And the font-lock-multiline
> property you add will make sure that those that were found will not
> disappear accidentally because of some later refontification.
>
>> I rather do that than turn on font-lock-multiline because I'm assuming
>> that font-lock-multiline is set to nil in most cases for
>> a good reason.
>
> Setting the `font-lock-multiline' variable to t has a performance cost.
>
>> I actually thought about properly parsing the regexp, but the effort to
>> do that could be spent on making multi-line patterns work properly, at
>> least if they don't span too many lines.
>
> If someone wants that, I have a parser that takes a regexp and turns it
> into something like `rx' syntax.  It uses my lex.el library (which
> takes an `rx'-like input syntax).
>
>> One more thing, multi-line regexp matches don't work properly even
>> with font-lock-multiline t when jit-lock is being used in a buffer
>> without syntactic fontification and using the default setting of
>> jit-lock-contextually, setting it to t gets multi-line fontification
>> to work.
>
> The `font-lock-multiline' variable only tells font-lock that if it ever
> bumps into a multiline element, it should mark it (with the
> font-lock-multiline property) so that it will not re-fontify it as
> a whole if it ever needs to refontify it.
>
> So it doesn't solve the problem of "how do I make sure that font-lock
> indeed finds the multiline element".  Multiline elements can only be
> found when font-locking a large enough piece of text, which tends to
> only happen during the initial fontification, or during background or
> contextual refontification, or during an explicit call such as in the
> above while loop.
>
>
>         Stefan