Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman <lennart.borgman@gmail.com>]

On Tue, Aug 11, 2009 at 3:50 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:
> On Tue, 2009-08-11 at 00:19 +0200, Lennart Borgman wrote:
>> On Tue, Aug 11, 2009 at 12:06 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:
>>
>> Hi Eric,
>>
>> >  The concept of using the Semantic parser/generator framework for
>> > improving font-locking accuracy has come up many times.  No-one to my
>> > knowledge has attempted to mix the two.
>>
>>
>> Maybe that can easier be done if Semantic parser use
>> font-lock/JIT-lock timers and marking to keep track of what need to be
>> reparsed? (It is just a wild idea perhaps.)
>
> I'm not certain of how the font/jit lock works.  Semantic works by
> tracking edits (after-change-functions) and then on it's own timer, it
> coalesces the changes into parsable units.  It then reparses those
> units.


This is precisely what font/jit lock does too.

More precisely this is done by setting the text property 'fontified to
nil (if I remember correctly). JIT-lock will then pick up this and
refontify those parts.


> Font lock can refontify based on fairly small subsections of a buffer,
> such as a single code line, or a comment section.  Semantic's
> subsections are the size of functions, variables, and datatypes (ie, the
> tags it creates.)


Font/jit lock may also need bigger sections. In the case of mumamo it
does, for example. A change like adding a new major mode chunk
boundary may make it necessary to refontify the whole rest of the
buffer.


>> So you do a first pass with coarse parsing and then you look in the
>> sub-sections for details? Is this strictly necessary? I guess you are
>> looking for top level definitions in the first pass?
>>
>> Could that pass have its own state and continue upon demand (when an
>> item is not recognized) or is such a logic impossible?
>
> It could, but I have not done so.  Tagging information is not generally
> needed right away, so just waiting for the user to either ask for it, or
> sit idle for a while works pretty well.  The overhead of such an
> incremental parser isn't really needed.


The benefits coudl be that Emacs behaved much more smoothly. Both
because the background parsing could be interrupted and restarted
easier and because it could be easier to handle precomutation
(perhaps).

The overhead might perhaps be rather small if you could say "please
give me a JIT framework that computes these values: ... (list of
functions)... and runs according to scheme ...(JIT computation
style)..."? Just loose ideas of course, but...


> The needs between the tagging parser and the font-lock parser are
> different.  Font lock needs to colorize arbitrary blocks of text, and a
> tagging parser needs to parse everything, but only needs the data
> periodically.


A very good point that maybe can be translated to strucures in a JIT framework.


> Converting a tagging parser to a colorizing parser would be challenging
> because of these different uses.


They might use similar JIT frameworks but not the same timers (but
maybe cooperating dito).


>> >  I would imagine that the parsing engine in Semantic, if it is deemed
>> > critical by the maintainers, will get faster if key components are
>> > integrated into the C code.
>>
>> Is that part stable?
>
> Yes.  Not much is going on there.


Stefan seems positive to getting this kind of things into C code. To
me it seems very reasonable too.


>> >  Lastly, as David Engster stated, CEDET has decoration tools that
>> > decorate entire tags in some way, such as putting a line on top of
>> > functions.  This is a separate decoration activity not related to font
>> > lock, and something font lock would not be able to do reliably.
>>
>> Why not if it asks the parser?
>
> Font lock runs long before the parser bothers trying to catch up.  Font
> lock would needs hooks for after the parser runs.
> problems.


Perhaps it can be handled like this:

- The parser puts a tag on the parts where it wants special faces.
- It also sets 'fontified to nil on those regions. This will tell JIT
lock to refontify those parts.
- Then font-lock keywords functions supplied by the parser can set the
actual fontification when called by font/JIT-lock.

This will avoid the problem that the special parser fontification gets
thrown away when scrolling that Joakim mentioned.


> While font lock and semantic share a need for a parsing infrastructure,
> the where/when of the parsing is quite different.  It is possible to
> conceptually mix and match the parsers vs the schedulers.  In practice,
> the two tools have their own lengthy histories that will make that
> challenging.  Before tackling such a project, it would be wise to take
> multi-mode (or similar tool) into account.

Yes.

Thanks for the good explanations.


> Eric
>