"Font-lock is limited to text matching" is a myth

[Daniel Colascione <danc@merrillprint.com>]

On Monday 10 August 2009, Xah Lee wrote:
> emacs's syntax coloring is based on text pattern matching. It might work
> faster or more smooth right now, but from technology point of view, its one
> generation behind.

This is a common misconception I'd like to dispel. font-lock is *NOT* limited to using regular expressions. Yes, regular expressions are its default mode of operation, and for most languages, they are sufficient. But font-lock has the ability to mix and match several different kinds of fontification. Let's not reinvent the wheel.

There are two approaches to go beyond regular expression matching in font-lock: first, one can use a function as a font-lock matcher instead of a regular expression. That function can be arbitrarily complex, and can look at as much of the buffer as is necessary to fontify correctly. When this function locates a pattern to match, it returns control to font-lock, which fontifies it like any other interesting tidbit it found.

If that's not good enough, you can also have your font-lock "matcher" actually perform the fontification itself. cc-mode, espresso-mode, and nxml-mode use this technique. The basic idea is to find interesting patterns in the region of interest, then call font-lock-apply-highlight on them to perform the actual fontification. Because the "matcher" function is still using the font-lock infrastructure, font-lock will respect the fontification and behave correctly in other respects.

Using one of these two techniques allows a mode to use arbitrarily-complex fontification while still integrating with user-supplied keywords (e.g., from font-lock-add-keywords), various font-lock enhancement modes like jit-lock, and all other normal Emacs infrastructure.

It's *really* not necessary to bypass font-lock to get fontification to be as complex as you want. Just look at cc-mode.

> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
> cedet, can be leads in this direction and see how things work out.

It already does. It's called font-lock. If you want some kind of infrastructure that uses a grammar to perform fontification, I believe CEDET, which will be included in Emacs, can be wired up that way [I think? I'm not quite sure whether Semantic's tags are generic enough for fontification].

But using font-lock is not "a generation behind". Really, most editors' hard-coded highlighting features are a generation behind font-lock!