Re: State-machine based syntax highlighting

["Robert Thorpe" <rthorpe@realworldtech.com>]

spamfilteraccount@gmail.com wrote:
> Tim X wrote:
> >
> > The problem with parse based analysis is that you need an in-built
> > parser for all the languages that the editor is used to develop in and
> > this is not a trivial task. I suspect some sort of plugin architecture
> > that is able to use stand-alone parses for some language of interest
> > would probably be the way to go as it is unlikely even a small subset
> > of the languages devleoped within an emacs environment can have a
> > parser developed in elisp which is readily maintained.
>
> I think too that some kind of bridge or plugin architecture is the
> answer.
>
> Lots of languages provide access to syntax trees in some form (python,
> java, etc.), so it would be much simpler to use their native
> implementation than reinveinting everything in elisp.

That isn't really appropriate though.

Consider the following.  When I open a project I generally open all
files in the directory by doing something like C-x C-x project_foo/*.c
.  I also use save-places, so point appears in each file wherever I
left it last.  I think both of these are quite common ways to use
Emacs.

Doing this with normal parsing technology is difficult.  If the editor
just feeds every file into the external parser then back into the
editor then this will be a lot of work.  It would be similar to the
work of a compiler doing a full rebuild.  In fact it would be less
because parsing for font-locking involves nothing similar to compiler
optimization or code generation.  But it would still be a big task.  A
much better strategy is to start parsing at point in each file and only
parse a screenful at a time, doing this with an external parser would
be very hard.

There are other problems.  What if a part of the code is incorrect?
Imagine, in C for example, if a function were written "foo (;" on line
10.  The effect of the error would propagate down far away from where
it occurs, even line 300 might be treated wrongly.  The parser would
have to cope with this eventuality.

Also, in many languages there are bits of the meaning that depend on
the names used.  In C for example the code " (foo) (bar)" means
something different if foo is a type than it does if it's an
identifier.  The C compiler can cope with this because it tracks all
typedefs and identifiers through not only the current file but those
included in it with #include.   The only way for a font-lock system
based on a normal parser to deal with this situation would be for it to
read all the include files, which may not even be present.

Compiler parsers and font-locking/navigating code have different
intentions.  Compiler parsers must be fast when handling a whole file,
and they must generate accurate error messages.  Font-locking code must
be fast when starting at any arbitrary part of the code, and it must
tolerate incomplete information and errors.