Fontification using a syntax tree

[Hugo Thunnissen <devel@hugot.nl>]

Hi all,

In the past weeks I've been improving phpinspect.el (my php
parser/completion package) to a point where the completion is functional
in most general cases for OOP code.

At the moment the parsing is done fairly "dumb" in the sense that the
entire buffer is parsed until the current point every time an eldoc
string or a completion needs to be provided. This is not a problem in
100-1000 line files, but once you're editing the last function in a 2000
line PHP class, you're bound to get a little annoyed by the
hiccups. For reference, parsing a 2000 line class takes about 0.3s on
my ryzen 5 3600, while a 400 line class takes only 0.06s.

To optimize this process, I am going to store my syntax tree and the
point -start and -end positions for its tokens in between parser
invocations. That will allow me to invalidate my syntax tree starting
from the token that is enclosing the start point of the edited region,
and "refresh" the invalidated part of the tree by parsing from that
point onwards.

Now, since I am going to store start and end positions of tokens, I was
thinking that from a performance standpoint it might be beneficial to
also use this information to provide fontification. My question to you
is, how should I go about doing this? From what I understand, font-lock
works with syntax tables, but if I use a syntax table for font-lock, I'm
letting font-lock take care of the parsing while I have a perfectly fine
syntax tree ready to use, right? Theoretically, with my stored tokens,
fontification would be as simple as: (pseudocode)

    ;; buffer-local alist with token objects as key.
    (setq phpinspect--token-positions `((,token . (start . end)) ...))

    (dolist (token-cons phpinspect--token-positions)
        (let ((token (car token-cons))
              (point-start  (cadr token-cons))
              (point-end (cddr token-cons)))
           (when (is-an-eligible-token-for-fontification-p token)
             (put-text-property point-start point-end
                                '(whatever property for this token)))))


Before I look deeper into this, is there a way to make this work with
font-lock? Or would I have to implement my own fontification mode? At
that point, is it even a thing that I should want to be doing? Is it a
bad idea / bad practice to not use font-lock?  Any advice or thoughts
are most welcome.

-Hugo