Fontification using a syntax tree

Fontification using a syntax tree

[Hugo Thunnissen]

Hi all,

In the past weeks I've been improving phpinspect.el (my php
parser/completion package) to a point where the completion is functional
in most general cases for OOP code.

At the moment the parsing is done fairly "dumb" in the sense that the
entire buffer is parsed until the current point every time an eldoc
string or a completion needs to be provided. This is not a problem in
100-1000 line files, but once you're editing the last function in a 2000
line PHP class, you're bound to get a little annoyed by the
hiccups. For reference, parsing a 2000 line class takes about 0.3s on
my ryzen 5 3600, while a 400 line class takes only 0.06s.

To optimize this process, I am going to store my syntax tree and the
point -start and -end positions for its tokens in between parser
invocations. That will allow me to invalidate my syntax tree starting
from the token that is enclosing the start point of the edited region,
and "refresh" the invalidated part of the tree by parsing from that
point onwards.

Now, since I am going to store start and end positions of tokens, I was
thinking that from a performance standpoint it might be beneficial to
also use this information to provide fontification. My question to you
is, how should I go about doing this? From what I understand, font-lock
works with syntax tables, but if I use a syntax table for font-lock, I'm
letting font-lock take care of the parsing while I have a perfectly fine
syntax tree ready to use, right? Theoretically, with my stored tokens,
fontification would be as simple as: (pseudocode)

    ;; buffer-local alist with token objects as key.
    (setq phpinspect--token-positions `((,token . (start . end)) ...))

    (dolist (token-cons phpinspect--token-positions)
        (let ((token (car token-cons))
              (point-start  (cadr token-cons))
              (point-end (cddr token-cons)))
           (when (is-an-eligible-token-for-fontification-p token)
             (put-text-property point-start point-end
                                '(whatever property for this token)))))


Before I look deeper into this, is there a way to make this work with
font-lock? Or would I have to implement my own fontification mode? At
that point, is it even a thing that I should want to be doing? Is it a
bad idea / bad practice to not use font-lock?  Any advice or thoughts
are most welcome.

-Hugo

Re: Fontification using a syntax tree

[Yuan Fu]

> 
> Before I look deeper into this, is there a way to make this work with
> font-lock? Or would I have to implement my own fontification mode? At
> that point, is it even a thing that I should want to be doing? Is it a
> bad idea / bad practice to not use font-lock?  Any advice or thoughts
> are most welcome.

You can bind font-lock-fontify-region-function and friends to your custom functions.

Yuan

Re: Fontification using a syntax tree

[Hugo Thunnissen]

Yuan Fu <casouri@gmail.com> writes:

> You can bind font-lock-fontify-region-function and friends to your custom functions.
>
> Yuan

Ah, I see. I didn't realise that this would override syntax-table
related behavior in all scenarios. Thanks!

Re: Fontification using a syntax tree

[Clément Pit-Claudel]

On 9/18/21 12:34 PM, Hugo Thunnissen wrote:
> Yuan Fu <casouri@gmail.com> writes:
> 
>> You can bind font-lock-fontify-region-function and friends to your custom functions.
>>
>> Yuan
> 
> Ah, I see. I didn't realise that this would override syntax-table
> related behavior in all scenarios. Thanks!

Check out js2-mode, too: it does that for Javascript, so reading through the sources should be instructive.  http://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html

Re: Fontification using a syntax tree

[Stefan Monnier]

> is, how should I go about doing this? From what I understand, font-lock
> works with syntax tables, but if I use a syntax table for font-lock, I'm

font-lock has various parts.  One part uses the syntax table to
highlight comments and strings.  This can be disabled by setting
`font-lock-keywords-only`.

Another part is the one that obeys `font-lock-keywords` and these
happily accept rules that consist of delegating the work to a function.

IOW you can just say

    (defconst foo-font-lock-keywords
      `((,#foo--fontify-upto)))

    (defun foo--fontify-upto (limit)
      ... Fontify between point and LIMIT ...
      ;; Return nil to tell font-lock that you're done.
      nil)

    (define-derived-mode ...
      ...
      (setq-local font-lock-defaults '(foo-font-lock-keywords t))
      ...)


-- Stefn