Re: Initial fontification in sh-mode with tree-sittter

[Yuan Fu <casouri@gmail.com>]

> On Nov 2, 2022, at 12:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
>> Date: Wed, 2 Nov 2022 12:34:56 -0600
>> 
>> Yes, the fontification in the buffer is still not updated correctly.
>> 
>> With the step below, the definition of function foo is not re-fontified and remains with string-face until C-x x f is
>> executed.
>> 
>> emacs -Q from top of the feature/tree-sitter branch
>> M-: (require 'treesit)
>> M-x customize-variable treesit-settings RET
>> Set "Activate" to "Yes" and apply the change.
>> C-x b sample.py RET
>> M-x python-mode
>> Write the following program:
>> 
>> def main():
>>    return 0
>> 
>> M-<
>> """
>> M->
>> """ (at this point everything is in string-face, as expected)
>> C-a
>> C-k (everything is still fontified as string)
>> C-x x f (leading """ is not fontified and definition of foo is correctly fontified)
>> 
>> I am interested in understanding what is causing this as a similar thing happens with heredocs in sh-mode &
>> tree-sitter.
> 
> Yuan, can you look into this?  It sounds like some general problem
> with integration of tree-sitter with jit-lock.  

Yeah, this is what I’ve been working in the past two days. I just pushed a change f331be1f074d68e7e5cdbac324419e07c186492a which should fix it. [ I just saw that I missed a function in the commit message, I guess I can’t fix it now :-( ]

If you are interested to know what’s going on:

The problem is as I described earlier, when the user inserts the starting quote (“””), the parse tree is incomplete and there is no string node. Only when the user inserts the ending quote is there now a string node to be captured by the fontification rules. 

For example, in this snippet there are two regions A and B

"""          ---+
def main():     | Region A
    return 0 ---+
             ---+ 
"""             | Region B
             ---+

when user inserts the “”” in B, and jit-lock fontifies region B, it captures the string node and the part of the string in region A needs to be updated. If we fontify the whole string in string face, redisplay does not immediately reflect the change in region A (maybe due to some optimization? jit-lock is definitely aware of this, see jit-lock-force-redisplay).

Redisplay not reflecting the change in face is just the surface problem, and can be fixed by setting fontified to t on region A, which seems to trigger redisplay to reflect changes immediately (this is what jit-lock-force-redisplay does). The deeper problem is, if there is some regex-based-font-lock face in region A (applied when Emacs fontified region A), eg, highlighted TODO keywords in a docstring, they will be overwritten by the string face, if we just apply string face to the whole string and trigger redisplay.

Maybe we can apply string face and re-run regex-based font-lock on the whole string, but that works against jit-lock. If the string is long regexp-font-lock might take a long time.

What I ended up doing is to set jit-lock-context-unfontify-pos to the beginning of the string node (aka beginning of region A). Then in a timer jit-lock-context will refontify everything after that position. And I have some measure to break possible infinite recursion (fontify region -> set jit-lock-context-unfontify-pos -> cause refontification -> fontify region -> …).

The alternative, where we put everything in string face when we see an opening “””, is not really feasible. It’s kind of feasible in python, where an opening “”” alone looks like (ERROR “””), aka the quote node inside an error node. But if you insert /* in javascript, you get (ERROR “/“) and (ERROR (regexp_pattern))—tree-sitter can’t tell that they are opening comment (which is fair). Plus this approach requires non-trivial involvement from each major mode: each major mode needs to somehow tell Emacs what is a “potential opening comment/string”.

> Or maybe something is
> missing from the way tree-sitter nodes are mapped into face text
> properties.  Are the faces actually being put on the relevant text,
> for starters?

They are. Fortunately this is not the problem.

Yuan