bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

[Yuan Fu <casouri@gmail.com>]

Yuan Fu <casouri@gmail.com> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Yuan? Just making sure you got this message.
>
> Sorry for the delay :-)
>
>> On 10/01/2023 16:10, Dmitry Gutov wrote:
>>> Perhaps Yuan has some further ideas. There are some strong oddities here:
>>> - Some time into debugging and repeating the benchmark again and
>>> again, I get the "Pure Lisp storage overflowed" message. Just once
>>> per Emacs session. It doesn't seem to change much, so it might be
>>> unimportant.
>
> That sounds like 60653. The next time you encounter it, could you record
> the output of M-x memory-usage and M-x memory-report? 
>
>>> - The profiler output looks like this:
>>>    18050  75%                    -
>>> font-lock-fontify-syntactically-region
>>>    15686  65%                     - treesit-font-lock-fontify-region
>>>     3738  15% treesit--children-covering-range-recurse
>>>      188   0%                        treesit-fontify-with-override
>>> - When running the benchmark for the first time in a buffer (such as
>>> ruby.rb), the variable treesit--font-lock-fast-mode is usually
>>> changed to t. In one Emacs session, after I changed it to nil and
>>> re-ran the benchmark, the variable stayed nil, and the benchmark ran
>>> much faster (like 10s vs 36s).
>>> In the next session, after I restarted Emacs, that didn't happen: it
>>> always stayed at t, even if I reset it to nil between runs. But if I
>>> comment out the block in treesit-font-lock-fontify-region that uses
>>> it
>>>      ;; (when treesit--font-lock-fast-mode
>>>      ;;   (setq nodes (treesit--children-covering-range-recurse
>>>      ;;                (car nodes) start end (* 4 jit-lock-chunk-size))))
>>> and evaluate the defun, the benchmark runs much faster again: 11s.
>>> (But then I brought it all back, and re-ran the tests, and the
>>> variable stayed nil that time around; to sum up: the way it's turned
>>> on is unstable.)
>>> Should treesit--font-lock-fast-mode be locally bound inside that
>>> function, so that it's reset between chunks? Or maybe the condition
>>> for its enabling should be tweaked? E.g. I don't think there are any
>>> particularly large or deep nodes in ruby.rb's parse tree. It's a
>>> very shallow file.
>
> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C
> function that checks the maximum depth of a parse tree and the maximum
> node span, and turn on the fast-mode if the depth is too large or a node
> is too wide. And we do that check once before doing any fontification.
>
> I’ll report back once I add it.

I wrote that function. But I didn’t end up using it. Instead I added a
"grace count", so that the query time has to be longer than the
threshold 5 times before we switch on the fast mode instead of 1.

My main worry is that simply looking at the parse tree would not catch
all the case where there will be expensive queries.

Could you try the latest commit and see if the fast mode still switches
on when it shouldn’t?

Yuan