bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

[Yuan Fu <casouri@gmail.com>]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Yuan? Just making sure you got this message.

Sorry for the delay :-)

> On 10/01/2023 16:10, Dmitry Gutov wrote:
>> Perhaps Yuan has some further ideas. There are some strong oddities here:
>> - Some time into debugging and repeating the benchmark again and
>> again, I get the "Pure Lisp storage overflowed" message. Just once
>> per Emacs session. It doesn't seem to change much, so it might be
>> unimportant.

That sounds like 60653. The next time you encounter it, could you record
the output of M-x memory-usage and M-x memory-report? 

>> - The profiler output looks like this:
>>    18050  75%                    -
>> font-lock-fontify-syntactically-region
>>    15686  65%                     - treesit-font-lock-fontify-region
>>     3738  15% treesit--children-covering-range-recurse
>>      188   0%                        treesit-fontify-with-override
>> - When running the benchmark for the first time in a buffer (such as
>> ruby.rb), the variable treesit--font-lock-fast-mode is usually
>> changed to t. In one Emacs session, after I changed it to nil and
>> re-ran the benchmark, the variable stayed nil, and the benchmark ran
>> much faster (like 10s vs 36s).
>> In the next session, after I restarted Emacs, that didn't happen: it
>> always stayed at t, even if I reset it to nil between runs. But if I
>> comment out the block in treesit-font-lock-fontify-region that uses
>> it
>>      ;; (when treesit--font-lock-fast-mode
>>      ;;   (setq nodes (treesit--children-covering-range-recurse
>>      ;;                (car nodes) start end (* 4 jit-lock-chunk-size))))
>> and evaluate the defun, the benchmark runs much faster again: 11s.
>> (But then I brought it all back, and re-ran the tests, and the
>> variable stayed nil that time around; to sum up: the way it's turned
>> on is unstable.)
>> Should treesit--font-lock-fast-mode be locally bound inside that
>> function, so that it's reset between chunks? Or maybe the condition
>> for its enabling should be tweaked? E.g. I don't think there are any
>> particularly large or deep nodes in ruby.rb's parse tree. It's a
>> very shallow file.

Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C
function that checks the maximum depth of a parse tree and the maximum
node span, and turn on the fast-mode if the depth is too large or a node
is too wide. And we do that check once before doing any fontification.

I’ll report back once I add it.

Yuan