bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

[Dmitry Gutov <dgutov@yandex.ru>]

On 12/01/2023 23:58, Yuan Fu wrote:
> 
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Yuan? Just making sure you got this message.
> 
> Sorry for the delay :-)
> 
>> On 10/01/2023 16:10, Dmitry Gutov wrote:
>>> Perhaps Yuan has some further ideas. There are some strong oddities here:
>>> - Some time into debugging and repeating the benchmark again and
>>> again, I get the "Pure Lisp storage overflowed" message. Just once
>>> per Emacs session. It doesn't seem to change much, so it might be
>>> unimportant.
> 
> That sounds like 60653. The next time you encounter it, could you record
> the output of M-x memory-usage and M-x memory-report?

Managed to reproduce this after running the test in a couple of 
different files.

But 'M-x memory-usage' says no such command, and 'M-x memory-report' 
ends up with this error:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
   memory-report--gc-elem(nil strings)
   memory-report--garbage-collect()
   memory-report()
   funcall-interactively(memory-report)
   #<subr call-interactively>(memory-report record nil)
   apply(#<subr call-interactively> memory-report (record nil))
   call-interactively@ido-cr+-record-current-command(#<subr 
call-interactively> memory-report record nil)
   apply(call-interactively@ido-cr+-record-current-command #<subr 
call-interactively> (memory-report record nil))
   call-interactively(memory-report record nil)
   command-execute(memory-report record)
   execute-extended-command(nil "memory-report" nil)
   funcall-interactively(execute-extended-command nil "memory-report" nil)
   #<subr call-interactively>(execute-extended-command nil nil)
   apply(#<subr call-interactively> execute-extended-command (nil nil))
   call-interactively@ido-cr+-record-current-command(#<subr 
call-interactively> execute-extended-command nil nil)
   apply(call-interactively@ido-cr+-record-current-command #<subr 
call-interactively> (execute-extended-command nil nil))
   call-interactively(execute-extended-command nil nil)
   command-execute(execute-extended-command)

garbage-collect's docstring says:

   However, if there was overflow in pure space, and Emacs was dumped
   using the "unexec" method, ‘garbage-collect’ returns nil, because
   real GC can’t be done.

I don't know if my Emacs was dumped using "unexec", though. ./configure 
says I'm using pdumper.

In case that matters, I'm testing the emacs-29 branch.

>>> - The profiler output looks like this:
>>>     18050  75%                    -
>>> font-lock-fontify-syntactically-region
>>>     15686  65%                     - treesit-font-lock-fontify-region
>>>      3738  15% treesit--children-covering-range-recurse
>>>       188   0%                        treesit-fontify-with-override
>>> - When running the benchmark for the first time in a buffer (such as
>>> ruby.rb), the variable treesit--font-lock-fast-mode is usually
>>> changed to t. In one Emacs session, after I changed it to nil and
>>> re-ran the benchmark, the variable stayed nil, and the benchmark ran
>>> much faster (like 10s vs 36s).
>>> In the next session, after I restarted Emacs, that didn't happen: it
>>> always stayed at t, even if I reset it to nil between runs. But if I
>>> comment out the block in treesit-font-lock-fontify-region that uses
>>> it
>>>       ;; (when treesit--font-lock-fast-mode
>>>       ;;   (setq nodes (treesit--children-covering-range-recurse
>>>       ;;                (car nodes) start end (* 4 jit-lock-chunk-size))))
>>> and evaluate the defun, the benchmark runs much faster again: 11s.
>>> (But then I brought it all back, and re-ran the tests, and the
>>> variable stayed nil that time around; to sum up: the way it's turned
>>> on is unstable.)
>>> Should treesit--font-lock-fast-mode be locally bound inside that
>>> function, so that it's reset between chunks? Or maybe the condition
>>> for its enabling should be tweaked? E.g. I don't think there are any
>>> particularly large or deep nodes in ruby.rb's parse tree. It's a
>>> very shallow file.
> 
> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C
> function that checks the maximum depth of a parse tree and the maximum
> node span, and turn on the fast-mode if the depth is too large or a node
> is too wide. And we do that check once before doing any fontification.
> 
> I’ll report back once I add it.

Thanks!

And if the check can be fast enough, we could probably do it in the 
beginning of fontifying every chunk.