Re: treesitter local parser: huge slowdown and memory usage in a long file

[Yuan Fu <casouri@gmail.com>]

> On May 5, 2024, at 7:04 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> Hi Yuan,
> 
> Sorry if I'm being too pedantic here.
> 
> On 20/04/2024 05:18, Yuan Fu wrote:
> 
>> I believe I’ve found a good way to solve this problem. I pushed the changes to master.
>> Basically I added a function treesit-parser-changed-ranges that can directly return the change ranges from last reparse. This means we don’t need to use notifiers to get those change ranges anymore. Then in treesit-pre-redisplay, we reparse the primary parser and get the changed ranges from it.
>> Once we have the changed ranges, we update other non-primary parser’s ranges, but only within the changed ranges. Originally we were updating those parser’s ranges on the whole buffer, which led to the slowdown. Then we had to use some workaround to solve this. Now the workaround isn’t needed anymore.
> 
> The essence of the change (querying fewer ranges) looks good.
> 
> I'm a bit uneasy about the new function and how it's supposed to be used. treesit-parser-changed-ranges returns the ranges changes during the last reparse. That seems to imply that all of its callers must have the up-to-date information about the state of the buffer before that reparse, and thus basically follow the parser's updates through some mechanism. The implementation also saves some information during every reparse, whether somebody is going to call treesit-parser-changed-ranges or not.
> 
> To take our new code as an example, the only client of treesit-parser-changed-ranges now is treesit--pre-redisplay, which is called from syntax-propertize-extend-region-functions and pre-redisplay-functions.
> 
> Is it possible that there would occur multiple changes and reparses between some firings of the above hooks? For example, some new feature might go over the buffer's text with an automated multi-step transformation, calling the parser (but not syntax-ppss) on each step.
> In such a scenario it seems treesit--pre-redisplay might miss intermediate range updates. Would that be okay?

I think you’re right. The chance of it actually go wrong will be slim, but anything that’s possible to go wrong will eventually go wrong. 

The remaining question is how. I’m thinking of keeping a history of updated ranges, each marked with the parser timestamp. The parser timestamp is already there, it’s incremented every time the parser reparses. And treesit-parser-changed-ranges will return the timestamp along with the updated ranges. Then in the next iteration, the consumer can pass the last timestamp to treesit-parser-changed-ranges, which tells it to return all the changed ranges since that timestamp.

The only problem is to decide how long a history of updated ranges do we keep for each parser. The 100% correct approach is to maintain a separate history for each consumer, and never throw away old ranges until the consumer consumes them. But then you risk wasting memory if some consumer never consumes the ranges. To handle that we can add a hard limit. But then this hard limit might be too low for some edge case… We can make this hard limit configurable, and if we ever encountered a case where this hard limit is not enough and there’s no way around it (unlikely), we can instruct users or lisp program to increase it.

Yuan