bug#67262: python-ts-mode cannot identify triple-quoted-strings

[Yuan Fu <casouri@gmail.com>]

On 11/26/23 4:05 PM, Dmitry Gutov wrote:
> On 27/11/2023 01:43, Yuan Fu wrote:
>>
>> On 11/26/23 6:58 AM, Dmitry Gutov wrote:
>>> On 26/11/2023 04:04, Dmitry Gutov wrote:
>>>> As for what to do about this one -- probably something involving 
>>>> syntax-propertize-extend-region-functions, adding an entry which 
>>>> would initialize the parser, but not call syntax-ppss-flush-cache 
>>>> directly (or at least not just that). It would signal the earlier 
>>>> position to extend to through some dynamic variable. This is 
>>>> getting tricky enough to move from the individual major modes into 
>>>> treesit.el proper, I think.
>>>
>>> Alternatively, we'd trigger updates eagerly from within 
>>> treesit_record_change -- that would make it slower, invalidating the 
>>> comment above it. Not sure by how much, though.
>>
>> It seems to me that what we need is to force a re-parse at the 
>> beginning of syntax-propertize or in syntax-ppss-flush-cache; the 
>> re-parse would cause the notifier to run, which runs 
>> python--treesit-parser-after-change.
>
> syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
> seems like it will be odd to have execution the other way around 
> and/or add some hook into it which would call the re-parse and extend 
> the region to be invalidated.
>
> syntax-propertize could have another hook added, yes. Or an advice.
>
> But it seems better to reuse some of the existing hooks, such as 
> syntax-propertize-extend-region-functions. It treesit.c provided a way 
> to fetch the newly-invalidated region, the treesit-major-mode-setup 
> could add a new function to syntax-propertize-extend-region-functions 
> which would invoke that feature. But even now it can instantiate the 
> parse, which would call treesit-force-reparse internally, and then 
> collect the info from the callbacks.
syntax-propertize-extend-region-functions looks perfect. We just need to 
force a reparse in it and the notifier will do the rest.
>
> And yet another way - is to extend the region to be propertized from 
> inside the major mode's syntax-propertize-function, invalidating some 
> earlier entries too. The main problem with that, I think, is that 
> every ts mode will have to repeat that trick. And that authors would 
> have to know to do that. How to make that easier and more obvious, is 
> a question.
>
> Finally, if I'm right that bug#66732 has a similar cause, then a 
> shared solution that can be reused by syntax and font-lock (or 
> preferably just fix both in the same place) would be ideal.
>
>> I'm not quite sure about how do we cause this re-parse. The 
>> straightforward approach would be calling treesit-force-reparse[1] in 
>> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to 
>> keep tree-sitter transparent for syntax.el. Maybe we can add a hook 
>> in syntax-propertize/syntax-ppss-flush-cache.
>>
>> [1] This function doesn't exist yet, but it's easy to define in lisp.
>
> treesit-parser-root-node calls it anyway and does little else, so we 
> could get by with just using it.
>
Yep.

Yuan