bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov <dgutov@yandex.ru>]

On 26/03/2023 13:01, Eli Zaretskii wrote:
>> Date: Sun, 26 Mar 2023 12:25:30 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 26/03/2023 08:04, Eli Zaretskii wrote:
>>>> Date: Sun, 26 Mar 2023 00:57:22 +0200
>>>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>>
>>>>> How does that work with features such as font-lock, which do widen?
>>>>
>>>> Using font-lock-dont-widen.
>>>
>>> That's only for font-lock.  Parsing was not on the table when that was
>>> introduced, so it doesn't have a similar mechanism.
>>
>> Parsing is on-demand, by font-lock and other features.
> 
> So you are suggesting to introduce kludges like font-lock-dont-widen
> in all of those places?

font-lock-dont-widen is a kludge, but that's largely determined by the 
way it's defined and used.

If we take indent-for-tab-command, for example, it doesn't have such a 
variable, and doesn't really need to: the top-level command calls 
'widen', and then indent-line-function (set by major mode and altered by 
e.g. mmm-mode) is free to impose its specific bounds.

I think, as far the immediate problem is concerned, blink-matching-paren 
could use the same route. And could even try to reuse 
show-paren-data-function instead of creating its own customization point.

> I don't even see how we will find them all in
> advance, let alone fix them or make sure they do what we want.

Just do it step-by-step.

The "grand unified theory of mixed major modes" has been attempted a few 
times in the past, and never reached anything practical. That's not to 
say nobody should try again, but they should keep that in mind.

>>> Again, I'm talking about using a parser library.  We may need to
>>> introduce a way of limiting the parser to a certain range of buffer
>>> text positions, independently of narrowing.
>>
>> Except it's already limited by narrowing.
> 
> Which is a fragile, semi-broken means, as we all know.

What is a broken mess, is user-level narrowing. And how the downstream 
code can never guess the purpose the narrowing was applied for.

>>> As we all know, narrowing
>>> is a problematic feature to use in Lisp programs, so maybe we should
>>> do this better in the case of parsers.  Then problems like this one
>>> could be solved more cleanly and simply.
>>
>> Narrowing problematic to use in Lisp?
> 
> Yes, because users can easily change narrowing.  We've had problems
> with that many times, and even some attempts at solving it, so why are
> you pretending you don't know about those deficiencies?

Because I suggested deprecating certain uses of it several times, and 
always got a "no" for an answer.

>>>> And anyway, I like I mentioned, this will break this common pattern as well:
>>>>
>>>>      (save-restriction
>>>>        (narrow-to-region ... some-limit-position)
>>>>        (forward-sexp))
>>>>
>>>> I've used it in ruby-syntax-propertize-percent-literal, for example.
>>>> Except with 'forward-list' rather than 'forward-sexp', but others can
>>>> use the latter.
>>>
>>> You want to repeat all the arguments we already brought up?
>>
>> You might choose to ignore a third-party mode, but breaking a common
>> pattern seems more dangerous.
> 
> ??? How does that follow from what I said?
> 
> Look, I'm trying to see how we could come up with an infrastructure
> that will support multiple modes and other similar features in the
> same buffer without relying on narrowing, thus bypassing the
> disadvantages and difficulties that come with narrowing.  I think we
> have a good chance here to come up with such a solution, specifically
> for features that us a parsing library.  If you aren't interested in
> discussing that, and think we should stick to narrowing, then this
> goes nowhere, and I'd rather bow out of it.

What I've seen here so far is you suggesting we go ahead and break the 
existing convention and then let "them" (third-party authors including 
myself) come up with a new working one.

My stance here is we shouldn't break it before we create a new one. And 
I'm happy to discuss it, of course. Just please don't expect me to take 
the initiative at this point.