treesit indentation "blinking"

treesit indentation "blinking"

[Daniel Colascione]

Is there a general-purpose through which we can avoid line indentation
oscillating as the user types when the AST is temporarily invalid,
e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
one of the more disconcerting things is the current line's indentation
changing rapidly as I type. Is it feasible to create ERROR recovery
indentation rules for every conceivable situation?

Re: treesit indentation "blinking"

[Yuan Fu]

> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
> 
> Is there a general-purpose through which we can avoid line indentation
> oscillating as the user types when the AST is temporarily invalid,
> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
> one of the more disconcerting things is the current line's indentation
> changing rapidly as I type. Is it feasible to create ERROR recovery
> indentation rules for every conceivable situation?
> 

Yes, but in reality, I think all we need is a couple special case for the unmatched ( and {’s. Can you think of other cases of blinking indentations?

Yuan

Re: treesit indentation "blinking"

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]



On March 22, 2023 20:00:23 Yuan Fu <casouri@gmail.com> wrote:

>> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>
>> Is there a general-purpose through which we can avoid line indentation
>> oscillating as the user types when the AST is temporarily invalid,
>> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
>> one of the more disconcerting things is the current line's indentation
>> changing rapidly as I type. Is it feasible to create ERROR recovery
>> indentation rules for every conceivable situation?
>
> Yes, but in reality, I think all we need is a couple special case for the 
> unmatched ( and {’s. Can you think of other cases of blinking indentations?
>
> Yuan

But TS reacts to missing closing brackets by clarifying the whole nearby 
expression as ERROR. It's not, as would be more useful, saying "here's a 
stray (, and everything else is normal and parsed as if that ( were absent"


[-- Attachment #2: Type: text/html, Size: 1918 bytes --]

Re: treesit indentation "blinking"

[Yuan Fu]

> On Mar 22, 2023, at 5:07 PM, Daniel Colascione <dancol@dancol.org> wrote:
> 
> 
> 
> On March 22, 2023 20:00:23 Yuan Fu <casouri@gmail.com> wrote:
> 
>>> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>> 
>>> Is there a general-purpose through which we can avoid line indentation
>>> oscillating as the user types when the AST is temporarily invalid,
>>> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
>>> one of the more disconcerting things is the current line's indentation
>>> changing rapidly as I type. Is it feasible to create ERROR recovery
>>> indentation rules for every conceivable situation?
>>> 
>> 
>> Yes, but in reality, I think all we need is a couple special case for the unmatched ( and {’s. Can you think of other cases of blinking indentations?
>> 
>> Yuan
> 
> But TS reacts to missing closing brackets by clarifying the whole nearby expression as ERROR. It's not, as would be more useful, saying "here's a stray (, and everything else is normal and parsed as if that ( were absent”

We can just look at the buffer text directly: if there’s an ERROR and the previous char (after skipping whitespace chars) is ( or {, we know what to do.

Yuan

Re: treesit indentation "blinking"

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]

On March 22, 2023 21:03:29 Yuan Fu <casouri@gmail.com> wrote:

>> On Mar 22, 2023, at 5:07 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>
>>
>>
>> On March 22, 2023 20:00:23 Yuan Fu <casouri@gmail.com> wrote:
>>
>>>> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>>>
>>>> Is there a general-purpose through which we can avoid line indentation
>>>> oscillating as the user types when the AST is temporarily invalid,
>>>> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
>>>> one of the more disconcerting things is the current line's indentation
>>>> changing rapidly as I type. Is it feasible to create ERROR recovery
>>>> indentation rules for every conceivable situation?
>>>
>>> Yes, but in reality, I think all we need is a couple special case for the 
>>> unmatched ( and {’s. Can you think of other cases of blinking indentations?
>>>
>>> Yuan
>>
>> But TS reacts to missing closing brackets by clarifying the whole nearby 
>> expression as ERROR. It's not, as would be more useful, saying "here's a 
>> stray (, and everything else is normal and parsed as if that ( were absent”
>
> We can just look at the buffer text directly: if there’s an ERROR and the 
> previous char (after skipping whitespace chars) is ( or {, we know what to do

Do we know what to do? That ERROR might be arbitrarily far up the parse 
tree. I don't think it's as easy as you think it might be. One strategy 
that might work is to see whether adding a "(" introduced an error, and, if 
so, temporarily replacing that "(" with whitespace, reparsing, and then 
using the resulting parse tree instead of the one with the "(" to do 
indentation and fontification. This way, I think you'd end up without the 
random jumping around we see today in TS modes.

[-- Attachment #2: Type: text/html, Size: 3207 bytes --]

Re: treesit indentation "blinking"

[Yuan Fu]

> On Mar 22, 2023, at 9:51 PM, Daniel Colascione <dancol@dancol.org> wrote:
> 
> On March 22, 2023 21:03:29 Yuan Fu <casouri@gmail.com> wrote:
> 
>>> On Mar 22, 2023, at 5:07 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>> 
>>> 
>>> 
>>> On March 22, 2023 20:00:23 Yuan Fu <casouri@gmail.com> wrote:
>>> 
>>>>> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>>>> 
>>>>> Is there a general-purpose through which we can avoid line indentation
>>>>> oscillating as the user types when the AST is temporarily invalid,
>>>>> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
>>>>> one of the more disconcerting things is the current line's indentation
>>>>> changing rapidly as I type. Is it feasible to create ERROR recovery
>>>>> indentation rules for every conceivable situation?
>>>>> 
>>>> 
>>>> Yes, but in reality, I think all we need is a couple special case for the unmatched ( and {’s. Can you think of other cases of blinking indentations?
>>>> 
>>>> Yuan
>>> 
>>> But TS reacts to missing closing brackets by clarifying the whole nearby expression as ERROR. It's not, as would be more useful, saying "here's a stray (, and everything else is normal and parsed as if that ( were absent”
>> 
>> We can just look at the buffer text directly: if there’s an ERROR and the previous char (after skipping whitespace chars) is ( or {, we know what to do
> 
> 
> Do we know what to do? That ERROR might be arbitrarily far up the parse tree. I don't think it's as easy as you think it might be. One strategy that might work is to see whether adding a "(" introduced an error, and, if so, temporarily replacing that "(" with whitespace, reparsing, and then using the resulting parse tree instead of the one with the "(" to do indentation and fontification. This way, I think you'd end up without the random jumping around we see today in TS modes.

We can place this special rule at the end of our rule list, and previous rules not matching should indicated “error” by itself. Of course, I can’t prove it by using this method to fix the blinking indent, but I don’t quite have the time for it right now.

Yuan

Re: treesit indentation "blinking"

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]



On March 23, 2023 16:04:16 Yuan Fu <casouri@gmail.com> wrote:

>> On Mar 22, 2023, at 9:51 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>
>> On March 22, 2023 21:03:29 Yuan Fu <casouri@gmail.com> wrote:
>>
>>>> On Mar 22, 2023, at 5:07 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>>>
>>>>
>>>>
>>>> On March 22, 2023 20:00:23 Yuan Fu <casouri@gmail.com> wrote:
>>>>
>>>>>> On Mar 22, 2023, at 1:49 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>>>>>
>>>>>> Is there a general-purpose through which we can avoid line indentation
>>>>>> oscillating as the user types when the AST is temporarily invalid,
>>>>>> e.g. after '(' or '{'? I'm checking out the C++ tree-sitter mode, and
>>>>>> one of the more disconcerting things is the current line's indentation
>>>>>> changing rapidly as I type. Is it feasible to create ERROR recovery
>>>>>> indentation rules for every conceivable situation?
>>>>>
>>>>> Yes, but in reality, I think all we need is a couple special case for the 
>>>>> unmatched ( and {’s. Can you think of other cases of blinking indentations?
>>>>>
>>>>> Yuan
>>>>
>>>> But TS reacts to missing closing brackets by clarifying the whole nearby 
>>>> expression as ERROR. It's not, as would be more useful, saying "here's a 
>>>> stray (, and everything else is normal and parsed as if that ( were absent”
>>>
>>> We can just look at the buffer text directly: if there’s an ERROR and the 
>>> previous char (after skipping whitespace chars) is ( or {, we know what to do
>>
>>
>> Do we know what to do? That ERROR might be arbitrarily far up the parse 
>> tree. I don't think it's as easy as you think it might be. One strategy 
>> that might work is to see whether adding a "(" introduced an error, and, if 
>> so, temporarily replacing that "(" with whitespace, reparsing, and then 
>> using the resulting parse tree instead of the one with the "(" to do 
>> indentation and fontification. This way, I think you'd end up without the 
>> random jumping around we see today in TS modes.
>
> We can place this special rule at the end of our rule list, and previous 
> rules not matching should indicated “error” by itself. Of course, I can’t 
> prove it by using this method to fix the blinking indent, but I don’t quite 
> have the time for it right now.


Editing can cause all sorts of transient nonsense in the AST, and it's 
impossible to predict in a general manner what this nonsense might be. The 
wrong kind of bracket can cause the entire rest of the file to be parsed as 
nonsense. This or that error recovery rule isn't going to solve the 
problem: such a strategy is a fragile whack a mole. There needs to be some 
general solution to prevent indentation blinking. This blinking makes TS 
modules unusable for me.


[-- Attachment #2: Type: text/html, Size: 4673 bytes --]

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 23/03/2023 23:10, Daniel Colascione wrote:
> Editing can cause all sorts of transient nonsense in the AST, and it's 
> impossible to predict in a general manner what this nonsense might be. 
> The wrong kind of bracket can cause the entire rest of the file to be 
> parsed as nonsense. This or that error recovery rule isn't going to 
> solve the problem: such a strategy is a fragile whack a mole. There 
> needs to be some general solution to prevent indentation blinking. This 
> blinking makes TS modules unusable for me.

It seems to me it will only be feasible to support a number of 
"incomplete" syntax constructs. Maybe it will constitute a majority of them.

Have you tried using electric-pair-mode, though? I'd hate to make it our 
strong recommendation for all ts modes, but it does reduce the odds of 
incomplete code a lot.

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> CC: <emacs-devel@gnu.org>
> Date: Thu, 23 Mar 2023 17:10:53 -0400
> 
>  We can place this special rule at the end of our rule list, and previous rules not matching should
>  indicated “error” by itself. Of course, I can’t prove it by using this method to fix the blinking indent, but I
>  don’t quite have the time for it right now.
> 
> Editing can cause all sorts of transient nonsense in the AST, and it's impossible to predict in a general
> manner what this nonsense might be. The wrong kind of bracket can cause the entire rest of the file to be
> parsed as nonsense. This or that error recovery rule isn't going to solve the problem: such a strategy is a
> fragile whack a mole. There needs to be some general solution to prevent indentation blinking. This blinking
> makes TS modules unusable for me.

Daniel, could you please post a recipe, starting from "emacs -Q", to
reproduce the "blinking" during editing you described up-thread?
Preferably while editing some code that one can meet reasonably
frequently in practice.  I'd like to play with the example and see how
serious the problem is.  TIA.

Re: treesit indentation "blinking"

[João Távora]

On Thu, Mar 23, 2023 at 9:24 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 23/03/2023 23:10, Daniel Colascione wrote:
> > Editing can cause all sorts of transient nonsense in the AST, and it's
> > impossible to predict in a general manner what this nonsense might be.
> > The wrong kind of bracket can cause the entire rest of the file to be
> > parsed as nonsense. This or that error recovery rule isn't going to
> > solve the problem: such a strategy is a fragile whack a mole. There
> > needs to be some general solution to prevent indentation blinking. This
> > blinking makes TS modules unusable for me.
>
> It seems to me it will only be feasible to support a number of
> "incomplete" syntax constructs. Maybe it will constitute a majority of them.

I don't think this problem has anything to do with tree-sitter.  It
would happen just as well in regular c++-mode if electric-indent-mode
were on by default there (but it isn't).

So it's just the fact that electric-indent-mode is on by default
_and_ c-ts-mode.el does this:

(setq-local electric-indent-chars (append "{}():;,#" electric-indent-chars))

so people are getting what the mode author (and Emacs defaults) ask for:
electric indentation doing its thing.

> Have you tried using electric-pair-mode, though? I'd hate to make it our
> strong recommendation for all ts modes, but it does reduce the odds of
> incomplete code a lot.

Maybe you'd hate it, but it would at least be consistent with the
setting for electric-indent-mode, which is much more jarring.

See also bug#62412 which is basically the same problem with a recipe.

João

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 25/03/2023 11:05, João Távora wrote:
> On Thu, Mar 23, 2023 at 9:24 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>> On 23/03/2023 23:10, Daniel Colascione wrote:
>>> Editing can cause all sorts of transient nonsense in the AST, and it's
>>> impossible to predict in a general manner what this nonsense might be.
>>> The wrong kind of bracket can cause the entire rest of the file to be
>>> parsed as nonsense. This or that error recovery rule isn't going to
>>> solve the problem: such a strategy is a fragile whack a mole. There
>>> needs to be some general solution to prevent indentation blinking. This
>>> blinking makes TS modules unusable for me.
>>
>> It seems to me it will only be feasible to support a number of
>> "incomplete" syntax constructs. Maybe it will constitute a majority of them.
> 
> I don't think this problem has anything to do with tree-sitter.  It
> would happen just as well in regular c++-mode if electric-indent-mode
> were on by default there (but it isn't).

The mode is globally on by default, but indeed the contents of 
electric-indent-chars are much shorter there.

> So it's just the fact that electric-indent-mode is on by default
> _and_ c-ts-mode.el does this:
> 
> (setq-local electric-indent-chars (append "{}():;,#" electric-indent-chars))
> 
> so people are getting what the mode author (and Emacs defaults) ask for:
> electric indentation doing its thing.

Good point: perhaps either this list should be revisited, or predicated 
(at runtime) somehow on electric-pair-mode being enabled.

>> Have you tried using electric-pair-mode, though? I'd hate to make it our
>> strong recommendation for all ts modes, but it does reduce the odds of
>> incomplete code a lot.
> 
> Maybe you'd hate it

Not really, just a figure of speech.

But not everybody will like it, I suspect.

, but it would at least be consistent with the
> setting for electric-indent-mode, which is much more jarring.
> 
> See also bug#62412 which is basically the same problem with a recipe.

Yep.

Re: treesit indentation "blinking"

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 14:42:50 +0200
> Cc: Daniel Colascione <dancol@dancol.org>, Yuan Fu <casouri@gmail.com>,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I don't think this problem has anything to do with tree-sitter.  It
> > would happen just as well in regular c++-mode if electric-indent-mode
> > were on by default there (but it isn't).
> 
> The mode is globally on by default, but indeed the contents of 
> electric-indent-chars are much shorter there.
> 
> > So it's just the fact that electric-indent-mode is on by default
> > _and_ c-ts-mode.el does this:
> > 
> > (setq-local electric-indent-chars (append "{}():;,#" electric-indent-chars))
> > 
> > so people are getting what the mode author (and Emacs defaults) ask for:
> > electric indentation doing its thing.
> 
> Good point: perhaps either this list should be revisited, or predicated 
> (at runtime) somehow on electric-pair-mode being enabled.

Maybe the list of electric characters should be a defcustom?

Re: treesit indentation "blinking"

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 25/03/2023 11:05, João Távora wrote:
>> On Thu, Mar 23, 2023 at 9:24 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>>
>>> On 23/03/2023 23:10, Daniel Colascione wrote:
>>>> Editing can cause all sorts of transient nonsense in the AST, and it's
>>>> impossible to predict in a general manner what this nonsense might be.
>>>> The wrong kind of bracket can cause the entire rest of the file to be
>>>> parsed as nonsense. This or that error recovery rule isn't going to
>>>> solve the problem: such a strategy is a fragile whack a mole. There
>>>> needs to be some general solution to prevent indentation blinking. This
>>>> blinking makes TS modules unusable for me.
>>>
>>> It seems to me it will only be feasible to support a number of
>>> "incomplete" syntax constructs. Maybe it will constitute a majority of them.
>> I don't think this problem has anything to do with tree-sitter.  It
>> would happen just as well in regular c++-mode if electric-indent-mode
>> were on by default there (but it isn't).
>
> The mode is globally on by default, but indeed the contents of
> electric-indent-chars are much shorter there.
>
>> So it's just the fact that electric-indent-mode is on by default
>> _and_ c-ts-mode.el does this:
>> (setq-local electric-indent-chars (append "{}():;,#"
>> electric-indent-chars))
>> so people are getting what the mode author (and Emacs defaults) ask
>> for:
>> electric indentation doing its thing.
>
> Good point: perhaps either this list should be revisited, or
> predicated (at runtime) somehow on electric-pair-mode being enabled.

At first, I thought this was a good idea.  electric-pair-chars could be
a function returning the characters to use, and c++-ts-mode could set it
to a function that returns the extended set if electric-pair-mode is on.

But OTOH, extra chars in electric-indent-chars is of limited use if
electric-pair-mode is on, because when the buffer is balanced,
newline-inserting commands already do the correct indent.

So, it would probably solve this problem, but not much beyond not
messing with electric-indent-chars at all in the first place.

João

Re: treesit indentation "blinking"

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 25 Mar 2023 14:42:50 +0200
>> Cc: Daniel Colascione <dancol@dancol.org>, Yuan Fu <casouri@gmail.com>,
>>  emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> > I don't think this problem has anything to do with tree-sitter.  It
>> > would happen just as well in regular c++-mode if electric-indent-mode
>> > were on by default there (but it isn't).
>> 
>> The mode is globally on by default, but indeed the contents of 
>> electric-indent-chars are much shorter there.
>> 
>> > So it's just the fact that electric-indent-mode is on by default
>> > _and_ c-ts-mode.el does this:
>> > 
>> > (setq-local electric-indent-chars (append "{}():;,#" electric-indent-chars))
>> > 
>> > so people are getting what the mode author (and Emacs defaults) ask for:
>> > electric indentation doing its thing.
>> 
>> Good point: perhaps either this list should be revisited, or predicated 
>> (at runtime) somehow on electric-pair-mode being enabled.
>
> Maybe the list of electric characters should be a defcustom?

Maybe, but then major-modes like c++-ts-mode shouldn't mess with it.  Or
do you mean a electric-indent-chars-for-c++-ts-mode-specifically
defcustom.  If so, that's a bit odd.  What if the user then also wants
to set electric-indent-chars herself?

I think it's easier that c++-ts-mode doesn't touch that variable at all
and leave it set to "\n".  I can't see it being much use, with or
without electric-pair-mode in the mix.

João

Re: treesit indentation "blinking"

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> I think it's easier that c++-ts-mode doesn't touch that variable at all
> and leave it set to "\n".  I can't see it being much use, with or
> without electric-pair-mode in the mix.

Following up on this, I have now tried c++-ts-mode for a while now and
can confirm this jumping-around-while-typing is terrible without
electric-pair-mode, and still pretty bad even with electric-pair-mode.

Here's just an example with emacs -Q + electric-pair-mode + c++-ts-mode

Start typing

i n t SPC m a i n ( ) SPC { RET

The buffer becomes

int main() {
  <- point here
}

So far so good.  Point is correctly indented at column 2 (because
electric-pair-mode has balanced the buffer and electric-indent-mode did
its thing on RET).

Type 'std'.  Nothing surpising happens, good.  Now type one ':' and see
the 'std:' be electrically indented to column 1, type the other ':' and
the 'std::' now jumps to column 2 again.

In bug#62412, Theo said is was OK with killing the custom setting of
electric-indent-chars in c-ts-mode.el and leave it set to the default.

Though a number of indenting problems would remain after that, at least
this one clear annoyance would be solved.  So if there are no
objections, I propose to apply this patch.

diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
index 59eb9fc23e6..88360716381 100644
--- a/lisp/progmodes/c-ts-mode.el
+++ b/lisp/progmodes/c-ts-mode.el
@@ -956,10 +956,6 @@ c-ts-base-mode
   ;; Comment
   (c-ts-common-comment-setup)
 
-  ;; Electric
-  (setq-local electric-indent-chars
-              (append "{}():;,#" electric-indent-chars))
-
   ;; Imenu.
   (setq-local treesit-simple-imenu-settings
               (let ((pred #'c-ts-mode--defun-valid-p))

In master?  In emacs-29?

João

Re: treesit indentation "blinking"

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

Thanks. I hadn't gotten around to a recipe yet. How do other editors handle 
this situation?

On March 28, 2023 18:09:11 João Távora <joaotavora@gmail.com> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
>> I think it's easier that c++-ts-mode doesn't touch that variable at all
>> and leave it set to "\n".  I can't see it being much use, with or
>> without electric-pair-mode in the mix.
>
> Following up on this, I have now tried c++-ts-mode for a while now and
> can confirm this jumping-around-while-typing is terrible without
> electric-pair-mode, and still pretty bad even with electric-pair-mode.
>
> Here's just an example with emacs -Q + electric-pair-mode + c++-ts-mode
>
> Start typing
>
> i n t SPC m a i n ( ) SPC { RET
>
> The buffer becomes
>
> int main() {
>  <- point here
> }
>
> So far so good.  Point is correctly indented at column 2 (because
> electric-pair-mode has balanced the buffer and electric-indent-mode did
> its thing on RET).
>
> Type 'std'.  Nothing surpising happens, good.  Now type one ':' and see
> the 'std:' be electrically indented to column 1, type the other ':' and
> the 'std::' now jumps to column 2 again.
>
> In bug#62412, Theo said is was OK with killing the custom setting of
> electric-indent-chars in c-ts-mode.el and leave it set to the default.
>
> Though a number of indenting problems would remain after that, at least
> this one clear annoyance would be solved.  So if there are no
> objections, I propose to apply this patch.
>
> diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
> index 59eb9fc23e6..88360716381 100644
> --- a/lisp/progmodes/c-ts-mode.el
> +++ b/lisp/progmodes/c-ts-mode.el
> @@ -956,10 +956,6 @@ c-ts-base-mode
>   ;; Comment
>   (c-ts-common-comment-setup)
>
> -  ;; Electric
> -  (setq-local electric-indent-chars
> -              (append "{}():;,#" electric-indent-chars))
> -
>   ;; Imenu.
>   (setq-local treesit-simple-imenu-settings
>               (let ((pred #'c-ts-mode--defun-valid-p))
>
> In master?  In emacs-29?
>
> João


[-- Attachment #2: Type: text/html, Size: 4095 bytes --]

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  dancol@dancol.org,  casouri@gmail.com,
>   emacs-devel@gnu.org, theo@thornhill.no
> Date: Tue, 28 Mar 2023 23:11:05 +0100
> 
> Though a number of indenting problems would remain after that, at least
> this one clear annoyance would be solved.  So if there are no
> objections, I propose to apply this patch.
> 
> diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
> index 59eb9fc23e6..88360716381 100644
> --- a/lisp/progmodes/c-ts-mode.el
> +++ b/lisp/progmodes/c-ts-mode.el
> @@ -956,10 +956,6 @@ c-ts-base-mode
>    ;; Comment
>    (c-ts-common-comment-setup)
>  
> -  ;; Electric
> -  (setq-local electric-indent-chars
> -              (append "{}():;,#" electric-indent-chars))
> -
>    ;; Imenu.
>    (setq-local treesit-simple-imenu-settings
>                (let ((pred #'c-ts-mode--defun-valid-p))
> 
> In master?  In emacs-29?

I suggest to do this in emacs-29, but conditionally, with a
defcustom.  This will allow users to try both ways and maybe we will
have some feedback regarding what is the best way.

I'm also interested to know what other editors do, as Daniel points
out.

Re: treesit indentation "blinking"

[João Távora]

On Wed, Mar 29, 2023 at 3:26 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Cc: Dmitry Gutov <dgutov@yandex.ru>,  dancol@dancol.org,  casouri@gmail.com,
> >   emacs-devel@gnu.org, theo@thornhill.no
> > Date: Tue, 28 Mar 2023 23:11:05 +0100
> >
> > Though a number of indenting problems would remain after that, at least
> > this one clear annoyance would be solved.  So if there are no
> > objections, I propose to apply this patch.
> >
> > diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
> > index 59eb9fc23e6..88360716381 100644
> > --- a/lisp/progmodes/c-ts-mode.el
> > +++ b/lisp/progmodes/c-ts-mode.el
> > @@ -956,10 +956,6 @@ c-ts-base-mode
> >    ;; Comment
> >    (c-ts-common-comment-setup)
> >
> > -  ;; Electric
> > -  (setq-local electric-indent-chars
> > -              (append "{}():;,#" electric-indent-chars))
> > -
> >    ;; Imenu.
> >    (setq-local treesit-simple-imenu-settings
> >                (let ((pred #'c-ts-mode--defun-valid-p))
> >
> > In master?  In emacs-29?
>
> I suggest to do this in emacs-29, but conditionally, with a
> defcustom.  This will allow users to try both ways and maybe we will
> have some feedback regarding what is the best way.

I'm not fond of creating a defcustom to work around what I consider
as a simple bug and to configure something which the user can already
configure with a two-line mode hook addition. So I won't do that
change myself.

> I'm also interested to know what other editors do, as Daniel points
> out.

I don't use many other editors but I can tell you that online editors
like the ones found at hackerrank.com, which I strongly suspect are
based on LSP + treesitter behave as if electric-pair-mode was on
and electric-indent-chars is just '(?\n).  I.e. they auto-indent
on newline and don't bounce around when symbols such as ':',')' or
';' are typed.  As far as I can tell, auto-indenting on characters
other than newline is an Emacs invention that only works well if a
mode has near-perfect predictive powers of indentation, which
c++-ts-mode clearly doesn't have (yet).

And then my personal opinion is that it is an annoying
feature to have on by default as it whole lines about.
Having electric-indent-chars set to '(?\n), like c++-mode
has, is fine.

João

Re: treesit indentation "blinking"

[Herman, Géza]

João Távora <joaotavora@gmail.com> writes:

> And then my personal opinion is that it is an annoying
> feature to have on by default as it whole lines about.
> Having electric-indent-chars set to '(?\n), like c++-mode
> has, is fine.
While c++-mode has only ?\n in electric-indent-chars, it does electric
indentation by other means. For example, ":" is bound to c-electric-colon,
which behaves the same as you described c++-ts-mode previously: the
"std" keeps dancing around as you put one and then two colons after it.

In my opinion, this issue has two different parts:

1.  design problem which cannot be really solved.  Like this "std::"
thing.  The editor cannot read the programmer's mind (whether they will
put a second colon or not).  And because c++-mode behaves the same, I
don't really think this is a bug.  Or at least it is just a small one.

2.  tree-sitter related problem, where tree-sitter has the wrong idea of
the correct indentation of a line.  c++-ts-mode fails for one of the
most simple thing: write "int foo() {" into an empty c++-ts-mode buffer,
and press RET.  You'll notice that the cursor won't be indented.  Also,
TAB doesn't work either.  You need to manually press multiple spaces to
have indentation.  c++-mode doesn't have this problem.  Note, I
understand that calling this "ts has the wrong idea of correct
indentation" may be unfair, but for a user the correct indentation is
pretty clear in this situation.  In my opinion this is a bug (that's why
I reported #62412).

Géza

Re: treesit indentation "blinking"

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2875 bytes --]

On Wed, Mar 29, 2023, 6:29 PM João Távora <joaotavora@gmail.com> wrote:

> On Wed, Mar 29, 2023 at 3:26 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Cc: Dmitry Gutov <dgutov@yandex.ru>,  dancol@dancol.org,
> casouri@gmail.com,
> > >   emacs-devel@gnu.org, theo@thornhill.no
> > > Date: Tue, 28 Mar 2023 23:11:05 +0100
> > >
> > > Though a number of indenting problems would remain after that, at least
> > > this one clear annoyance would be solved.  So if there are no
> > > objections, I propose to apply this patch.
> > >
> > > diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
> > > index 59eb9fc23e6..88360716381 100644
> > > --- a/lisp/progmodes/c-ts-mode.el
> > > +++ b/lisp/progmodes/c-ts-mode.el
> > > @@ -956,10 +956,6 @@ c-ts-base-mode
> > >    ;; Comment
> > >    (c-ts-common-comment-setup)
> > >
> > > -  ;; Electric
> > > -  (setq-local electric-indent-chars
> > > -              (append "{}():;,#" electric-indent-chars))
> > > -
> > >    ;; Imenu.
> > >    (setq-local treesit-simple-imenu-settings
> > >                (let ((pred #'c-ts-mode--defun-valid-p))
> > >
> > > In master?  In emacs-29?
> >
> > I suggest to do this in emacs-29, but conditionally, with a
> > defcustom.  This will allow users to try both ways and maybe we will
> > have some feedback regarding what is the best way.
>
> I'm not fond of creating a defcustom to work around what I consider
> as a simple bug and to configure something which the user can already
> configure with a two-line mode hook addition. So I won't do that
> change myself.
>
> > I'm also interested to know what other editors do, as Daniel points
> > out.
>
> I don't use many other editors but I can tell you that online editors
> like the ones found at hackerrank.com, which I strongly suspect are
> based on LSP + treesitter behave as if electric-pair-mode was on
> and electric-indent-chars is just '(?\n).  I.e. they auto-indent
> on newline and don't bounce around when symbols such as ':',')' or
> ';' are typed.  As far as I can tell, auto-indenting on characters
> other than newline is an Emacs invention that only works well if a
> mode has near-perfect predictive powers of indentation, which
> c++-ts-mode clearly doesn't have (yet).
>
> And then my personal opinion is that it is an annoying
> feature to have on by default as it whole lines about.
> Having electric-indent-chars set to '(?\n), like c++-mode
> has, is fine.
>

I would think the advantage of using tree-sitter is triggering formatting
based on lexemes and grammar symbols rather than raw characters.  So
instead of ';' a formatting action might be triggered by the parser
deducing a lexeme/symbol STATEMENT-SEPARATOR, as opposed to a ';'
separating clauses in a for statement.

Lynn

[-- Attachment #2: Type: text/html, Size: 4430 bytes --]

Re: treesit indentation "blinking"

[João Távora]

On Thu, Mar 30, 2023 at 12:13 AM Herman, Géza <geza.herman@gmail.com> wrote:
>
>
> João Távora <joaotavora@gmail.com> writes:
>
> > And then my personal opinion is that it is an annoying
> > feature to have on by default as it whole lines about.
> > Having electric-indent-chars set to '(?\n), like c++-mode
> > has, is fine.
> While c++-mode has only ?\n in electric-indent-chars, it does electric
> indentation by other means. For example, ":" is bound to c-electric-colon,
> which behaves the same as you described c++-ts-mode previously: the
> "std" keeps dancing around as you put one and then two colons after it.
>
> In my opinion, this issue has two different parts:
>
> 1.  design problem which cannot be really solved.  Like this "std::"
> thing.  The editor cannot read the programmer's mind (whether they will
> put a second colon or not).  And because c++-mode behaves the same, I
> don't really think this is a bug.  Or at least it is just a small one.

If we're going to use c++-mode as a reference, then, without
electric-pair-mode you get another class of "bigger" bugs.  Just type

emacs -Q thingy.cpp -f c++-ts-mode

int main () { RET for (;;) {printf("infloop"); RET }

it still bounces around.  c++-mode doesn't.  Sure, you may say
"oh that's because the c++-ts-mode indenting is off".  OK, then I say
"when/if it's ever fixed, then we can add these extra
electric-indent-chars".

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 29 Mar 2023 22:30:29 +0000
> Cc: dgutov@yandex.ru, dancol@dancol.org, casouri@gmail.com, 
> 	emacs-devel@gnu.org, theo@thornhill.no
> 
> > > diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
> > > index 59eb9fc23e6..88360716381 100644
> > > --- a/lisp/progmodes/c-ts-mode.el
> > > +++ b/lisp/progmodes/c-ts-mode.el
> > > @@ -956,10 +956,6 @@ c-ts-base-mode
> > >    ;; Comment
> > >    (c-ts-common-comment-setup)
> > >
> > > -  ;; Electric
> > > -  (setq-local electric-indent-chars
> > > -              (append "{}():;,#" electric-indent-chars))
> > > -
> > >    ;; Imenu.
> > >    (setq-local treesit-simple-imenu-settings
> > >                (let ((pred #'c-ts-mode--defun-valid-p))
> > >
> > > In master?  In emacs-29?
> >
> > I suggest to do this in emacs-29, but conditionally, with a
> > defcustom.  This will allow users to try both ways and maybe we will
> > have some feedback regarding what is the best way.
> 
> I'm not fond of creating a defcustom to work around what I consider
> as a simple bug and to configure something which the user can already
> configure with a two-line mode hook addition.

I think this conclusion is at least inaccurate, if not incorrect.
IOW, this is not "a simple bug", it could be an issue with different
personal preferences or something else.  See below for details.

> So I won't do that change myself.

That's a strange stance.  Discussion of solutions to issues can
legitimately conclude that some solution could be subject to personal
preferences, some of which you don't share.  Refusing to install a
change you yourself suggested with such minor adjustments, just
because it doesn't fit your personal preferences in editing C/C++
code, sounds at least unfriendly.

> And then my personal opinion is that it is an annoying
> feature to have on by default as it whole lines about.
> Having electric-indent-chars set to '(?\n), like c++-mode
> has, is fine.

Let's back up a notch and revisit the evidence, okay?

First, I don't agree with your conclusion that the local setting of
electric-indent-chars in c-ts-base-mode causes divergent behavior wrt
its CC mode equivalents.  In particular, the example you brought up in

  https://lists.gnu.org/archive/html/emacs-devel/2023-03/msg00939.html

behaves the same in c++-mode, at least in "emacs -Q": "std:" (with a
single colon) causes reindentation as if this were a label, and adding
another colon indents back to column 2.  So at least in this example
the behavior I see is the same in both modes.

Also, at least some of the examples for a different behavior between
c++-mode and c++-ts-mode explicitly turned OFF electric-indent-mode.
For example, see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14

When electric-indent-mode is turned OFF, setting electric-indent-chars
cannot possibly make any difference, right?

(Btw, I cannot reproduce what the first example there says: if I turn
OFF electric-indent-mode, RET doesn't correctly indent the next line;
instead, the next line is not indented at all.  So I wonder what is
being missed here.)

More generally, the claim that CC mode doesn't alter
electric-indent-chars doesn't really explain the differences in
behavior, because while the claim is factually true, CC mode gives
some characters electric behavior that is independent of
electric-indent-chars, see the node "Electric Keys" in the CC Mode
manual.  What happens here is that keys like '#', '*', '<', '(', '{',
':', and others are bound to CC mode specific commands like
c-electric-pound and c-electric-brace, which reindent the code if
electric-indent-mode is turned on in the buffer.

Bottom line: I think both this thread and the discussion in bug#62412
have a lot of misunderstandings and examples at least I cannot
reproduce, and therefore the conclusions reached there, including your
suggestion to remove addition to electric-indent-chars, might very
well be based on those misunderstandings and/or unreproducible
results.  Therefore, to make progress in this issue, we should revisit
the examples, decide whether we are discussing behavior with or
without electric-indent-mode, and then take this from there, after
making sure we are on the same page.

For now, I see no reason to remove the line in c-ts-base-mode which
adds to electric-indent-chars, with or without a defcustom to control
that.

Re: treesit indentation "blinking"

[Herman, Géza]

João Távora <joaotavora@gmail.com> writes:

> If we're going to use c++-mode as a reference, then, without
> electric-pair-mode you get another class of "bigger" bugs.  Just type
>
> emacs -Q thingy.cpp -f c++-ts-mode
>
> int main () { RET for (;;) {printf("infloop"); RET }
>
> it still bounces around.  c++-mode doesn't.  Sure, you may say
> "oh that's because the c++-ts-mode indenting is off".  OK, then I say
> "when/if it's ever fixed, then we can add these extra
> electric-indent-chars".
>
> João

For me, your example doesn't produce any bouncing (with or without
adding an additional RET before printf, I'm not sure whether you
accidentally missed to put a RET there).

Everything is just put unindented.

What could be the cause of the difference? I'm using the latest
tree-sitter-cpp (03fa93db133d6048a77d4de154a7b17ea8b9d076), and latest
emacs-29 branch (several-day-old master also behaves the same).

Géza

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 30/03/2023 10:43, Eli Zaretskii wrote:

> That's a strange stance.  Discussion of solutions to issues can
> legitimately conclude that some solution could be subject to personal
> preferences, some of which you don't share.  Refusing to install a
> change you yourself suggested with such minor adjustments, just
> because it doesn't fit your personal preferences in editing C/C++
> code, sounds at least unfriendly.

Whatever the conclusion regarding the value of electric-indent-chars in 
c-ts-mode (I'd also suggest to try removing ":" but keeping the rest; 
maybe remove "," as well, as it's also likely to indicate incomplete 
expression), I also don't think adding a defcustom to this particular 
mode is a great idea: it will be odd and inconsistent to have it for 
this major mode but not any others.

electric-indent-chars can be adjusted in a major mode hook anyway, so 
the users who will want to tweak it can do it already.

Re: treesit indentation "blinking"

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 6739 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 29 Mar 2023 22:30:29 +0000
>> Cc: dgutov@yandex.ru, dancol@dancol.org, casouri@gmail.com, 
>> 	emacs-devel@gnu.org, theo@thornhill.no
>> 
>> > > diff --git a/lisp/progmodes/c-ts-mode.el b/lisp/progmodes/c-ts-mode.el
>> > > index 59eb9fc23e6..88360716381 100644
>> > > --- a/lisp/progmodes/c-ts-mode.el
>> > > +++ b/lisp/progmodes/c-ts-mode.el
>> > > @@ -956,10 +956,6 @@ c-ts-base-mode
>> > >    ;; Comment
>> > >    (c-ts-common-comment-setup)
>> > >
>> > > -  ;; Electric
>> > > -  (setq-local electric-indent-chars
>> > > -              (append "{}():;,#" electric-indent-chars))
>> > > -
>> > >    ;; Imenu.
>> > >    (setq-local treesit-simple-imenu-settings
>> > >                (let ((pred #'c-ts-mode--defun-valid-p))
>> > >
>> > > In master?  In emacs-29?
>> >
>> > I suggest to do this in emacs-29, but conditionally, with a
>> > defcustom.  This will allow users to try both ways and maybe we will
>> > have some feedback regarding what is the best way.
>> 
>> I'm not fond of creating a defcustom to work around what I consider
>> as a simple bug and to configure something which the user can already
>> configure with a two-line mode hook addition.
>
> I think this conclusion is at least inaccurate, if not incorrect.
> IOW, this is not "a simple bug", it could be an issue with different
> personal preferences or something else.  See below for details.
>
>> So I won't do that change myself.
>
> That's a strange stance.  Discussion of solutions to issues can
> legitimately conclude that some solution could be subject to personal
> preferences, some of which you don't share.  Refusing to install a
> change you yourself suggested with such minor adjustments, just
> because it doesn't fit your personal preferences in editing C/C++
> code, sounds at least unfriendly.

Oh dear.  "at least unfriendly"?

Eli, I'm just trying to help. I don't use c++-ts-mode yet.  It's still
very immature IMO.  Someone in this thread mentioned electric-pair-mode,
a mode I authored, and its (positive) effects on bouncing indentation.

Apparently noone had looked into electric-indent-chars for the cause, or
provide a recipe.  I did and proposed a trivial solution for anreal
problem that I wasn't the first or even the second to witness.

If you don't like my patch, it's fine.  Use the results of my
experiments as you see fit.  But just don't want to figure out a
defcustom name, a default value, etc and contribute to an Emacs with
more defcustoms for areas where I personally believe defcustom aren't
the answer.  I won't stop anyone from adding one.  Is this "unfriendly"?
Geez.

>> And then my personal opinion is that it is an annoying
>> feature to have on by default as it whole lines about.
>> Having electric-indent-chars set to '(?\n), like c++-mode
>> has, is fine.
>
> Let's back up a notch and revisit the evidence, okay?
>
> First, I don't agree with your conclusion that the local setting of
> electric-indent-chars in c-ts-base-mode causes divergent behavior wrt
> its CC mode equivalents.  In particular, the example you brought up in
>
>   https://lists.gnu.org/archive/html/emacs-devel/2023-03/msg00939.html
>
> behaves the same in c++-mode, at least in "emacs -Q": "std:" (with a
> single colon) causes reindentation as if this were a label, and adding
> another colon indents back to column 2.  So at least in this example
> the behavior I see is the same in both modes.

I didn't realize that, because I use c++-mode with its electric features
off.  But I've already given a different MRE for bounciness.

I think within 5 minutes of editing, someone used to c++-mode -- even
with its default electricity -- will start to feel unconfortable with
c++-ts-mode.  I saw bouncing in lots of other places, bouncing that I
know I just don't see with c++-mode.  If I ever pick it up again, I'll
let the ts people know.

I also see "Mismatched parenthesis" warnings where none exist, there's
and MRE for that also below.

> Also, at least some of the examples for a different behavior between
> c++-mode and c++-ts-mode explicitly turned OFF electric-indent-mode.
> For example, see
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14
>
> When electric-indent-mode is turned OFF, setting electric-indent-chars
> cannot possibly make any difference, right?
>
> (Btw, I cannot reproduce what the first example there says: if I turn
> OFF electric-indent-mode, RET doesn't correctly indent the next line;
> instead, the next line is not indented at all.  So I wonder what is
> being missed here.)

What's being missed is that perhaps you're mixing electric-pair-mode
with electric-indent-mode? In these tests I never turned off e-i-m,
because it's on by default in Emacs.  In these tests I do turn e-p-m on
and off to investigate claims by other users that it helps and measure
by how much.

> More generally, the claim that CC mode doesn't alter
> electric-indent-chars doesn't really explain the differences in
> behavior, because while the claim is factually true, CC mode gives
> some characters electric behavior that is independent of
> electric-indent-chars, see the node "Electric Keys" in the CC Mode
> manual.  What happens here is that keys like '#', '*', '<', '(', '{',
> ':', and others are bound to CC mode specific commands like
> c-electric-pound and c-electric-brace, which reindent the code if
> electric-indent-mode is turned on in the buffer.

Sure, I've already said that I missed this electricity.  I turn off
c++-mode electricity, and so that example was flawed.  I'm deeply, truly
sorry.  But I've already provided a better example.

> Bottom line: I think both this thread and the discussion in bug#62412
> have a lot of misunderstandings and examples at least I cannot
> reproduce, 

Maybe because of the mix-up between e-p-m and e-i-m mentioned above?

I already gave this alternative MRE of bouncing behaviour.

   emacs -Q `mktemp`.cpp -f c++-ts-mode  

   i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;  

Can you reproduce this bouncing?  Now try the same with c++-mode. Do you
confirm that it doesn't bounce?  Also in c++-ts-mode, add a closing `}`.
See the "mismatched parenthesis"?

Now I'm going to give another example.  In the same file, go back to
c++-ts-mode.  Say you spotted a mistake and dont want an infloop after
all.  Go and delete the two ';;' in the for expressions, leaving, say
just the parenthesis.  Start typing another set of expressions.  See the
bouncing back and forth?  I've even made a gif of this.


[-- Attachment #2: bouncing.gif --]
[-- Type: image/gif, Size: 70095 bytes --]

[-- Attachment #3: Type: text/plain, Size: 9 bytes --]


João

Re: treesit indentation "blinking"

[João Távora]

On Thu, Mar 30, 2023 at 9:58 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 30/03/2023 10:43, Eli Zaretskii wrote:
>
> > That's a strange stance.  Discussion of solutions to issues can
> > legitimately conclude that some solution could be subject to personal
> > preferences, some of which you don't share.  Refusing to install a
> > change you yourself suggested with such minor adjustments, just
> > because it doesn't fit your personal preferences in editing C/C++
> > code, sounds at least unfriendly.
>
> Whatever the conclusion regarding the value of electric-indent-chars in
> c-ts-mode (I'd also suggest to try removing ":" but keeping the rest;
> maybe remove "," as well, as it's also likely to indicate incomplete
> expression), I also don't think adding a defcustom to this particular
> mode is a great idea: it will be odd and inconsistent to have it for
> this major mode but not any others.

Right.  And see my updated MRE where bouncing happens also on ';'.
I think at least ';' should also be added to that set of chars to
be removed from electric-indent-chars.

João

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 30/03/2023 12:06, João Távora wrote:
> I already gave this alternative MRE of bouncing behaviour.
> 
>     emacs -Q `mktemp`.cpp -f c++-ts-mode
> 
>     i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;
> 
> Can you reproduce this bouncing?  Now try the same with c++-mode. Do you
> confirm that it doesn't bounce?

FWIW, I'm unable to repro bouncing here. If anything, what's problematic 
here, is I get 0 spaces of indentation after the first RET (when 
electric-pair-mode is disabled, of course). But no bouncing when typing 
the first two semicolons, and the last one puts indentation where it 
should be.

> Also in c++-ts-mode, add a closing `}`.
> See the "mismatched parenthesis"?

I do see that, no idea why that happens (show-paren-mode highlights the 
matching parens correctly).

Re: treesit indentation "blinking"

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 30/03/2023 12:06, João Távora wrote:
>> I already gave this alternative MRE of bouncing behaviour.
>>     emacs -Q `mktemp`.cpp -f c++-ts-mode
>>     i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;
>> Can you reproduce this bouncing?  Now try the same with c++-mode. Do
>> you
>> confirm that it doesn't bounce?
>
> FWIW, I'm unable to repro bouncing here. If anything, what's
> problematic here, is I get 0 spaces of indentation after the first RET
> (when electric-pair-mode is disabled, of course). But no bouncing when
> typing the first two semicolons, and the last one puts indentation
> where it should be.

This problematic already counts as "bouncing" to me, for some meaning of
"bouncing". c++-mode doesn't behave like that because indentation is
already where it is supposed to be if you type that sequence of
keystrokes.

And did you try the final example that I showed in the animated gif?
That's even more "bouncy", at least in my book.

João

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 30/03/2023 12:28, João Távora wrote:
> This problematic already counts as "bouncing" to me, for some meaning of
> "bouncing". c++-mode doesn't behave like that because indentation is
> already where it is supposed to be if you type that sequence of
> keystrokes.

Okay, if that's what you meant.

I think this one (indentation after RET in an incomplete function 
definition) should be fixed in the indentation rules. The contents of 
electric-indent-chars won't fix it either way.

> And did you try the final example that I showed in the animated gif?
> That's even more "bouncy", at least in my book.

Yep, I see that. Also seems like something that could be improved in the 
indentation rules. When there is just one or no semis inside for's 
parens, we get ERROR nodes. I guess some rule is hit that should not be hit.

Re: treesit indentation "blinking"

[João Távora]

On Thu, Mar 30, 2023 at 10:36 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 30/03/2023 12:28, João Távora wrote:
> > This problematic already counts as "bouncing" to me, for some meaning of
> > "bouncing". c++-mode doesn't behave like that because indentation is
> > already where it is supposed to be if you type that sequence of
> > keystrokes.
>
> Okay, if that's what you meant.
>
> I think this one (indentation after RET in an incomplete function
> definition) should be fixed in the indentation rules. The contents of
> electric-indent-chars won't fix it either way.

This is all down to indentation rules, that's how electric-indent-mode
decides what to do.  My point is that having electric-indent-chars be
this ambitious with "broken" indentation rules isn't a good place to
be.

What counts as "broken" indentation is also arguable though. When dealing
with invalid programs, there is really no "right" or "wrong" indentation.
See my message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14 where
I show cases where c++-ts-mode's answer to indenting an invalid program
makes more sense than c++-mode's answer.

Whatever the indentation rules, the current bouncing is so jarring
that it really doesn't encourage people to try switching to
c++-ts-mode, get used to its set of indentation rules, and then perhaps
experience its other benefits like, say, performance or simplicity.

At least it didn't for me.  I'm back to c++-mode atm.

In my opinion electric-indent-char should be reduced to the default
and should be added criteriously as the indentation rules they trigger
are fixed.

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: dgutov@yandex.ru,  dancol@dancol.org,  casouri@gmail.com,
>   emacs-devel@gnu.org,  theo@thornhill.no
> Date: Thu, 30 Mar 2023 10:06:19 +0100
> 
> Eli, I'm just trying to help. I don't use c++-ts-mode yet.  It's still
> very immature IMO.  Someone in this thread mentioned electric-pair-mode,
> a mode I authored, and its (positive) effects on bouncing indentation.
> 
> Apparently noone had looked into electric-indent-chars for the cause, or
> provide a recipe.  I did and proposed a trivial solution for anreal
> problem that I wasn't the first or even the second to witness.

FWIW, I don't see any problem that needs fixing, not yet.

> If you don't like my patch, it's fine.  Use the results of my
> experiments as you see fit.  But just don't want to figure out a
> defcustom name, a default value, etc and contribute to an Emacs with
> more defcustoms for areas where I personally believe defcustom aren't
> the answer.  I won't stop anyone from adding one.

I suggested a defcustom because I blindly believed your description
and your analysis.  Now that I have tried reproducing the issues you
describe, and cannot see the problems you describe, I no longer think
there's a problem that justifies removal of electric-indent-chars
customization by c++-ts-mode, with or without the defcustom.

> Is this "unfriendly"?  Geez.

What _is_ unfriendly is to refuse to install a change that you
yourself consider required, when asked to do that conditionally, so
users who still want the electricity, even though it "blinks", could
still have it.  I think the request is reasonable, especially since
you don't use this mode, and so can easily overlook some useful
behavior that your proposed change could disable or break.

> > First, I don't agree with your conclusion that the local setting of
> > electric-indent-chars in c-ts-base-mode causes divergent behavior wrt
> > its CC mode equivalents.  In particular, the example you brought up in
> >
> >   https://lists.gnu.org/archive/html/emacs-devel/2023-03/msg00939.html
> >
> > behaves the same in c++-mode, at least in "emacs -Q": "std:" (with a
> > single colon) causes reindentation as if this were a label, and adding
> > another colon indents back to column 2.  So at least in this example
> > the behavior I see is the same in both modes.
> 
> I didn't realize that, because I use c++-mode with its electric features
> off.

If you turn electric features off, then electric-indent-chars will
have no effect whatsoever, and all this discussion is moot.

> I think within 5 minutes of editing, someone used to c++-mode -- even
> with its default electricity -- will start to feel unconfortable with
> c++-ts-mode.

We shall see, okay?  You could be right or you could be wrong.  The
purpose of releasing these modes in Emacs 29 is to collect user
feedback, so we know in which direction(s) to develop them further.
Your opinions are noted, but let's give others chance to voice theirs,
okay?

> I saw bouncing in lots of other places, bouncing that I
> know I just don't see with c++-mode.  If I ever pick it up again, I'll
> let the ts people know.

Please report any problems you see when you see them, so that we could
fix those that are still there.

> > Also, at least some of the examples for a different behavior between
> > c++-mode and c++-ts-mode explicitly turned OFF electric-indent-mode.
> > For example, see
> >
> >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14
> >
> > When electric-indent-mode is turned OFF, setting electric-indent-chars
> > cannot possibly make any difference, right?
> >
> > (Btw, I cannot reproduce what the first example there says: if I turn
> > OFF electric-indent-mode, RET doesn't correctly indent the next line;
> > instead, the next line is not indented at all.  So I wonder what is
> > being missed here.)
> 
> What's being missed is that perhaps you're mixing electric-pair-mode
> with electric-indent-mode? In these tests I never turned off e-i-m,
> because it's on by default in Emacs.

The above message as part of bug#62412 clearly says "Let's assume you
turn off electric-indent-mode."  I interpreted that as meaning that
electric-indent-mode is to be turned off for the rest of the examples
to do what you mean them to do.

> I already gave this alternative MRE of bouncing behaviour.
> 
>    emacs -Q `mktemp`.cpp -f c++-ts-mode  
> 
>    i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;  
> 
> Can you reproduce this bouncing?

No, I cannot.  All I see is that the semi-colon after "foo()" indents
the line, just once, so that it has the correct indentation
(previously it had no indentation at all).

> Now try the same with c++-mode. Do you confirm that it doesn't
> bounce?

The only difference I see is that c++-mode indents the line with "for"
right from the start, after I type RET.  Which is better indeed, but
what c++-ts-mode does is not a catastrophe, either.

> Also in c++-ts-mode, add a closing `}`.  See the "mismatched
> parenthesis"?

No, I don't.

Are you using Emacs 29 or Emacs 30?  I'm using the former.

Also, which version of the tree-sitter C++ grammar library do you have
installed?

> Now I'm going to give another example.  In the same file, go back to
> c++-ts-mode.  Say you spotted a mistake and dont want an infloop after
> all.  Go and delete the two ';;' in the for expressions, leaving, say
> just the parenthesis.  Start typing another set of expressions.

Please state exactly what to type, otherwise we will again be talking
past each other.

> See the bouncing back and forth?  I've even made a gif of this.

I see it in your GIF, but not on my system.

Re: treesit indentation "blinking"

[Herman, Géza]

Eli Zaretskii <eliz@gnu.org> writes:

> [...]  Which is better indeed, but
> what c++-ts-mode does is not a catastrophe, either.

Maybe it's not a catastrophe, but it's far from behaving well.

Type this example into a c++-ts-mode buffer (I used "emacs -Q"):

--8<---------------cut here---------------start------------->8---
int main() {
for (;;) {
printf("Hello world\n");
}
}
--8<---------------cut here---------------end--------------->8---

This is how it will be indented as I wrote it here. c++-ts-mode doesn't
re-indent anything during typing, even though it had the chance to do
that, because electric-indent-mode is enabled by default, and
electric-indent-chars contains the necessary characters.

Or, another example. Put the "void foo() { }" part first into a
c++-ts-mode buffer, then write the main function:

--8<---------------cut here---------------start------------->8---
int main() {
int a = 0;
for (;;) {
printf("Hello!\n");
}
}

void foo() {
}
--8<---------------cut here---------------end--------------->8---

Again, c++-ts-mode doesn't indent anything.

If you change the example to contain "void foo();" instead of "void
foo() { }", then indenting happens during typing the main function.

If you try adding "int a = 0;" into the first example, then it will be
indented at typing the ";". But then for loop is still not get indented,
and the buffer will look like this:

--8<---------------cut here---------------start------------->8---
int main() {
  int a = 0;
for (;;) {
printf("Hello world\n");
}
}
--8<---------------cut here---------------end--------------->8---

If you start by adding an empty comment, and then write the first
example above the empty comment, the buffer will look like this:

--8<---------------cut here---------------start------------->8---
int main() {
  for (;;) {
printf("Hello world\n");
  }
}

/**/
--8<---------------cut here---------------end--------------->8---

Can you reproduce these? These happen both with emacs-29 and
several-day-old master with the latest tree-sitter-cpp (but it also
happens with a ~1 month-old tree-sitter-cpp).

Géza

Re: treesit indentation "blinking"

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> What _is_ unfriendly is to refuse to install a change that you
> yourself consider required, when asked to do that conditionally, so
> users who still want the electricity, even though it "blinks", could
> still have it.  I think the request is reasonable, especially since
> you don't use this mode, and so can easily overlook some useful
> behavior that your proposed change could disable or break.

You don't need me to install changes, do you?  It's not like I'm
refusing to fix a bug that I added: I didn't write any of this.  Users
that do (or don't) want electricity already have good ways to fine-tune
that preference.  I don't think this particular idea of your is very
good, so I politely stated I'd rather not work on that possibility.  If
someone does, I won't object.  I think that's friendly enough.

>> I didn't realize that, because I use c++-mode with its electric features
>> off.
>
> If you turn electric features off, then electric-indent-chars will
> have no effect whatsoever, and all this discussion is moot.

No.  I use c++ mode with its bindings like c-electric-paren deactivated.
I still use electric-indent-mode with its very reasonable default value
for of (?\n)

>> I think within 5 minutes of editing, someone used to c++-mode -- even
>> with its default electricity -- will start to feel unconfortable with
>> c++-ts-mode.
>
> We shall see, okay?  You could be right or you could be wrong.  The
> purpose of releasing these modes in Emacs 29 is to collect user
> feedback, so we know in which direction(s) to develop them further.
> Your opinions are noted, but let's give others chance to voice theirs,
> okay?

I find this suggestion that I'm somehow shooshing other's opinions.  So
far I don't think anyone here has said they _like_ the bouncing.  But
maybe someone does..  I wasn't even the first to report this.  Daniel
did in this list and Geza Herman did in #61412.

> The above message as part of bug#62412 clearly says "Let's assume you
> turn off electric-indent-mode."  I interpreted that as meaning that
> electric-indent-mode is to be turned off for the rest of the examples
> to do what you mean them to do.

I see, my bad.  Should have written electric-pair-mode instead.

>> I already gave this alternative MRE of bouncing behaviour.
>> 
>>    emacs -Q `mktemp`.cpp -f c++-ts-mode  
>> 
>>    i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;  
>> 
>> Can you reproduce this bouncing?
>
> No, I cannot.  All I see is that the semi-colon after "foo()" indents
> the line, just once, so that it has the correct indentation
> (previously it had no indentation at all).

We have different ideas of bouncing.

>> Now try the same with c++-mode. Do you confirm that it doesn't
>> bounce?
>
> The only difference I see is that c++-mode indents the line with "for"
> right from the start, after I type RET.  Which is better indeed, but
> what c++-ts-mode does is not a catastrophe, either.

Never said it's a "catastrophe".  Just annoying/jarring/suprising
behaviour that you don't get with vanilla c++-mode or modes based on
cc-mode.  I think that electric-indent-chars was designed with those
modes in mind.

>> Also in c++-ts-mode, add a closing `}`.  See the "mismatched
>> parenthesis"?
>
> No, I don't.
> Are you using Emacs 29 or Emacs 30?  I'm using the former.

I'm using the latest master (560c27a3) and starting it with 

src/emacs -Q `mktemp`.cpp -f c++-ts-mode

> Also, which version of the tree-sitter C++ grammar library do you have
> installed?

I don't know how to answer that.  I used M-x
treesit-install-language-grammar RET "cpp" and accepted the default in
all prompts.  I presume it installs the latest version of the Git repo.
I've just reinstalled today.

I have ~/.emacs.d/tree-sitter/libtree-sitter-cpp.so I don't think it
stores the version number there.

>> Now I'm going to give another example.  In the same file, go back to
>> c++-ts-mode.  Say you spotted a mistake and dont want an infloop after
>> all.  Go and delete the two ';;' in the for expressions, leaving, say
>> just the parenthesis.  Start typing another set of expressions.
>
> Please state exactly what to type, otherwise we will again be talking
> past each other.

You can _see_ in the GIF what I type.  I've just using normal keys for
self-insert.  Then C-p, and some C-f to position my cursor before the
';;', then C-d to delete the two ';;'. And then add them back again with
two ';;'.  After the first ';' the line goes back, then the second one,
and the line goes forward ';'.

Just confirmed it happens in Emacs 29 too, commit
d2e82817a3f341e426c220e98048e1784d1e3076.

Also see the original recipe of bug#62142, which is quite easy to
follow, for more bouncing.  Can't you reproduce this either?

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: Herman, Géza <geza.herman@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,
>  dgutov@yandex.ru, dancol@dancol.org,
>  casouri@gmail.com, theo@thornhill.no, emacs-devel@gnu.org
> Date: Thu, 30 Mar 2023 12:26:16 +0200
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > [...]  Which is better indeed, but
> > what c++-ts-mode does is not a catastrophe, either.
> 
> Maybe it's not a catastrophe, but it's far from behaving well.
> 
> Type this example into a c++-ts-mode buffer (I used "emacs -Q"):
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
> for (;;) {
> printf("Hello world\n");
> }
> }
> --8<---------------cut here---------------end--------------->8---
> 
> This is how it will be indented as I wrote it here. c++-ts-mode doesn't
> re-indent anything during typing, even though it had the chance to do
> that, because electric-indent-mode is enabled by default, and
> electric-indent-chars contains the necessary characters.
> 
> Or, another example. Put the "void foo() { }" part first into a
> c++-ts-mode buffer, then write the main function:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
> int a = 0;
> for (;;) {
> printf("Hello!\n");
> }
> }
> 
> void foo() {
> }
> --8<---------------cut here---------------end--------------->8---
> 
> Again, c++-ts-mode doesn't indent anything.
> 
> If you change the example to contain "void foo();" instead of "void
> foo() { }", then indenting happens during typing the main function.
> 
> If you try adding "int a = 0;" into the first example, then it will be
> indented at typing the ";". But then for loop is still not get indented,
> and the buffer will look like this:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
>   int a = 0;
> for (;;) {
> printf("Hello world\n");
> }
> }
> --8<---------------cut here---------------end--------------->8---
> 
> If you start by adding an empty comment, and then write the first
> example above the empty comment, the buffer will look like this:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
>   for (;;) {
> printf("Hello world\n");
>   }
> }
> 
> /**/
> --8<---------------cut here---------------end--------------->8---
> 
> Can you reproduce these? These happen both with emacs-29 and
> several-day-old master with the latest tree-sitter-cpp (but it also
> happens with a ~1 month-old tree-sitter-cpp).

Thanks for the examples, but how are they related to the issue at
hand?  We are discussing the possibly adverse effects that
electric-indent-mode and the customization of electric-indent-chars by
c-ts-base-mode might have on user experience while typing C/C++ code.
Are you saying that removing the customization of
electric-indent-chars by c-ts-base-mode solves the problems you
mention?  If not, then I think your examples should go into a separate
bug report, which should be investigated separately.

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: dgutov@yandex.ru,  dancol@dancol.org,  casouri@gmail.com,
>   emacs-devel@gnu.org,  theo@thornhill.no, geza.herman@gmail.com
> Date: Thu, 30 Mar 2023 12:05:35 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What _is_ unfriendly is to refuse to install a change that you
> > yourself consider required, when asked to do that conditionally, so
> > users who still want the electricity, even though it "blinks", could
> > still have it.  I think the request is reasonable, especially since
> > you don't use this mode, and so can easily overlook some useful
> > behavior that your proposed change could disable or break.
> 
> You don't need me to install changes, do you?

Yes, I do.  If I had to install all the changes myself, Emacs could
never be moving forward as fast is it does today, and wouldn't be what
we see now.

This is a community project; without active collaboration of all the
contributors it wouldn't be such a success.  It should go without
saying that when someone proposes a change and the discussion then
concludes the change needs small adaptations, the person who proposed
the change will then go ahead and install those small adaptations.
Without such minimal collaboration, we as a project will get nowhere.

> >> I didn't realize that, because I use c++-mode with its electric features
> >> off.
> >
> > If you turn electric features off, then electric-indent-chars will
> > have no effect whatsoever, and all this discussion is moot.
> 
> No.  I use c++ mode with its bindings like c-electric-paren deactivated.
> I still use electric-indent-mode with its very reasonable default value
> for of (?\n)

I suspect that this is the root cause why you find c++-ts-mode so
jarring: since you disable all the electric behavior in C++ mode
except the one bound to newline, you are not used to electric
reindentation, except when you type RET.  But that is not the default.
I do use the default settings of all the CC Mode electric characters
(with very few minor customizations), and so I do see the electric
reindentation in c++-mode all the time, and am used to it.  If most
users don't disable the default electric behavior in CC Mode, they
will not see c++-ts-mode behavior as "jarring".

The difference between c++-mode and c++-ts-mode is that the latter
doesn't (yet) let you customize each electric character separately.
This could be a subject of future extensions, but for now, as Dmitry
points out, it is easy to override the value of electric-indent-chars
in a mode hook, and leave only the newline in it -- this should give
you a better approximation to what you are used to in c++-mode, I
think.

> So far I don't think anyone here has said they _like_ the bouncing.

It depends on how much it does that.  What I saw until now is not
different from what I see in CC mode with the default electric
behavior, in which case I'm okay with that.

> But maybe someone does..  I wasn't even the first to report this.
> Daniel did in this list and Geza Herman did in #61412.

Maybe Daniel also disables the electric behavior of CC mode?  In any
case, customizing electric-indent-chars could solve Daniel's problems
as well.

> >>    emacs -Q `mktemp`.cpp -f c++-ts-mode  
> >> 
> >>    i n t SPC m a i n ( ) { RET for ( ; ; ) SPC b l a ( ) ;  
> >> 
> >> Can you reproduce this bouncing?
> >
> > No, I cannot.  All I see is that the semi-colon after "foo()" indents
> > the line, just once, so that it has the correct indentation
> > (previously it had no indentation at all).
> 
> We have different ideas of bouncing.

No, I don't think so.  We just perceive it differently.  Also, I think
your description said that each semi-colon inside "for" causes
reindentation, and I don't see that

> >> Now try the same with c++-mode. Do you confirm that it doesn't
> >> bounce?
> >
> > The only difference I see is that c++-mode indents the line with "for"
> > right from the start, after I type RET.  Which is better indeed, but
> > what c++-ts-mode does is not a catastrophe, either.
> 
> Never said it's a "catastrophe".  Just annoying/jarring/suprising
> behaviour that you don't get with vanilla c++-mode or modes based on
> cc-mode.  I think that electric-indent-chars was designed with those
> modes in mind.

Didn't we just establish that you modify the vanilla behavior of
c++-mode by disabling the electric behavior of all the characters
except the newline?  If so, users who don't disable will not see the
electric reindentation as jarring.

> I'm using the latest master (560c27a3) and starting it with 
> 
> src/emacs -Q `mktemp`.cpp -f c++-ts-mode
> 
> > Also, which version of the tree-sitter C++ grammar library do you have
> > installed?
> 
> I don't know how to answer that.  I used M-x
> treesit-install-language-grammar RET "cpp" and accepted the default in
> all prompts.  I presume it installs the latest version of the Git repo.
> I've just reinstalled today.

Then I don't know why we don't see the same behavior.  Beats me.

> You can _see_ in the GIF what I type.  I've just using normal keys for
> self-insert.  Then C-p, and some C-f to position my cursor before the
> ';;', then C-d to delete the two ';;'. And then add them back again with
> two ';;'.  After the first ';' the line goes back, then the second one,
> and the line goes forward ';'.

Similar things happen with CC mode.  Perhaps not in this particular
situation, but in others.

> Also see the original recipe of bug#62142, which is quite easy to
> follow, for more bouncing.  Can't you reproduce this either?

I'm confused by the description there, and we just discovered that you
said there to disable electric-indent-mode when you really meant
electric-pair-mode.  So maybe post a revised example there (or here),
this time without any mistakes, and let's take it from there.

Re: treesit indentation "blinking"

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> This is a community project; without active collaboration of all the
> contributors it wouldn't be such a success.  It should go without
> saying that when someone proposes a change and the discussion then
> concludes the change needs small adaptations, the person who proposed
> the change will then go ahead and install those small adaptations.
> Without such minimal collaboration, we as a project will get nowhere.

But "the discussion" _didn't_ conclude that.  You concluded that.  I'm
not interesting in designing a defcustom, figuring out if it should be a
boolean or a string, what the default should be, the docstring, the
NEWS, etc for something which I think is a bug.

I've never insisted other people to modify their work in ways they
aren't confident in.  They would probably politely refuse it and maybe I
would be slightly disappointed, but I could always do those changes
myself if they are indeed easy. "Unfriendly" would be a really bizarre
judgement to bestow on someone who volunteered time and code to my
project and happened not to agree on a detail.

> (with very few minor customizations), and so I do see the electric
> reindentation in c++-mode all the time, and am used to it.  If most
> users don't disable the default electric behavior in CC Mode, they
> will not see c++-ts-mode behavior as "jarring".

Users coming from other editors like Microsoft Visual Studio will.  You
asked for feedback on other editors, I'm giving it to you.  And there's
also the fact that c++-ts-mode does _not_ behave like c++-mode w.r.t to
indentation to start with.  There are many indentation differences, even
without the electricity part.

>> But maybe someone does..  I wasn't even the first to report this.
>> Daniel did in this list and Geza Herman did in #61412.
>
> Maybe Daniel also disables the electric behavior of CC mode?  In any
> case, customizing electric-indent-chars could solve Daniel's problems
> as well.

But why subject new users to the same strangeness?  Presuming 

>> > Also, which version of the tree-sitter C++ grammar library do you have
>> > installed?
>> 
>> I don't know how to answer that.  I used M-x
>> treesit-install-language-grammar RET "cpp" and accepted the default in
>> all prompts.  I presume it installs the latest version of the Git repo.
>> I've just reinstalled today.
>
> Then I don't know why we don't see the same behavior.  Beats me.

Dmitry has reproduced it.  It's not a "sometimes only" thing.  Happens
consistently.

  >> JT: And did you try the final example that I showed in the animated gif?
  >> JT:That's even more "bouncy", at least in my book.
  > DG: Yep, I see that. 

My opinion is if this is a problem with the indentation rules, then we
should either fix those indentation rules or turn off the electric
features that rely on those temporarily broken indentation rules.  As
the rules get fixed, then adding the features back makes sense.  The
current situation is inconsistent, IMO.

>> You can _see_ in the GIF what I type.  I've just using normal keys for
>> self-insert.  Then C-p, and some C-f to position my cursor before the
>> ';;', then C-d to delete the two ';;'. And then add them back again with
>> two ';;'.  After the first ';' the line goes back, then the second one,
>> and the line goes forward ';'.
>
> Similar things happen with CC mode.  Perhaps not in this particular
> situation, but in others.

If you think this indentation back and forth as you type something as
innocent as two ';;' is acceptable behaviour, then fine.  I though there
was consensus that this was undesired in the new mode and volunteered my
analysis and a fix.  If there isn't such a consensus, then I'll see
myself out.

>> Also see the original recipe of bug#62142, which is quite easy to
>> follow, for more bouncing.  Can't you reproduce this either?
>
> I'm confused by the description there, and we just discovered that you
> said there to disable electric-indent-mode when you really meant
> electric-pair-mode.  So maybe post a revised example there (or here),
> this time without any mistakes, and let's take it from there.

I didn't mean _my_ description, I meant Geza's example.  Thus I wrote
"original recipe" of that bug. IOW, the post that created the bug#62412.
That's the one you could look intro reproducing, if you're interested.

But if you really want to focus on that other particular message of
mine, it is only describing the different indentation behaviour of
c++-ts-mode vs c++-mode, identifying the strengths and weaknesses of
each, in ways that aren't particularly critical of one vs the other.

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: dgutov@yandex.ru,  dancol@dancol.org,  casouri@gmail.com,
>   emacs-devel@gnu.org,  theo@thornhill.no,  geza.herman@gmail.com
> Date: Thu, 30 Mar 2023 15:43:49 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This is a community project; without active collaboration of all the
> > contributors it wouldn't be such a success.  It should go without
> > saying that when someone proposes a change and the discussion then
> > concludes the change needs small adaptations, the person who proposed
> > the change will then go ahead and install those small adaptations.
> > Without such minimal collaboration, we as a project will get nowhere.
> 
> But "the discussion" _didn't_ conclude that.  You concluded that.  I'm
> not interesting in designing a defcustom, figuring out if it should be a
> boolean or a string, what the default should be, the docstring, the
> NEWS, etc for something which I think is a bug.

I asked you to become interested, but only in the defcustom.  I never
said anything about NEWS etc.  It's a simple request that should have
taken just a few moments of your time.  I was astonished by your flat
refusal.

And if that doesn't drive the point home, then I have nothing else to
say in this matter.

> I've never insisted other people to modify their work in ways they
> aren't confident in.

Sorry, but part of _my_ job here is to "insist".

> They would probably politely refuse it and maybe I
> would be slightly disappointed, but I could always do those changes
> myself if they are indeed easy. "Unfriendly" would be a really bizarre
> judgement to bestow on someone who volunteered time and code to my
> project and happened not to agree on a detail.

Most, if not all people whom I ask to make such small amendments (and
also similar other ones, like reword the doc strings, rename
variables, make some behavior optional, etc.) just do it.  I'm asked
to do similar things in other projects, where I'm just one of many
contributors.  Why you think it's unreasonable is beyond me.

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: Herman, Géza <geza.herman@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,
>  dgutov@yandex.ru, dancol@dancol.org,
>  casouri@gmail.com, theo@thornhill.no, emacs-devel@gnu.org
> Date: Thu, 30 Mar 2023 12:26:16 +0200
> 
> Type this example into a c++-ts-mode buffer (I used "emacs -Q"):
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
> for (;;) {
> printf("Hello world\n");
> }
> }
> --8<---------------cut here---------------end--------------->8---
> 
> This is how it will be indented as I wrote it here. c++-ts-mode doesn't
> re-indent anything during typing, even though it had the chance to do
> that, because electric-indent-mode is enabled by default, and
> electric-indent-chars contains the necessary characters.
> 
> Or, another example. Put the "void foo() { }" part first into a
> c++-ts-mode buffer, then write the main function:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
> int a = 0;
> for (;;) {
> printf("Hello!\n");
> }
> }
> 
> void foo() {
> }
> --8<---------------cut here---------------end--------------->8---
> 
> Again, c++-ts-mode doesn't indent anything.
> 
> If you change the example to contain "void foo();" instead of "void
> foo() { }", then indenting happens during typing the main function.
> 
> If you try adding "int a = 0;" into the first example, then it will be
> indented at typing the ";". But then for loop is still not get indented,
> and the buffer will look like this:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
>   int a = 0;
> for (;;) {
> printf("Hello world\n");
> }
> }
> --8<---------------cut here---------------end--------------->8---
> 
> If you start by adding an empty comment, and then write the first
> example above the empty comment, the buffer will look like this:
> 
> --8<---------------cut here---------------start------------->8---
> int main() {
>   for (;;) {
> printf("Hello world\n");
>   }
> }
> 
> /**/
> --8<---------------cut here---------------end--------------->8---

Yuan and Theo, can you please look into these examples?  A simple
"C-c C-q" fixes the indentation, but the question is why it doesn't
happen while typing?

Re: treesit indentation "blinking"

[Herman, Géza]

Eli Zaretskii <eliz@gnu.org> writes:

>> [ examples snipped ]
>>
>> Can you reproduce these? These happen both with emacs-29 and
>> several-day-old master with the latest tree-sitter-cpp (but it also
>> happens with a ~1 month-old tree-sitter-cpp).
>
> Thanks for the examples, but how are they related to the issue at
> hand?  We are discussing the possibly adverse effects that
> electric-indent-mode and the customization of electric-indent-chars by
> c-ts-base-mode might have on user experience while typing C/C++ code.

These are related because the root cause of the bouncing issue is that
tree sitter calculates the wrong intentation during the code is being
written (i.e., not complete).  If this were fixed, most of the bouncing
issue would go away.  Not everything, of course.  If a case causes
bouncing with c++-mode, then it will likely bounce with c++-ts-mode as
well.

I wrote this in a previous email:

"In my opinion, this issue has two different parts:

1.  design problem which cannot be really solved.  Like this "std::"
thing.  The editor cannot read the programmer's mind (whether they will
put a second colon or not).  And because c++-mode behaves the same, I
don't really think this is a bug.  Or at least it is just a small one.

2.  tree-sitter related problem, where tree-sitter has the wrong idea of
the correct indentation of a line.  c++-ts-mode fails for one of the
most simple thing: write "int foo() {" into an empty c++-ts-mode buffer,
and press RET.  You'll notice that the cursor won't be indented.  Also,
TAB doesn't work either.  You need to manually press multiple spaces to
have indentation.  c++-mode doesn't have this problem.  Note, I
understand that calling this "ts has the wrong idea of correct
indentation" may be unfair, but for a user the correct indentation is
pretty clear in this situation.  In my opinion this is a bug (that's why
I reported #62412)."

> Are you saying that removing the customization of
> electric-indent-chars by c-ts-base-mode solves the problems you
> mention?  If not, then I think your examples should go into a separate
> bug report, which should be investigated separately.

No, the root cause doesn't have any relation to electric-indent-chars.

Géza

Re: treesit indentation "blinking"

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> But "the discussion" _didn't_ conclude that.  You concluded that.  I'm
>> not interesting in designing a defcustom, figuring out if it should be a
>> boolean or a string, what the default should be, the docstring, the
>> NEWS, etc for something which I think is a bug.
>
> I asked you to become interested, but only in the defcustom.  I never
> said anything about NEWS etc.  It's a simple request that should have
> taken just a few moments of your time.

There are multiple ways to do this defcustom, I can't guess what you
want it to be called, or its type, or its default value.  So maybe "a
few moments" of typing the code, but definitely not of design.

> I was astonished by your flat refusal.

Would you rather I had rather silently ignored you?  I gave reasons for
being against the change, just as Dmitry did when he also expressed
opposition.  Then I added "...so I won't do that change myself." meaning
that while I don't think it's a very good idea, I won't try to stop
anyone else from doing it.  If anything, I was trying to make _less_
attrition by writing that.

> Most, if not all people whom I ask to make such small amendments (and
> also similar other ones, like reword the doc strings, rename
> variables, make some behavior optional, etc.) just do it.  I'm asked
> to do similar things in other projects, where I'm just one of many
> contributors.  Why you think it's unreasonable is beyond me.

It's _not_ unreasonable, it's perfectly fine to suggest changes.  In
fact I do it all the time.  What's unreasonable, and frankly a bit
saddening (for me) is call others "unfriendly" when encountering the
smallest justified opposition to a given suggestion.

João

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 30/03/2023 13:00, João Távora wrote:
> On Thu, Mar 30, 2023 at 10:36 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>> On 30/03/2023 12:28, João Távora wrote:
>>> This problematic already counts as "bouncing" to me, for some meaning of
>>> "bouncing". c++-mode doesn't behave like that because indentation is
>>> already where it is supposed to be if you type that sequence of
>>> keystrokes.
>>
>> Okay, if that's what you meant.
>>
>> I think this one (indentation after RET in an incomplete function
>> definition) should be fixed in the indentation rules. The contents of
>> electric-indent-chars won't fix it either way.
> 
> This is all down to indentation rules, that's how electric-indent-mode
> decides what to do.  My point is that having electric-indent-chars be
> this ambitious with "broken" indentation rules isn't a good place to
> be.

In this instance, removing chars off electric-indent-chars won't conceal 
the problem: the user can still type RET or press TAB and see unexpected 
indentation where Emacs should have been able to guess the correct one.

> What counts as "broken" indentation is also arguable though. When dealing
> with invalid programs, there is really no "right" or "wrong" indentation.
> See my message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14 where
> I show cases where c++-ts-mode's answer to indenting an invalid program
> makes more sense than c++-mode's answer.

It's not cut-and-dried indeed, but historically, with the "native" major 
modes we, wittingly or not, have used the principle that code fully 
typed until point is considered "decidable", even if some missing code 
after point makes the it incomplete. Even though, on rare occasions, 
continuing to type might change the indentation again (e.g. for "case 
labels", if the indent style dedents them).

> Whatever the indentation rules, the current bouncing is so jarring
> that it really doesn't encourage people to try switching to
> c++-ts-mode, get used to its set of indentation rules, and then perhaps
> experience its other benefits like, say, performance or simplicity.
> 
> At least it didn't for me.  I'm back to c++-mode atm.
> 
> In my opinion electric-indent-char should be reduced to the default
> and should be added criteriously as the indentation rules they trigger
> are fixed.

We should probably revisit this after an honest attempt to fix these two 
cases.

Re: treesit indentation "blinking"

[João Távora]

On Thu, Mar 30, 2023 at 5:29 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> In this instance, removing chars off electric-indent-chars won't conceal
> the problem: the user can still type RET or press TAB and see unexpected
> indentation where Emacs should have been able to guess the correct one.

electric-indent-chars exacerbates an existing problem.

IOW, it's much easier to live with incorrect indentation for temporarily
invalid programs if it's not shuffling the ground under your feed as
you type, which I think is exactly what was reported here.

It's perfectly analogous with electric-pair-mode.  It only makes sense
to turn it on if the syntax code counting openers and closers is doing
its job.  Else it's just a nuisance.

So it only makes sense to have an ambitious
electric-indent-chars or even electric-indent-mode at all if the indentation
rules are solid and predictable.  Which they are to a certain extent in c+
+-mode and to much lesser extent in c++-ts-mode.

And even if c++-ts-mode's could be made easily predictable, making them
match c++-mode's exactly probably still another job.  One that I
personally wouldn't bother with.  I don't know if that's the goal.

> > What counts as "broken" indentation is also arguable though. When dealing
> > with invalid programs, there is really no "right" or "wrong" indentation.
> > See my message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62412#14 where
> > I show cases where c++-ts-mode's answer to indenting an invalid program
> > makes more sense than c++-mode's answer.
>
> It's not cut-and-dried indeed, but historically, with the "native" major
> modes we, wittingly or not, have used the principle that code fully
> typed until point is considered "decidable", even if some missing code
> after point makes the it incomplete. Even though, on rare occasions,
> continuing to type might change the indentation again (e.g. for "case
> labels", if the indent style dedents them).

Yes, true.  But as Stefan once pointed out, that's just an arbitrary
decision.  Though I agree it's a fairly consistent one.  Probably because
I'm also used to it.  It could be the other way around and look backwards
leaning towards facilitating editing of code instead of typing new code.

> > Whatever the indentation rules, the current bouncing is so jarring
> > that it really doesn't encourage people to try switching to
> > c++-ts-mode, get used to its set of indentation rules, and then perhaps
> > experience its other benefits like, say, performance or simplicity.
> >
> > At least it didn't for me.  I'm back to c++-mode atm.
> >
> > In my opinion electric-indent-char should be reduced to the default
> > and should be added criteriously as the indentation rules they trigger
> > are fixed.
>
> We should probably revisit this after an honest attempt to fix these two
> cases.

Works for me.  My point is that I won't be using much c++-ts-mode in the
meantime. My guess is many users may be silently switching to it and
then back away to c++-mode in part due to this misbehaving electricity.
My point is that the mode should be more inviting to try out.

João

Re: treesit indentation "blinking"

[Yuan Fu]

> On Mar 30, 2023, at 7:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Herman, Géza <geza.herman@gmail.com>
>> Cc: João Távora <joaotavora@gmail.com>,
>> dgutov@yandex.ru, dancol@dancol.org,
>> casouri@gmail.com, theo@thornhill.no, emacs-devel@gnu.org
>> Date: Thu, 30 Mar 2023 12:26:16 +0200
>> 
>> Type this example into a c++-ts-mode buffer (I used "emacs -Q"):
>> 
>> --8<---------------cut here---------------start------------->8---
>> int main() {
>> for (;;) {
>> printf("Hello world\n");
>> }
>> }
>> --8<---------------cut here---------------end--------------->8---
>> 
>> This is how it will be indented as I wrote it here. c++-ts-mode doesn't
>> re-indent anything during typing, even though it had the chance to do
>> that, because electric-indent-mode is enabled by default, and
>> electric-indent-chars contains the necessary characters.
>> 
>> Or, another example. Put the "void foo() { }" part first into a
>> c++-ts-mode buffer, then write the main function:
>> 
>> --8<---------------cut here---------------start------------->8---
>> int main() {
>> int a = 0;
>> for (;;) {
>> printf("Hello!\n");
>> }
>> }
>> 
>> void foo() {
>> }
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Again, c++-ts-mode doesn't indent anything.
>> 
>> If you change the example to contain "void foo();" instead of "void
>> foo() { }", then indenting happens during typing the main function.
>> 
>> If you try adding "int a = 0;" into the first example, then it will be
>> indented at typing the ";". But then for loop is still not get indented,
>> and the buffer will look like this:
>> 
>> --8<---------------cut here---------------start------------->8---
>> int main() {
>>  int a = 0;
>> for (;;) {
>> printf("Hello world\n");
>> }
>> }
>> --8<---------------cut here---------------end--------------->8---
>> 
>> If you start by adding an empty comment, and then write the first
>> example above the empty comment, the buffer will look like this:
>> 
>> --8<---------------cut here---------------start------------->8---
>> int main() {
>>  for (;;) {
>> printf("Hello world\n");
>>  }
>> }
>> 
>> /**/
>> --8<---------------cut here---------------end--------------->8---
> 
> Yuan and Theo, can you please look into these examples?  A simple
> "C-c C-q" fixes the indentation, but the question is why it doesn't
> happen while typing?

I’ll have a look!

Yuan

Re: treesit indentation "blinking"

[Yuan Fu]

> On Apr 1, 2023, at 12:39 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Mar 30, 2023, at 7:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>>> From: Herman, Géza <geza.herman@gmail.com>
>>> Cc: João Távora <joaotavora@gmail.com>,
>>> dgutov@yandex.ru, dancol@dancol.org,
>>> casouri@gmail.com, theo@thornhill.no, emacs-devel@gnu.org
>>> Date: Thu, 30 Mar 2023 12:26:16 +0200
>>> 
>>> Type this example into a c++-ts-mode buffer (I used "emacs -Q"):
>>> 
>>> --8<---------------cut here---------------start------------->8---
>>> int main() {
>>> for (;;) {
>>> printf("Hello world\n");
>>> }
>>> }
>>> --8<---------------cut here---------------end--------------->8---
>>> 
>>> This is how it will be indented as I wrote it here. c++-ts-mode doesn't
>>> re-indent anything during typing, even though it had the chance to do
>>> that, because electric-indent-mode is enabled by default, and
>>> electric-indent-chars contains the necessary characters.
>>> 
>>> Or, another example. Put the "void foo() { }" part first into a
>>> c++-ts-mode buffer, then write the main function:
>>> 
>>> --8<---------------cut here---------------start------------->8---
>>> int main() {
>>> int a = 0;
>>> for (;;) {
>>> printf("Hello!\n");
>>> }
>>> }
>>> 
>>> void foo() {
>>> }
>>> --8<---------------cut here---------------end--------------->8---
>>> 
>>> Again, c++-ts-mode doesn't indent anything.
>>> 
>>> If you change the example to contain "void foo();" instead of "void
>>> foo() { }", then indenting happens during typing the main function.
>>> 
>>> If you try adding "int a = 0;" into the first example, then it will be
>>> indented at typing the ";". But then for loop is still not get indented,
>>> and the buffer will look like this:
>>> 
>>> --8<---------------cut here---------------start------------->8---
>>> int main() {
>>> int a = 0;
>>> for (;;) {
>>> printf("Hello world\n");
>>> }
>>> }
>>> --8<---------------cut here---------------end--------------->8---
>>> 
>>> If you start by adding an empty comment, and then write the first
>>> example above the empty comment, the buffer will look like this:
>>> 
>>> --8<---------------cut here---------------start------------->8---
>>> int main() {
>>> for (;;) {
>>> printf("Hello world\n");
>>> }
>>> }
>>> 
>>> /**/
>>> --8<---------------cut here---------------end--------------->8---
>> 
>> Yuan and Theo, can you please look into these examples?  A simple
>> "C-c C-q" fixes the indentation, but the question is why it doesn't
>> happen while typing?
> 
> I’ll have a look!

Ok, the reason is that while you were typing, because the closing bracket was missing, the parser couldn’t produce a “good” parse tree. OTOH, the C parser can reasonably fix the source and produce something easy to use by our indenting rules (perhaps due to its simpler grammar) so you don’t see this problem in c-ts-mode.

If you enable electric-pair-mode so that the brackets are always balanced, then both c-ts-mode and c++-ts-mode should work fine.

Trying to solve this with heuristics is going against to grain, IMO. I think it’s acceptable to say that users of ts-modes should enabled electric-pair-mode, since it’s based on a parser, after all.

As for the blinking, I’ll let Theo to assess the solutions mentioned earlier (removing some indentation rules set on error nodes). 

Yuan

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 1 Apr 2023 18:49:46 -0700
> Cc: theodor thornhill <theo@thornhill.no>,
>  geza.herman@gmail.com,
>  Daniel Colascione <dancol@dancol.org>,
>  emacs-devel@gnu.org
> 
> If you enable electric-pair-mode so that the brackets are always balanced, then both c-ts-mode and c++-ts-mode should work fine.
> 
> Trying to solve this with heuristics is going against to grain, IMO. I think it’s acceptable to say that users of ts-modes should enabled electric-pair-mode, since it’s based on a parser, after all.

I'm okay with that conclusion.  An alternative is to use "C-c C-q"
after typing the final brace.

Re: treesit indentation "blinking"

[Alan Mackenzie]

Hello, Yuan.

On Sat, Apr 01, 2023 at 18:49:46 -0700, Yuan Fu wrote:


> > On Apr 1, 2023, at 12:39 PM, Yuan Fu <casouri@gmail.com> wrote:



> >> On Mar 30, 2023, at 7:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> >>> From: Herman, Géza <geza.herman@gmail.com>
> >>> Cc: João Távora <joaotavora@gmail.com>,
> >>> dgutov@yandex.ru, dancol@dancol.org,
> >>> casouri@gmail.com, theo@thornhill.no, emacs-devel@gnu.org
> >>> Date: Thu, 30 Mar 2023 12:26:16 +0200

> >>> Type this example into a c++-ts-mode buffer (I used "emacs -Q"):

> >>> --8<---------------cut here---------------start------------->8---
> >>> int main() {
> >>> for (;;) {
> >>> printf("Hello world\n");
> >>> }
> >>> }
> >>> --8<---------------cut here---------------end--------------->8---

> >>> This is how it will be indented as I wrote it here. c++-ts-mode doesn't
> >>> re-indent anything during typing, even though it had the chance to do
> >>> that, because electric-indent-mode is enabled by default, and
> >>> electric-indent-chars contains the necessary characters.

> >>> Or, another example. Put the "void foo() { }" part first into a
> >>> c++-ts-mode buffer, then write the main function:

> >>> --8<---------------cut here---------------start------------->8---
> >>> int main() {
> >>> int a = 0;
> >>> for (;;) {
> >>> printf("Hello!\n");
> >>> }
> >>> }

> >>> void foo() {
> >>> }
> >>> --8<---------------cut here---------------end--------------->8---

> >>> Again, c++-ts-mode doesn't indent anything.

> >>> If you change the example to contain "void foo();" instead of "void
> >>> foo() { }", then indenting happens during typing the main function.

> >>> If you try adding "int a = 0;" into the first example, then it will be
> >>> indented at typing the ";". But then for loop is still not get indented,
> >>> and the buffer will look like this:

> >>> --8<---------------cut here---------------start------------->8---
> >>> int main() {
> >>> int a = 0;
> >>> for (;;) {
> >>> printf("Hello world\n");
> >>> }
> >>> }
> >>> --8<---------------cut here---------------end--------------->8---

> >>> If you start by adding an empty comment, and then write the first
> >>> example above the empty comment, the buffer will look like this:

> >>> --8<---------------cut here---------------start------------->8---
> >>> int main() {
> >>> for (;;) {
> >>> printf("Hello world\n");
> >>> }
> >>> }

> >>> /**/
> >>> --8<---------------cut here---------------end--------------->8---

> >> Yuan and Theo, can you please look into these examples?  A simple
> >> "C-c C-q" fixes the indentation, but the question is why it doesn't
> >> happen while typing?

> > I’ll have a look!

> Ok, the reason is that while you were typing, because the closing
> bracket was missing, the parser couldn’t produce a “good” parse tree.
> OTOH, the C parser can reasonably fix the source and produce something
> easy to use by our indenting rules (perhaps due to its simpler grammar)
> so you don’t see this problem in c-ts-mode.

> If you enable electric-pair-mode so that the brackets are always
> balanced, then both c-ts-mode and c++-ts-mode should work fine.

That is not a solution.  Some people dislike electric-pair-mode, and they
would be faced with the unwelcome choice of having to put up with e-p-m
or having source code not indenting (or jumping all over the place) on
the text line being input.

> Trying to solve this with heuristics is going against to grain, IMO.

Does the parser not produce adequate information for source code which
isn't syntactically valid (which is most of the time when editing is in
progress)?

> I think it’s acceptable to say that users of ts-modes should enable
> electric-pair-mode, since it’s based on a parser, after all.

electric-pair-mode is a user option.  We shouldn't be mandating such
things to users, they should be individual choices.  I've had to use a
proprietary editor where e-p-m couldn't be disabled (or at least I didn't
know how to), and I hated it.  Emacs should be better than such editors.

That electric-pair-mode isn't needed to get reasonable electric
indentation is demonstrated by CC Mode, where electric indentation occurs
when needed, but rarely happens more than once per source line.

> As for the blinking, I’ll let Theo to assess the solutions mentioned
> earlier (removing some indentation rules set on error nodes). 

> Yuan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: treesit indentation "blinking"

[João Távora]

On Sun, Apr 2, 2023 at 3:26 PM Alan Mackenzie <acm@muc.de> wrote:

> > I think it’s acceptable to say that users of ts-modes should enable
> > electric-pair-mode, since it’s based on a parser, after all.
>
> electric-pair-mode is a user option.  We shouldn't be mandating such
> things to users, they should be individual choices.

Fair enough.  But so is electric-indent-mode and its electric-indent-chars
which are problematic in c++-ts-mode and they _are_ enabled by default.

electric-pair-mode not only is unproblematic in c++-ts-mode (at least, as
far as we know) but is proven to be a good (though not perfect) defense
against the real problems posed by the default value of electric-indent-mode
and the default value of electric-indent-chars in c++-ts-mode specifically.

So it makes sense to either have both e-p-m and e-i-m or none (or at least
less of the second as has been suggested).  At least until the presumed
indentation bugs (if in fact they are bugs at all) are fixed (if in fact
there is an easy fix for them).

The current default state of c++-ts-mode makes little sense to me, it's
very uninviting and strange when compared to c++-mode _or_ other editors.

( Admittedly, not only for this reason, the C-M-f/C-M-b/C-M-a behaviour
is unlike anything I've ever seen in Emacs. )

> I've had to use a
> proprietary editor where e-p-m couldn't be disabled (or at least I didn't
> know how to), and I hated it.  Emacs should be better than such editors.

This comparison doesn't make sense to me because in Emacs it's easy to
disable it.

João

Re: treesit indentation "blinking"

[Alan Mackenzie]

Hello, João.

On Sun, Apr 02, 2023 at 16:48:55 +0100, João Távora wrote:
> On Sun, Apr 2, 2023 at 3:26 PM Alan Mackenzie <acm@muc.de> wrote:

> > > I think it’s acceptable to say that users of ts-modes should enable
> > > electric-pair-mode, since it’s based on a parser, after all.

> > electric-pair-mode is a user option.  We shouldn't be mandating such
> > things to users, they should be individual choices.

> Fair enough.  But so is electric-indent-mode and its electric-indent-chars
> which are problematic in c++-ts-mode and they _are_ enabled by default.

Yes.  Users should be able to chose from amongst these options.  They
shouldn't be "problematic", and that was the point of my last post.

> electric-pair-mode not only is unproblematic in c++-ts-mode (at least, as
> far as we know) but is proven to be a good (though not perfect) defense
> against the real problems posed by the default value of electric-indent-mode
> and the default value of electric-indent-chars in c++-ts-mode specifically.

It's not a "good defense" for somebody, like me, who doesn't like it.
It's not good for Emacs for options only to work when distinct other
options are also enabled.

> So it makes sense to either have both e-p-m and e-i-m or none (or at least
> less of the second as has been suggested).

Not from a user's point of view.  These two modes are wholly independent
of each other (from that user's point of view), and she should be able
to enable either or both, as desired, and have them work properly.

> At least until the presumed indentation bugs (if in fact they are bugs
> at all) are fixed (if in fact there is an easy fix for them).

I think it is clear there are bugs here.  Electric indentation isn't
working.

[ .... ]

> > I've had to use a proprietary editor where e-p-m couldn't be
> > disabled (or at least I didn't know how to), and I hated it.  Emacs
> > should be better than such editors.

> This comparison doesn't make sense to me because in Emacs it's easy to
> disable it.

But disabling it hits the bugs in electric indentation.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: treesit indentation "blinking"

[João Távora]

On Sun, Apr 2, 2023 at 6:04 PM Alan Mackenzie <acm@muc.de> wrote:

> > > electric-pair-mode is a user option.  We shouldn't be mandating such
> > > things to users, they should be individual choices.
>
> > Fair enough.  But so is electric-indent-mode and its electric-indent-chars
> > which are problematic in c++-ts-mode and they _are_ enabled by default.
>
> Yes.  Users should be able to chose from amongst these options.  They
> shouldn't be "problematic", and that was the point of my last post.

And my point was underlying that electric-indent-chars and electric-indent
mode is currently being "mandated", so to be consistent with your principle
we might as well "mandate" neither of them.

> > electric-pair-mode not only is unproblematic in c++-ts-mode (at least, as
> > far as we know) but is proven to be a good (though not perfect) defense
> > against the real problems posed by the default value of electric-indent-mode
> > and the default value of electric-indent-chars in c++-ts-mode specifically.
>
> It's not a "good defense" for somebody, like me, who doesn't like it.
> It's not good for Emacs for options only to work when distinct other
> options are also enabled.
>
> > So it makes sense to either have both e-p-m and e-i-m or none (or at least
> > less of the second as has been suggested).
>
> Not from a user's point of view.  These two modes are wholly independent
> of each other (from that user's point of view), and she should be able
> to enable either or both, as desired, and have them work properly.

But they don't, at least not at the moment.  That's a fact.

Of the two modes, one of them is working wholly inaccurately and not doing
any good, and that mode is electric-indent-mode.  Because of a combination
of the ambitious value of electric-indent-chars _and_ the underlying
indentation rules problems.

So my initial idea was to tone down electric-indent-chars, at least
for the moment.  And Dmitry's idea was to make electric-indent-chars
be ambitious _only_ if electric-pair-mode is enabled (by the user).
Maybe we should bring back that idea, and it seems the least bad of the
bunch right now.

> > At least until the presumed indentation bugs (if in fact they are bugs
> > at all) are fixed (if in fact there is an easy fix for them).
>
> I think it is clear there are bugs here.  Electric indentation isn't
> working.

Yes.

If we take c++-mode to be the absolute reference, no it isn't, and
it's unlikely that it ever will.  If we take a more modest stance
and think of other editors that don't indent exactly like c++-mode,
then the issue is more subtle.

It's important to state that AFAIK when the buffer is correctly balanced,
c++-ts-mode's indentation is flawless.  We're talking about invalid
situations here, and there it's harder to know what the "good" indentation
is, but a nice rule-of-thumb could be "don't do anything".  Another
rule would be: do exactly what c++-mode does, but that's a taller order
and I don't think we should be aiming for it.

> > This comparison doesn't make sense to me because in Emacs it's easy to
> > disable it.
>
> But disabling it hits the bugs in electric indentation.

Yes! :-) So you'd tone down electric-indent-chars, but that's been turned
down.  So options are disable electric-indent-mode entirely in c++-ts-mode,
enable electric-pair-mode, or live with poor ergonomics in emacs-29
until the bugs are fixed.

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 2 Apr 2023 18:23:57 +0100
> Cc: Yuan Fu <casouri@gmail.com>, Eli Zaretskii <eliz@gnu.org>, 
> 	theodor thornhill <theo@thornhill.no>, geza.herman@gmail.com, 
> 	Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> 
> So you'd tone down electric-indent-chars, but that's been turned
> down.

Doing that _by_default_ was turned down.  Users can still do that in
their mode hooks if they want.  So nothing is lost, since this seems
to be heavily affected by personal preferences and by whether or not
electric-pair-mode is turned on.

Re: treesit indentation "blinking"

[João Távora]

On Sun, Apr 2, 2023 at 6:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sun, 2 Apr 2023 18:23:57 +0100
> > Cc: Yuan Fu <casouri@gmail.com>, Eli Zaretskii <eliz@gnu.org>,
> >       theodor thornhill <theo@thornhill.no>, geza.herman@gmail.com,
> >       Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> >
> > So you'd tone down electric-indent-chars, but that's been turned
> > down.
>
> Doing that _by_default_ was turned down.  Users can still do that in
> their mode hooks if they want.  So nothing is lost, since this seems
> to be heavily affected by personal preferences and by whether or not
> electric-pair-mode is turned on.

I am precisely talking about the default state of c++-ts-mode.
I don't think there is anyone who really likes it at the moment.
This is regardless of personal preference w.r.t. electric indentation:
c++-mode's default state in that area is much more consistent.

Maybe I'm wrong, or maybe when these bugs are fixed, stock/default
c++-ts-mode will be better.  Currently it is not so nice.

João

Re: treesit indentation "blinking"

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 2 Apr 2023 19:04:42 +0100
> Cc: acm@muc.de, casouri@gmail.com, theo@thornhill.no, geza.herman@gmail.com, 
> 	dancol@dancol.org, emacs-devel@gnu.org, dgutov@yandex.ru
> 
> On Sun, Apr 2, 2023 at 6:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Sun, 2 Apr 2023 18:23:57 +0100
> > > Cc: Yuan Fu <casouri@gmail.com>, Eli Zaretskii <eliz@gnu.org>,
> > >       theodor thornhill <theo@thornhill.no>, geza.herman@gmail.com,
> > >       Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> > >
> > > So you'd tone down electric-indent-chars, but that's been turned
> > > down.
> >
> > Doing that _by_default_ was turned down.  Users can still do that in
> > their mode hooks if they want.  So nothing is lost, since this seems
> > to be heavily affected by personal preferences and by whether or not
> > electric-pair-mode is turned on.
> 
> I am precisely talking about the default state of c++-ts-mode.
> I don't think there is anyone who really likes it at the moment.

We need more opinions and feedback before we know whether and how to
change the defaults.

> This is regardless of personal preference w.r.t. electric indentation:
> c++-mode's default state in that area is much more consistent.

c++-mode's default state is not relevant to this discussion.  It's a
different mode with different design and different advantages and
disadvantages.  It also has much more history behind it.

> Maybe I'm wrong, or maybe when these bugs are fixed, stock/default
> c++-ts-mode will be better.  Currently it is not so nice.

Your opinions are noted.

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 02/04/2023 20:23, João Távora wrote:
> So my initial idea was to tone down electric-indent-chars, at least
> for the moment.  And Dmitry's idea was to make electric-indent-chars
> be ambitious_only_  if electric-pair-mode is enabled (by the user).
> Maybe we should bring back that idea, and it seems the least bad of the
> bunch right now.

Alternatively, we only perform "electric indent" (aside from after RET) 
when the parse tree does not contain errors. Using the hook 
electric-indent-functions instead of altering electric-indent-chars.

Re: treesit indentation "blinking"

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is regardless of personal preference w.r.t. electric indentation:
>> c++-mode's default state in that area is much more consistent.
>
> c++-mode's default state is not relevant to this discussion.  It's a
> different mode with different design and different advantages and
> disadvantages.  

I drew this comparison because earlier in the conversation you cited
c++-mode's defaults w.r.t. electric indenting to justify c++-ts-mode's
defaults regarding the same functionality.

I have no opinion on what the defaults of this new mode should be except
that they should be congruent between themselves.

João

Re: treesit indentation "blinking"

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 02/04/2023 20:23, João Távora wrote:
>> So my initial idea was to tone down electric-indent-chars, at least
>> for the moment.  And Dmitry's idea was to make electric-indent-chars
>> be ambitious _only_ if electric-pair-mode is enabled (by the user).
>> Maybe we should bring back that idea, and it seems the least bad of
>> the bunch right now.
>
> Alternatively, we only perform "electric indent" (aside from after
> RET) when the parse tree does not contain errors. Using the hook
> electric-indent-functions instead of altering electric-indent-chars.

If easy to do, that's even better.  Then electric-pair-mode doesn't need
to enter the picture at all.

João

Re: treesit indentation "blinking"

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
> On 02/04/2023 20:23, João Távora wrote:
> > So my initial idea was to tone down electric-indent-chars, at least
> > for the moment.  And Dmitry's idea was to make electric-indent-chars
> > be ambitious_only_  if electric-pair-mode is enabled (by the user).
> > Maybe we should bring back that idea, and it seems the least bad of the
> > bunch right now.

> Alternatively, we only perform "electric indent" (aside from after RET) 
> when the parse tree does not contain errors.

That is NOT electric indentation.  The whole point about electric
indentation is for it to take effect whilst point is still on the line
being edited.  Thus, for example, you can see whether or not the line
needs breaking, or whether there's room for a short comment at the end
of the line.

What you're proposing is something which would almost never trigger,
since a line being edited will not have a parse tree without errors (if
I've understood that properly).  If it did trigger at some point, that
would likely cause annoyance and puzzlement.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: treesit indentation "blinking"

[João Távora]

On Mon, Apr 3, 2023 at 10:59 AM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, Dmitry.
>
> On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
> > On 02/04/2023 20:23, João Távora wrote:
> > > So my initial idea was to tone down electric-indent-chars, at least
> > > for the moment.  And Dmitry's idea was to make electric-indent-chars
> > > be ambitious_only_  if electric-pair-mode is enabled (by the user).
> > > Maybe we should bring back that idea, and it seems the least bad of the
> > > bunch right now.
>
> > Alternatively, we only perform "electric indent" (aside from after RET)
> > when the parse tree does not contain errors.
>
> That is NOT electric indentation.  The whole point about electric
> indentation is for it to take effect whilst point is still on the line
> being edited.

I can't speak for Dmitry, but I think that's precisely what is in mind
here.  Say I have this buffer and point is on the end of the second line:

int foo() {
something()
}

As soon as I type the ';' at the end of "something", the parse tree is
valid and electric-indent kicks in.  In this buffer:

int foo() {
  for() something();
}

Then "blinking" would not happen if you add two ;; inside the for's
parens, as it does now, because as soon as soon as I add one, the
parse tree becomes invalid (presumably) and electric-indent does not
kick in.

João

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 03/04/2023 12:59, Alan Mackenzie wrote:
> Hello, Dmitry.
> 
> On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
>> On 02/04/2023 20:23, João Távora wrote:
>>> So my initial idea was to tone down electric-indent-chars, at least
>>> for the moment.  And Dmitry's idea was to make electric-indent-chars
>>> be ambitious_only_  if electric-pair-mode is enabled (by the user).
>>> Maybe we should bring back that idea, and it seems the least bad of the
>>> bunch right now.
> 
>> Alternatively, we only perform "electric indent" (aside from after RET)
>> when the parse tree does not contain errors.
> 
> That is NOT electric indentation.  The whole point about electric
> indentation is for it to take effect whilst point is still on the line
> being edited.  Thus, for example, you can see whether or not the line
> needs breaking, or whether there's room for a short comment at the end
> of the line.

Wouldn't you know whether the line needs breaking, as long as the line 
was indented correctly when you opened it with RET?

> What you're proposing is something which would almost never trigger,
> since a line being edited will not have a parse tree without errors (if
> I've understood that properly).  If it did trigger at some point, that
> would likely cause annoyance and puzzlement.

That's a fair assessment, but it's going to trigger in a lot of cases 
still: after ;, or after a paren or brace being closed.

This is not a substitute for enabling electric-pair-mode, alas. Just a 
less hackish way to check the same thing rather than check for this 
particular mode being enabled.

I would also prefer c++-ts-mode supported indentation with closing 
braces missing, but we're really limited by what the parser provides. 
E.g. for this code

   int foo() {
     for (;;) {

we get this parse tree:

   (ERROR (primitive_type)
    (function_declarator declarator: (identifier)
     parameters: (parameter_list ( )))
     { for ( ; ; ) {)

where the "for" statement isn't even present (the separate tokens are 
parsed as leaf nodes, and that's it). It's difficult to write meaningful 
indentation rules that would match this.

Re: treesit indentation "blinking"

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Apr 03, 2023 at 15:07:11 +0300, Dmitry Gutov wrote:
> On 03/04/2023 12:59, Alan Mackenzie wrote:
> > Hello, Dmitry.

> > On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
> >> On 02/04/2023 20:23, João Távora wrote:
> >>> So my initial idea was to tone down electric-indent-chars, at least
> >>> for the moment.  And Dmitry's idea was to make electric-indent-chars
> >>> be ambitious_only_  if electric-pair-mode is enabled (by the user).
> >>> Maybe we should bring back that idea, and it seems the least bad of the
> >>> bunch right now.

> >> Alternatively, we only perform "electric indent" (aside from after RET)
> >> when the parse tree does not contain errors.

> > That is NOT electric indentation.  The whole point about electric
> > indentation is for it to take effect whilst point is still on the line
> > being edited.  Thus, for example, you can see whether or not the line
> > needs breaking, or whether there's room for a short comment at the end
> > of the line.

> Wouldn't you know whether the line needs breaking, as long as the line 
> was indented correctly when you opened it with RET?

Maybe sometimes, but often not.

> > What you're proposing is something which would almost never trigger,
> > since a line being edited will not have a parse tree without errors (if
> > I've understood that properly).  If it did trigger at some point, that
> > would likely cause annoyance and puzzlement.

> That's a fair assessment, but it's going to trigger in a lot of cases 
> still: after ;, or after a paren or brace being closed.

OK.

> This is not a substitute for enabling electric-pair-mode, alas.

As you've no doubt gathered, I particularly dislike electric-pair-mode.
I'm likely far from rare in that respect.  I think we'll be doing our
users a disservice if indentation only works when e-p-m is enabled.

> Just a less hackish way to check the same thing rather than check for
> this particular mode being enabled.

> I would also prefer c++-ts-mode supported indentation with closing 
> braces missing, but we're really limited by what the parser provides. 
> E.g. for this code

>    int foo() {
>      for (;;) {

> we get this parse tree:

>    (ERROR (primitive_type)
>     (function_declarator declarator: (identifier)
>      parameters: (parameter_list ( )))
>      { for ( ; ; ) {)

> where the "for" statement isn't even present (the separate tokens are 
> parsed as leaf nodes, and that's it). It's difficult to write meaningful 
> indentation rules that would match this.

Why do we get a parse tree with ERROR in it when the source isn't
erroneous?  It is merely incomplete.  Tree sitter was surely designed for
editors, where source code being entered is typically incomplete, not
just for things like code formatters.  Why do we not get a full valid
parse tree indicating the current (incomplete) state?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: treesit indentation "blinking"

[Dmitry Gutov]

Hi Alan,

On 03/04/2023 15:56, Alan Mackenzie wrote:
>>> That is NOT electric indentation.  The whole point about electric
>>> indentation is for it to take effect whilst point is still on the line
>>> being edited.  Thus, for example, you can see whether or not the line
>>> needs breaking, or whether there's room for a short comment at the end
>>> of the line.
> 
>> Wouldn't you know whether the line needs breaking, as long as the line
>> was indented correctly when you opened it with RET?
> 
> Maybe sometimes, but often not.

I think it's the exceptions that are relatively rare, mostly labels 
(goto and switch/case). I'm no C++ expert, though.

>> This is not a substitute for enabling electric-pair-mode, alas.
> 
> As you've no doubt gathered, I particularly dislike electric-pair-mode.
> I'm likely far from rare in that respect.  I think we'll be doing our
> users a disservice if indentation only works when e-p-m is enabled.

Be that as it may, improving the state of affairs looks difficult. It 
will likely involve taking part in C and C++'s grammars development, and 
even then full success is not guaranteed.

I've tried to make indentation's behavior better in ruby-ts-mode is 
similar circumstances, but the sole difference between it and c++ indent 
rules is that is has a "catch all" rule which, for code like

   def foo
     if asd

, does indent 'if' one level. And Ruby code is usually more nested anyway.

Also note that the C++ example starts working much better as soon as 
there is at least one closing brace, e.g.

int foo() {
   for (;;) {
     |

}

because in this situation both the declaration's bounds and the for 
node's bounds are more or less easily determined.

In practice, it should mean that as long as the top-level definition is 
"closed" (perhaps manually, without electric-pair-mode), you can type 
inside an enjoy much better auto-indentation. Although maybe some 
exceptions will remain still (some further research could be useful).

>> I would also prefer c++-ts-mode supported indentation with closing
>> braces missing, but we're really limited by what the parser provides.
>> E.g. for this code
> 
>>     int foo() {
>>       for (;;) {
> 
>> we get this parse tree:
> 
>>     (ERROR (primitive_type)
>>      (function_declarator declarator: (identifier)
>>       parameters: (parameter_list ( )))
>>       { for ( ; ; ) {)
> 
>> where the "for" statement isn't even present (the separate tokens are
>> parsed as leaf nodes, and that's it). It's difficult to write meaningful
>> indentation rules that would match this.
> 
> Why do we get a parse tree with ERROR in it when the source isn't
> erroneous?  It is merely incomplete.

It is incomplete and hard to make sense of, apparently. But perhaps 
someone could contribute improvements to the parser that would make 
things better.

> Tree sitter was surely designed for
> editors, where source code being entered is typically incomplete, not
> just for things like code formatters.  Why do we not get a full valid
> parse tree indicating the current (incomplete) state?

The parser's tolerance has limits, aparently. But see my latest example: 
that code is still incomplete, and yet it parses much better.

Most code editing is additive, rather writing functions from scratch. So 
I'm guessing it might work out, most of the time.

parser error recovery algorithm vs treesit indentation "blinking"

[Stephen Leake]

Alan Mackenzie <acm@muc.de> writes:

> Does the <tree-sitter> parser not produce adequate information for source code which
> isn't syntactically valid (which is most of the time when editing is in
> progress)?

It often does, but also often does not. The error recovery algorithm in
tree-sitter is apparently not good enough for this use case (indentation
with a missing close block symbol). That's because the tree-sitter
algorithm does not insert symbols, it only skips them.

The error recovery algorithm in the wisitoken parser used by ada-mode
does work well in this case; it inserts the missing close block symbol
(and in general inserts other missing symbols).

It would be interesting to convert the C++ grammar used by tree-sitter
to a wisitoken grammar, and see how well that works. I don't have time
or energy for that, but I could support someone else doing it.

I am currently working on using tree-sitter as an ada-mode backend
parser, so we will be able to compare the two parser's indent behavior
using Ada test cases.

-- 
-- Stephe

Re: treesit indentation "blinking"

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

On April 3, 2023 08:07:17 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 03/04/2023 12:59, Alan Mackenzie wrote:
>> Hello, Dmitry.
>>
>> On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
>>> On 02/04/2023 20:23, João Távora wrote:
>>>> So my initial idea was to tone down electric-indent-chars, at least
>>>> for the moment.  And Dmitry's idea was to make electric-indent-chars
>>>> be ambitious_only_  if electric-pair-mode is enabled (by the user).
>>>> Maybe we should bring back that idea, and it seems the least bad of the
>>>> bunch right now.
>>
>>> Alternatively, we only perform "electric indent" (aside from after RET)
>>> when the parse tree does not contain errors.
>>
>> That is NOT electric indentation.  The whole point about electric
>> indentation is for it to take effect whilst point is still on the line
>> being edited.  Thus, for example, you can see whether or not the line
>> needs breaking, or whether there's room for a short comment at the end
>> of the line.
>
> Wouldn't you know whether the line needs breaking, as long as the line
> was indented correctly when you opened it with RET?
>
>> What you're proposing is something which would almost never trigger,
>> since a line being edited will not have a parse tree without errors (if
>> I've understood that properly).  If it did trigger at some point, that
>> would likely cause annoyance and puzzlement.
>
> That's a fair assessment, but it's going to trigger in a lot of cases
> still: after ;, or after a paren or brace being closed.

Silly question: can't we make a mode with c++-mode's indentation (and 
folding etc.) and c++-ts-mode's fontification? Such a thing would also 
preserve compatibility with the numerous ad hoc c++-mode styles out there.

[-- Attachment #2: Type: text/html, Size: 3620 bytes --]

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 04/04/2023 00:59, Daniel Colascione wrote:
> On April 3, 2023 08:07:17 Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
>> On 03/04/2023 12:59, Alan Mackenzie wrote:
>>> Hello, Dmitry.
>>>
>>> On Mon, Apr 03, 2023 at 00:21:18 +0300, Dmitry Gutov wrote:
>>>> On 02/04/2023 20:23, João Távora wrote:
>>>>> So my initial idea was to tone down electric-indent-chars, at least
>>>>> for the moment.  And Dmitry's idea was to make electric-indent-chars
>>>>> be ambitious_only_  if electric-pair-mode is enabled (by the user).
>>>>> Maybe we should bring back that idea, and it seems the least bad of the
>>>>> bunch right now.
>>>
>>>> Alternatively, we only perform "electric indent" (aside from after RET)
>>>> when the parse tree does not contain errors.
>>>
>>> That is NOT electric indentation.  The whole point about electric
>>> indentation is for it to take effect whilst point is still on the line
>>> being edited.  Thus, for example, you can see whether or not the line
>>> needs breaking, or whether there's room for a short comment at the end
>>> of the line.
>>
>> Wouldn't you know whether the line needs breaking, as long as the line
>> was indented correctly when you opened it with RET?
>>
>>> What you're proposing is something which would almost never trigger,
>>> since a line being edited will not have a parse tree without errors (if
>>> I've understood that properly).  If it did trigger at some point, that
>>> would likely cause annoyance and puzzlement.
>>
>> That's a fair assessment, but it's going to trigger in a lot of cases
>> still: after ;, or after a paren or brace being closed.
> 
> Silly question: can't we make a mode with c++-mode's indentation (and 
> folding etc.) and c++-ts-mode's fontification? Such a thing would also 
> preserve compatibility with the numerous ad hoc c++-mode styles out there.

Sure, we could: inherit from c++-mode and add then add most of 
c++-ts-mode's definition, just without indent-related variables. 
python-ts-mode is an example of such mode.

Whatever performance complaints people had about c++-mode, though, those 
will stay around, and perhaps get a little bit worse due to additional 
work of maintaining the tree-sitter parse tree.

And at least some of CC Mode's latest bug reports related to problems 
"parsing" the code (in the way that CC Mode does that). Such problems 
would be harder to spot when using the new hybrid mode because they 
won't affect syntax highlighting, but they could still affect indentation.

Re: treesit indentation "blinking"

[João Távora]

On Mon, Apr 3, 2023 at 10:59 PM Daniel Colascione <dancol@dancol.org> wrote:

> Silly question: can't we make a mode with c++-mode's indentation (and folding etc.) and c++-ts-mode's fontification? Such a thing would also preserve compatibility with the numerous ad hoc c++-mode styles out there.

To get folding to work (presuming you mean hideshow.el) just set
forward-sexp-function to nil, and then C-c @ C-c works.  C-M-f
C-M-b and C-M-a will also start working as they do in most cc-modes
and similar modes.  I don't know why c++-ts-mode has this special
forward-sexp-function.  This is bug#62302 by Herman, which has
no replies yet.

João

Re: parser error recovery algorithm vs treesit indentation "blinking"

[John Yates]

On Mon, Apr 3, 2023 at 5:49 PM Stephen Leake
<stephen_leake@stephe-leake.org> wrote:
>
> That's because the tree-sitter
> algorithm does not insert symbols, it only skips them.

Is this a fundamental architectural limitation of tree-sitter's parsing
scheme?  Was it a design decision that trying insertions would be
too costly?  Or is it an improvement that should be explored?

Has this been discussed in the wider tree-sitter community?  I would
be surprised if emacs is the first to encounter this weakness.

Re: parser error recovery algorithm vs treesit indentation "blinking"

[Dmitry Gutov]

On 04/04/2023 15:01, John Yates wrote:
> On Mon, Apr 3, 2023 at 5:49 PM Stephen Leake
> <stephen_leake@stephe-leake.org>  wrote:
>> That's because the tree-sitter
>> algorithm does not insert symbols, it only skips them.
> Is this a fundamental architectural limitation of tree-sitter's parsing
> scheme?  Was it a design decision that trying insertions would be
> too costly?  Or is it an improvement that should be explored?
> 
> Has this been discussed in the wider tree-sitter community?  I would
> be surprised if emacs is the first to encounter this weakness.

I think the answer is "it varies". E.g. this unfinished snippet actually 
parses without errors:

   int foo() {

Even creating a "virtual" closing brace node, 0 characters in length.

But insert this snippet twice (maybe with a different function name) - 
and you get errors in the parse tree.

So there is some mechanism for virtual insertion.

One could say that our indentation mechanism could be at fault here (a 
little), given that the first step is to find the node enclosing the 
current position of point. But the "virtual" closer is positioned to be 
at the end of the previous line.

It's a matter of perspective, which side the fault lies at: maybe the 
virtual closer should be positioned at EOB, then our code would work 
fine with this example. This would make a lot of sense from my POV.

But maybe we should adjust the logic in this particular case: when 
between point and EOB is only whitespace, it could look for nodes at the 
end of the last non-whitespace character, and consider that the current 
node. This will need a fair amount of testing, I think, to make sure we 
don't get false positives this way.

Here's a relevant discussion, but there's nothing about positions in 
there: https://github.com/tree-sitter/tree-sitter/issues/224

Re: parser error recovery algorithm vs treesit indentation "blinking"

[Stephen Leake]

John Yates <john@yates-sheets.org> writes:

> On Mon, Apr 3, 2023 at 5:49 PM Stephen Leake
> <stephen_leake@stephe-leake.org> wrote:
>>
>> That's because the tree-sitter
>> algorithm does not insert symbols, it only skips them.
>
> Is this a fundamental architectural limitation of tree-sitter's parsing
> scheme?  

Possibly. In the wisitoken parser, allowing insertions during error
correction significantly complicates the edit step of incremental
parsing - so much so that there are still bugs in the wisitoken parser
that I'm having a very hard time squashing.

> Was it a design decision that trying insertions would be too costly?

I don't know. There are some references on error correction on the
tree-sitter website, but they don't discuss the possibility of
insertion, and they don't discuss implications for incremental parsing.

> Or is it an improvement that should be explored?

Yes.

> Has this been discussed in the wider tree-sitter community?  I would
> be surprised if emacs is the first to encounter this weakness.

I raised it a couple times; the only response I got was "tree-sitter has
excellent error correction".

Perhaps other users of tree-sitter don't use it for indenting complex
languages.

-- 
-- Stephe

Re: parser error recovery algorithm vs treesit indentation "blinking"

[Dmitry Gutov]

On 04/04/2023 16:50, Stephen Leake wrote:
>> Has this been discussed in the wider tree-sitter community?  I would
>> be surprised if emacs is the first to encounter this weakness.
> I raised it a couple times; the only response I got was "tree-sitter has
> excellent error correction".
> 
> Perhaps other users of tree-sitter don't use it for indenting complex
> languages.

This one also looks relevant: 
https://github.com/tree-sitter/tree-sitter/issues/1870

Re: parser error recovery algorithm vs treesit indentation "blinking"

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Here's a relevant discussion, but there's nothing about positions in
> there: https://github.com/tree-sitter/tree-sitter/issues/224

Ah. So tree-sitter does do some insertion, but not as much as wisitoken
wisitoken is also GLR, and uses the same approach to multiple error
correction solutions, although the cost of insert/delete depends on the
token; for example, it's cheaper to insert or delete parentheses, since
they are often missing or extra while editing.

I've posted a draft paper on the wisitoken error correction algorithm;
https://stephe-leake.org/ada/error_correction_algorithm.pdf

In the full algorithm used by ada-mode, there is some consideration
about whether to insert tokens before or after new-lines, to address the
indent issue you describe.

-- 
-- Stephe

Re: treesit indentation "blinking"

[Daniel Martín]

Daniel Colascione <dancol@dancol.org> writes:

>
> Silly question: can't we make a mode with c++-mode's indentation (and
> folding etc.) and c++-ts-mode's fontification? Such a thing would also
> preserve compatibility with the numerous ad hoc c++-mode styles out
> there.

Note that c++-ts-mode's fontification can also "blink":

i n t  m a i n ( ) { RET i n (now "main" is not fontified) t i ; (now
"main" is fontified again).

In general, I'd suggest taking a look at Neovim's Tree-sitter
implementation and perhaps reuse (part of) their .scm Tree-sitter syntax
highlight or indentation queries:
https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries/c
Neovim's implementation doesn't "blink" in the scenarios presented here,
but I'm not sure about how it performs in other tricky cases.

Now that text editors have access to the same programming language
grammars, I'm surprised the Tree-sitter community hasn't started any
cross-team effort to work on canonical queries (or slighter richer
abstractions) per programming language.  Text editor developers are
working and rediscovering complex language problems independently.

Re: treesit indentation "blinking"

[Dmitry Gutov]

On 07/04/2023 17:20, Daniel Martín wrote:
> In general, I'd suggest taking a look at Neovim's Tree-sitter
> implementation and perhaps reuse (part of) their .scm Tree-sitter syntax
> highlight or indentation queries:
> https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries/c
> Neovim's implementation doesn't "blink" in the scenarios presented here,
> but I'm not sure about how it performs in other tricky cases.

Looking at that file, it does contain queries that look for ERROR nodes, 
e.g.

(
   ERROR
     "for" "(" @indent.begin ";" ";" ")" @indent.end)

So perhaps we should revisit that approach too.

> Now that text editors have access to the same programming language
> grammars, I'm surprised the Tree-sitter community hasn't started any
> cross-team effort to work on canonical queries (or slighter richer
> abstractions) per programming language.  Text editor developers are
> working and rediscovering complex language problems independently.

We could do an indent implementation that just uses these indents.scm 
files. Not exactly sure why we didn't start out this way, but note that 
our Lisp based solution is a lot more flexible.

E.g. there doesn't seem to be an obvious way to support "indentation 
styles" with these.

Re: treesit indentation "blinking"

[Yuan Fu]

> On Apr 7, 2023, at 6:32 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 07/04/2023 17:20, Daniel Martín wrote:
>> In general, I'd suggest taking a look at Neovim's Tree-sitter
>> implementation and perhaps reuse (part of) their .scm Tree-sitter syntax
>> highlight or indentation queries:
>> https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries/c
>> Neovim's implementation doesn't "blink" in the scenarios presented here,
>> but I'm not sure about how it performs in other tricky cases.
> 
> Looking at that file, it does contain queries that look for ERROR nodes, e.g.
> 
> (
>  ERROR
>    "for" "(" @indent.begin ";" ";" ")" @indent.end)
> 
> So perhaps we should revisit that approach too.

Yeah, some more specific error handling can be only beneficial.

> 
>> Now that text editors have access to the same programming language
>> grammars, I'm surprised the Tree-sitter community hasn't started any
>> cross-team effort to work on canonical queries (or slighter richer
>> abstractions) per programming language.  Text editor developers are
>> working and rediscovering complex language problems independently.
> 
> We could do an indent implementation that just uses these indents.scm files. Not exactly sure why we didn't start out this way, but note that our Lisp based solution is a lot more flexible.
> 
> E.g. there doesn't seem to be an obvious way to support "indentation styles" with these.

The indent.scm way basically queries the whole file with all the queries, build a hash map out of the captured nodes, then get the node at the current line and go up the tree; at each parent, check if it’s a node that was captured, if so, add an indentation level.

To be honest, this will work fine in even largish files, but it’ll be slow when you indent a region instead of a line (in a large file). And as you said, it’s a bit more work to support styles. But, there are ways to make this method more efficient, by using some caching and localizing, so maybe it could work.

indent.scm definitely looks neater and more convenient to write than the rules we use (plus it already exists). So it would be nice to somehow make it compatible. I’ve been trying to conceive some way to convert the normal query into something more flexible, so that it can be used for more than querying “top-down” from a node. For example, verifying that a node would be captured as xxx in a query.

Ideas and patches are welcome :-)

Yuan

Re: treesit indentation "blinking"

[Daniel Martín]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 07/04/2023 17:20, Daniel Martín wrote:
>> In general, I'd suggest taking a look at Neovim's Tree-sitter
>> implementation and perhaps reuse (part of) their .scm Tree-sitter syntax
>> highlight or indentation queries:
>> https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries/c
>> Neovim's implementation doesn't "blink" in the scenarios presented here,
>> but I'm not sure about how it performs in other tricky cases.
>
> Looking at that file, it does contain queries that look for ERROR
> nodes, e.g.
>
> (
>   ERROR
>     "for" "(" @indent.begin ";" ";" ")" @indent.end)
>
> So perhaps we should revisit that approach too.
>
>> Now that text editors have access to the same programming language
>> grammars, I'm surprised the Tree-sitter community hasn't started any
>> cross-team effort to work on canonical queries (or slighter richer
>> abstractions) per programming language.  Text editor developers are
>> working and rediscovering complex language problems independently.
>
> We could do an indent implementation that just uses these indents.scm
> files. Not exactly sure why we didn't start out this way, but note
> that our Lisp based solution is a lot more flexible.
>
> E.g. there doesn't seem to be an obvious way to support "indentation
> styles" with these.

I think it may be possible to have different query files, say,
"GNU.scm", "K&R.scm", "C.scm" and add to the beginning of "GNU.scm" a
Lisp comment like "; import C.scm".  We would need to do manual parsing
of the .scm files, though.

Another question is how flexible this mechanism is.  If I change
something in an .scm file, would I need to restart Emacs to see the
results?  We could turn the .scm files into proper data Emacs Lisp files
that could be evaluated on the fly, I don't know how over-engineered
that'd be.  Also, I don't know if we could even satisfy advanced CC mode
users who are used to modify their style by just invoking C-c C-o on the
line whose indentation they want to change.

Even if we don't use external .scm files, perhaps we could add the
queries in them to our Tree-sitter rules?  I think we'd need a new
Tree-sitter matcher to match them properly, right?