newline-and-indent vs. electric-indent-mode

newline-and-indent vs. electric-indent-mode

[Harald Jörg]

Hi all,

in a debugging session for some indenting bugs I noticed with some
surprise how often the mode-specific indenting function is called.
There seems to be a systematic overlap between the keybinding of RET and
electric-indent-mode.

Many (almost all?) modes bind RET to newline-and-indent, but '(?\n) is
also the default value of electric-indent-chars.  So, whenever a newline
is entered, there are three calls to the mode-specific indenting
function:

 - one call for the current line, caused by electric-indent-mode.  This
   makes some sense because the line's content might suggest a different
   indentation than what could be guessed when the line started out as a
   new empty line.  It is annoying, however, when the mode gets the
   indentation wrong (which occasionally happens).  It is also
   superfluous if the character which causes a change in indentation
   (for example "}") is either itself in electric-indent-chars (as in
   perl-mode) or handled by the mode's keymap (as in cperl-mode), both
   resulting in a call to the indenting function.

 - two calls for the following, empty line.  One is caused by '(?\n)
   being in electric-indent-chars, the other by the current command
   being newline-AND-INDENT.  This doesn't make any sense.

Should electric-indent-mode be switched off by modes which map RET to
newline-and-indent? Or should the modes refrain from mapping RET?
-- 
Cheers,
haj

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

> Many (almost all?) modes bind RET to newline-and-indent,

Any mode which does that should be fixed.  Whether RET indents or not is
a user preference, not something that should depend on the kind of
language you're editing.

> So, whenever a newline is entered,

By that I assume you simply mean whenever `newline-and-indent` is executed?

> there are three calls to the mode-specific indenting function:
>
>  - one call for the current line, caused by electric-indent-mode.  This
>    makes some sense because the line's content might suggest a different
>    indentation than what could be guessed when the line started out as a
>    new empty line.  It is annoying, however, when the mode gets the
>    indentation wrong (which occasionally happens).  It is also
>    superfluous if the character which causes a change in indentation
>    (for example "}") is either itself in electric-indent-chars (as in
>    perl-mode) or handled by the mode's keymap (as in cperl-mode), both
>    resulting in a call to the indenting function.
>
>  - two calls for the following, empty line.  One is caused by '(?\n)
>    being in electric-indent-chars, the other by the current command
>    being newline-AND-INDENT.  This doesn't make any sense.

It sounds like a bug indeed.  I think both having two calls (one for
each line) or having one call (for the new line) could arguably be
correct, but three calls is indeed an error.

> Or should the modes refrain from mapping RET?

Very much so, yes (unless there's a good language-related reason why RET
should behave differently for that specific language).  But that doesn't
change the fact that `newline-and-indent` should work right, including
not calling the indent function redundantly.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 22.01.2021 16:49, Stefan Monnier wrote:
> I think both having two calls (one for
> each line)

I would like it to stop reindenting the previous line.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>> I think both having two calls (one for each line)
> I would like it to stop reindenting the previous line.

"it" being `newline-and-indent` (when used together with
`electric-indent-mode`) or `electric-indent-mode` in general?

For `electric-indent-mode` have you tried
(setq-default electric-indent-inhibit t) ?


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 22.01.2021 17:09, Stefan Monnier wrote:
>>> I think both having two calls (one for each line)
>> I would like it to stop reindenting the previous line.
> 
> "it" being `newline-and-indent` (when used together with
> `electric-indent-mode`) or `electric-indent-mode` in general?

The former.

> For `electric-indent-mode` have you tried
> (setq-default electric-indent-inhibit t) ?

That would disable the effects of electric-indent-functions, and in 
particular, of ruby--electric-indent-p.

But with your latest change, seems like I can bind `newline-and-indent` 
to RET and it will behave as expected.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>> For `electric-indent-mode` have you tried
>> (setq-default electric-indent-inhibit t) ?
> That would disable the effects of electric-indent-functions, and in
> particular, of ruby--electric-indent-p.

I don't see why.  AFAIK it should only inhibit the "reindent original
line when inserting \n".  It should affect indentation of the line after
the inserted \n nor should it affect indentation when inserting
other chars.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 23.01.2021 00:56, Stefan Monnier wrote:
>> That would disable the effects of electric-indent-functions, and in
>> particular, of ruby--electric-indent-p.
> I don't see why.  AFAIK it should only inhibit the "reindent original
> line when inserting \n".  It should affect indentation of the line after
> the inserted \n nor should it affect indentation when inserting
> other chars.

It also affect "reindent original line when inserting something other 
than \n", which is what ruby--electric-indent-p is all about (e.g. I 
type 'd' finishing the token 'end', and the line is reindented).

It's just that in my mental model \n doesn't belong to the current line, 
only to the next one. So it shouldn't reindent the original line.

Perhaps others feel differently.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>>> That would disable the effects of electric-indent-functions, and in
>>> particular, of ruby--electric-indent-p.
>> I don't see why.  AFAIK it should only inhibit the "reindent original
>> line when inserting \n".  It should affect indentation of the line after
>> the inserted \n nor should it affect indentation when inserting
>> other chars.
> It also affect "reindent original line when inserting something other than
> \n", which is what ruby--electric-indent-p is all about (e.g. I type 'd'
> finishing the token 'end', and the line is reindented).

Hmm... indeed I now see that the code also inhibits reindentation in
that case.  Weird!
Could you open a bug report for this?

> It's just that in my mental model \n doesn't belong to the current line,
> only to the next one. So it shouldn't reindent the original line.

It's often useful for me, as in typing

    foo RET else RET blabla

where the else benefits from being reindented upon the second RET.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 23.01.2021 01:16, Stefan Monnier wrote:
>>>> That would disable the effects of electric-indent-functions, and in
>>>> particular, of ruby--electric-indent-p.
>>> I don't see why.  AFAIK it should only inhibit the "reindent original
>>> line when inserting \n".  It should affect indentation of the line after
>>> the inserted \n nor should it affect indentation when inserting
>>> other chars.
>> It also affect "reindent original line when inserting something other than
>> \n", which is what ruby--electric-indent-p is all about (e.g. I type 'd'
>> finishing the token 'end', and the line is reindented).
> 
> Hmm... indeed I now see that the code also inhibits reindentation in
> that case.  Weird!
> Could you open a bug report for this?

I'm not sure how it is a bug. It's "reindentation" in both cases, right?

And electric-indent-inhibit's docstring refers to reindentation.

>> It's just that in my mental model \n doesn't belong to the current line,
>> only to the next one. So it shouldn't reindent the original line.
> 
> It's often useful for me, as in typing
> 
>      foo RET else RET blabla
> 
> where the else benefits from being reindented upon the second RET.

I see. Well, ruby--electric-indent-p covers this scenario already.

Perhaps your approach is simpler, but always reindenting the original 
line gets annoying if the indentation function sometimes produces 
suboptimal results. Or, you know, just results that disagree with the 
style guide used in a project. Then I have to go back and change the 
indentation manually again, or undo the change and go with C-o C-n TAB.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

> I'm not sure how it is a bug. It's "reindentation" in both cases, right?
> And electric-indent-inhibit's docstring refers to reindentation.

But with the current behavior, a char other than \n into
`electric-indent-chars` can never have any effect when
`electric-indent-inhibit` is non-nil.  So I'd argue that a char other
than \n should take precedence over `electric-indent-inhibit`,
especially because such a char can only be there if it's been
explicitly added.

>>> It's just that in my mental model \n doesn't belong to the current line,
>>> only to the next one. So it shouldn't reindent the original line.
>> It's often useful for me, as in typing
>>      foo RET else RET blabla
>> where the else benefits from being reindented upon the second RET.
> I see. Well, ruby--electric-indent-p covers this scenario already.

Indeed, this approach can work as well.
[ BTW, here's another one where ruby-mode is saved by the reindentation
  done by `newline`: foo RET endl DEL RET  ;-)  ]

> Perhaps your approach is simpler, but always reindenting the original line
> gets annoying if the indentation function sometimes produces suboptimal
> results.

I think `electric-indent-mode` is annoying in any case if the
indentation code disagrees with your style.  There are lots of knobs to
play with deciding when and were indentation happens, and it's hard to
know what the statistically optimal heuristic.  So I won't try to argue
that the current choice is best, but I think it's equally hard to argue
that some other choice is best.

If we could automatically detect when a reindentation is immediately
undone by the user, maybe we could collect statistics.

> again, or undo the change and go with C-o C-n TAB.

I think `C-j TAB` is quicker (but I must admit that my muscle memory
often makes me do `C-q C-j TAB`)


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 23.01.2021 05:16, Stefan Monnier wrote:
>> I'm not sure how it is a bug. It's "reindentation" in both cases, right?
>> And electric-indent-inhibit's docstring refers to reindentation.
> 
> But with the current behavior, a char other than \n into
> `electric-indent-chars` can never have any effect when
> `electric-indent-inhibit` is non-nil.  So I'd argue that a char other
> than \n should take precedence over `electric-indent-inhibit`,
> especially because such a char can only be there if it's been
> explicitly added.

All right, filed as bug#46064.

I suppose of a user wants to have those over vars not have effect, they 
can set those to nil in the major mode hook.

>>>> It's just that in my mental model \n doesn't belong to the current line,
>>>> only to the next one. So it shouldn't reindent the original line.
>>> It's often useful for me, as in typing
>>>       foo RET else RET blabla
>>> where the else benefits from being reindented upon the second RET.
>> I see. Well, ruby--electric-indent-p covers this scenario already.
> 
> Indeed, this approach can work as well.
> [ BTW, here's another one where ruby-mode is saved by the reindentation
>    done by `newline`: foo RET endl DEL RET  ;-)  ]

This one can be also solved by 'undo' instead of DEL, but yes.

>> Perhaps your approach is simpler, but always reindenting the original line
>> gets annoying if the indentation function sometimes produces suboptimal
>> results.
> 
> I think `electric-indent-mode` is annoying in any case if the
> indentation code disagrees with your style.

True, but my present complaint is about it being annoying *twice* for 
the same line. And if it's being annoying while point is still on that 
line, it's marginally easier to fix.

> There are lots of knobs to
> play with deciding when and were indentation happens, and it's hard to
> know what the statistically optimal heuristic.  So I won't try to argue
> that the current choice is best, but I think it's equally hard to argue
> that some other choice is best.

Agreed.

> If we could automatically detect when a reindentation is immediately
> undone by the user, maybe we could collect statistics.

That might help with indentation defaults (if the statistics is sent 
back to us somehow), but not with specific cases, I think. And the 
indentation code has to be capable of producing the needed indentation 
patterns, at least with some values of indentation-related options.

>> again, or undo the change and go with C-o C-n TAB.
> 
> I think `C-j TAB` is quicker (but I must admit that my muscle memory
> often makes me do `C-q C-j TAB`)

Oh, right.

In the meantime, I'll try running with RET bound to newline-and-indent, 
like in good old days.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

> All right, filed as bug#46064.

Thanks.

>> I think `electric-indent-mode` is annoying in any case if the
>> indentation code disagrees with your style.
> True, but my present complaint is about it being annoying *twice* for the
> same line. And if it's being annoying while point is still on that line,
> it's marginally easier to fix.

BTW, I just want to clarify that while I'm to be blamed for the current
behavior, I'm perfectly happy if someone wants to change it.

My main goal was to consolidate all those major modes's ad-hoc
auto-indent (typically by binding RET to `newline-and-indent`) into
a global user config, and that's done.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 24.01.2021 07:29, Stefan Monnier wrote:
>> All right, filed as bug#46064.
> 
> Thanks.
> 
>>> I think `electric-indent-mode` is annoying in any case if the
>>> indentation code disagrees with your style.
>> True, but my present complaint is about it being annoying *twice* for the
>> same line. And if it's being annoying while point is still on that line,
>> it's marginally easier to fix.
> 
> BTW, I just want to clarify that while I'm to be blamed for the current
> behavior, I'm perfectly happy if someone wants to change it.
> 
> My main goal was to consolidate all those major modes's ad-hoc
> auto-indent (typically by binding RET to `newline-and-indent`) into
> a global user config, and that's done.

I've considered what default we could change, but if we wanted to set 
electric-indent-inhibit to t by default (after resolving bug#46064), 
most/all major modes should first get electric-indent-functions similar 
to #'ruby--electric-indent-p to compensate for it, and that's a 
non-trivial effort, not to mention the change in user habits.

Though we probably could generate something based on SMIE grammar, in 
modes based on it.

Re: newline-and-indent vs. electric-indent-mode

[Madhu]

* Stefan Monnier <jwva6t0s6vj.fsf-monnier+emacs@gnu.org> :
Wrote on Fri, 22 Jan 2021 22:16:39 -0500:

> I think `electric-indent-mode` is annoying in any case if the
> indentation code disagrees with your style.  There are lots of knobs to
> play with deciding when and were indentation happens, and it's hard to
> know what the statistically optimal heuristic.  So I won't try to argue
> that the current choice is best, but I think it's equally hard to argue
> that some other choice is best.
>
> If we could automatically detect when a reindentation is immediately
> undone by the user, maybe we could collect statistics.

As a data point, I find every design choice you have made in this area
has directly attacked my usage.  Typically In C-like modes turning it on
leads to spurious whitespace when I want to insert a new line. Which
means I have to *always* what electric indent does.  This may be an
excuse for introducing Telemetry in organizations like mozilla, but I
hope emacs development isn't driven by those devils.

I used to be benefit from emacs indentation by binding RET to
newline-and-indent and remain in control of indentation.  Since electric
indent the *only* I can use emacs is to globally turn electric mode off.
IN ALL CASES what it guesses for me is exactly what I do not want it to
do.

Collecting the changes in a central place means that all modes are
affected.  I am unable to use any mode effectively which SM has touched.

It is a lost case when the behaviour that I have been using for years is
now broken and there is no path to fix it short of throwing all SMIE
code out of the window.  The problem is the new code is hostile by
design and only cares about enslaving me to its ideology and to change
my established work style and work behaviour. Invariably I have to give
up what I wanted to do and curse, and chalk it up as another victory of
the devil over me

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 25.01.2021 03:56, Madhu wrote:
> I used to be benefit from emacs indentation by binding RET to
> newline-and-indent and remain in control of indentation.

How was it different?

newline-and-indent can lead to "spurious whitespace" just the same.

Re: newline-and-indent vs. electric-indent-mode

[Madhu]

* Dmitry Gutov <801ef866-4212-5b74-350e-9942953174fe@yandex.ru> :
Wrote on Mon, 25 Jan 2021 04:29:55 +0200:

> On 25.01.2021 03:56, Madhu wrote:
>> I used to be benefit from emacs indentation by binding RET to
>> newline-and-indent and remain in control of indentation.
>
> How was it different?
> newline-and-indent can lead to "spurious whitespace" just the same.

This is true now, and I'm not able to say with certainty what the
difference was 10 years ago (if it indeed was different)

IIRC the way to deal with it was that if the next event after a
newline-and-indent is *not* a self-insert-command, then empty whitespace
on that line should have be cleaned up.

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 25.01.2021 12:45, Madhu wrote:
> * Dmitry Gutov <801ef866-4212-5b74-350e-9942953174fe@yandex.ru> :
> Wrote on Mon, 25 Jan 2021 04:29:55 +0200:
> 
>> On 25.01.2021 03:56, Madhu wrote:
>>> I used to be benefit from emacs indentation by binding RET to
>>> newline-and-indent and remain in control of indentation.
>>
>> How was it different?
>> newline-and-indent can lead to "spurious whitespace" just the same.
> 
> This is true now, and I'm not able to say with certainty what the
> difference was 10 years ago (if it indeed was different)

It wasn't different before electric-indent-mode was introduced.

> IIRC the way to deal with it was that if the next event after a
> newline-and-indent is *not* a self-insert-command, then empty whitespace
> on that line should have be cleaned up.

I've always had to use a third-party mode to clean up that whitespace, 
such as https://github.com/purcell/whitespace-cleanup-mode (most recently).

But what you're suggesting should be easy enough to implement in your 
personal config too, creating a wrapper for newline-and-indent.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

> IIRC the way to deal with it was that if the next event after a
> newline-and-indent is *not* a self-insert-command, then empty whitespace
> on that line should have be cleaned up.

I don't think there's ever been such a behavior in Emacs's defaults.
But I'd welcome patches which implement that functionality (tho
I don't think it should special case `self-insert-command`).

Dmitry mentions:
> I've always had to use a third-party mode to clean up that whitespace, such
> as https://github.com/purcell/whitespace-cleanup-mode (most recently).

The problem with `whitespace-cleanup-mode` is that it's global.
I'd like something that only operates locally, i.e. only touches lines
you've modified.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 25.01.2021 16:36, Stefan Monnier wrote:
> The problem with `whitespace-cleanup-mode` is that it's global.
> I'd like something that only operates locally, i.e. only touches lines
> you've modified.

That would be https://github.com/lewang/ws-butler.

whitespace-cleanup-mode is working out just fine for me, though, because 
by default is only cleans files that were clean previously.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>> The problem with `whitespace-cleanup-mode` is that it's global.
>> I'd like something that only operates locally, i.e. only touches lines
>> you've modified.
> That would be https://github.com/lewang/ws-butler.

That's better, indeed, tho if you look at the implementation you see
that it has to deal with lots of corner cases, so it ends up "ugly and
brittle": I'm not sure "cleanup on save" is the better option.
I don't mean to say that Le Wang did a bad job, BTW: the issues aren't
caused by a bad design or implementation, but by a hard problem.

I was thinking instead of something that removes the trailing space(s)
when point leaves the line (like Madhu suggested).  Of course that's
also fraught with danger (e.g. you don't want that whitespace to
disappear just because you did `C-x C-x`), so the result might end up
just as ugly&brittle.

> whitespace-cleanup-mode is working out just fine for me, though, because by
> default is only cleans files that were clean previously.

I'd like a feature that is also active in non-clean files.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Rudolf Schlatte]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I was thinking instead of something that removes the trailing space(s)
> when point leaves the line (like Madhu suggested).  Of course that's
> also fraught with danger (e.g. you don't want that whitespace to
> disappear just because you did `C-x C-x`), so the result might end up
> just as ugly&brittle.

Even here there are corner cases: in markdown, two or more spaces at the
end of a line denote a line break.

Re: newline-and-indent vs. electric-indent-mode

[Dmitry Gutov]

On 25.01.2021 17:15, Stefan Monnier wrote:
>>> The problem with `whitespace-cleanup-mode` is that it's global.
>>> I'd like something that only operates locally, i.e. only touches lines
>>> you've modified.
>> That would be https://github.com/lewang/ws-butler.
> 
> That's better, indeed, tho if you look at the implementation you see
> that it has to deal with lots of corner cases, so it ends up "ugly and
> brittle": I'm not sure "cleanup on save" is the better option.
> I don't mean to say that Le Wang did a bad job, BTW: the issues aren't
> caused by a bad design or implementation, but by a hard problem.

Indeed, there are a lot of hooks involved, but it seemed to work last 
time I tried it.

> I was thinking instead of something that removes the trailing space(s)
> when point leaves the line (like Madhu suggested).  Of course that's
> also fraught with danger (e.g. you don't want that whitespace to
> disappear just because you did `C-x C-x`), so the result might end up
> just as ugly&brittle.

Or if I just did C-n C-p, or something more complex to the same effect. 
But I suppose somebody could try creating a prototype, and then we'd see 
just how jarring this approach is (or not).

Alternatively, one could start with looking into how others editors do it.

>> whitespace-cleanup-mode is working out just fine for me, though, because by
>> default is only cleans files that were clean previously.
> 
> I'd like a feature that is also active in non-clean files.

I just clean such a file manually once, and then the feature is active. 
Or I don't, and use whitespace-mode to make sure I don't leave new 
trailing whitespace around.

Files like that are a minority in my experience, so it's not too big a 
price to pay for being able to rely on a simple implementation. YMMV, of 
course.

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

> Indeed, there are a lot of hooks involved, but it seemed to work last
> time I tried it.

Yes, that's good enough for an optional feature, but I'd like something
that has a fighting chance of being enabled by default

>>> whitespace-cleanup-mode is working out just fine for me, though, because by
>>> default is only cleans files that were clean previously.
>> I'd like a feature that is also active in non-clean files.
>
> I just clean such a file manually once, and then the feature is active. Or
> I don't, and use whitespace-mode to make sure I don't leave new trailing
> whitespace around.
>
> Files like that are a minority in my experience, so it's not too big a price
> to pay for being able to rely on a simple implementation. YMMV, of course.

I agree for my own use.  But that's not what I'm after ;-)


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[martin rudalics]

 >>> The problem with `whitespace-cleanup-mode` is that it's global.
 >>> I'd like something that only operates locally, i.e. only touches lines
 >>> you've modified.
 >> That would be https://github.com/lewang/ws-butler.
 >
 > That's better, indeed, tho if you look at the implementation you see
 > that it has to deal with lots of corner cases, so it ends up "ugly and
 > brittle":

It's the best you can get I would say.  You can't really postpone
trimming to 'kill-buffer' and anything before 'save-buffer' can be just
too early.  Yet, saving will irrevocably trim all whitespace so you
can't get it back for future editing.

 > I'm not sure "cleanup on save" is the better option.
 > I don't mean to say that Le Wang did a bad job, BTW: the issues aren't
 > caused by a bad design or implementation, but by a hard problem.
 >
 > I was thinking instead of something that removes the trailing space(s)
 > when point leaves the line (like Madhu suggested).  Of course that's
 > also fraught with danger (e.g. you don't want that whitespace to
 > disappear just because you did `C-x C-x`), so the result might end up
 > just as ugly&brittle.

We could put some phantom space there and restore it as soon as point
reenters that line.  And maybe handle the current line specially when
saving the buffer.

martin

Re: newline-and-indent vs. electric-indent-mode

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Mon, 25 Jan 2021 07:26:03 +0530
> 
> As a data point, I find every design choice you have made in this area
> has directly attacked my usage.  Typically In C-like modes turning it on
> leads to spurious whitespace when I want to insert a new line.

Please show a couple of examples of that.

> It is a lost case when the behaviour that I have been using for years is
> now broken and there is no path to fix it short of throwing all SMIE
> code out of the window.

What is broken, please?  I have electric-indent-mode turned on in my C
mode buffers, and if there was something broken in that, I'd pay
attention very quickly.

Really, such bitter complaints must always come with a lot of details,
including, but not limited, any customizations you have made.
Otherwise what do you expect us to do about your gripes?

Re: newline-and-indent vs. electric-indent-mode

[Harald Jörg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Many (almost all?) modes bind RET to newline-and-indent,
>
> Any mode which does that should be fixed.

Ouch... I see now that my "observation" was plain wrong.  CPerl mode
does it, but only as a user preference.  When I checked other modes with
C-h k, I did it in an Emacs instance where I had it globally remapped.
I failed to check the sources.  Sorry for the confusion.

> Whether RET indents or not is a user preference, not something that
> should depend on the kind of language you're editing.

For most programming and markup languages indenting makes sense, but
less so for other modes.  I understand that python-mode disables it, but
was a bit surprised that c-mode disables it, too (I understand it now
after following two lengthy threads on debbugs).

I guess it wouldn't hurt to add a sentence to the docstring of
electric-indent-mode how it should be managed for a single buffer.  The
method with an extra variable (electric-indent-inhibit, works only to
disable a globally enabled mode) or an extra mode
(electric-indent-local-mode, works both ways) is somewhat nonstandard
and the question seems to pop up occasionally on various platforms.

>> So, whenever a newline is entered,
>
> By that I assume you simply mean whenever `newline-and-indent` is executed?

Yes.  I thought that I meant "whenever I hit the enter key" but this was
only true because I had mapped it that way.

>> there are three calls to the mode-specific indenting function:
>>
>>  - one call for the current line, caused by electric-indent-mode.  [...]
>>
>>  - two calls for the following, empty line.  One is caused by '(?\n)
>>    being in electric-indent-chars, the other by the current command
>>    being newline-AND-INDENT.  This doesn't make any sense.
>
> It sounds like a bug indeed.  I think both having two calls (one for
> each line) or having one call (for the new line) could arguably be
> correct, but three calls is indeed an error.

So... I guess newline-and-indent could suppress the call to
indent-line-function for the new line if electric-indent-mode is t and
electric-indent-inhibit is nil and ?\n is in electric-indent-chars?

Just for the record: The results are correct, and the delay isn't
noticeable even with the convoluted indenting routines of CPerl mode.
It is just a bit annoying when you are tracing through the routines
trying to figure out where to fix a bug.

>> Or should the modes refrain from mapping RET?
>
> Very much so, yes (unless there's a good language-related reason why RET
> should behave differently for that specific language).

I now see that they actually don't do this.  Sorry again.

In CPerl mode, the remaining issue is actually the other way around.
You can activate cperl-electric-linefeed via customize to do
newline-and-indent.  However, when you don't set this option, you still
get newline ... and indent, thanks to electric-indent-mode.

So, that customization option is futile since whenever 2013-ish
electric-indent-mode became default.  I take the total lack of customer
protests and bug reports as a hint that Perl programmers actually are
quite fine with indenting as they type.
-- 
Cheers,
haj

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>>> Many (almost all?) modes bind RET to newline-and-indent,
>> Any mode which does that should be fixed.
> Ouch... I see now that my "observation" was plain wrong.

Yes and now: historically, it's been quite common for major modes to do
that kind of thing.  I've been fighting it for many years now (even
long before `electric-indent-mode`) since it's usually the reflection of
the major mode's author's preference, rather than something directly
linked to the major mode (there are several other examples of this
behavior: setting `comment-column` in the major mode is another similar
example).

>> Whether RET indents or not is a user preference, not something that
>> should depend on the kind of language you're editing.
> For most programming and markup languages indenting makes sense, but
> less so for other modes.

Concrete examples would be helpful and could be reported as bugs (I have
made efforts to setup `electric-indent-mode` such that it tries not to
get in the way in "plain text" modes, but of course that's no guarantee
that it catches all relevant cases).

> I guess it wouldn't hurt to add a sentence to the docstring of
> electric-indent-mode how it should be managed for a single buffer.  The
> method with an extra variable (electric-indent-inhibit, works only to
> disable a globally enabled mode) or an extra mode
> (electric-indent-local-mode, works both ways) is somewhat nonstandard
> and the question seems to pop up occasionally on various platforms.

`electric-indent-inhibit` doesn't inhibit auto-indentation.  It inhibits
auto-*re*indentation.

I know it takes many people by surprise (because the choices are more
refined than just "on or off" and they don't expect that), but I find it
hard to improve the docs to guide the users/programmers.

>> It sounds like a bug indeed.  I think both having two calls (one for
>> each line) or having one call (for the new line) could arguably be
>> correct, but three calls is indeed an error.
> So... I guess newline-and-indent could suppress the call to
> indent-line-function for the new line if electric-indent-mode is t and
> electric-indent-inhibit is nil and ?\n is in electric-indent-chars?

That would be one way, tho I find it fairly ugly.  Another might be to
temporarily disable `electric-indent-mode`.  The more I think about it,
the more I think this is the better choice.

> Just for the record: The results are correct, and the delay isn't
> noticeable even with the convoluted indenting routines of CPerl mode.

Usually indentation of a single line is very quick so doing twice is
indeed lost in the noise.  Of course, there are corner cases where
indentation can take a non-negligible amount of time, in which case
doubling it can be noticed (but still isn't a main worry: even if it
increases the runtime from 30s to 1min it's not that significant since
30s is really not all that much better than 1min).

Still counts as a performance bug to me.

> It is just a bit annoying when you are tracing through the routines
> trying to figure out where to fix a bug.

I suspect that there are cases where it really does introduce bugs
beyond a (minor) performance impact, e.g. in some of the major modes
where repeated TABs cycle between different indentation points.

In any case, I pushed a change to `master` to disable
`electric-indent-mode` in `newline-and-indent`, so this should hopefully
be fixed now.

> In CPerl mode, the remaining issue is actually the other way around.
> You can activate cperl-electric-linefeed via customize to do
> newline-and-indent.

More or less.  `cperl-mode` intends to offer basically 3 different
options: plain newline, newline+indent, and newline+indent+fancystuff.
And these are bound be default resp. to RET, LFD, and C-c LFD.
Then `cperl-electric-linefeed` lets you swap the last two.  Of course,
with `electric-indent-mode` RET ends up doing (more or less) what
LFD does.

IMO keybindings is more harmful than anything here, so a better choice
would be to offer only plain newline and newline+indent+fancystuff, bind
them to RET and LFD, let `electric-indent-mode` control which of RET and
LFD does which, and let `cperl-electric-linefeed` control whether
fancystuff is done at all.


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Harald Jörg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Many (almost all?) modes bind RET to newline-and-indent,
>>> Any mode which does that should be fixed.
>> Ouch... I see now that my "observation" was plain wrong.
>
> Yes and now: historically, it's been quite common for major modes to do
> that kind of thing.  I've been fighting it for many years now (even
> long before `electric-indent-mode`) since it's usually the reflection of
> the major mode's author's preference, rather than something directly
> linked to the major mode ...

Guilty, Your Honour.

It would never occur to me to use any other than the RET key to advance
to the next line, plus DWIM (Do What I Mean).

I'm not the mode's author, though.

>>> Whether RET indents or not is a user preference, not something that
>>> should depend on the kind of language you're editing.
>> For most programming and markup languages indenting makes sense, but
>> less so for other modes.
>
> Concrete examples would be helpful and could be reported as bugs ...

I don't think these are bugs, but my personal user preference is to have
RET indent in programming modes but not in text modes.  Org mode doesn't
indent, which is fine, but it has its own binding of RET (in a table it
does many fancy things, amongst them add a newline).

> `electric-indent-inhibit` doesn't inhibit auto-indentation.  It inhibits
> auto-*re*indentation.

Ah, thanks for the clarification.

> I know it takes many people by surprise (because the choices are more
> refined than just "on or off" and they don't expect that), but I find it
> hard to improve the docs to guide the users/programmers.

I admit that the whole electric-indent stuff is new to me.  I saw it
happening but never checked *why* it is happening.  First time I noticed
it explicitly was in the backtrace leading to my original post.

>>> It sounds like a bug indeed.  I think both having two calls (one for
>>> each line) or having one call (for the new line) could arguably be
>>> correct, but three calls is indeed an error.
>> So... I guess newline-and-indent could suppress the call to
>> indent-line-function for the new line if electric-indent-mode is t and
>> electric-indent-inhibit is nil and ?\n is in electric-indent-chars?

> That would be one way, tho I find it fairly ugly.

Yeah, that has a certain smell. 

> Another might be to temporarily disable `electric-indent-mode`.  The
> more I think about it, the more I think this is the better choice.

Wouldn't that result in `newline` re-indenting both the new and the
previous line (as per electric-indent-mode), but `newline-and-indent`
*not* re-indenting the previous line?  That would seem a bit surprising.

First experiments suggest that the patch does indeed change the behavior
when a line contains just a closing "]" or ")" - neither
(newline-and-indent) nor (cperl-linefeed) now re-indent that line (which
they should) - only a plain RET does.

`newline-and-indent` could also temporarily enforce electric-indent-mode
with electric-indent-chars set to ?\n, let that minor mode do its job
while inserting the newline char, and never call indent-line-function by
itself.

>> [...]
> IMO keybindings is more harmful than anything here, so a better choice
> would be to offer only plain newline and newline+indent+fancystuff, bind
> them to RET and LFD, let `electric-indent-mode` control which of RET and
> LFD does which, and let `cperl-electric-linefeed` control whether
> fancystuff is done at all.

That sounds good... it would need some unraveling to prevent deep
recursion.  `electric-indent-mode` calls the mode-specific indentation
function, which would optionally call fancystuff, which in turn calls
both newline-and-indent _and_ the mode-specific indentation function.
-- 
Cheers,
haj

Re: newline-and-indent vs. electric-indent-mode

[Stefan Monnier]

>> Concrete examples would be helpful and could be reported as bugs ...
> I don't think these are bugs, but my personal user preference is to have
> RET indent in programming modes but not in text modes.

You might like to try removing that customization and then complain
about the cases that don't behave like you want.  I don't guarantee I'll
agree, but since `electric-indent-mode` is globally enabled by default,
I think it's important to try and make sure it's not too annoying in
those modes where it's not helpful.

>> I know it takes many people by surprise (because the choices are more
>> refined than just "on or off" and they don't expect that), but I find it
>> hard to improve the docs to guide the users/programmers.
> I admit that the whole electric-indent stuff is new to me.  I saw it
> happening but never checked *why* it is happening.  First time I noticed
> it explicitly was in the backtrace leading to my original post.

Yes, it happened because I think it's important to consolidate the
needs shared by all major modes.  It's surprisingly hard work, because
every major mode tends to do it slightly differently, so the
consolidated version is never "quite the same", thus encountering a lot
of resistance to change.

> Wouldn't that result in `newline` re-indenting both the new and the
> previous line (as per electric-indent-mode), but `newline-and-indent`
> *not* re-indenting the previous line?

Yes.

> That would seem a bit surprising.

Then again, the users have the choice of either calling
`newline-and-indent` or `reindent-then-newline-and-indent`, so
presumably when they choose `newline-and-indent` it's because they don't
want the first line to be reindented.

It also brings back the behavior they had before `electric-indent-mode`.

> First experiments suggest that the patch does indeed change the behavior
> when a line contains just a closing "]" or ")" - neither
> (newline-and-indent) nor (cperl-linefeed) now re-indent that line (which
> they should)

This behavior is the behavior that `cperl-linefeed` had when it was
written, so I disagree with "they should".

>> IMO keybindings is more harmful than anything here, so a better choice
>> would be to offer only plain newline and newline+indent+fancystuff, bind
>> them to RET and LFD, let `electric-indent-mode` control which of RET and
>> LFD does which, and let `cperl-electric-linefeed` control whether
>> fancystuff is done at all.
>
> That sounds good... it would need some unraveling to prevent deep
> recursion.  `electric-indent-mode` calls the mode-specific indentation
> function, which would optionally call fancystuff, which in turn calls
> both newline-and-indent _and_ the mode-specific indentation function.

I haven't even looked at what it would take in terms of coding.
Hell!  I don't even know half of what the "fancystuff" does, so maybe
I'm just plain wrong about what should be done.
In other words: I won't be touching this code any time soon (I'd much
rather add some of the "fancystuff" features to `perl-mode`, where
I wouldn't need to worry about breaking old cperl-mode users's habits).


        Stefan

Re: newline-and-indent vs. electric-indent-mode

[Harald Jörg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Concrete examples would be helpful and could be reported as bugs ...
>> I don't think these are bugs, but my personal user preference is to have
>> RET indent in programming modes but not in text modes.
>
> You might like to try removing that customization ...

I've already done that.  It was part of my ancient customization stuff I
wasn't aware of any more.  I guess that was all that needed to be done
to fix my doubts.

>> I admit that the whole electric-indent stuff is new to me.  I saw it
>> happening but never checked *why* it is happening.  First time I noticed
>> it explicitly was in the backtrace leading to my original post.
>
> Yes, it happened because I think it's important to consolidate the
> needs shared by all major modes.

Agreed, I also consider this very valuable.

> Then again, the users have the choice of either calling
> `newline-and-indent` or `reindent-then-newline-and-indent`, so
> presumably when they choose `newline-and-indent` it's because they don't
> want the first line to be reindented.

Well... actually, as a user, I don't want to be bothered with such
choices while I'm typing.  I want to hit RET and have the mode figure
out what needs to be done so that the code follows some coding
convention.  But that's only me, so "anecdotal evidence".

This choice might be of interest for mode maintainers, though.  They
might want to check how new user settings or global minor modes affect
their mode's behavior.  Maybe mode authors should wrap
‘electric-indent-local-mode’ with a mode-specific command (in
particular: a mode-specific docstring) because users shouldn't be
supposed to figure out this particular's mode's settings for
‘electric-indent-functions’ and ‘electric-indent-chars’, which is all
they get from the docstring of the global ‘electric-indent-mode’
command.

> It also brings back the behavior they had before `electric-indent-mode`.

I'm sure this is correct.  I wasn't aware of it since I only used Emacs
sporadically in that era.

>> First experiments suggest that the patch does indeed change the behavior
>> when a line contains just a closing "]" or ")" - neither
>> (newline-and-indent) nor (cperl-linefeed) now re-indent that line (which
>> they should)
>
> This behavior is the behavior that `cperl-linefeed` had when it was
> written, so I disagree with "they should".

I should rephrase that: They should according to my user expectation
(basically Damian Conway's book Perl Best Practices, or perltidy
conventions) - regardless of whether the current implementation of
`cperl-linefeed` is up to this task.

> Hell!  I don't even know half of what the "fancystuff" does, so maybe
> I'm just plain wrong about what should be done.

CPerl mode is full of fancy stuff :) In particular, it can rewrite
(not just indent) existing code to make it conforming to styleguides.
Or just to "beautify" it.

That said, I don't use this often, and maybe it isn't that popular
overall.  My main interest isn't the fancy stuff.  I want to, however,
support features which have been added to Perl in the current century.
Therefore I struggle to understand how CPerl mode does things, and how
it probably should do things in contemporary Emacs.

So, bottom line: I've understood a lot about the intention of
electric-indent-mode, about the current implementation and about my own
blunders.  Thanks for your patience!
-- 
Cheers,
haj