electric-indent-mode: abolition of `newline' function is not the Right Thing.

electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

Hi, Emacs.

`electric-indent-post-self-insert-function' effectively redefines
`newline' to `newline-and-indent'.  So, whilst `electric-indent-mode' is
enabled, there is no easy way to invoke the traditional functionality of
`newline'.  This is a Bad Thing.

I discovered this whilst trying `electric-indent-mode' in Text Mode.
I have a command which starts a new paragraph thusly:
(i) It calls `newline' and inserts, e.g. "(vi) ".
(ii) It calls `fill-paragraph' for the text in the new paragraph.

Unfortunately, `newline' indents the "(vi) " to the level of the
old paragraph, so that the following `fill-paragraph' joins the old and
new paragraphs together.  This is bad.

`newline-and-indent' exists as its own command, traditionally bound to
C-j.  A better way for `electric-indent-mode' to achieve the effect it
wants is to get users to rebind <CR> to `newline-and-indent' or suggest
the use of C-j.

`newline' must continue to exist, both as a command and as a lisp
function.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> I discovered this whilst trying `electric-indent-mode' in Text Mode.
> I have a command which starts a new paragraph thusly:
> (i) It calls `newline' and inserts, e.g. "(vi) ".

Use (insert "\n") rather than (newline).


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

'morning, Stefan!

On Sun, Oct 13, 2013 at 09:23:49AM -0400, Stefan Monnier wrote:
> > I discovered this whilst trying `electric-indent-mode' in Text Mode.
> > I have a command which starts a new paragraph thusly:
> > (i) It calls `newline' and inserts, e.g. "(vi) ".

> Use (insert "\n") rather than (newline).

:-)  Of course there are workarounds, but that's not the point.  There
are approximately 198 calls to `newline' in the Emacs Lisp sources, and
an unknown further number in other Lisp files and users' .emacsen.  Some
of these calls are going to get broken by the effective change to
`newline'.

Redefining `newline' as `newline-and-indent' whilst
`electric-indent-mode' is active is not the Right Thing to do.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> :-)  Of course there are workarounds, but that's not the point.  There
> are approximately 198 calls to `newline' in the Emacs Lisp sources, and
> an unknown further number in other Lisp files and users' .emacsen.  Some
> of these calls are going to get broken by the effective change to
> `newline'.

Right.  But the problem is that this is not specific to
electric-indent-mode: newline calls self-insert-command, which runs
expand-abbrev and (since Emacs24) post-self-insert-hook.  Any caller
which only wants a \n and no extra doo-dads should not call `newline'.

electric-indent-mode just makes this problem more apparent.


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Oct 13, 2013 at 12:22:33PM -0400, Stefan Monnier wrote:
> > :-)  Of course there are workarounds, but that's not the point.
> > There are approximately 198 calls to `newline' in the Emacs Lisp
> > sources, and an unknown further number in other Lisp files and users'
> > .emacsen.  Some of these calls are going to get broken by the
> > effective change to `newline'.

> Right.  But the problem is that this is not specific to
> electric-indent-mode: newline calls self-insert-command, which runs
> expand-abbrev and (since Emacs24) post-self-insert-hook.  Any caller
> which only wants a \n and no extra doo-dads should not call `newline'.

> electric-indent-mode just makes this problem more apparent.

Sorry Stefan, but that is disingenuous.  If you look again at
`electric-indent-post-self-insert-function', you will see that that
function deliberately converts `newline' into `newline-and-indent'.  It
even says so in the comments.  _This_ is the problem.

What is the justification for this extension in functionality of
`newline' to do the same as `newline-and-indent'?

And people calling `newline' do not simply want a \n.  They want
abbreviations to be processed, they want auto-fill to be run, trailing
space to be removed from otherwise blank lines, and so on.  What they
_don't_ want is indentation inserted on the new line.

Please take this indent-newline functionality out of
e-i-post-s-i-function (or let me do it).

Incidentally, each call of e-i-post-s-i-function creates a new marker,
bound to the let variable `before'.  This is not set to point nowhere
before `before' becomes unbound.  I think it probably should.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> And people calling `newline' do not simply want a \n.  They want
> abbreviations to be processed, they want auto-fill to be run, trailing
> space to be removed from otherwise blank lines, and so on.  What they
> _don't_ want is indentation inserted on the new line.

Not sure who is your "they".  In my experience, what I've seen users
want is "auto-indentation" and by that they mean that the text should be
kept indented when they hit RET and a few other keys.
E.g. a question I've seen many times is

  So I tried this Emacs thingy which claims to do indentation for me,
  but all my text stays at the left-margin.  What am I doing wrong.

and the answer I generally see given is:

  You want to bind RET to newline-and-indent.

Are these the majority, users?  I don't know.  But so far, you're the
only person who's complained about ?\n being in the default value of
electric-indent-chars, so I see no evidence that your preference in this
regard is representative.

> Please take this indent-newline functionality out of
> e-i-post-s-i-function (or let me do it).

Very easy: remove ?\n from electric-indent-chars.
IOW (setq-default electric-indent-chars nil) should do that for you.

> Incidentally, each call of e-i-post-s-i-function creates a new marker,
> bound to the let variable `before'.  This is not set to point nowhere
> before `before' becomes unbound.  I think it probably should.

Let's see if/when this leads to a concrete performance problem, and at
that point we'll see how to fix it.  Making a marker point nowhere is an
annoyance and is itself a costly operation (O(N) for N markers), so
I don't much like using the workaround you suggest.


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Oct 14, 2013 at 08:56:25AM -0400, Stefan Monnier wrote:
> > And people calling `newline' do not simply want a \n.  They want
> > abbreviations to be processed, they want auto-fill to be run, trailing
> > space to be removed from otherwise blank lines, and so on.  What they
> > _don't_ want is indentation inserted on the new line.

> Not sure who is your "they".

The people who have written Emacs Lisp code which calls `newline' (as
contrasted with `newline-and-indent').

> In my experience, what I've seen users want is "auto-indentation" and
> by that they mean that the text should be kept indented when they hit
> RET and a few other keys.

That is a different topic from this thread's.  The question which should
be asked, should have been asked some while ago, is what is the best way
of giving these users what they want.  The answer is not, and can not be,
to have an Emacs without the traditional functionality of `newline'.

[ .... ]

> But so far, you're the only person who's complained about ?\n being in
> the default value of electric-indent-chars, so I see no evidence that
> your preference in this regard is representative.

Have I complained about \n being in electric-indent-chars?  I don't think
I have, but might be mistaken.  My complaint, and I've been perfectly
clear about it, is the abolition of the traditional functionality of the
`newline' function whenever electric-indent-mode is enabled.  You haven't
really addressed this issue in your past few posts.

> > Please take this indent-newline functionality out of
> > e-i-post-s-i-function (or let me do it).

> Very easy: remove ?\n from electric-indent-chars.
> IOW (setq-default electric-indent-chars nil) should do that for you.

Is that your solution to the ~198 uses of `newline' in Emacs and
countless others in other code and users' .emacsen?  Each caller of
`newline' should first ensure that \n is not in electric-indent-chars?

> > Incidentally, each call of e-i-post-s-i-function creates a new marker,
> > bound to the let variable `before'.  This is not set to point nowhere
> > before `before' becomes unbound.  I think it probably should.

> Let's see if/when this leads to a concrete performance problem, and at
> that point we'll see how to fix it.  Making a marker point nowhere is an
> annoyance and is itself a costly operation (O(N) for N markers), so
> I don't much like using the workaround you suggest.

Well, it's not really my suggestion, it's standard advice in the Elisp
manual on page "Overview of Markers".  Without doing this, a buffer's
performance is gradually going to worsen as a heap of useless markers
accumulates, until such time as garbage collection eventually gets rid of
them.  The cost of making a marker point nowhere, less than once per
keystroke, is hardly high.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[martin rudalics]

 > And people calling `newline' do not simply want a \n.  They want
 > abbreviations to be processed, they want auto-fill to be run, trailing
 > space to be removed from otherwise blank lines, and so on.  What they
 > _don't_ want is indentation inserted on the new line.

What I want is RET to always indent the new line.  And when I'm at the
beginning of a line I want that RET opens a new line here and indents
it.  And that's why I'm hopelessly lost when I have to work with emacs
-Q.

martin

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

Hello, Martin.

On Tue, Oct 15, 2013 at 08:28:22PM +0200, martin rudalics wrote:
>  > And people calling `newline' do not simply want a \n.  They want
>  > abbreviations to be processed, they want auto-fill to be run, trailing
>  > space to be removed from otherwise blank lines, and so on.  What they
>  > _don't_ want is indentation inserted on the new line.

> What I want is RET to always indent the new line.  And when I'm at the
> beginning of a line I want that RET opens a new line here and indents
> it.  And that's why I'm hopelessly lost when I have to work with emacs
> -Q.

With respect, that is not the topic of this thread.  That topic is that
with electric-indent-mode enabled, there is now no command, also no
elisp function which has the traditional functionality of `newline'.
That is a significant loss to Emacs.  The change in the functionality of
`newline' will break existing elisp code which calls `newline'.  Also,
there will be no way for a user to set up traditional Emacs behaviour
wrt <ret> and C-j.

For what you want, `newline-and-indent' exists, as you know.

> martin
> 

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> With respect, that is not the topic of this thread.  That topic is that
> with electric-indent-mode enabled, there is now no command, also no
> elisp function which has the traditional functionality of `newline'.

Quick, without looking at the source code of newline: what does
`newline' do that (insert "\n") or C-q C-j doesn't do?

> That is a significant loss to Emacs.

I don't know what that is supposed to mean.


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Oct 16, 2013 at 01:59:07PM -0400, Stefan Monnier wrote:
> > With respect, that is not the topic of this thread.  That topic is that
> > with electric-indent-mode enabled, there is now no command, also no
> > elisp function which has the traditional functionality of `newline'.

> Quick, without looking at the source code of newline: what does
> `newline' do that (insert "\n") or C-q C-j doesn't do?

The answer you're not looking for is "It inserts a newline, whilst doing
lots of Right Things on the previous line that I don't need to worry
about too much.", but it's my preferred answer.

Answering you literally, I think newline invokes filling, abbreviation
expansion and possibly other things, but (insert "\n") and C-q C-j
don't.

> > That is a significant loss to Emacs.

> I don't know what that is supposed to mean.

For one thing, it would prevent somebody using a mode with {e-i-m using
\n} and retaining the classical difference betweeen <ret> and C-j.

For another thing, it will stop lots of Emacs Lisp from working.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[chad]

On 16 Oct 2013, at 11:58, Alan Mackenzie <acm@muc.de> wrote:
> 
> For another thing, it will stop lots of Emacs Lisp from working.

What specific packages/libraries/etc break with the current
electric-indent-mode enabled?

~Chad

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Daniel Colascione]

On 10/16/13 1:55 PM, chad wrote:
> On 16 Oct 2013, at 11:58, Alan Mackenzie <acm@muc.de> wrote:
>>
>> For another thing, it will stop lots of Emacs Lisp from working.
>
> What specific packages/libraries/etc break with the current
> electric-indent-mode enabled?
>

FWIW, I've seen a lot of crufty site-specific elisp that barely works as 
it is. (Monkeypatching cc-mode seems perennially popular.) Can we please 
try not to break it further? I don't see why we have to change newline's 
behavior in order to make newline-and-indent behavior the default, which 
I agree we should do.

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Alan Mackenzie]

On Wed, Oct 16, 2013 at 01:55:07PM -0700, chad wrote:
> On 16 Oct 2013, at 11:58, Alan Mackenzie <acm@muc.de> wrote:

> > For another thing, it will stop lots of Emacs Lisp from working.

> What specific packages/libraries/etc break with the current
> electric-indent-mode enabled?

Haven't a clue.  Some will though.  One of my personal defuns breaks,
with the following sequence:

(newline)
(insert "(vi)")
(fill-paragraph)

Because of the unwanted indentation attached to `insert', which indents
the new "(vi)" to the same level as the previous line of text, the
`fill-paragraph' joins it to the previous paragraph.

> ~Chad

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stephen J. Turnbull]

Stefan Monnier writes:
 > > With respect, that is not the topic of this thread.  That topic is that
 > > with electric-indent-mode enabled, there is now no command, also no
 > > elisp function which has the traditional functionality of `newline'.
 > 
 > Quick, without looking at the source code of newline: what does
 > `newline' do that (insert "\n") or C-q C-j doesn't do?

    `newline' is an interactive built-in compiled Lisp function
      -- loaded from "/playpen/src/XEmacs/xemacs/lisp/simple.elc"
    (newline &optional N)

    Documentation:
    Insert a newline, and move to left margin of the new line if it's blank.
    The newline is marked with the text-property `hard'.
    With optional arg N, insert that many newlines.
    In Auto Fill mode, if no numeric arg, break the preceding line if it's
    long.

Look ma, no code browsing!  (N.B. The Emacs docstring is worded
differently but it says the same thing, modulo being more accurate
about `use-hard-newlines' and auto-fill-mode.)

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[martin rudalics]

> For what you want, `newline-and-indent' exists, as you know.

I've been talking about the command run by RET and that I'm lost when I
run emacs -Q and try to do some editing.  Otherwise, I don't care about
the standard behavior because I obviously have RET run my own command.
But if I were new to Emacs, I certainly would want RET to indent the new
line out of the box.

martin

Default behaviour of RET.

[Alan Mackenzie]

Hi, Martin.

On Wed, Oct 16, 2013 at 08:34:56PM +0200, martin rudalics wrote:
> > For what you want, `newline-and-indent' exists, as you know.

> I've been talking about the command run by RET and that I'm lost when I
> run emacs -Q and try to do some editing.  Otherwise, I don't care about
> the standard behavior because I obviously have RET run my own command.
> But if I were new to Emacs, I certainly would want RET to indent the new
> line out of the box.

I might agree with you, at least for programming modes.  I'm not so sure
about things like Text Mode.  But there must also be a ready way of doing
what RET currently does, inserting a new line without indenting it.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Default behaviour of RET.

[Eli Zaretskii]

> Date: Wed, 16 Oct 2013 19:26:42 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> But there must also be a ready way of doing what RET currently does,
> inserting a new line without indenting it.

What's wrong with "C-q C-j"?

Default behaviour of RET.

[Stephen J. Turnbull]

Alan Mackenzie writes:
 > On Wed, Oct 16, 2013 at 08:34:56PM +0200, martin rudalics wrote:

 > > I've been talking about the command run by RET

Which is easy enough to change.

 > I might agree with you, at least for programming modes.  I'm not so sure
 > about things like Text Mode.

I suspect I would find it really annoying, by analogy with the overly
aggressive quote-prefixing that is done by `filladapt'.

 > But there must also be a ready way of doing what RET currently
 > does, inserting a new line without indenting it.

You're enabling the confusaholics.  If the user doesn't like what RET
does, the the user can bind it to a different function (or, more
likely, invoke a mode that does that for her and perhaps customize the
mode).  Despite the subject, the interesting issue is "what should
`newline' do when invoked from code?"

The traditional docstring says that it moves to the left margin and
handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
that, and it's not what `newline' does when corrupted by
`electric-shock-mode'.  But I think it's useful behavior, and I think
programs should be able to rely on it (as opposed to users who can
modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
removing ?\n, or not invoke the mode in the first place).

Of course Stefan is correct that in Emacs, nothing is reliable.  I
don't understand why he thinks it's a virtue, though.

Re: Default behaviour of RET.

[Stefan Monnier]

> The traditional docstring says that it moves to the left margin and
> handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
> that, and it's not what `newline' does when corrupted by
> `electric-shock-mode'.  But I think it's useful behavior, and I think
> programs should be able to rely on it (as opposed to users who can
> modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
> removing ?\n, or not invoke the mode in the first place).

I can't remember ever seeing a piece of code which wants "the Emacs-23
newline behavior".  Usually it either wants to (insert "\n") or it wants
to simulate hitting RET.

This discussion would benefit from actual examples of code that
fall into neither "do whatever RET does" nor "insert \n".


        Stefan

Re: Default behaviour of RET.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Oct 16, 2013 at 08:47:04PM -0400, Stefan Monnier wrote:
> > The traditional docstring says that it moves to the left margin and
> > handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
> > that, and it's not what `newline' does when corrupted by
> > `electric-shock-mode'.  But I think it's useful behavior, and I think
> > programs should be able to rely on it (as opposed to users who can
> > modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
> > removing ?\n, or not invoke the mode in the first place).

> I can't remember ever seeing a piece of code which wants "the Emacs-23
> newline behavior".  Usually it either wants to (insert "\n") or it wants
> to simulate hitting RET.

I gave an example of such code in the post which opened this thread.  To
recap:

(newline)
(insert "(vi)")
(fill-paragraph)

The `-and-indent' behaviour messes up the indentation of "(vi)" and
causes it to get re-attached to the previous paragraph.

> This discussion would benefit from actual examples of code that
> fall into neither "do whatever RET does" nor "insert \n".

See above.  Any code which is interested in filling (or, possibly, even
margins, if anything actually uses these) will get broken by the
`-and-indent'.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Default behaviour of RET.

[Stefan Monnier]

>> > The traditional docstring says that it moves to the left margin and
>> > handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
>> > that, and it's not what `newline' does when corrupted by
>> > `electric-shock-mode'.  But I think it's useful behavior, and I think
>> > programs should be able to rely on it (as opposed to users who can
>> > modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
>> > removing ?\n, or not invoke the mode in the first place).
>> I can't remember ever seeing a piece of code which wants "the Emacs-23
>> newline behavior".  Usually it either wants to (insert "\n") or it wants
>> to simulate hitting RET.
> I gave an example of such code in the post which opened this thread.  To
> recap:

> (newline)
> (insert "(vi)")
> (fill-paragraph)

> The `-and-indent' behaviour messes up the indentation of "(vi)" and
> causes it to get re-attached to the previous paragraph.

I see why the above doesn't want the "simulate RET" behavior.
But I don't see why it wouldn't work just fine with the (insert
"\n") behavior?

>> This discussion would benefit from actual examples of code that
>> fall into neither "do whatever RET does" nor "insert \n".
> See above.  Any code which is interested in filling (or, possibly, even
> margins, if anything actually uses these) will get broken by the
> `-and-indent'.

"any code which..." is not concrete.


        Stefan

Re: Default behaviour of RET.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Oct 18, 2013 at 03:52:22PM -0400, Stefan Monnier wrote:
> >> > The traditional docstring says that it moves to the left margin and
> >> > handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
> >> > that, and it's not what `newline' does when corrupted by
> >> > `electric-shock-mode'.  But I think it's useful behavior, and I think
> >> > programs should be able to rely on it (as opposed to users who can
> >> > modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
> >> > removing ?\n, or not invoke the mode in the first place).

> >> I can't remember ever seeing a piece of code which wants "the Emacs-23
> >> newline behavior".  Usually it either wants to (insert "\n") or it wants
> >> to simulate hitting RET.

> > I gave an example of such code in the post which opened this thread.  To
> > recap:

> > (newline)
> > (insert "(vi)")
> > (fill-paragraph)

> > The `-and-indent' behaviour messes up the indentation of "(vi)" and
> > causes it to get re-attached to the previous paragraph.

> I see why the above doesn't want the "simulate RET" behavior.
> But I don't see why it wouldn't work just fine with the (insert
> "\n") behavior?

The `newline' tidies up the previous paragraph, in particular it fills
it.  It might also cause abbreviations to be expanded.

I could easily adapt this piece of code to cope with the `-and-indent'
which has been attached to `newline', but that's not the point.

The point is that there are a near infinite number of invocations of
`newline' in the wild, and an unknown proportion of these, possibly
quite high, will depend on `newline' doing precisely what its doc string
says.  Their existence makes the utility of electric-indent-mode, at the
moment, questionable.

> >> This discussion would benefit from actual examples of code that
> >> fall into neither "do whatever RET does" nor "insert \n".

> > See above.  Any code which is interested in filling (or, possibly, even
> > margins, if anything actually uses these) will get broken by the
> > `-and-indent'.

> "any code which..." is not concrete.

I've given you a concrete example of such code, several times.  It's not
a particularly bizarre piece of code, either.

electric-indent-mode's indentation of a new line is broken.  Please take
this behaviour out.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Default behaviour of RET.

[Stefan Monnier]

> The point is that there are a near infinite number of invocations of
> `newline' in the wild, and an unknown proportion of these, possibly
> quite high, will depend on `newline' doing precisely what its doc string

but also possibly quite low.

> says.  Their existence makes the utility of electric-indent-mode, at the
> moment, questionable.

As long as we're in the "unknown proportion", we can't tell.

>> >> This discussion would benefit from actual examples of code that
>> >> fall into neither "do whatever RET does" nor "insert \n".
>> > See above.  Any code which is interested in filling (or, possibly, even
>> > margins, if anything actually uses these) will get broken by the
>> > `-and-indent'.
>> "any code which..." is not concrete.
> I've given you a concrete example of such code, several times.

Right, but I said "examples" not "example".  So far I've seen one
concrete example, from your ~/.emacs (i.e. a problem that will affect
O(1) user).  That's not very compelling.


        Stefan

Re: Default behaviour of RET.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Oct 18, 2013 at 09:59:54PM -0400, Stefan Monnier wrote:
> > The point is that there are a near infinite number of invocations of
> > `newline' in the wild, and an unknown proportion of these, possibly
> > quite high, will depend on `newline' doing precisely what its doc string

> but also possibly quite low.

Just how low does it have to be before we can ignore it?

> > says.  Their existence makes the utility of electric-indent-mode, at the
> > moment, questionable.

> As long as we're in the "unknown proportion", we can't tell.

> >> >> This discussion would benefit from actual examples of code that
> >> >> fall into neither "do whatever RET does" nor "insert \n".

> >> > See above.  Any code which is interested in filling (or, possibly, even
> >> > margins, if anything actually uses these) will get broken by the
> >> > `-and-indent'.

> >> "any code which..." is not concrete.

> > I've given you a concrete example of such code, several times.

> Right, but I said "examples" not "example".  So far I've seen one
> concrete example, from your ~/.emacs (i.e. a problem that will affect
> O(1) user).  That's not very compelling.

OK.  I've done a find-grep in .../lisp for all *.el matching
'^[^;\n]*(newline\([ )]\|$\)'.  There are 199 matches.

There are 27 occurences of the exact string "(or (bolp) (newline))",
which looks like making sure point is at BOL in an output file.  For
example, in uce.el, we have the following code at L359:

        ;; If message doesn't end with a newline, insert it.
	(goto-char (point-max))
	(or (bolp) (newline)))

. In org-datetree.el, we have this, at L83:

      (goto-char (point-max))
      (or (bolp) (newline))
      (org-datetree-insert-line year))

.  This same idiom appears two other times in this file alone.  I looked
at quite a few other occurences of "(or (bolp) (newline))" and all the
ones I could understand instantly were about ensuring point is at BOL, as
indeed was my own personal code.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: Default behaviour of RET.

[Drew Adams]

> OK.  I've done a find-grep in .../lisp for all *.el matching
> '^[^;\n]*(newline\([ )]\|$\)'.  There are 199 matches.
> There are 27 occurences of the exact string "(or (bolp) (newline))",
> which looks like making sure point is at BOL in an output file.

I haven't been following this thread; sorry.  I don't have any calls
to `newline' in my code, FWIW.

Is the following a reasonable summary of the question & positions?

1. One position wants to change the behavior of `newline', so that
all uses of it automatically benefit from the new behavior.

2. The other position wants to keep `newline' as it is, perhaps for
some existing calls and in any case for some future calls.  This
position holds that the "old" behavior can be useful in some contexts.

Sounds like two different functions are in order, for the old and new
behaviors - au choix.  If so, it's not a big deal to update the Emacs
sources one way or the other.

But in case there is 3rd-party code that uses `newline', it seems
like the prudent choice would be to keep the existing name for the
existing behavior and give the new behavior a new name.  3rd-party
code that wants to "upgrade" to the new behavior can do that.  And
there would be no surprises.

Is there more to it that this?

Re: Default behaviour of RET.

[Alan Mackenzie]

[ personal mail ]

Hi, Drew.

On Sat, Oct 19, 2013 at 08:07:35AM -0700, Drew Adams wrote:
> > OK.  I've done a find-grep in .../lisp for all *.el matching
> > '^[^;\n]*(newline\([ )]\|$\)'.  There are 199 matches.
> > There are 27 occurences of the exact string "(or (bolp) (newline))",
> > which looks like making sure point is at BOL in an output file.

> I haven't been following this thread; sorry.  I don't have any calls
> to `newline' in my code, FWIW.

> Is the following a reasonable summary of the question & positions?

> 1. One position wants to change the behavior of `newline', so that
> all uses of it automatically benefit from the new behavior.

> 2. The other position wants to keep `newline' as it is, perhaps for
> some existing calls and in any case for some future calls.  This
> position holds that the "old" behavior can be useful in some contexts.

More or less.

> Sounds like two different functions are in order, for the old and new
> behaviors - au choix.  If so, it's not a big deal to update the Emacs
> sources one way or the other.

I think I've managed to persuade SM of that, now.  But I'm not convinced
his latest patch is the right thing.  It's probably the best I'll manage,
though.

> But in case there is 3rd-party code that uses `newline', it seems
> like the prudent choice would be to keep the existing name for the
> existing behavior and give the new behavior a new name.  3rd-party
> code that wants to "upgrade" to the new behavior can do that.  And
> there would be no surprises.

That command already exists, and is called `newline-and-indent'.

> Is there more to it that this?

'Fraid so!  The mechanism by which `newline' was changed was by adding a
function to `post-self-insert-hook' when electric-indent-mode is active,
and this function checked whether a newline had been added, and indented
the new line if it had.  My task, which seems to have been inordinately
difficult, was to persuade SM that this roundabout means of turning
`newline' into `newline-and-indent' is just as bad as the direct means.
It's been exhausting.  I hope the argument has now been won.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Default behaviour of RET.

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

I don't think the default behavior of RET should be changed, even in
programming modes.  We have two commands, RET and C-j, which insert a newline.
C-j indents, and RET doesn't.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Default behaviour of RET.

[Eli Zaretskii]

> Date: Sun, 20 Oct 2013 18:26:31 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: rudalics@gmx.at, stephen@xemacs.org, monnier@iro.umontreal.ca,
> 	drew.adams@oracle.com, emacs-devel@gnu.org
> 
> I don't think the default behavior of RET should be changed, even in
> programming modes.  We have two commands, RET and C-j, which insert a newline.
> C-j indents, and RET doesn't.

The days when keyboards had an LFD key that produced C-j are long
gone.  Nowadays, keyboards have only the RET key, and users expect it
to indent in programming modes, at least by default.  As things are
now, I need to customize every programming mode I use to do that.  I
think it's time for a change.

Re: Default behaviour of RET.

[Stefan Monnier]

>> but also possibly quite low.
> Just how low does it have to be before we can ignore it?

That was not the question.

Anyway, it does look like there are indeed many incorrect uses of
newline (most of those (or (bolp) (newline)) will break sooner or later
because of unexpected filling and/or abbrev/skeleton expansion).

So I installed a patch which makes `newline' only run
post-self-insert-hook when called interactively, so as to try and
preserve compatibility with those dubious uses.


        Stefan

Re: Default behaviour of RET.

[Alan Mackenzie]

Hello, Stefan.

On Sat, Oct 19, 2013 at 06:20:24PM -0400, Stefan Monnier wrote:
> >> but also possibly quite low.
> > Just how low does it have to be before we can ignore it?

> That was not the question.

> Anyway, it does look like there are indeed many incorrect uses of
> newline (most of those (or (bolp) (newline)) will break sooner or later
> because of unexpected filling and/or abbrev/skeleton expansion).

We don't know that the filling, etc., is unexpected.  The crucial point
is that all these secondary actions of `newline' happen on the _previous_
line, leaving the main point of newline: "Insert a newline, and move to
left margin of the new line ...." unaffected.

> So I installed a patch which makes `newline' only run
> post-self-insert-hook when called interactively, so as to try and
> preserve compatibility with those dubious uses.

Thanks.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Default behaviour of RET.

[Alan Mackenzie]

Hi, Stephen.

On Thu, Oct 17, 2013 at 08:17:01AM +0900, Stephen J. Turnbull wrote:
> Alan Mackenzie writes:
>  > On Wed, Oct 16, 2013 at 08:34:56PM +0200, martin rudalics wrote:

>  > > I've been talking about the command run by RET

> Which is easy enough to change.

>  > I might agree with you, at least for programming modes.  I'm not so sure
>  > about things like Text Mode.

> I suspect I would find it really annoying, by analogy with the overly
> aggressive quote-prefixing that is done by `filladapt'.

>  > But there must also be a ready way of doing what RET currently
>  > does, inserting a new line without indenting it.

> You're enabling the confusaholics.  If the user doesn't like what RET
> does, the the user can bind it to a different function (or, more
> likely, invoke a mode that does that for her and perhaps customize the
> mode).  Despite the subject, the interesting issue is "what should
> `newline' do when invoked from code?"

Indeed, that was in the title of the original post in the thread.  But
you seem to have misunderstood the situation: the current
electric-indent-mode effectively transforms `newline' into
`newline-and-indent', leaving no command for the user who doesn't like
this to bind RET to.

> The traditional docstring says that it moves to the left margin and
> handles auto-filling.  Eli's suggestion of `(insert "\n")' doesn't do
> that, and it's not what `newline' does when corrupted by
> `electric-shock-mode'.  But I think it's useful behavior, and I think
> programs should be able to rely on it (as opposed to users who can
> modify the behavior of `One-Flew-Over-the-Cuckoos-Nest-mode' by
> removing ?\n, or not invoke the mode in the first place).

Yes.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Matthias Meulien]

> (...) I discovered this whilst trying `electric-indent-mode' in 
> Text Mode. 

For the same reason it is impossible to write Python code with 
both `python-skeleton-autoinsert' set to t and Electric Indent 
mode enabled...

Say a buffer contains the following three lines:

class base(object): 
    pass 

-!-

Now you hit c l a s s SPC d e r i v e d RET b a s e RET RET
which results in:

class base(object): 
    pass 

    class derived(base):
        """-!-""" 

Unexpected isn't it?
-- 
Matthias Meulien

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> For the same reason it is impossible to write Python code with both
> python-skeleton-autoinsert' set to t and Electric Indent mode enabled...

> Say a buffer contains the following three lines:

> class base(object):  pass 

> -!-

> Now you hit c l a s s SPC d e r i v e d RET b a s e RET RET
> which results in:

> class base(object):  pass 

>    class derived(base):
>        """-!-""" 

> Unexpected isn't it?

I can't reproduce it on trunk, probably thanks to the new
electric-indent-inhibit, which is set in python-mode.


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Matthias Meulien]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

> (...) I can't reproduce it on trunk, probably thanks to the new
> electric-indent-inhibit, which is set in python-mode.

Good news! I encounter the problem I described with 24.3.1. Note that my
comment was misleading: It has nothing to do with
`python-skeleton-autoinsert'... And my post was badly formatted (I don't
know why): The `pass' keyword must be on its own indented line, like this:

class base(object):
>       pass
>

Sorry for the noise and many thanks!
-- 
Matthias

[-- Attachment #2: Type: text/html, Size: 763 bytes --]

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

>> (...) I can't reproduce it on trunk, probably thanks to the new
>> electric-indent-inhibit, which is set in python-mode.

> Good news! I encounter the problem I described with 24.3.1. Note that my
> comment was misleading: It has nothing to do with
> `python-skeleton-autoinsert'... And my post was badly formatted (I don't
> know why): The `pass' keyword must be on its own indented line, like this:

> class base(object):
>> pass
>> 

Hmm, then I don't know if it happens in trunk or not, because I don't
really know how to reproduce it or what result would be correct or not
(I don't know/use Python).


        Stefan

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Davis Herring]

>> class base(object):
>>> pass
> 
> Hmm, then I don't know if it happens in trunk or not, because I don't
> really know how to reproduce it or what result would be correct or not
> (I don't know/use Python).

I can help with the basic idea: most colons require a following
statement, either on the same line (one statement) or on all the
following lines indented more than the line with the colon.  (We're only
concerned with the latter case, which Python calls a suite.)

"pass" is a no-op used in such a context where no (real) statement is
desired.  Therefore, it makes sense for an Emacs Python mode to notice a
"pass" (along with certain other choices like "return") and presume that
the user wants it to end the suite -- that is, to use one fewer level of
indentation on the next line.  (Sometimes the user wants even less
indentation, so they type p a s s C-j DEL or so.)

The bug (if it exists) is that newline/newline-and-indent/whatever are
not implementing this behavior properly, resulting in blocks that the
user "obviously" means to be siblings being nested instead (when no DEL
is used).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Matthias Meulien]

>>> (...) I can't reproduce it on trunk, probably thanks to the 
>>> new electric-indent-inhibit, which is set in python-mode.

You are right: I've compiled the trunk (git revision: b670693) and it
works as expected!
-- 
Matthias

Re: electric-indent-mode: abolition of `newline' function is not the Right Thing.

[Stefan Monnier]

> You are right: I've compiled the trunk (git revision: b670693) and it
> works as expected!

Thank you for confirming,


        Stefan