Fixing post-self-insert-hook.

Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Emacs.

For some while, there's been a problem with post-self-insert-hook.  Not
the hook itself, but what is sometimes done with it.

What is sometimes done with it is to effect buffer changes additional to
the prime change caused by self-insert-function.  This is the case, for
example, with electric-pair-post-self-insert-function.  Usually, when
self-insert-function is called from a key binding, this is harmless.  The
key gets inserted, then the auxiliary buffer changes take place.  Fine.

What isn't fine is when self-insert-function is called from Lisp, as it
is 293 times from our sources, including from cc-cmds.el.  The calling
Lisp function expects (usually) exactly one copy of the pressed key to be
inserted into the buffer.  With the pertinent post-self-insert-hook
functions in place, this certainty is lost.  There could be none, one,
two, or even many characters inserted.  This can wreck the functionality
of the Lisp function.

This is precisely what happened in quite a few c-electric-* functions,
and the uneasy workaround was to bind post-self-insert-hook to nil around
calls to self-insert-function, giving certainty back to the processing,
and try to compensate for this elsewhere.

This is clearly unsatisfactory.

#########################################################################

Given that it is now (at least politically) impossible to ban buffer
changing post-self-insert-hook functions, I propose to change the time at
which the hook gets called.

Instead of getting called straight after the self-insert-command, it
should be called at the end of the command which called
self-insert-command.  Just before post-command-hook, perhaps.  Yes there
are details to be worked out.

This will make no difference to the usual self-insert-command call.  It
will, however, restore the certainty of processing to Lisp code such as
c-electric-brace without having to resort to ugly workarounds.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[Stefan Monnier]

> What is sometimes done with it is to effect buffer changes additional to
> the prime change caused by self-insert-function.

FWIW, I think the above "sometimes" really means "always" or "almost
always" ;-)
[ That was the primary motivation for the addition of this hook.  ]

> What isn't fine is when self-insert-function is called from Lisp, as it
> is 293 times from our sources, including from cc-cmds.el.

The question here is why those effects are undesirable while the other
effects (like auto-fill or abbrev expansion) aren't.

I suspect that those 293 uses fall into roughly 3 different camps:

- Those that really do want the full `self-insert-command` effects.
- Those that call `self-insert-command` mostly because the author didn't
  know better and they should really call `insert` instead.
- The rest that wants more than `insert` but less than
  `self-insert-command`.

The last group might indeed deserve a new function.


        Stefan


PS: I do have one regret regarding `post-self-insert-hook`: I should
    have defined a `self-insert-function` variable instead.  This is
    because some of the `post-self-insert-hook` functions would be
    cleaner if they could be turned into (add-function :around
    self-insert-function ...).
    Sadly, `add-function` didn't exist back then :-(

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Sep 17, 2021 at 16:04:58 -0400, Stefan Monnier wrote:
> > What is sometimes done with it is to effect buffer changes additional
> > to the prime change caused by self-insert-function.

> FWIW, I think the above "sometimes" really means "always" or "almost
> always" ;-)
> [ That was the primary motivation for the addition of this hook.  ]

> > What isn't fine is when self-insert-function is called from Lisp, as it
> > is 293 times from our sources, including from cc-cmds.el.

Actually, it was just 111, not 293.  Sorry.

> The question here is why those effects are undesirable while the other
> effects (like auto-fill or abbrev expansion) aren't.

It's because it's like making a buffer change in an after-change function
(that is an insertion or a deletion, not just a text-property change).
There's no way for the calling function to keep track of what's done.
This was the cause of bug #33794 in January 2019.

> I suspect that those 293 uses fall into roughly 3 different camps:

> - Those that really do want the full `self-insert-command` effects.
> - Those that call `self-insert-command` mostly because the author didn't
>   know better and they should really call `insert` instead.
> - The rest that wants more than `insert` but less than
>   `self-insert-command`.

> The last group might indeed deserve a new function.

That may be so, but the point is, an indeterminate part of these 111
calls is currently undefined.  The effect of self-insert-function called
from Lisp is wholly dependent on what happens to be in post-s-i-h.  You
might get no characters inserted, you might get 1 or 2, you might get
many.  You just can't know at programming time.

However, if the call to post-self-insert-hook were to be postponed to the
end of the function, all the 111 Lisp calls would be defined again, and
they would do the same as they did before post-s-i-h came along.

>         Stefan


> PS: I do have one regret regarding `post-self-insert-hook`: I should
>     have defined a `self-insert-function` variable instead.  This is
>     because some of the `post-self-insert-hook` functions would be
>     cleaner if they could be turned into (add-function :around
>     self-insert-function ...).
>     Sadly, `add-function` didn't exist back then :-(

I actually disagree with this (or I'm lacking some knowledge).  With M-:
post-self-insert-function<CR> you can see what's in the hook, and with a
little bit of delq, and so on, can change it for testing purposes.  I
don't think you can do any of that with an add-function type structure.
Please tell me if I'm wrong, here.  Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[João Távora]

Alan Mackenzie <acm@muc.de> writes:

> Actually, it was just 111, not 293.  Sorry.

Are you familiar with the GitHub platform? It's averse to the FSF and
software freedom in some ways, but it also hosts much source code and
has a moderately useful search tool for code "in the wild".

Here's a pertinent search "self-insert-command":

   https://github.com/search?p=6&q=self-insert-command&type=Code

It lists almost 83 thousand hits in emacs lisp, many in forks of Emacs,
and many more in user configurations or packages we don't know about.  I
found it curious that it doesn't take much page turning to find users
undoing problematic bindings in c-mode-base-map back to
self-insert-command.  I do this myself but seem to be joined by others.

I find also interesting how, given such pervasive usage of
self-insert-command and reasonable popularity of electric-pair-mode (8k
hits in the same search), the on-default of electric-indent-mode, etc,
our bug tracker isn't overflowing with bug reports about electric modes,
post-self-insert-hook or their "undefined behaviour" or "race condition"
(as you describe them).  The only ones I know about are about cc-mode.el

Best regards,
João

PS: If people know of electric-pair-mode related bugs for other major
modes (_not cc-mode_), please forward them to me for analysis.  Thanks
in advance.

Re: Fixing post-self-insert-hook.

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

On Fri, Sep 17, 2021 at 10:45:19PM +0100, João Távora wrote:
> Alan Mackenzie <acm@muc.de> writes:
> 
> > Actually, it was just 111, not 293.  Sorry.
> 
> Are you familiar with the GitHub platform? It's averse to the FSF and
> software freedom in some ways, but it also hosts much source code and
> has a moderately useful search tool for code "in the wild".
> 
> Here's a pertinent search "self-insert-command":
> 
>    https://github.com/search?p=6&q=self-insert-command&type=Code
> 
> It lists almost 83 thousand hits in emacs lisp, many in forks of Emacs,

It may well be that those numbers are inflated by the fact that Github
emphasises fork for things one would just do a branch (or just a local
clone which then languishes unproductively).

This emphasis on fork is probably fueled by Github's marketing strategy 
"We haz 329 gazillions repozitoriez!1!!. Which at some point handsomely
paid off.

IOW, I'd say many "forks" in Github have probably never seen a compile,
let alone an install. Those can't produce bug reports.

Watch the denominator in your statistics ;-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Fixing post-self-insert-hook.

[João Távora]

On Sat, Sep 18, 2021 at 7:09 AM <tomas@tuxteam.de> wrote:
>
> On Fri, Sep 17, 2021 at 10:45:19PM +0100, João Távora wrote:
> > Alan Mackenzie <acm@muc.de> writes:
> >
> > > Actually, it was just 111, not 293.  Sorry.
> >
> > Are you familiar with the GitHub platform? It's averse to the FSF and
> > software freedom in some ways, but it also hosts much source code and
> > has a moderately useful search tool for code "in the wild".
> It may well be that those numbers are inflated by the fact that Github
> emphasises fork for things one would just do a branch (or just a local
> clone which then languishes unproductively).

But you can't be sure of that, and that's part of the point.

> Watch the denominator in your statistics ;-)

This is why I called the tool "moderately useful" ;-)

My point in providing numbers is to point out that there are
more than the occurrences found in Emacs source tree. I'll go
out on a limb and speculate "quite more".  For making that
point, I think those figures are just fine.

João

Re: Fixing post-self-insert-hook.

[Stefan Monnier]

>> The question here is why those effects are undesirable while the other
>> effects (like auto-fill or abbrev expansion) aren't.
> It's because it's like making a buffer change in an after-change function
> (that is an insertion or a deletion, not just a text-property change).

I still don't see how the changes performed by auto-fill and abbrev
expansions are different in this respect (both of them are internal to
`self-insert-command` rather than being on `post-self-insert-hook`, for
historical reasons).  They aren't just making text-property changes.

> There's no way for the calling function to keep track of what's done.

That's usually the point: the calling function doesn't want to insert
a given char, but instead it wants to faithfully reproduce all the magic
things that happen when you hit the corresponding key.

After all, if the code just wanted to insert a given char, it
should/would have used `insert`, which is both shorter and
more efficient.

> That may be so, but the point is, an indeterminate part of these 111
> calls is currently undefined.

And I'm pretty sure that among those a significant fraction is already
similarly broken/undefined even when `post-self-insert-hook` is nil
because it would also break in the face of some auto-fill or
abbrev expansions.

> The effect of self-insert-function called from Lisp is wholly
> dependent on what happens to be in post-s-i-h.  You might get no
> characters inserted, you might get 1 or 2, you might get many.
> You just can't know at programming time.

Indeed.  If they want/need to know, then why do they call
`self-insert-command` instead of `insert`?
Until we know that, we can't know what's the better fix.

> However, if the call to post-self-insert-hook were to be postponed to the
> end of the function, all the 111 Lisp calls would be defined again, and
> they would do the same as they did before post-s-i-h came along.

[ I assume by "end of the function" you meant "after the command" such as
  via `post-command-hook`.  ]

It would introduce its own lot of regressions, of course.

> I actually disagree with this (or I'm lacking some knowledge).  With M-:
> post-self-insert-function<CR> you can see what's in the hook, and with a
> little bit of delq, and so on, can change it for testing purposes.  I
> don't think you can do any of that with an add-function type structure.
> Please tell me if I'm wrong, here.  Thanks!

Not wrong, no.  `M-:` indeed currently gives you the raw unreadable
bytecode, but if you do `C-h o foo-function` or if you use
`foo-function` in `M-x ielm`, you'll get something like:

    ELISP> (defvar foo-function #'ignore)
    foo-function
    ELISP> (add-function :around foo-function #'fun1)
    #f(advice-wrapper :around ignore fun1)
    
    ELISP> foo-function
    #f(advice-wrapper :around ignore fun1)
    
    ELISP> 

It's more verbose than the values we have on hooks, admittedly, but it's
somewhat readable.  This "passably readable" form is displayed whenever
the value is printed by `cl-print`.


        Stefan

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Stefan.

On Sat, Sep 18, 2021 at 10:15:02 -0400, Stefan Monnier wrote:
> >> The question here is why those effects are undesirable while the other
> >> effects (like auto-fill or abbrev expansion) aren't.
> > It's because it's like making a buffer change in an after-change function
> > (that is an insertion or a deletion, not just a text-property change).

> I still don't see how the changes performed by auto-fill and abbrev
> expansions are different in this respect (both of them are internal to
> `self-insert-command` rather than being on `post-self-insert-hook`, for
> historical reasons).  They aren't just making text-property changes.

They're defined.  The caller of self-insert-function can take account of
them.  Also only things like SPACE or LFD trigger these functionalities.

> > There's no way for the calling function to keep track of what's done.

> That's usually the point: the calling function doesn't want to insert
> a given char, but instead it wants to faithfully reproduce all the magic
> things that happen when you hit the corresponding key.

> After all, if the code just wanted to insert a given char, it
> should/would have used `insert`, which is both shorter and
> more efficient.

OK.  We're back at the point of speculating what all the
self-insert-commands are there for.  `insert' isn't actually fewer
characters to type, because one hast to extract the keypress from
this-command-keys, but I take the point.

> > That may be so, but the point is, an indeterminate part of these 111
> > calls is currently undefined.

> And I'm pretty sure that among those a significant fraction is already
> similarly broken/undefined even when `post-self-insert-hook` is nil
> because it would also break in the face of some auto-fill or
> abbrev expansions.

Maybe.

> > The effect of self-insert-function called from Lisp is wholly
> > dependent on what happens to be in post-s-i-h.  You might get no
> > characters inserted, you might get 1 or 2, you might get many.
> > You just can't know at programming time.

> Indeed.  If they want/need to know, then why do they call
> `self-insert-command` instead of `insert`?
> Until we know that, we can't know what's the better fix.

I suspect it's for commands which do something extra as well as
self-insert-function.  The c-electric-... commands fall into this
category.  Maybe the thinking is compatibility with self-i-c.

> > However, if the call to post-self-insert-hook were to be postponed to the
> > end of the function, all the 111 Lisp calls would be defined again, and
> > they would do the same as they did before post-s-i-h came along.

> [ I assume by "end of the function" you meant "after the command" such as
>   via `post-command-hook`.  ]

Sorry, yes, that's exactly what I meant.

> It would introduce its own lot of regressions, of course.

It might, I'm not convinced either way.

Right now, commands like c-electric-brace get by by binding
post-self-insert-hook to nil (thus getting predictability) then
explicitly calling electric-pair-post-self-insert-function a bit later.
This isn't good, since e-p-p-s-i-function is really an internal function
in elec-pair.el, without any guaranteed functionality.  I really need a
defined external function in elec-pair.el which would do the same thing
as e-p-p-s-i-function, but in addition return information about what
it's just done.  I've asked João about such an interface over the last
few years, now and then, but such has not yet been forthcoming.

> > I actually disagree with this (or I'm lacking some knowledge).  With M-:
> > post-self-insert-function<CR> you can see what's in the hook, and with a
> > little bit of delq, and so on, can change it for testing purposes.  I
> > don't think you can do any of that with an add-function type structure.
> > Please tell me if I'm wrong, here.  Thanks!

> Not wrong, no.  `M-:` indeed currently gives you the raw unreadable
> bytecode, but if you do `C-h o foo-function` or if you use
> `foo-function` in `M-x ielm`, you'll get something like:

>     ELISP> (defvar foo-function #'ignore)
>     foo-function
>     ELISP> (add-function :around foo-function #'fun1)
>     #f(advice-wrapper :around ignore fun1)
    
>     ELISP> foo-function
>     #f(advice-wrapper :around ignore fun1)
    
>     ELISP> 

> It's more verbose than the values we have on hooks, admittedly, but it's
> somewhat readable.  This "passably readable" form is displayed whenever
> the value is printed by `cl-print`.

Thanks.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[Stefan Monnier]

> They're defined.  The caller of self-insert-function can take account of
> them.  Also only things like SPACE or LFD trigger these functionalities.

abbrev expansion can take place for other keys than SPC and auto-fill
can happen for any char in `auto-fill-char`, so it's not limited to SPC
and LFD: it can happen virtually with any key, depending on the
user's config.

Indeed, in practice it tends to happen only for a few keys like SPC, but
if a code doesn't want `post-self-insert-hook` modifications, and it
presumes that auto-fill or abbrev expansion won't happen, then we're
back at the question about why does it call `self-insert-command` rather
than `insert`.

>> After all, if the code just wanted to insert a given char, it
>> should/would have used `insert`, which is both shorter and
>> more efficient.
> OK.  We're back at the point of speculating what all the
> self-insert-commands are there for.  `insert' isn't actually fewer
> characters to type, because one hast to extract the keypress from
> this-command-keys, but I take the point.

In my experience, there isn't a general solution to this problem (and
yes, I agree it's a problem; what I disagree with is the
characterization that it's specifically linked to
`post-self-insert-hook`).  We need to fix it on a case by case basis,
and I think it's OK to do that lazily by only fixing the problems when
we encounter them.

>> > The effect of self-insert-function called from Lisp is wholly
>> > dependent on what happens to be in post-s-i-h.  You might get no
>> > characters inserted, you might get 1 or 2, you might get many.
>> > You just can't know at programming time.
>
>> Indeed.  If they want/need to know, then why do they call
>> `self-insert-command` instead of `insert`?
>> Until we know that, we can't know what's the better fix.
>
> I suspect it's for commands which do something extra as well as
> self-insert-function.  The c-electric-... commands fall into this
> category.  Maybe the thinking is compatibility with self-i-c.

It's difficult to combine several extra behaviors in
`self-insert-command` in a modular way.  For that reason
`electric-indent-mode`, `electric-pair-mode`, and `electric-layout-mode`
know about each other to some extent (tho I strove to design them such
that they are independent).

For that same reason, `c-electric-*` needs to know about those others as
well :-(, tho it also has the option of just breaking them by using
plain `insert`.

>> It would introduce its own lot of regressions, of course.
> It might, I'm not convinced either way.

I can see a few "obvious" interactions with other post-command-hooks,
and based on the history of the subtle issues we've had with interactions
between `electric-*-mode`s I'm pretty sure it would take a few
iterations before getting something that works as well as what we have.

And of course, the effects that are currently undesired may still
be undesired when postponed to `post-command-hook`.

I think postponing to `post-command-hook` will just make the behavior
yet more complex and yet harder to control (e.g. you won't have the easy
option of let-binding `electric-*-mode` around the call to
`self-insert-command`).

> Right now, commands like c-electric-brace get by by binding
> post-self-insert-hook to nil (thus getting predictability) then
> explicitly calling electric-pair-post-self-insert-function a bit later.
> This isn't good, since e-p-p-s-i-function is really an internal function
> in elec-pair.el, without any guaranteed functionality.  I really need a
> defined external function in elec-pair.el which would do the same thing
> as e-p-p-s-i-function, but in addition return information about what
> it's just done.  I've asked João about such an interface over the last
> few years, now and then, but such has not yet been forthcoming.

Of course, IMO (and I believe João has the same position) the better
path forward is to drop that `c-electric-brace` code altogether and rely
on `electric-pair-mode` instead like all other modes do, so the need for
such extra API is definitely not very high in my list of priorities ;-)


        Stefan

Re: Fixing post-self-insert-hook.

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> it's just done.  I've asked João about such an interface over the last
> few years, now and then, but such has not yet been forthcoming.

I've made my opinion clear now and then, over the years.  But here I go
again:

'electric-pair-post-self-insert-function' (aka e-p-p-s-i-f) existed
before I have laid my programmer eyes on it.  It's not an internal
function, it has a defined protocol and exists to be a member of
'post-self-insert-hook'.  Its purpose is -- and has always been -- to
insert aditional delimiters related to the one just typed or,
alternatively, to delete the character just inserted when it happens to
skip a closing delimiter.  In Emacs, as you know, a delimiter is a well
defined thing, thanks to syntax tables.  e-p-m's functionality, known
sometimes as auto-pairing, is implemented in numerous editors.  I first
saw it in TextMate around 2008.  In Emacs, apart from e-p-m, extensions
such as autopair, smartparens, paredit (and probably others) do the
same.

The only significant thing I did in e-p-m -- besides a boring code
refactor coordinated with then-maintainer Stefan -- was add the ability
to choose which action to take based on the information conveyed by
syntax-ppss, so that a degree of delimiter balance is maintained.  As
you known, syntax-ppss, relies again on that fundamental Emacs
abstraction -- the syntax-table -- that buffers have.  This works very
well in practice and is a testament to how useful simple syntax tables
are.

Nevertheless, if you want to extract things from e-p-p-s-i-f and use
them in your function, you can refactor its innards.  As long as you
don't break anything (fortunately there are good tests) or convolute
elec-pair.el's code beyond recognition.  Don't expect me to spend my
time any time in this effort: I won't.  I think it would be a very gross
waste of time.  _My_ time, at least.  You may convince someone else of
that idea.

My alternative suggestion, which I've offered now and then, over the
years, is for you to study how cc-mode behaves when you bind the
delimiter keys simply to 'self-insert-command' and refrain from
re-inventing 'electric-pair-mode' inside cc-mode.  I do this in my
.emacs, and I see that other users have settled on the same practice.
If you were to try that, you could theoretically arrive at the
conclusion that it is possible to do everything cc-mode does today
w.r.t. electricity via the existing interfaces of 'electric-pair-mode',
'electric-layout-mode' and 'electric-indent-mode'.

I must be honest.  I don't really expect you arrive at that conclusion
or even try that experiment.  That's OK, I can't force anyone to reason
the way I do.  My personal hopes for C/C++/Java editing in Emacs are a
completely new major mode where syntax is handled by TreeSitter and all
the keys are bound to 'self-insert-command'.  I'd bet good money that
'electric-pair-mode' is going to work there fine and with 0
configuration, like it does in all other major modes.

With best regards,
João

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, João

On Sat, Sep 18, 2021 at 23:55:10 +0100, João Távora wrote:
> João Távora <joaotavora@gmail.com> writes:

> > it's just done.  I've asked João about such an interface over the last
> > few years, now and then, but such has not yet been forthcoming.

> I've made my opinion clear now and then, over the years.  But here I go
> again:

> 'electric-pair-post-self-insert-function' (aka e-p-p-s-i-f) existed
> before I have laid my programmer eyes on it.  It's not an internal
> function, it has a defined protocol ....

It doesn't even have a doc string.

> .... and exists to be a member of 'post-self-insert-hook'.

Exactly.  There is a need for a stable functional interface to
electric-pair-mode which CC Mode can use.  Despite how little effort it
would be to create and formalise this, you have refused to do so over
the years.

> Its purpose is -- and has always been -- to insert aditional
> delimiters related to the one just typed or, alternatively, to delete
> the character just inserted when it happens to skip a closing
> delimiter.

Yes.  Would you please write this into a doc string for
electric-pair-post-self-insert-function, together with information about
where that function gets its information from, and what it does with
point.

[ .... ]

> Nevertheless, if you want to extract things from e-p-p-s-i-f and use
> them in your function, you can refactor its innards.  As long as you
> don't break anything (fortunately there are good tests) or convolute
> elec-pair.el's code beyond recognition.  Don't expect me to spend my
> time any time in this effort: I won't.  I think it would be a very gross
> waste of time.  _My_ time, at least.  You may convince someone else of
> that idea.

What a wonderful attitude for a maintainer.  ;-(

You will perhaps recall bug #33794 "26.1; electric-pair-mode breaks
auto-newline minor mode of cc-mode".  Despite a large part of the
problem lying within electric-pair-mode, I don't think you changed a
single line of your code to fix the problem.  As a result, I was forced
to implement ugly workarounds, which are still there almost three years
later, the alternative being leaving the bug unfixed.

> My alternative suggestion, which I've offered now and then, over the
> years, is for you to study how cc-mode behaves when you bind the
> delimiter keys simply to 'self-insert-command' ....

You know full well that this is problematic.

> and refrain from re-inventing 'electric-pair-mode' inside cc-mode.

I sometimes wonder if such a reimplementation inside CC Mode might have
been less work than all the email exchange with you trying to get you to
fix things.

> I do this in my .emacs, and I see that other users have settled on the
> same practice.  If you were to try that, you could theoretically
> arrive at the conclusion that it is possible to do everything cc-mode
> does today w.r.t. electricity via the existing interfaces of
> 'electric-pair-mode', 'electric-layout-mode' and
> 'electric-indent-mode'.

Sure, anything's possible.  But doing so whilst retaining functionality
would fragment the code, and lead inevitably to bugs in what is
currently an exceptionally stable part of CC Mode.

> I must be honest.  I don't really expect you arrive at that conclusion
> or even try that experiment.

No.  I have a user base to consider.

> That's OK, I can't force anyone to reason the way I do.  My personal
> hopes for C/C++/Java editing in Emacs are a completely new major mode
> where syntax is handled by TreeSitter and all the keys are bound to
> 'self-insert-command'.  I'd bet good money that 'electric-pair-mode'
> is going to work there fine and with 0 configuration, like it does in
> all other major modes.

> With best regards,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Sun, Sep 19, 2021, 13:14 Alan Mackenzie <acm@muc.de> wrote:

> > I must be honest.  I don't really expect you arrive at that conclusion
> > or even try that experiment.
>
> No.  I have a user base to consider.
>

In fact, I shouldn't even have expected you to read what I wrote, otherwise
you
wouldn't have replied with such a bizarre conclusion. I'll write a
docstring for e-p-p-s-i-f.

Best regards,
João



>

[-- Attachment #2: Type: text/html, Size: 1063 bytes --]

Re: Fixing post-self-insert-hook.

[Stefan Monnier]

>> My alternative suggestion, which I've offered now and then, over the
>> years, is for you to study how cc-mode behaves when you bind the
>> delimiter keys simply to 'self-insert-command' ....
>
> You know full well that this is problematic.

I can't speak for João, but no I don't.

>> and refrain from re-inventing 'electric-pair-mode' inside cc-mode.
> I sometimes wonder if such a reimplementation inside CC Mode might have

I don't understand the "might".  You do have such a "reimplementation"
(and you even implemented it before `electric-*-mode`).

The difficulty you're facing is due to the face that contrary to all
other major modes which used to have such features, you decided to keep
your implementation and try to make it interact correctly with
`electric-*-mode`s.  There's no doubt that this is hard.  I do doubt
whether it's useful, OTOH.

I understand your desire to preserve exactly the featureset you designed
for CC-mode, rather than rely on the `electric-*-mode`s features which
aren't exactly equivalent and aren't configured in the same way.
So you gave more importance to preserving compatibility with older
Emacs/CC-mode, whereas I give more importance to harmonization of
configuration and behavior across major modes.

> been less work than all the email exchange with you trying to get you to
> fix things.

There is a simpler solution at hand.  You say it's "problematic", but
all solutions have their downsides.

>> I must be honest.  I don't really expect you arrive at that conclusion
>> or even try that experiment.
> No.  I have a user base to consider.

Some of your user base would appreciate not having to do things
differently in CC-modes than in other modes.


        Stefan

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Sep 19, 2021 at 08:59:59 -0400, Stefan Monnier wrote:
> >> My alternative suggestion, which I've offered now and then, over the
> >> years, is for you to study how cc-mode behaves when you bind the
> >> delimiter keys simply to 'self-insert-command' ....

> > You know full well that this is problematic.

> I can't speak for João, but no I don't.

OK, I'm telling you now it is problematic.

> >> and refrain from re-inventing 'electric-pair-mode' inside cc-mode.
> > I sometimes wonder if such a reimplementation inside CC Mode might have

> I don't understand the "might".  You do have such a "reimplementation"
> (and you even implemented it before `electric-*-mode`).

And we're now reduced to silly discussions about the precise meanings of
words.  There is no implementation of the electric pair mode
functionality in CC Mode.  There is a call to an unofficial function in
elec-pair.el which currently happens to work.  There is no better
interface available, and João seems determined that there won't be one.

> The difficulty you're facing is due to the face that contrary to all
> other major modes which used to have such features, ....

There were no other such modes.  Or were there?  Which modes at the time
had electric indentation, or auto newline?

> .... you decided to keep your implementation and try to make it
> interact correctly with `electric-*-mode`s.

More precisely, the difficulties were caused by electric-*-mode being
developed in such a way as to be incompatible with existing major modes,
in particular CC Mode modes.  Perhaps you might like to say something
about why you did this.  It is (at least in hindsight) obvious there
would be incompatibilities with CC Mode.

I don't recall there being any discussion on emacs-devel at the time
these electric-*-modes were being planned.  If there had been, the
friction which has happened since might well have been avoided.

So, why did you design these new modes in such an incompatible fashion?

It's perhaps worthwhile to come back to a main incompatibility
introduced, and that is the original topic of this thread,
post-self-insert-hook.  This broke self-insert-function, at least as
used in CC Mode.  This is what has forced the ugly workarounds on CC
Mode.  Again, the question why?

[ .... ]

> I understand your desire to preserve exactly the featureset you designed
> for CC-mode, ....

More the features introduced by Barry Warsaw, Martin Stjernholm and
their predecessors.  They're good features, not obsolete, and change for
change's sake is never something I've been keen on.

> .... rather than rely on the `electric-*-mode`s features which aren't
> exactly equivalent and aren't configured in the same way.

They're worse features, from CC Mode's point of view.  They're more
complicated, more fragmented (you yourself admitted to cross coupling
between the mechanisms of the different electric-*-modes), more
difficult to debug, and so on.  All this compared with a collection of
straightforwardly written commands, c-electric-brace and friends, which
just work, and have done for several decades, and would be exceptionally
easy to debug if that were ever needed.

> So you gave more importance to preserving compatibility with older
> Emacs/CC-mode, whereas I give more importance to harmonization of
> configuration and behavior across major modes.

If you really wanted such harmonization, why did you create these modes
with such gross incompatibilities with CC Mode?  Why didn't you discuss
these things with me or Martin Stjernholm first, so that we could have
developed things co-operatively rather than you going your own way and
trying to force the result on CC Mode?

> > been less work than all the email exchange with you trying to get
> > you to fix things.

> There is a simpler solution at hand.  You say it's "problematic", but
> all solutions have their downsides.

It's problematic, and anything but simple.

> >> I must be honest.  I don't really expect you arrive at that conclusion
> >> or even try that experiment.
> > No.  I have a user base to consider.

> Some of your user base would appreciate not having to do things
> differently in CC-modes than in other modes.

That's pure speculation.  Also speculation: CC Mode users appreciate not
having to report bugs for c-electric-brace.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[Stefan Monnier]

>> Some of your user base would appreciate not having to do things
>> differently in CC-modes than in other modes.
> That's pure speculation.

Definitely not.  I have no idea how important that part of the user base
is, but it is a fact that it is not the empty set.


        Stefan

Re: Fixing post-self-insert-hook.

[João Távora]

Alan Mackenzie <acm@muc.de> writes:

> What isn't fine is when self-insert-function is called from Lisp, as it
> is 293 times from our sources, including from cc-cmds.el.  The calling
[...]
> Instead of getting called straight after the self-insert-command, it
> should be called at the end of the command which called
> self-insert-command.  Just before post-command-hook, perhaps.  Yes there
> are details to be worked out.

Let me get this straight: are you are proposing that 283 +
who-knows-how-many third-party references to 'self-insert-command' --
which all have been working fine as far as we know -- should _change_ to
accomodate the 10 problematic calls that are found in cc-cmds.el?

Best regards,
João

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, João.

On Fri, Sep 17, 2021 at 21:15:33 +0100, João Távora wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > What isn't fine is when self-insert-function is called from Lisp, as it
> > is 293 times from our sources, including from cc-cmds.el.  The calling
> [...]
> > Instead of getting called straight after the self-insert-command, it
> > should be called at the end of the command which called
> > self-insert-command.  Just before post-command-hook, perhaps.  Yes there
> > are details to be worked out.

> Let me get this straight: are you are proposing that 283 +
> who-knows-how-many third-party references to 'self-insert-command' --
> which all have been working fine as far as we know -- should _change_ to
> accomodate the 10 problematic calls that are found in cc-cmds.el?

Actually, I miscounted, sorry, it's 111 calls to self-insert-command.
But that's still a lot.

The point is, these calls are currently undefined.  Pretty much every
last one of them, apart from the ones in cc-cmds.el, which have been
made defined again.

These 111 calls might have been working, they might not, they might have
sort of been working as long as electric-pair-mode isn't enabled.  Who
knows?  They're currently in a sort of race condition.

But most of these calls, I guess, were there long before
post-self-insert-hook, and they were working then.  They would work
again if the call to p-s-i-h was delayed.

So, yes, you're basically right.  We should move the call to p-s-i-h not
just to fix the interface to electric-pair-mode and friends, but to
bring sanity back to Emacs with respect to this matter.

You have recently said you don't want to change the mechanism of the
electric-... functionality.  This is one way to fix it without you
having to make any such changes.  Can you see any concrete problems with
making this change?

> Best regards,
> João

--  
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[João Távora]

On Fri, Sep 17, 2021 at 9:35 PM Alan Mackenzie <acm@muc.de> wrote:

> > Let me get this straight: are you are proposing that 283 +
> > who-knows-how-many third-party references to 'self-insert-command' --
> > which all have been working fine as far as we know -- should _change_ to
> > accomodate the 10 problematic calls that are found in cc-cmds.el?
>
> So, yes, you're basically right.

Thanks.  Just checking.

Best regards,
João

Re: Fixing post-self-insert-hook.

[Eli Zaretskii]

> Date: Fri, 17 Sep 2021 20:35:50 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> The point is, these calls are currently undefined.  Pretty much every
> last one of them, apart from the ones in cc-cmds.el, which have been
> made defined again.
> 
> These 111 calls might have been working, they might not, they might have
> sort of been working as long as electric-pair-mode isn't enabled.  Who
> knows?  They're currently in a sort of race condition.

If they aren't working, where are the bug reports about that?  We only
have bug reports about CC mode so far.

> So, yes, you're basically right.  We should move the call to p-s-i-h not
> just to fix the interface to electric-pair-mode and friends, but to
> bring sanity back to Emacs with respect to this matter.

I'm against such changes, as you well know.  With Emacs, "sane" is
mostly defined as the status quo, because Emacs "mostly works".  Any
significant change in such a low-level facility is, therefore,
"insane", IMNSHO, unless the problem is also general.  And we don't
have evidence that the problem is global.

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Eli.

On Sat, Sep 18, 2021 at 08:59:54 +0300, Eli Zaretskii wrote:
> > Date: Fri, 17 Sep 2021 20:35:50 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: emacs-devel@gnu.org

> > The point is, these calls are currently undefined.  Pretty much every
> > last one of them, apart from the ones in cc-cmds.el, which have been
> > made defined again.

> > These 111 calls might have been working, they might not, they might have
> > sort of been working as long as electric-pair-mode isn't enabled.  Who
> > knows?  They're currently in a sort of race condition.

> If they aren't working, where are the bug reports about that?

They don't work, at least some of them.  That is evident from examining
the source code and mechanisms involved.  As for bug reports, there are
many thousand open ones which Lars is heroically working through, some
of which don't have obvious causes.

> We only have bug reports about CC mode so far.

The bugs are, however, not in CC Mode and can't be fixed there.  They
can only be worked around, which is what has been done since bug #33794
at the end of 2018.  This workaround is ugly and caused bad feeling at
the time.

> > So, yes, you're basically right.  We should move the call to p-s-i-h not
> > just to fix the interface to electric-pair-mode and friends, but to
> > bring sanity back to Emacs with respect to this matter.

> I'm against such changes, as you well know.

Indeed!  I can only respect that whilst disagreeing strongly with it.
I've never been a fan of "it mostly works, so don't touch it!".

> With Emacs, "sane" is mostly defined as the status quo, because Emacs
> "mostly works".  Any significant change in such a low-level facility
> is, therefore, "insane", IMNSHO, unless the problem is also general.

The problem is indeed general, in the sense that self-insert-function is
general.  The effects of the problem have only been felt in the
electric-... minor modes so far.

> And we don't have evidence that the problem is global.

We have a complete understanding of how self-insert-function is broken.
This has had damaging effects in the interface between
electric-pair-mode and CC Mode for which workarounds were installed in
CC Mode.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[Eli Zaretskii]

> Date: Fri, 17 Sep 2021 19:37:27 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: João Távora <joaotavora@gmail.com>
> 
> Given that it is now (at least politically) impossible to ban buffer
> changing post-self-insert-hook functions, I propose to change the time at
> which the hook gets called.
> 
> Instead of getting called straight after the self-insert-command, it
> should be called at the end of the command which called
> self-insert-command.  Just before post-command-hook, perhaps.  Yes there
> are details to be worked out.
> 
> This will make no difference to the usual self-insert-command call.  It
> will, however, restore the certainty of processing to Lisp code such as
> c-electric-brace without having to resort to ugly workarounds.

If CC Mode has problem with these hooks, it could bind them to nil
around the call to self-insert-command, couldn't it?  That'd be much
better than making any globally-visible change in behavior, for which
we cannot possibly know the unintended consequences.

In any case, please let's not make this change before the emacs-28
branch is cut, as it can potentially disrupt many places.

Re: Fixing post-self-insert-hook.

[Alan Mackenzie]

Hello, Eli.

On Sat, Sep 18, 2021 at 08:50:26 +0300, Eli Zaretskii wrote:
> > Date: Fri, 17 Sep 2021 19:37:27 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: João Távora <joaotavora@gmail.com>

> > Given that it is now (at least politically) impossible to ban buffer
> > changing post-self-insert-hook functions, I propose to change the time at
> > which the hook gets called.

> > Instead of getting called straight after the self-insert-command, it
> > should be called at the end of the command which called
> > self-insert-command.  Just before post-command-hook, perhaps.  Yes there
> > are details to be worked out.

> > This will make no difference to the usual self-insert-command call.  It
> > will, however, restore the certainty of processing to Lisp code such as
> > c-electric-brace without having to resort to ugly workarounds.

> If CC Mode has problem with these hooks, it could bind them to nil
> around the call to self-insert-command, couldn't it?

That has indeed been done since early 2019.  It's not nice.  It involves
c-electric-brace knowing that one of the entries in post-self-insert-hook
is electric-pair-post-self-insert-function, and calling it explicitly.
It couples the electric-... minor modes with CC Mode, and blocks out any
other functionality on the hook from CC Mode.

> That'd be much better than making any globally-visible change in
> behavior, for which we cannot possibly know the unintended
> consequences.

The mechanism is currently broken.  Do we stick with this known breakage
for fear of an unknown, not particularly likely one, or do we fix it?

> In any case, please let's not make this change before the emacs-28
> branch is cut, as it can potentially disrupt many places.

Yes.  But surely we have enough time before Emacs 29 for any problems it
might cause to come to light.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Fixing post-self-insert-hook.

[Eli Zaretskii]

> Date: Sat, 18 Sep 2021 09:57:19 +0000
> Cc: emacs-devel@gnu.org, joaotavora@gmail.com
> From: Alan Mackenzie <acm@muc.de>
> 
> > If CC Mode has problem with these hooks, it could bind them to nil
> > around the call to self-insert-command, couldn't it?
> 
> That has indeed been done since early 2019.  It's not nice.  It involves
> c-electric-brace knowing that one of the entries in post-self-insert-hook
> is electric-pair-post-self-insert-function, and calling it explicitly.
> It couples the electric-... minor modes with CC Mode, and blocks out any
> other functionality on the hook from CC Mode.
> 
> > That'd be much better than making any globally-visible change in
> > behavior, for which we cannot possibly know the unintended
> > consequences.
> 
> The mechanism is currently broken.  Do we stick with this known breakage
> for fear of an unknown, not particularly likely one, or do we fix it?

The former, I hope.

> > In any case, please let's not make this change before the emacs-28
> > branch is cut, as it can potentially disrupt many places.
> 
> Yes.  But surely we have enough time before Emacs 29 for any problems it
> might cause to come to light.

I don't know.  We don't even have any clear idea of what problems it
causes in practice, only some vague fears of such problems.