(insert ?\n) spuriously calls before-change-functions twice. Help!

(insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

Hi, Emacs.

The following is a partial diagnosis of a bug reported by an AWK user.
(AWK Mode throws an error when require-final-newline is t).

In emacs -Q, in an AWK buffer, do M-: (insert ?\n).  The following
happens:
1. before-change-functions is invoked with arguments (beg end) (16 16).
2. The NL is inserted at position 16.
3. before-change-functions is invoked with arguments (16 17).
This all happens before after-change-functions is invoked.

However, M-: (insert "\n") just does the Right Thing.

This second invocation of before-change-functions is surely a bug.  I
can make this happen in a Fundamental Mode buffer in my normal Emacs
session, but not in Emacs -Q.

What is causing this spurious invocation?  Does anybody have any idea?
I've perused the C source for `insert', without finding anything.  Could
it be anything to do with text-properties?

Thanks for the help!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Stefan Monnier]

> In emacs -Q, in an AWK buffer, do M-: (insert ?\n).  The following
[...]
> However, M-: (insert "\n") just does the Right Thing.

That's curious indeed.

> This second invocation of before-change-functions is surely a bug.  I
> can make this happen in a Fundamental Mode buffer in my normal Emacs
> session, but not in Emacs -Q.

Can you try to figure out which diference between "emacs -Q" and your
"normal Emacs session" triggers the problem?


        Stefan

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

Hi, Stefan,

On Tue, Jan 05, 2010 at 10:22:26AM -0500, Stefan Monnier wrote:
> > In emacs -Q, in an AWK buffer, do M-: (insert ?\n).  The following
> [...]
> > However, M-: (insert "\n") just does the Right Thing.

> That's curious indeed.

> > This second invocation of before-change-functions is surely a bug.  I
> > can make this happen in a Fundamental Mode buffer in my normal Emacs
> > session, but not in Emacs -Q.

> Can you try to figure out which diference between "emacs -Q" and your
> "normal Emacs session" triggers the problem?

I think I've tracked down what's happening.  After inserting the ?\n (any
other character is just the same), Emacs clears any text properties from
the new character by calling set_properties.  This, in its turn, invokes
before-change-functions.  A more detailed call stack is:

Finsert calls
  general_insert_function (insert, insert_from_string, 0, nargs, args) calls
    (*insert_func) (str, len), namely insert (str, len), calls
      insert_1_both (string, len, nbytes, 0, 1, 0), calls
        prepare_to_modify_buffer, calls
        signal_before_change.  This does the first invocation of b_c_f. <======

        insert_1_both inserts the character ?\n into the buffer.

        insert_1_both (conditionally on BUF_INTERVALS) calls
          set_text_properties on the newly inserted character, calls
            modify_region, calls
              prepare_to_modify_buffer, calls
                signal_before_change.  This does the second invocation of
                                       b_c_f.    <=============================

There doesn't seem to be any system in where the two change hooks are
invoked.  For example, insert directly calls signal_after_change, yet not
signal_before_change; surely invocations of these should be paired.

I think there needs to be two levels of these primitive: a "user level",
which invokes the change hooks, and calls the "internal level" to do the
actual work.  Other primitives would call only the "internal level".  Or
something like that.

For CC Mode, I will put in a test for two consecutive invocations of
before-change-functions without an after-change-functions between them.
In such a case, I then ignore the second invocation.  I think.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Stefan Monnier]

> I think I've tracked down what's happening.  After inserting the ?\n (any
> other character is just the same), Emacs clears any text properties from
> the new character by calling set_properties.  This, in its turn, invokes
> before-change-functions.  A more detailed call stack is:

Sounds like a bug indeed.

> There doesn't seem to be any system in where the two change hooks are
> invoked.  For example, insert directly calls signal_after_change, yet not
> signal_before_change; surely invocations of these should be paired.

Agreed.  Patch very welcome,

> For CC Mode, I will put in a test for two consecutive invocations of
> before-change-functions without an after-change-functions between them.
> In such a case, I then ignore the second invocation.  I think.

Even if we fix the bug, there can be any number of reasons why the two
hooks may occasionally not be paired, so your hooks should be robust
against such situations.  IOW it's best to try and avoid relying in the
after-hook on info passed from the before hook.


        Stefan

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

Hi, Stefan,

On Tue, Jan 05, 2010 at 02:52:37PM -0500, Stefan Monnier wrote:
> > I think I've tracked down what's happening.  After inserting the ?\n (any
> > other character is just the same), Emacs clears any text properties from
> > the new character by calling set_properties.  This, in its turn, invokes
> > before-change-functions.  A more detailed call stack is:

> Sounds like a bug indeed.

> > There doesn't seem to be any system in where the two change hooks are
> > invoked.  For example, insert directly calls signal_after_change, yet not
> > signal_before_change; surely invocations of these should be paired.

> Agreed.  Patch very welcome,

I'll look at this.  It looks impractical to change the source to do this.
signal_after_change seems usually to be paired with modify_region or
prepare_to_modify_buffer.  But I'll certainly fix the current bug, and
check (as far as I can) for other circumstances it might happen.

> > For CC Mode, I will put in a test for two consecutive invocations of
> > before-change-functions without an after-change-functions between
> > them.  In such a case, I then ignore the second invocation.  I think.

> Even if we fix the bug, there can be any number of reasons why the two
> hooks may occasionally not be paired, .....

Really?  I would have thought they should be rigorously paired, without
any "recursive" invocations in-between (barring elisp code doing this
deliberately).

> .... so your hooks should be robust against such situations.  IOW it's
> best to try and avoid relying in the after-hook on info passed from the
> before hook.

Haven't we discussed this before at some time?  ;-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Stefan Monnier]

>> Even if we fix the bug, there can be any number of reasons why the two
>> hooks may occasionally not be paired, .....
> Really?  I would have thought they should be rigorously paired, without
> any "recursive" invocations in-between (barring elisp code doing this
> deliberately).

There's combine-after-change-functions for one.  Or there's the case
where a signal is thrown in between the two.  Of course, there's also
the nesting case.

>> .... so your hooks should be robust against such situations.  IOW it's
>> best to try and avoid relying in the after-hook on info passed from the
>> before hook.
> Haven't we discussed this before at some time?  ;-)

You mean I should just have said "Told ya!"?


        Stefan

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

Hi, Stefan!

On Tue, Jan 05, 2010 at 02:52:37PM -0500, Stefan Monnier wrote:
> > I think I've tracked down what's happening.  After inserting the ?\n (any
> > other character is just the same), Emacs clears any text properties from
> > the new character by calling set_properties.  This, in its turn, invokes
> > before-change-functions.  A more detailed call stack is:

> Sounds like a bug indeed.

> > There doesn't seem to be any system in where the two change hooks are
> > invoked.  For example, insert directly calls signal_after_change, yet not
> > signal_before_change; surely invocations of these should be paired.

> Agreed.  Patch very welcome,

DONE.  Specifically, revision 99376, fixing set_text_properties in
textprop.c.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

>> > There doesn't seem to be any system in where the two change hooks are
>> > invoked.  For example, insert directly calls signal_after_change, yet not
>> > signal_before_change; surely invocations of these should be paired.
>
>> Agreed.  Patch very welcome,
>
> DONE.  Specifically, revision 99376, fixing set_text_properties in
> textprop.c.

Please commit a ChangeLog entry as well.

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

Hi, Yidong,

On Tue, Jan 19, 2010 at 10:11:47AM -0500, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> >> > There doesn't seem to be any system in where the two change hooks are
> >> > invoked.  For example, insert directly calls signal_after_change, yet not
> >> > signal_before_change; surely invocations of these should be paired.

> >> Agreed.  Patch very welcome,

> > DONE.  Specifically, revision 99376, fixing set_text_properties in
> > textprop.c.

> Please commit a ChangeLog entry as well.

There is a ChangeLog entry, just that it's way down at 2010-01-08 (when I
made the change on my own PC).  I really ought to have "updated" it to
show today's date, I think.  With bzr, there's no longer a unique date of
committing.

I'll change this around to put my ChangeLog entry with and at today's
date.

Sorry for not thinking this through myself.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

>> Please commit a ChangeLog entry as well.
>
> There is a ChangeLog entry, just that it's way down at 2010-01-08 (when I
> made the change on my own PC).  I really ought to have "updated" it to
> show today's date, I think.  With bzr, there's no longer a unique date of
> committing.
>
> I'll change this around to put my ChangeLog entry with and at today's
> date.

Ah, I see.  Thanks.

Re: (insert ?\n) spuriously calls before-change-functions twice. Help!

[Alan Mackenzie]

On Tue, Jan 19, 2010 at 01:02:23PM -0500, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> >> Please commit a ChangeLog entry as well.

> > There is a ChangeLog entry, just that it's way down at 2010-01-08 (when I
> > made the change on my own PC).  I really ought to have "updated" it to
> > show today's date, I think.  With bzr, there's no longer a unique date of
> > committing.

> > I'll change this around to put my ChangeLog entry with and at today's
> > date.

> Ah, I see.  Thanks.

DONE.

-- 
Alan Mackenzie (Nuremberg, Germany).