Unbalanced change hooks (part 2)

Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Emacs.

There are several functions in .../src/insdel.c which have a parameter
`prepare'.  The meaning of `prepare' is "direct the function to call
prepare_to_modify_buffer" (YUCK!).  Thus, whether or not
before-change-functions is invoked is determined by the `prepare'
parameter.

The pertinent functions are insert_1_both, replace_range, del_range_1,
del_range_byte, and del_range_both.  These functions fall into two
groups, the first two in group 1 and the rest in group 2.  Those in group
2 always call signal_after_change, those in group 1 never do.

There are invocations of group 2 functions with `prepare' false.  This
wrongly invokes after-change-functions without having invoked
before-change-functions.

All the invocations of group 1 functions I've checked so far balance a
true `prepare' with an explicit call of signal_after_change.

At the heart of the problem is the lack of coherence in
prepare_to_modify_buffer.  This function has no definite coherent
function, the best description that can be given it being "perform some
of the things which need doing before a buffer change".  It is unclear
which of these things is unwanted when `prepare' is set to false.

The group 2 functions always invoke after-change-functions, so they must
always invoke before-change-functions.  In Finsert_file_contents there
are instances of del_range_byte with `prepare' false.  The spurious lack
of the before-change-functions invocation fouls up CC Mode, causing bug
#240[79]4.

I propose that the call to signal_before_change should be removed from
prepare_to_modify_buffer, and that it should be called explicitly from
the places it is needed.  This would allow bug #240[79]4 to be fixed.

Comments?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Noam Postavsky]

On Sun, Jul 31, 2016 at 8:16 AM, Alan Mackenzie <acm@muc.de> wrote:
>
> The pertinent functions are insert_1_both, replace_range, del_range_1,
> del_range_byte, and del_range_both.  These functions fall into two
> groups, the first two in group 1 and the rest in group 2.  Those in group
> 2 always call signal_after_change, those in group 1 never do.

replace_range should be in group 2 as well, no?

>
> There are invocations of group 2 functions with `prepare' false.  This
> wrongly invokes after-change-functions without having invoked
> before-change-functions.

replace_range has one call with prepare = false, it's in
Fsubst_char_in_region. I think the idea is that the caller takes care
of calling both before (by calling modify_text) and after (by explicit
call and/or(??) by replace_range) change functions, though the code is
sufficiently convuluted that I can't really say whether it works
correctly or not.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 12:16:42 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: Óscar Fuentes <ofv@wanadoo.es>,
> 	Richard Copley <rcopley@gmail.com>
> 
> I propose that the call to signal_before_change should be removed from
> prepare_to_modify_buffer, and that it should be called explicitly from
> the places it is needed.  This would allow bug #240[79]4 to be fixed.
> 
> Comments?

Some.

First, I don't agree with your conclusion that calls to
before-change-functions and after-change-functions must always be
balanced.  For starters, they aren't, and never have been.  Neither is
it written anywhere that they should, and your interpretation of the
ELisp manual seems to take the desired world for the real one.

My interpretation of these two variables is that they are two hooks
provided for different kinds of features, because hooking into both
should not generally be required.  It should suffice for an
application that wants to react to a change in a buffer to hook into
after-change-functions only.

But even if I agree, for the sake of argument, with your
interpretation, let's put this issue into a proper perspective, okay?
We have a single major mode (albeit a very important one, one that I
use every day) that under very specific conditions apparently fails to
discard its caches, because its design assumed calls to these two
hooks will always be balanced.  As a solution, you are proposing a
change in a very low-level and delicate part of Emacs, which means
significant changes to code that was basically unchanged for more than
20, if not 30 years.  This code is involved in almost every operation
on buffer text, and so changes you propose will necessarily cause
ripples all over Emacs.  Just look what the code you propose to change
does:

  . call barf-if-buffer-read-only
  . set the buffer's redisplay flag
  . affect the undo recording
  . save the region when select-active-regions is non-nil
  . sets deactivate-mark non-nil

Can you imagine how much Lisp code and basic features will be affected
by calling this where we never did before?  All that for the benefit
of a single major mode that fails in a single, albeit important use
case?

I'm sorry, but that makes very little sense to me.

So I suggest to take one or two steps back, and try to find a safer
solution for the problem.  Doing so will also greatly enhance the
chances of Emacs users to see the solution in the next Emacs release
(whereas doing what you suggest will push it back to Emacs 26 or maybe
even later).

Questions:

1) Why does CC Mode need both hooks?  What does it do in each one of
them, in the specific use case reported in the named bugs?  And why
isn't it enough to make only the change you proposed in part 1 of your
report?

2) Can this problem be fixed in CC Mode itself, without touching
either insert-file-contents or insdel.c functions?

3) If the answer to 2) above is a categorical NO (but please provide
enough arguments so we could be very sure it is that), then I _might_
be convinced to consider changes to insert-file-contents only, and
only for the case when both VISIT and REPLACE are non-nil.  That's
because I think this combination is only used by revert-buffer and
similar functionality (if someone knows about other callers that do
this, please holler), so the ripples will be not as wide as if we mess
with insdel.c.  (Still, even this possibility scares the living s**t
out of me; it should do that to you as well.)  To this end, please
tell which calls to del_range_byte in insert-file-contents are
involved with this use case and cause trouble to CC Mode, and maybe we
could change only those, in addition to the 'if' clause that guards
the call to before-change-functions.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Noam.

On Sun, Jul 31, 2016 at 09:58:50AM -0400, Noam Postavsky wrote:
> On Sun, Jul 31, 2016 at 8:16 AM, Alan Mackenzie <acm@muc.de> wrote:

> > The pertinent functions are insert_1_both, replace_range, del_range_1,
> > del_range_byte, and del_range_both.  These functions fall into two
> > groups, the first two in group 1 and the rest in group 2.  Those in group
> > 2 always call signal_after_change, those in group 1 never do.

> replace_range should be in group 2 as well, no?

Er, yes, it should.  ;-)  So we have a group of 4 and a single function.
Thanks for spotting that.

> > There are invocations of group 2 functions with `prepare' false.  This
> > wrongly invokes after-change-functions without having invoked
> > before-change-functions.

> replace_range has one call with prepare = false, it's in
> Fsubst_char_in_region. I think the idea is that the caller takes care
> of calling both before (by calling modify_text) and after (by explicit
> call and/or(??) by replace_range) change functions, though the code is
> sufficiently convoluted that I can't really say whether it works
> correctly or not.

By taking signal_after_change out of prepare_to_modify_buffer, I'm
rather hoping to reduce the convolution.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Sun, Jul 31, 2016 at 06:03:19PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 31 Jul 2016 12:16:42 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: Óscar Fuentes <ofv@wanadoo.es>,
> > 	Richard Copley <rcopley@gmail.com>

> > I propose that the call to signal_before_change should be removed from
> > prepare_to_modify_buffer, and that it should be called explicitly from
> > the places it is needed.  This would allow bug #240[79]4 to be fixed.
 
> > Comments?

> Some.

> First, I don't agree with your conclusion that calls to
> before-change-functions and after-change-functions must always be
> balanced.  For starters, they aren't, and never have been.  Neither is
> it written anywhere that they should, and your interpretation of the
> ELisp manual seems to take the desired world for the real one.

The pertinent section of Elisp ("Change Hooks") says:

    These hook variables let you arrange to take notice of ALL changes
    in ALL buffers.  [My emphasis]

Your interpretation of that seems to be that each buffer change will
call _at least_ one of before-... and after-..., but not necessarily
both.  That doesn't seem sensible to me.

Further down, in `before_change_functions' (and we have the same thing
in `after_change_functions') we have:

    This variable holds a list of functions to call before any buffer
    modification.

I can't see any sensible interpretation of "any ... modification" other
than "each and every modification".  I've been relying on this for many
years.

Looked at another way, how can it be sensible to call just one of these
hooks?  Like, if you were redesigning Emacs from scratch, are there any
categories of buffer modifications in which you would call only
after-change-functions, and not before-c-f?

> My interpretation of these two variables is that they are two hooks
> provided for different kinds of features, because hooking into both
> should not generally be required.  It should suffice for an
> application that wants to react to a change in a buffer to hook into
> after-change-functions only.

No.  A program (such as CC Mode) reacts to a _change_, not merely to
what a buffer looks like after a change.  A buffer change loses
information, so in the general case, any required info needs to be
recorded by before-change-functions before it gets lost in the change,
so that it can be used by after-change-functions.

As a concrete example of this, suppose we have this in a buffer:

    #define foo() first line \
                  second line \
	          third line \
	          fourth line

, and the change consists of deleting the backslash on L2.
before-change-functions will have noted that all four lines are in the
"active area".  after-change-functions unassisted would merely see that
the "active area" consists of L1 and L2.  The info from before-change
enables after-change to operate on the entire four lines.

> But even if I agree, for the sake of argument, with your
> interpretation, let's put this issue into a proper perspective, okay?
> We have a single major mode (albeit a very important one, one that I
> use every day) that under very specific conditions apparently fails to
> discard its caches, ....

Being precise, two critical variables were dirty store from the previous
invocation of c-after-change, not having been set in the missing call to
before-change-functions.

> .... because its design assumed calls to these two hooks will always
> be balanced.  As a solution, you are proposing a change in a very
> low-level and delicate part of Emacs, which means significant changes
> to code that was basically unchanged for more than 20, if not 30
> years.

prepare_to_modify buffer has been modified lots of times in recent
years.  (I count six modifications since 2014-01.)

> This code is involved in almost every operation on buffer
> text, and so changes you propose will necessarily cause ripples all
> over Emacs.  Just look what the code you propose to change does:

>   . call barf-if-buffer-read-only
>   . set the buffer's redisplay flag
>   . affect the undo recording
>   . save the region when select-active-regions is non-nil
>   . sets deactivate-mark non-nil

I'm not proposing to change any of that at all.  What I'm proposing is
strictly limited, and could be accompished in two steps, as follows:

1. (Pure refactoring).  Remove the call to signal_before_change from
prepare_to_modify_buffer.  Insert an explicit call to s_b_c after each
invocation of p_t_m_b.  Restore the check on inhibit_modification_hooks
in s_b_c.

2. Where a call to signal_buffer_change is within an "if (prepare) ..."
construct, move it to after that construct, so that

    if (prepare)
      {
        prepare_to_modify_buffer (...);
	signal_after_change (...);
      }

would become

    if (prepare)
      prepare_to_modify_buffer (...);
    signal_after_change (...);

.

> Can you imagine how much Lisp code and basic features will be affected
> by calling this where we never did before?  All that for the benefit
> of a single major mode that fails in a single, albeit important use
> case?

> I'm sorry, but that makes very little sense to me.

Please reconsider carefully the concrete proposal I've made (above) and
judge whether it will really affect very much at all.  I can foresee
that nothing will happen except the desired invocations of
before-change-functions being made.

I'm not entirely sure that only CC Mode will be helped here.  The value
of before-change-functions in many major modes contains the function
syntax-ppss-flush-cache.  There is a good chance that some call of, say,
del_range_byte is going to leave the syntax-ppss cache stale and
invalid, a difficult situation to diagnose.

> So I suggest to take one or two steps back, and try to find a safer
> solution for the problem.  Doing so will also greatly enhance the
> chances of Emacs users to see the solution in the next Emacs release
> (whereas doing what you suggest will push it back to Emacs 26 or maybe
> even later).

> Questions:

> 1) Why does CC Mode need both hooks?

I think I've already answered this above.

> What does it do in each one of them, in the specific use case reported
> in the named bugs?

It should calculate the two variables c-new-BEG and c-new-END (which
bound the active region) in c-before-change, and these get modified in
c-after-change.  In the bug situation, these variables simply held stale
values from the previous c-after-change call (which was from
transpose-lines).  Specifically, c-after-change modified c-new-END by
subtracting old-len.  This happened to make c-new-END < c-new-BEG.  So a
subsequent re-search-forward invocation had the limit on the wrong side
of point and threw an error.

> And why isn't it enough to make only the change you proposed in part 1
> of your report?

I tried that, but it didn't work.  With the help of gdb, it was clear
that the missing before-change-functions came from an invocation of
del_range_byte with `prepare' false, rather than directly from
Finsert_file_contents.

> 2) Can this problem be fixed in CC Mode itself, without touching
> either insert-file-contents or insdel.c functions?

I honestly don't know.  That would require extensive redesign if it's
possible.  That would need something like creating a new cache to
hold the information for the entire buffer which is currently determined
in c-before-change.  A simple workaround would be to ignore (i.e. not
process) an out of sequence invocation of c-after-change, but that's
hardly ideal.  I suspect I'm going to have to do that anyway in
standalone CC Mode.

> 3) If the answer to 2) above is a categorical NO (but please provide
> enough arguments so we could be very sure it is that), then I _might_
> be convinced to consider changes to insert-file-contents only, and
> only for the case when both VISIT and REPLACE are non-nil.  That's
> because I think this combination is only used by revert-buffer and
> similar functionality (if someone knows about other callers that do
> this, please holler), ....

The actual operation was C-x C-f where the file was already visited in
Emacs, but had been modified outside of Emacs, so Finsert_file_contents
went through the code which attempts to preserve markers, etc.

> .... so the ripples will be not as wide as if we mess with insdel.c.
> (Still, even this possibility scares the living s**t out of me; it
> should do that to you as well.)  To this end, please tell which calls
> to del_range_byte in insert-file-contents are involved with this use
> case and cause trouble to CC Mode, and maybe we could change only
> those, in addition to the 'if' clause that guards the call to
> before-change-functions.

The del_range_byte call which I caught with gdb last night was, I think,
the one at fileio.c L4037.  I don't know the code well enough to say
that that's the only one which might trigger.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 17:28:04 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
>     This variable holds a list of functions to call before any buffer
>     modification.
> 
> I can't see any sensible interpretation of "any ... modification" other
> than "each and every modification".  I've been relying on this for many
> years.

You are forgetting that "buffer modifications" is not rigorously
defined in Emacs.  Let me remind you that in so many places we pretend
that no changes were made to a buffer, while in fact there were.

Given this fact, "any modification" has no meaning, beyond the
tautological "modifications that Emacs developers decided to treat as
such".

IOW, what the code does, and has been doing for many moons, is in this
case a de-facto _definition_ of what should be done.

> Looked at another way, how can it be sensible to call just one of these
> hooks?

I gave one such interpretation.

> Like, if you were redesigning Emacs from scratch, are there any
> categories of buffer modifications in which you would call only
> after-change-functions, and not before-c-f?

But we are not redesigning Emacs.  At this stage, we would be nuts
trying to.  Emacs, at the level of insdel.c, is stable, has been for a
long time.  We should only make there localized changes in response to
localized problems that are revealed at that level.  In the case in
point, the problem is revealed at a much higher level, so solving it
in insdel.c is simply wrong, because that will destabilize Emacs for
years to come.  We cannot afford that, as the "Emacs cloud" is too
large, and the number of people who understand these internals is too
small.

> > My interpretation of these two variables is that they are two hooks
> > provided for different kinds of features, because hooking into both
> > should not generally be required.  It should suffice for an
> > application that wants to react to a change in a buffer to hook into
> > after-change-functions only.
> 
> No.  A program (such as CC Mode) reacts to a _change_, not merely to
> what a buffer looks like after a change.  A buffer change loses
> information, so in the general case, any required info needs to be
> recorded by before-change-functions before it gets lost in the change,
> so that it can be used by after-change-functions.

And yet you had no problems until now.  This particular use case is
not about some arbitrary buffer change, it is about replacing the
whole buffer with an updated contents of the file.  We don't need to
consider any other use cases, because we are not aware of problems in
those cases.

> As a concrete example of this, suppose we have this in a buffer:
> 
>     #define foo() first line \
>                   second line \
> 	          third line \
> 	          fourth line
> 
> , and the change consists of deleting the backslash on L2.
> before-change-functions will have noted that all four lines are in the
> "active area".  after-change-functions unassisted would merely see that
> the "active area" consists of L1 and L2.  The info from before-change
> enables after-change to operate on the entire four lines.

You could simply discard the information for all the 4 lines, and be
done.  It is more important to be correct than to be optimal.

But anyway, I believe the case in point has nothing to do with this
example.  Right?

> > But even if I agree, for the sake of argument, with your
> > interpretation, let's put this issue into a proper perspective, okay?
> > We have a single major mode (albeit a very important one, one that I
> > use every day) that under very specific conditions apparently fails to
> > discard its caches, ....
> 
> Being precise, two critical variables were dirty store from the previous
> invocation of c-after-change, not having been set in the missing call to
> before-change-functions.

I still don't understand why you couldn't clear all such variables in
this case.  The entire buffer text has been replaced, why keep
anything you had before?

> > .... because its design assumed calls to these two hooks will always
> > be balanced.  As a solution, you are proposing a change in a very
> > low-level and delicate part of Emacs, which means significant changes
> > to code that was basically unchanged for more than 20, if not 30
> > years.
> 
> prepare_to_modify buffer has been modified lots of times in recent
> years.  (I count six modifications since 2014-01.)

I said "basically unchanged".  Trivial refactoring doesn't count.  And
any changes in this function were in response to issues common to all
the callers.  The case in point isn't such a case.

> > This code is involved in almost every operation on buffer
> > text, and so changes you propose will necessarily cause ripples all
> > over Emacs.  Just look what the code you propose to change does:
> 
> >   . call barf-if-buffer-read-only
> >   . set the buffer's redisplay flag
> >   . affect the undo recording
> >   . save the region when select-active-regions is non-nil
> >   . sets deactivate-mark non-nil
> 
> I'm not proposing to change any of that at all.  What I'm proposing is
> strictly limited, and could be accompished in two steps, as follows:
> 
> 1. (Pure refactoring).  Remove the call to signal_before_change from
> prepare_to_modify_buffer.  Insert an explicit call to s_b_c after each
> invocation of p_t_m_b.  Restore the check on inhibit_modification_hooks
> in s_b_c.
> 
> 2. Where a call to signal_buffer_change is within an "if (prepare) ..."
> construct, move it to after that construct, so that
> 
>     if (prepare)
>       {
>         prepare_to_modify_buffer (...);
> 	signal_after_change (...);
>       }
> 
> would become
> 
>     if (prepare)
>       prepare_to_modify_buffer (...);
>     signal_after_change (...);

See, that's the part that scares me.  You are proposing to run code
where we didn't run it before.  Did you look at what
signal_after_change does?  It doesn't just call
after-change-functions, it does much more.

> > Can you imagine how much Lisp code and basic features will be affected
> > by calling this where we never did before?  All that for the benefit
> > of a single major mode that fails in a single, albeit important use
> > case?
> 
> > I'm sorry, but that makes very little sense to me.
> 
> Please reconsider carefully the concrete proposal I've made (above) and
> judge whether it will really affect very much at all.

Your proposal doesn't take care of my fears.

> I can foresee that nothing will happen except the desired
> invocations of before-change-functions being made.

We will have to agree to disagree about this.

> I'm not entirely sure that only CC Mode will be helped here.  The value
> of before-change-functions in many major modes contains the function
> syntax-ppss-flush-cache.  There is a good chance that some call of, say,
> del_range_byte is going to leave the syntax-ppss cache stale and
> invalid, a difficult situation to diagnose.

I would have to see very specific evidence to that, not just
theoretical possibilities.  And even then, the proposed change touches
much more than just before-change-functions.

> > What does it do in each one of them, in the specific use case reported
> > in the named bugs?
> 
> It should calculate the two variables c-new-BEG and c-new-END (which
> bound the active region) in c-before-change, and these get modified in
> c-after-change.

Sorry, I don't understand.  Why can't you compute these two values in
the c-after-change?  And if you can't, why calling c-before-change
when REPLACE is non-nil didn't fix that?  Once again, we are talking
about reverting the entire buffer, so any intermediate deletions and
insertions aren't important.

> In the bug situation, these variables simply held stale values from
> the previous c-after-change call (which was from transpose-lines).
> Specifically, c-after-change modified c-new-END by subtracting
> old-len.  This happened to make c-new-END < c-new-BEG.  So a
> subsequent re-search-forward invocation had the limit on the wrong
> side of point and threw an error.

Why can't this error be caught and interpreted as a sign that the
cached values are wrong and need to be recomputed?

> > And why isn't it enough to make only the change you proposed in part 1
> > of your report?
> 
> I tried that, but it didn't work.  With the help of gdb, it was clear
> that the missing before-change-functions came from an invocation of
> del_range_byte with `prepare' false, rather than directly from
> Finsert_file_contents.

So if that single call is with 'prepare' set to true (when both VISIT
and REPLACE are non-nil), the problem would have been solved?

> > 2) Can this problem be fixed in CC Mode itself, without touching
> > either insert-file-contents or insdel.c functions?
> 
> I honestly don't know.  That would require extensive redesign if it's
> possible.  That would need something like creating a new cache to
> hold the information for the entire buffer which is currently determined
> in c-before-change.  A simple workaround would be to ignore (i.e. not
> process) an out of sequence invocation of c-after-change, but that's
> hardly ideal.  I suspect I'm going to have to do that anyway in
> standalone CC Mode.

If you are going to have to do that anyway, why do we have to consider
such low-level changes?

Btw, there's also a possibility for CC Mode to define its own
revert-buffer function.  That function could be a thin wrapper around
revert-buffer, and could do in the wrapping code whatever you think is
missing in insert-file-contents, like forcefully calling
before-change-functions.  That would still be better than messing with
insdel.c, IMO.

> > 3) If the answer to 2) above is a categorical NO (but please provide
> > enough arguments so we could be very sure it is that), then I _might_
> > be convinced to consider changes to insert-file-contents only, and
> > only for the case when both VISIT and REPLACE are non-nil.  That's
> > because I think this combination is only used by revert-buffer and
> > similar functionality (if someone knows about other callers that do
> > this, please holler), ....
> 
> The actual operation was C-x C-f where the file was already visited in
> Emacs, but had been modified outside of Emacs, so Finsert_file_contents
> went through the code which attempts to preserve markers, etc.

That's what revert-buffer does, it calls insert-file-contents with
both VISIT and REPLACE non-nil.  And that's the use case we are
talking about, right?

> > .... so the ripples will be not as wide as if we mess with insdel.c.
> > (Still, even this possibility scares the living s**t out of me; it
> > should do that to you as well.)  To this end, please tell which calls
> > to del_range_byte in insert-file-contents are involved with this use
> > case and cause trouble to CC Mode, and maybe we could change only
> > those, in addition to the 'if' clause that guards the call to
> > before-change-functions.
> 
> The del_range_byte call which I caught with gdb last night was, I think,
> the one at fileio.c L4037.  I don't know the code well enough to say
> that that's the only one which might trigger.

If changing that one alone solves the problem, we don't need to
consider any others.

Thanks.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 21:11:32 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> 
> > The del_range_byte call which I caught with gdb last night was, I think,
> > the one at fileio.c L4037.  I don't know the code well enough to say
> > that that's the only one which might trigger.
> 
> If changing that one alone solves the problem, we don't need to
> consider any others.

Actually, I think we need to change all 3 of those calls, since they
all handle the same situation, with minor differences.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>     These hook variables let you arrange to take notice of ALL changes
>     in ALL buffers.  [My emphasis]
>
> Your interpretation of that seems to be that each buffer change will
> call _at least_ one of before-... and after-..., but not necessarily
> both.  That doesn't seem sensible to me.

I fully agree that if you only use before-change-functions (or only use
after-change-functions), your hook function should see *all* changes.

This is an important property and if there are places where this is not
the case, we should fix them.

> No.  A program (such as CC Mode) reacts to a _change_, not merely to
> what a buffer looks like after a change.

But here we have a problem: I don't think that
before/after-change-functions should be always be properly paired, which
IIUC is a property that CC-mode relies on.


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 21:52:04 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> 
> > Date: Sun, 31 Jul 2016 21:11:32 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > 
> > > The del_range_byte call which I caught with gdb last night was, I think,
> > > the one at fileio.c L4037.  I don't know the code well enough to say
> > > that that's the only one which might trigger.
> > 
> > If changing that one alone solves the problem, we don't need to
> > consider any others.
> 
> Actually, I think we need to change all 3 of those calls, since they
> all handle the same situation, with minor differences.

Of course, I still think we should try solving this in CC Mode alone,
if reasonably possible.

Re: Unbalanced change hooks (part 2)

[Noam Postavsky]

On Sun, Jul 31, 2016 at 2:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 2. Where a call to signal_buffer_change is within an "if (prepare) ..."
>> construct, move it to after that construct, so that
>>
>>     if (prepare)
>>       {
>>         prepare_to_modify_buffer (...);
>>       signal_after_change (...);
>>       }
>>
>> would become
>>
>>     if (prepare)
>>       prepare_to_modify_buffer (...);
>>     signal_after_change (...);
>
> See, that's the part that scares me.  You are proposing to run code
> where we didn't run it before.  Did you look at what
> signal_after_change does?  It doesn't just call
> after-change-functions, it does much more.

Doesn't look like that much more to me. In addtion to
after-change-functions, it also calls overlay and text property
modification hooks. I don't see anything else.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 17:28:04 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > And why isn't it enough to make only the change you proposed in part 1
> > of your report?
> 
> I tried that, but it didn't work.

Answering myself here: that didn't work because the part you proposed
to change, i.e.

  if (NILP (visit) && total > 0)
    {
      if (!NILP (BVAR (current_buffer, file_truename))
	  /* Make binding buffer-file-name to nil effective.  */
	  && !NILP (BVAR (current_buffer, filename))
	  && SAVE_MODIFF >= MODIFF)
	we_locked_file = true;
      prepare_to_modify_buffer (PT, PT, NULL);
    }

is not executed when REPLACE is non-nil.  So adding the condition you
suggested there will have no effect at all.

The relevant code is indeed the 3 calls to del_range_byte.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 31 Jul 2016 14:55:32 -0400
> 
> >     These hook variables let you arrange to take notice of ALL changes
> >     in ALL buffers.  [My emphasis]
> >
> > Your interpretation of that seems to be that each buffer change will
> > call _at least_ one of before-... and after-..., but not necessarily
> > both.  That doesn't seem sensible to me.
> 
> I fully agree that if you only use before-change-functions (or only use
> after-change-functions), your hook function should see *all* changes.

The change in question is a replacement.  Emacs implements
replacements by a deletion followed by an insertion.  The question is:
do we need to call the before-change-functions for both the deletion
and the insertion, or do we need to call them only once for the
"replacement"?  We currently do the latter, i.e. call them only once.
If you think that's incorrect, please explain why.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Stefan.

On Sun, Jul 31, 2016 at 02:55:32PM -0400, Stefan Monnier wrote:
> >     These hook variables let you arrange to take notice of ALL changes
> >     in ALL buffers.  [My emphasis]

> > Your interpretation of that seems to be that each buffer change will
> > call _at least_ one of before-... and after-..., but not necessarily
> > both.  That doesn't seem sensible to me.

> I fully agree that if you only use before-change-functions (or only use
> after-change-functions), your hook function should see *all* changes.

Taking your parenthetical clause to be an essential part of your
sentence, this can only happen if all changes cause both before- and
after-change-functions to be called.  (A major mode "registering" which
one of before- or after- it wishes to see would be ludicrous.)

> This is an important property and if there are places where this is not
> the case, we should fix them.

I agree, it's what I'm proposing and trying to persuade Eli to accept.

> > No.  A program (such as CC Mode) reacts to a _change_, not merely to
> > what a buffer looks like after a change.

> But here we have a problem: I don't think that
> before/after-change-functions should be always be properly paired, which
> IIUC is a property that CC-mode relies on.

That appears to contradict your first paragraph.  CC Mode does indeed
rely on this proper pairing.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Sun, Jul 31, 2016 at 09:11:32PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 31 Jul 2016 17:28:04 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> >     This variable holds a list of functions to call before any buffer
> >     modification.

> > I can't see any sensible interpretation of "any ... modification" other
> > than "each and every modification".  I've been relying on this for many
> > years.

> You are forgetting that "buffer modifications" is not rigorously
> defined in Emacs.  Let me remind you that in so many places we pretend
> that no changes were made to a buffer, while in fact there were.

I've no problem with that, as long as each action either definitely is a
change or definitely isn't a change.  In the former case, it should run
the change hooks, in the latter case it shouldn't.  It's the halfway
house where half of the change hooks are run that I don't like.

> Given this fact, "any modification" has no meaning, beyond the
> tautological "modifications that Emacs developers decided to treat as
> such".

> IOW, what the code does, and has been doing for many moons, is in this
> case a de-facto _definition_ of what should be done.

Extending that principle ad absurdum, Emacs has no bugs at all, since
what it does is what it should do.

> > Looked at another way, how can it be sensible to call just one of these
> > hooks?

> I gave one such interpretation.

OK.  I don't agree with your principle, and I'm not convinced you do
either.  :-)

> > Like, if you were redesigning Emacs from scratch, are there any
> > categories of buffer modifications in which you would call only
> > after-change-functions, and not before-c-f?

> But we are not redesigning Emacs.  At this stage, we would be nuts
> trying to.  Emacs, at the level of insdel.c, is stable, has been for a
> long time.  We should only make there localized changes in response to
> localized problems that are revealed at that level.  In the case in
> point, the problem is revealed at a much higher level, so solving it
> in insdel.c is simply wrong, because that will destabilize Emacs for
> years to come.  We cannot afford that, as the "Emacs cloud" is too
> large, and the number of people who understand these internals is too
> small.

I'm in favour of fixing what is wrong, rather than implementing
workarounds somewhere else.  Workarounds are rarely fully satisfactory.
I think you would agree with me that having before- and after- hook
calls rigorously paired would be the ideal state - you haven't attempted
so far to argue that missing out a before-change-functions call is a
good thing.

Solving the problem in insdel.c is the Right Thing to do, because that
is where the cause of the problem lies.  You assert that the change I
propose would "destabilise Emacs for years to come", but you haven't
provided any analysis of my proposed change to support that assertion.
The hackers who built Emacs were and are of a higher quality than to
build such instability into Emacs.

> > > My interpretation of these two variables is that they are two hooks
> > > provided for different kinds of features, because hooking into both
> > > should not generally be required.  It should suffice for an
> > > application that wants to react to a change in a buffer to hook into
> > > after-change-functions only.
 
> > No.  A program (such as CC Mode) reacts to a _change_, not merely to
> > what a buffer looks like after a change.  A buffer change loses
> > information, so in the general case, any required info needs to be
> > recorded by before-change-functions before it gets lost in the change,
> > so that it can be used by after-change-functions.

> And yet you had no problems until now.  This particular use case is
> not about some arbitrary buffer change, it is about replacing the
> whole buffer with an updated contents of the file.

No.  The code does fancy things comparing the file to be revisited with
the current buffer contents, and only loading the minimum (in some
sense) portion of the file, so as to preserve markers, or something like
that.  In Richard Copley's recipe for bug #24094, an essential part was
the C-x C-t to transpose two lines before C-x C-s.  The complexities of
the transposition got remembered when reloading the file portion,
leading to the bug.

> We don't need to consider any other use cases, because we are not
> aware of problems in those cases.

If we fixed things the way I would like, we'd automatically fix these
other use cases without any effort.

> > As a concrete example of this, suppose we have this in a buffer:
> > 
> >     #define foo() first line \
> >                   second line \
> > 	          third line \
> > 	          fourth line
> > 
> > , and the change consists of deleting the backslash on L2.
> > before-change-functions will have noted that all four lines are in the
> > "active area".  after-change-functions unassisted would merely see that
> > the "active area" consists of L1 and L2.  The info from before-change
> > enables after-change to operate on the entire four lines.

> You could simply discard the information for all the 4 lines, and be
> done.  It is more important to be correct than to be optimal.

It is for correctness, not speed, that the information is recorded.
Over the last decade or so, there were quite a few bugs where jit-lock
delayed fontification messed up what had been (for 0.5s) correct, or
where typing a CR in the middle of construct messed fontification up, or
....  Please believe me, this recording of info in c-before-change was
not done just for its own sake.

For example, if a C++ template delimiter, which is marked with a
parenthesis syntax-table property, is about to be deleted, we ensure in
c-before-change that the matching delimiter is appropriately handled.

I'm not saying that CC Mode couldn't be redesigned not to use
before-change-functions, but it would be a massive amount of work which
wouldn't get us any closer to where we want CC Mode to be, and it would
be depressing unrewarding work.

> But anyway, I believe the case in point has nothing to do with this
> example.  Right?

No, the example offers a justification of why CC Mode needs both before-
and after-change-functions.

> > > But even if I agree, for the sake of argument, with your
> > > interpretation, let's put this issue into a proper perspective, okay?
> > > We have a single major mode (albeit a very important one, one that I
> > > use every day) that under very specific conditions apparently fails to
> > > discard its caches, ....

> > Being precise, two critical variables were dirty store from the previous
> > invocation of c-after-change, not having been set in the missing call to
> > before-change-functions.

> I still don't understand why you couldn't clear all such variables in
> this case.  The entire buffer text has been replaced, why keep
> anything you had before?

The entire buffer hasn't been replaced (see above).  Those variables
get initialised in c-before-change.  If the call to c-before-change is
missing, those variables will be dirty store.

> > > .... because its design assumed calls to these two hooks will always
> > > be balanced.  As a solution, you are proposing a change in a very
> > > low-level and delicate part of Emacs, which means significant changes
> > > to code that was basically unchanged for more than 20, if not 30
> > > years.
 
> > prepare_to_modify buffer has been modified lots of times in recent
> > years.  (I count six modifications since 2014-01.)

> I said "basically unchanged".  Trivial refactoring doesn't count.  And
> any changes in this function were in response to issues common to all
> the callers.  The case in point isn't such a case.

What I am proposing is only a little more than trivial refactoring.

> > > This code is involved in almost every operation on buffer
> > > text, and so changes you propose will necessarily cause ripples all
> > > over Emacs.  Just look what the code you propose to change does:

> > >   . call barf-if-buffer-read-only
> > >   . set the buffer's redisplay flag
> > >   . affect the undo recording
> > >   . save the region when select-active-regions is non-nil
> > >   . sets deactivate-mark non-nil
 
> > I'm not proposing to change any of that at all.  What I'm proposing is
> > strictly limited, and could be accompished in two steps, as follows:
 
> > 1. (Pure refactoring).  Remove the call to signal_before_change from
> > prepare_to_modify_buffer.  Insert an explicit call to s_b_c after each
> > invocation of p_t_m_b.  Restore the check on inhibit_modification_hooks
> > in s_b_c.
 
> > 2. Where a call to signal_buffer_change is within an "if (prepare) ..."
> > construct, move it to after that construct, so that
 
> >     if (prepare)
> >       {
> >         prepare_to_modify_buffer (...);
> > 	signal_after_change (...);
> >       }
 
> > would become
 
> >     if (prepare)
> >       prepare_to_modify_buffer (...);
> >     signal_after_change (...);

> See, that's the part that scares me.  You are proposing to run code
> where we didn't run it before.  Did you look at what
> signal_after_change does?  It doesn't just call
> after-change-functions, it does much more.

I'm proposing to run code designed to run before buffer changes before a
buffer change.  The code does a little more than call
before-change-functions but only a little:
(i) it runs first-change-hook;
(ii) it runs before-change-functions;
(iii) it runs report_overlay_modification (whatever that does).
(iv) it preserves *preserve_ptr as described in the comment.

I would be more worried that there are buffer modifications where
report_overlay_modification _doesn't_ get run.

> > > Can you imagine how much Lisp code and basic features will be affected
> > > by calling this where we never did before?  All that for the benefit
> > > of a single major mode that fails in a single, albeit important use
> > > case?

> > > I'm sorry, but that makes very little sense to me.
 
> > Please reconsider carefully the concrete proposal I've made (above) and
> > judge whether it will really affect very much at all.

> Your proposal doesn't take care of my fears.

I'm still trying to work out what they are, beyond a generalised
opposition to any change at this level of the code.

> > I can foresee that nothing will happen except the desired
> > invocations of before-change-functions being made.

> We will have to agree to disagree about this.

OK.  How about I implement what I've described, and we try it out?  If
it doesn't work, or it throws up non-trivial problems, it will be easy
enough to revert.

> > I'm not entirely sure that only CC Mode will be helped here.  The value
> > of before-change-functions in many major modes contains the function
> > syntax-ppss-flush-cache.  There is a good chance that some call of, say,
> > del_range_byte is going to leave the syntax-ppss cache stale and
> > invalid, a difficult situation to diagnose.

> I would have to see very specific evidence to that, not just
> theoretical possibilities.

Programs other than CC Mode use before-change-functions.  It would be
more effort to construct an actual failure than simply to fix the
problem.

> And even then, the proposed change touches much more than just
> before-change-functions.

> > > What does it do in each one of them, in the specific use case reported
> > > in the named bugs?

> > It should calculate the two variables c-new-BEG and c-new-END (which
> > bound the active region) in c-before-change, and these get modified in
> > c-after-change.

> Sorry, I don't understand.  Why can't you compute these two values in
> the c-after-change?

In the general case because information gets lost at a change; things
like where the beginning of a statement was, where deleted template
delimiters used to be, what the bounds of a macro or string used to be
(before backslashes were deleted).

Or are you suggesting I implement a special after-change handler for the
case when the before-change-functions was missing?  This would be
possible, but it would still be more work than ensuring that
before-change-functions actually gets called.

> And if you can't, why calling c-before-change
> when REPLACE is non-nil didn't fix that?  Once again, we are talking
> about reverting the entire buffer, so any intermediate deletions and
> insertions aren't important.

Again, the entire buffer wasn't replaced.  Read the comment in
Finsert_file_contents at ~L3697:

  /* If requested, replace the accessible part of the buffer
     with the file contents.  Avoid replacing text at the
     beginning or end of the buffer that matches the file contents;
     that preserves markers pointing to the unchanged parts.

> > In the bug situation, these variables simply held stale values from
> > the previous c-after-change call (which was from transpose-lines).
> > Specifically, c-after-change modified c-new-END by subtracting
> > old-len.  This happened to make c-new-END < c-new-BEG.  So a
> > subsequent re-search-forward invocation had the limit on the wrong
> > side of point and threw an error.

> Why can't this error be caught and interpreted as a sign that the
> cached values are wrong and need to be recomputed?

It's better (and less work) to determine them properly in the first
place.  Once we're in c-after-change, it's no longer possible to
calculate them properly in the general case.

> > > And why isn't it enough to make only the change you proposed in part 1
> > > of your report?

> > I tried that, but it didn't work.  With the help of gdb, it was clear
> > that the missing before-change-functions came from an invocation of
> > del_range_byte with `prepare' false, rather than directly from
> > Finsert_file_contents.

> So if that single call is with 'prepare' set to true (when both VISIT
> and REPLACE are non-nil), the problem would have been solved?

Yes, apart from all the other processing in prepare_to_modify_buffer
that the `prepare' flag being false is designed to avoid.

> > > 2) Can this problem be fixed in CC Mode itself, without touching
> > > either insert-file-contents or insdel.c functions?

> > I honestly don't know.  That would require extensive redesign if it's
> > possible.  That would need something like creating a new cache to
> > hold the information for the entire buffer which is currently determined
> > in c-before-change.  A simple workaround would be to ignore (i.e. not
> > process) an out of sequence invocation of c-after-change, but that's
> > hardly ideal.  I suspect I'm going to have to do that anyway in
> > standalone CC Mode.

> If you are going to have to do that anyway, why do we have to consider
> such low-level changes?

Because a proper fix is the Right Thing to do.  It's more satisfying,
works better, and eliminates other potential problems which haven't
shown up yet, or undiagnosed bugs caused by this which we haven't been
able to tie down.

> Btw, there's also a possibility for CC Mode to define its own
> revert-buffer function.  That function could be a thin wrapper around
> revert-buffer, and could do in the wrapping code whatever you think is
> missing in insert-file-contents, like forcefully calling
> before-change-functions.  That would still be better than messing with
> insdel.c, IMO.

Reverting the buffer is merely the recipe to reproduce the bug.  The bug
itself is c-before-change not being called.  It is likely to show up in
other circumstances if we don't fix it.

> > > 3) If the answer to 2) above is a categorical NO (but please provide
> > > enough arguments so we could be very sure it is that), then I _might_
> > > be convinced to consider changes to insert-file-contents only, and
> > > only for the case when both VISIT and REPLACE are non-nil.  That's
> > > because I think this combination is only used by revert-buffer and
> > > similar functionality (if someone knows about other callers that do
> > > this, please holler), ....
 
> > The actual operation was C-x C-f where the file was already visited in
> > Emacs, but had been modified outside of Emacs, so Finsert_file_contents
> > went through the code which attempts to preserve markers, etc.

> That's what revert-buffer does, it calls insert-file-contents with
> both VISIT and REPLACE non-nil.  And that's the use case we are
> talking about, right?

More or less.  C-x C-f also goes through these machinations in the case
described.

> > > .... so the ripples will be not as wide as if we mess with insdel.c.
> > > (Still, even this possibility scares the living s**t out of me; it
> > > should do that to you as well.)  To this end, please tell which calls
> > > to del_range_byte in insert-file-contents are involved with this use
> > > case and cause trouble to CC Mode, and maybe we could change only
> > > those, in addition to the 'if' clause that guards the call to
> > > before-change-functions.

> > The del_range_byte call which I caught with gdb last night was, I think,
> > the one at fileio.c L4037.  I don't know the code well enough to say
> > that that's the only one which might trigger.

> If changing that one alone solves the problem, we don't need to
> consider any others.

As I said, why not let me implement what I've suggested?  If it doesn't
work well enough, I can revert it.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>> I fully agree that if you only use before-change-functions (or only use
>> after-change-functions), your hook function should see *all* changes.
> The change in question is a replacement.  Emacs implements
> replacements by a deletion followed by an insertion.  The question is:
> do we need to call the before-change-functions for both the deletion
> and the insertion, or do we need to call them only once for the
> "replacement"?

Either way is fine, according to my interpretation of "should see *all*
changes".


        Stefan

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>> > As a concrete example of this, suppose we have this in a buffer:
>> > 
>> >     #define foo() first line \
>> >                   second line \
>> > 	          third line \
>> > 	          fourth line
>> > 
>> > , and the change consists of deleting the backslash on L2.
>> > before-change-functions will have noted that all four lines are in the
>> > "active area".  after-change-functions unassisted would merely see that
>> > the "active area" consists of L1 and L2.  The info from before-change
>> > enables after-change to operate on the entire four lines.

I think this can work at one condition: you need to make sure that it
also works if you get something like:

  b-c-f called on L2
  b-c-f called on L37
  a-c-f called on L2-L37

or

  b-c-f called on L2
  b-c-f called on L37
  a-c-f called on L2
  a-c-f called on L37

or

  b-c-f called on L2-L3
  a-c-f called on L2
  a-c-f called on L3

and other such variations.

> I'm not saying that CC Mode couldn't be redesigned not to use
> before-change-functions, but it would be a massive amount of work which
> wouldn't get us any closer to where we want CC Mode to be, and it would
> be depressing unrewarding work.

Switching to use syntax-propertize would likely solve this problem along
with various others and would be very rewarding.


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 31 Jul 2016 21:26:35 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > IOW, what the code does, and has been doing for many moons, is in this
> > case a de-facto _definition_ of what should be done.
> 
> Extending that principle ad absurdum, Emacs has no bugs at all, since
> what it does is what it should do.

Which is true.  Of course, since Emacs developers are reasonable
people (with the possible exception of yours truly), the result is
also reasonable.

So I suggest that we abandon this semi-philosophical sub-thread, and
instead focus on the practical issue at hand.

> I'm in favour of fixing what is wrong, rather than implementing
> workarounds somewhere else.  Workarounds are rarely fully satisfactory.

I'm not suggesting a workaround.  I'm suggesting a fix for a design
flaw in CC Mode -- its assumption that before-change-functions and
after-change-functions are always invoked in balanced pairs.  CC Mode
must not depend on that assumption, because it's false.

> I think you would agree with me that having before- and after- hook
> calls rigorously paired would be the ideal state - you haven't attempted
> so far to argue that missing out a before-change-functions call is a
> good thing.

It might be ideal, but it's next to impossible in practice.  It would
prohibit or greatly complicate code that deals with buffer
modifications, because for various reasons that code might do the
changes piecemeal.

In practice, we need to make sure that every modification triggers at
least one call to before-change-functions and at least one call to
after-change-functions.  The numbers of these calls do not have to
match.

> Solving the problem in insdel.c is the Right Thing to do, because that
> is where the cause of the problem lies.  You assert that the change I
> propose would "destabilise Emacs for years to come", but you haven't
> provided any analysis of my proposed change to support that assertion.

I had my share of hacking that code and its callers.  It's impossible
to convey everything I learned while doing that in a discussion such
as this one.  I say more about this below.  If that's still not
enough, you will have to trust me on that.

> The hackers who built Emacs were and are of a higher quality than to
> build such instability into Emacs.

"Instability" here means that various features that expect certain
things to happen will instead see unexpected situations.  Emacs will
still work and be stable as a program (i.e. won't crash and probably
won't corrupt your buffers).  But various functions will exhibit more
or less subtle bugs and misfeatures.

> The code does fancy things comparing the file to be revisited with
> the current buffer contents, and only loading the minimum (in some
> sense) portion of the file, so as to preserve markers, or something like
> that.

It's an internal optimization in insert-file-contents that CC Mode
doesn't have to follow, if doing that is painful.  Of course, if you
can reliably invalidate only the cached values that refer to the
region actually replaced, it's better.  But it's only an optimization.

> Please believe me, this recording of info in c-before-change was not
> done just for its own sake.

I believe you.  Just don't assume that there will be a c-before-change
call for each c-after-change call, that's all.

Specifically, in the case in point, I believe that the "unbalanced"
call to c-after-change clearly tells you that text has been deleted,
so I think it should be enough for you to do the job of the "missing"
c-before-change call as well.

> > I said "basically unchanged".  Trivial refactoring doesn't count.  And
> > any changes in this function were in response to issues common to all
> > the callers.  The case in point isn't such a case.
> 
> What I am proposing is only a little more than trivial refactoring.

No, it's far from that.  It fundamentally changes the list of things
that are done in many operations that modify buffers in various ways,
and the order in which these things are done.  We shouldn't make such
changes at this point in Emacs development, unless we identify a clear
bug that affects all the callers of these functions.

This case is not such a case.  It is a case of a single Lisp package
that made incorrect assumptions about the way the modification hooks
are called.  So the right place to fix this is in that Lisp package.

> > >     if (prepare)
> > >       prepare_to_modify_buffer (...);
> > >     signal_after_change (...);
> > 
> > See, that's the part that scares me.  You are proposing to run code
> > where we didn't run it before.  Did you look at what
> > signal_after_change does?  It doesn't just call
> > after-change-functions, it does much more.
> 
> I'm proposing to run code designed to run before buffer changes before a
> buffer change.

You are confusingly using signal_after_change and signal_before_change
interchangeably, so there's probably quite a lot of misunderstanding.

If you meant this:

     if (prepare)
       prepare_to_modify_buffer (...);
     signal_before_change (...);

then I don't see how it could work, because prepare_to_modify_buffer
computes one of the arguments to signal_before_change, so calling the
latter without calling the former is not really a good idea.  (This is
just one example of the many subtleties hiding in these seemingly
simple, almost innocently looking functions we have in insdel.c.)

> I'm still trying to work out what they are, beyond a generalised
> opposition to any change at this level of the code.

Without a good reason, yeah, that's pretty much it.

It's not some paranoia, mind you: I've been working on fixing quite a
few subtle bugs which originated due to such changes for the last few
years, so I know what I'm talking about.  Please trust my experience
and the conclusions I drew from it.

> > We will have to agree to disagree about this.
> 
> OK.  How about I implement what I've described, and we try it out?  If
> it doesn't work, or it throws up non-trivial problems, it will be easy
> enough to revert.

Sorry, no.  What I'd like you to try instead is eliminate from CC Mode
the incorrect assumption about pairwise calls to before- and
after-change-functions.  That is the correct way towards solving this
problem.  It is unwise to waste your time (and mine) on trying to
solve this by changing how modification hooks are called, because
that's not where the problem is.

> Programs other than CC Mode use before-change-functions.  It would be
> more effort to construct an actual failure than simply to fix the
> problem.

But before-change-functions _are_ being called in the scenario of the
bugs we are discussing.  They just aren't balanced with calls to
after-change-functions, that's all.  See, in this fragment of
insert-file-contents:

      if (same_at_end != same_at_start)
	{
	  invalidate_buffer_caches (current_buffer,
				    BYTE_TO_CHAR (same_at_start),
				    same_at_end_charpos);
	  del_range_byte (same_at_start, same_at_end, 0);
	  temp = GPT;
	  eassert (same_at_start == GPT_BYTE);
	  same_at_start = GPT_BYTE;
	}
      else
	{
	  temp = same_at_end_charpos;
	}
      /* Insert from the file at the proper position.  */
      SET_PT_BOTH (temp, same_at_start);
      same_at_start_charpos
	= buf_bytepos_to_charpos (XBUFFER (conversion_buffer),
				  same_at_start - BEGV_BYTE
				  + BUF_BEG_BYTE (XBUFFER (conversion_buffer)));
      eassert (same_at_start_charpos == temp - (BEGV - BEG));
      inserted_chars
	= (buf_bytepos_to_charpos (XBUFFER (conversion_buffer),
				   same_at_start + inserted - BEGV_BYTE
				  + BUF_BEG_BYTE (XBUFFER (conversion_buffer)))
	   - same_at_start_charpos);
      /* This binding is to avoid ask-user-about-supersession-threat
	 being called in insert_from_buffer (via in
	 prepare_to_modify_buffer).  */
      specbind (intern ("buffer-file-name"), Qnil);
      insert_from_buffer (XBUFFER (conversion_buffer),
			  same_at_start_charpos, inserted_chars, 0);

the before-change-functions are called from insert_from_buffer.  So
it's not like the replacement would be missed by the
before-change-functions.

IOW, your problem is not that before-change-functions are not called
in this specific scenario to indicate the replacement.  Your problem
is that the number of calls to before-change-functions doesn't match
the number of calls to after-change-functions, because del_range_byte
calls after-change-functions one extra time.  And that's the problem
which needs to be fixed, and it should be fixed in CC Mode, because
assuming these calls are balanced is simply wrong.

> > > It should calculate the two variables c-new-BEG and c-new-END (which
> > > bound the active region) in c-before-change, and these get modified in
> > > c-after-change.
> > 
> > Sorry, I don't understand.  Why can't you compute these two values in
> > the c-after-change?
> 
> In the general case because information gets lost at a change; things
> like where the beginning of a statement was, where deleted template
> delimiters used to be, what the bounds of a macro or string used to be
> (before backslashes were deleted).

I'm sure CC Mode is well equipped to recalculate all of these if
needed.  It does that when a file is first visited.

> Or are you suggesting I implement a special after-change handler for the
> case when the before-change-functions was missing?

Could be.  I don't know CC Mode well enough to propose specific
solutions.

> This would be possible, but it would still be more work than
> ensuring that before-change-functions actually gets called.

As I show above, before-change-functions _do_ get called by
insert-file-contents in this scenario.

> > If you are going to have to do that anyway, why do we have to consider
> > such low-level changes?
> 
> Because a proper fix is the Right Thing to do.

Well, we clearly disagree about that.  I think there's no problem in
insdel.c or its callers, and the cause of this bug is the incorrect
assumption in CC Mode that before- and after-change-functions will be
called in balanced pairs.  So from my POV, TRT would be to eliminate
that assumption, which should make CC Mode more robust in the long
run, as I'm not sure this is the only case where that assumption
breaks.

> Reverting the buffer is merely the recipe to reproduce the bug.  The bug
> itself is c-before-change not being called.

See above: it _is_ being called, from insert_from_buffer.

> As I said, why not let me implement what I've suggested?  If it doesn't
> work well enough, I can revert it.

Because it will waste time and efforts on a way towards a dead end.

Thanks.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sun, 31 Jul 2016 17:59:32 -0400
> 
> >> I fully agree that if you only use before-change-functions (or only use
> >> after-change-functions), your hook function should see *all* changes.
> > The change in question is a replacement.  Emacs implements
> > replacements by a deletion followed by an insertion.  The question is:
> > do we need to call the before-change-functions for both the deletion
> > and the insertion, or do we need to call them only once for the
> > "replacement"?
> 
> Either way is fine, according to my interpretation of "should see *all*
> changes".

Then I think we are fine, give or take a couple of minor changes that
might be needed in insert-file-contents.

Btw, it looks like insert-file-contents deliberately doesn't call
before-change-functions if nothing was inserted, even if some text was
deleted.  I wonder why.  The way the code is written, it cannot be a
coincidence.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> Btw, it looks like insert-file-contents deliberately doesn't call
> before-change-functions if nothing was inserted, even if some text was
> deleted.

Sounds like a bug.

> I wonder why.  The way the code is written, it cannot be a
> coincidence.

Coincidence or not, the b-c-f should we warned about the change.


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Mon, 01 Aug 2016 10:36:42 -0400
> Cc: emacs-devel@gnu.org
> 
> > Btw, it looks like insert-file-contents deliberately doesn't call
> > before-change-functions if nothing was inserted, even if some text was
> > deleted.
> 
> Sounds like a bug.
> 
> > I wonder why.  The way the code is written, it cannot be a
> > coincidence.
> 
> Coincidence or not, the b-c-f should we warned about the change.

But then why do we use a similar logic for after-change-functions?
Like this:

  if (inserted > 0 && total > 0
      && (NILP (visit) || !NILP (replace)))
    {
      signal_after_change (PT, 0, inserted);
      update_compositions (PT, PT, CHECK_BORDER);
    }

IOW, we don't call after-change-functions if nothing was inserted.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> But then why do we use a similar logic for after-change-functions?
> Like this:

>   if (inserted > 0 && total > 0
>       && (NILP (visit) || !NILP (replace)))
>     {
>       signal_after_change (PT, 0, inserted);
>       update_compositions (PT, PT, CHECK_BORDER);
>     }

> IOW, we don't call after-change-functions if nothing was inserted.

Sounds like a bug as well (unless it's called elsewhere in that case).
It's OK to skip calling a-c-f if we did not insert anything *and* did
not remove anything either.


        Stefan

Re: Unbalanced change hooks (part 2)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There are several functions in .../src/insdel.c which have a parameter
  > `prepare'.  The meaning of `prepare' is "direct the function to call
  > prepare_to_modify_buffer" (YUCK!).

Why do you say "yuck" to that?

I'm not saying it can't have problems, or that you can't
think of some change for the better.  However, to merit "yuck",
it would have to be grossly and conspicuously bad.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 01, 2016 at 04:07:50PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 31 Jul 2016 21:26:35 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > I'm in favour of fixing what is wrong, rather than implementing
> > workarounds somewhere else.  Workarounds are rarely fully satisfactory.

> I'm not suggesting a workaround.  I'm suggesting a fix for a design
> flaw in CC Mode -- its assumption that before-change-functions and
> after-change-functions are always invoked in balanced pairs.  CC Mode
> must not depend on that assumption, because it's false.

The flaw lies between the Emacs core and the Elisp manual.  The latter
quite clearly implies that each buffer change is accompanied by a single
before-change-functions call and a single after-change-functions call.

If Emacs is not going to be fixed to match its spec, then the doc will
need to be changed to match Emacs.  It will need to say, in some fashion,
that the two change hooks are called indeterminately.

[ .... ]

> In practice, we need to make sure that every modification triggers at
> least one call to before-change-functions and at least one call to
> after-change-functions.  The numbers of these calls do not have to
> match.

And in such calls, what are the parameters BEG, END, and OLD-LEN supposed
to mean?

> I had my share of hacking that code [insdel.c] and its callers.  It's
> impossible to convey everything I learned while doing that in a
> discussion such as this one.  I say more about this below.  If that's
> still not enough, you will have to trust me on that.

OK.  In exchange, I'm asking you to trust me when I say that the sort of
change you're expecting in CC Mode is impractical - months of work,
rather than weeks.

[ .... ]

> Just don't assume that there will be a c-before-change call for each
> c-after-change call, that's all.

What you seem to be saying is that before-change-functions isn't very
useful.  Since lots of packages use it, you seem to be saying that these
packages are broken, just as CC Mode is broken.

> Specifically, in the case in point, I believe that the "unbalanced"
> call to c-after-change clearly tells you that text has been deleted,
> so I think it should be enough for you to do the job of the "missing"
> c-before-change call as well.

No.  CC Mode also needs to know what was there before the change.  It
cannot guess.  I think it's fair to ask you to suggest a way for CC Mode
to get that deleted buffer section, somehow.

It is a perfectly reasonable requirement for a generic piece of Lisp code
to be able to get full details of any buffer change.

I've had a few wierd ideas myself on this, one of the first being that CC
Mode will maintain a shadow buffer for each of its buffers.  The shadow
will be identical to the main buffer in all ways, apart from when changes
are under way.  After a buffer change, CC Mode will consult the shadow
buffer to see what used to be there, and then copy the change from the
main buffer to keep it up to date.  I don't think this is a good way to
go, so I'm asking you to suggest something better.

> > > I said "basically unchanged".  Trivial refactoring doesn't count.  And
> > > any changes in this function were in response to issues common to all
> > > the callers.  The case in point isn't such a case.

> > What I am proposing is only a little more than trivial refactoring.

> No, it's far from that.  It fundamentally changes the list of things
> that are done in many operations that modify buffers in various ways,
> and the order in which these things are done.  We shouldn't make such
> changes at this point in Emacs development, unless we identify a clear
> bug that affects all the callers of these functions.

> This case is not such a case.  It is a case of a single Lisp package
> that made incorrect assumptions about the way the modification hooks
> are called.  So the right place to fix this is in that Lisp package.

Here is a list of lisp files which use before-change-functions.
Currently all of them are broken due to the unreliability of
before-change-functions:

./wid-edit.el:1292:  (add-hook 'before-change-functions 'widget-before-change nil t)
./emacs-lisp/syntax.el:495:             (add-hook 'before-change-functions
./whitespace.el:2184:    (add-hook 'before-change-functions #'whitespace-buffer-changed nil t)
./allout-widgets.el:594:        (add-hook 'before-change-functions 'allout-widgets-before-change-handler
./allout-widgets.el:741:          (add-hook 'before-change-functions
./obsolete/lazy-lock.el:591:    (add-hook 'before-change-functions 'lazy-lock-arrange-before-change nil t))
./org/org-table.el:3954:  (add-hook 'before-change-functions 'org-table-remove-rectangle-highlight))
./allout.el:2049:      (add-hook 'before-change-functions 'allout-before-change-handler nil t)
./progmodes/cc-mode.el:641:  (add-hook 'before-change-functions 'c-before-change nil t)
./progmodes/js.el:3740:  (add-hook 'before-change-functions #'js--flush-caches t t)
./progmodes/compile.el:2115:  (add-hook 'before-change-functions 'compilation--flush-parse nil t)
./cedet/semantic/grammar.el:1347:  (add-hook 'before-change-functions
./textmodes/tex-mode.el:707:      (add-hook 'before-change-functions
./textmodes/sgml-mode.el:926:        (add-hook 'before-change-functions
./emulation/viper-cmd.el:2307:  (add-hook 'before-change-functions 'viper-before-change-sentinel t)

[ .... ]

> > I'm still trying to work out what they are, beyond a generalised
> > opposition to any change at this level of the code.

> Without a good reason, yeah, that's pretty much it.

> It's not some paranoia, mind you: I've been working on fixing quite a
> few subtle bugs which originated due to such changes for the last few
> years, so I know what I'm talking about.  Please trust my experience
> and the conclusions I drew from it.

Please tell me how CC Mode can get full details of any buffer change,
then I'll be (reasonably) happy.

[ .... ]

> What I'd like you to try instead is eliminate from CC Mode the
> incorrect assumption about pairwise calls to before- and
> after-change-functions.  That is the correct way towards solving this
> problem.

CC Mode needs full details of all buffer changes.  Redesigning it so that
it doesn't doesn't bear thinking about - it would be far more work than
making the change hooks get called properly.

How about the following idea: we deprecate before-change-functions, which
is broken anyway, and isn't going to get fixed.  Functions in
after-change-functions would somehow get access to the old deleted
portion of the buffer, complete with text properties, overlays, ...?
This shouldn't be too hard, given that such stuff is stored in the undo
list anyway, even though in an unsuitable form.

> > Programs other than CC Mode use before-change-functions.  It would be
> > more effort to construct an actual failure than simply to fix the
> > problem.

> But before-change-functions _are_ being called in the scenario of the
> bugs we are discussing.  They just aren't balanced with calls to
> after-change-functions, that's all.

No.  In the scenario of bug #24094 there was no call to
before-change-functions whatsoever.  (Discovered by instrumenting
c-before-change and c-after-change for edebug.)

> See, in this fragment of insert-file-contents:

>       if (same_at_end != same_at_start)
> 	{
> 	  invalidate_buffer_caches (current_buffer,
> 				    BYTE_TO_CHAR (same_at_start),
> 				    same_at_end_charpos);
> 	  del_range_byte (same_at_start, same_at_end, 0);
> 	  temp = GPT;
> 	  eassert (same_at_start == GPT_BYTE);
> 	  same_at_start = GPT_BYTE;
> 	}
>       else
> 	{
> 	  temp = same_at_end_charpos;
> 	}
>       /* Insert from the file at the proper position.  */
>       SET_PT_BOTH (temp, same_at_start);
>       same_at_start_charpos
> 	= buf_bytepos_to_charpos (XBUFFER (conversion_buffer),
> 				  same_at_start - BEGV_BYTE
> 				  + BUF_BEG_BYTE (XBUFFER (conversion_buffer)));
>       eassert (same_at_start_charpos == temp - (BEGV - BEG));
>       inserted_chars
> 	= (buf_bytepos_to_charpos (XBUFFER (conversion_buffer),
> 				   same_at_start + inserted - BEGV_BYTE
> 				  + BUF_BEG_BYTE (XBUFFER (conversion_buffer)))
> 	   - same_at_start_charpos);
>       /* This binding is to avoid ask-user-about-supersession-threat
> 	 being called in insert_from_buffer (via in
> 	 prepare_to_modify_buffer).  */
>       specbind (intern ("buffer-file-name"), Qnil);
>       insert_from_buffer (XBUFFER (conversion_buffer),
> 			  same_at_start_charpos, inserted_chars, 0);

> the before-change-functions are called from insert_from_buffer.  So
> it's not like the replacement would be missed by the
> before-change-functions.

It was missed in the recipe for bug #24094.  For an atomic user action
(namely C-x C-f test.c++ followed by typing "yes" to the prompt to reload
the file) there was no invocation of before-change-functions and one of
after-change-functions.  This is the bug I am complaining of.

[ .... ]

> > > > It should calculate the two variables c-new-BEG and c-new-END (which
> > > > bound the active region) in c-before-change, and these get modified in
> > > > c-after-change.

> > > Sorry, I don't understand.  Why can't you compute these two values in
> > > the c-after-change?

> > In the general case because information gets lost at a change; things
> > like where the beginning of a statement was, where deleted template
> > delimiters used to be, what the bounds of a macro or string used to be
> > (before backslashes were deleted).

> I'm sure CC Mode is well equipped to recalculate all of these if
> needed.  It does that when a file is first visited.

An empty buffer is a fully defined state.  A buffer immediately following
a deletion is in an undefined state, which requires to know what was
deleted to make it defined again.

[ .... ]

> > Because a proper fix is the Right Thing to do.

> Well, we clearly disagree about that.  I think there's no problem in
> insdel.c or its callers, and the cause of this bug is the incorrect
> assumption in CC Mode that before- and after-change-functions will be
> called in balanced pairs.  So from my POV, TRT would be to eliminate
> that assumption, which should make CC Mode more robust in the long
> run, as I'm not sure this is the only case where that assumption
> breaks.

Please provide some means by which CC Mode has access to the full details
of a buffer change.  That's all I'm asking for now (even if I'm getting a
bit repetitive).

[ .... ]

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello again, Eli.

On Mon, Aug 01, 2016 at 04:53:23PM +0000, Alan Mackenzie wrote:

[ .... ]

> Please provide some means by which CC Mode has access to the full details
> of a buffer change.  That's all I'm asking for now (even if I'm getting a
> bit repetitive).

How about the following idea: we create a special purpose undo list
separate from the existing one.  Every change is recorded on this list,
regardless of whether or not main undo is disabled.  After each
invocation of after-change-functions the list is emptied.

    (defmacro with-change-undone (&rest forms) ...)

An after-change-function can invoke the macro `with-change-undone' which
would undo the changes, execute the &rest forms, then redo the changes.

Surely this would satisfy both of us, no?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 1 Aug 2016 16:53:23 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > I'm not suggesting a workaround.  I'm suggesting a fix for a design
> > flaw in CC Mode -- its assumption that before-change-functions and
> > after-change-functions are always invoked in balanced pairs.  CC Mode
> > must not depend on that assumption, because it's false.
> 
> The flaw lies between the Emacs core and the Elisp manual.  The latter
> quite clearly implies that each buffer change is accompanied by a single
> before-change-functions call and a single after-change-functions call.

I don't mind making the manual more explicit on this.  But doing so
won't fix the problem at hand.

> > In practice, we need to make sure that every modification triggers at
> > least one call to before-change-functions and at least one call to
> > after-change-functions.  The numbers of these calls do not have to
> > match.
> 
> And in such calls, what are the parameters BEG, END, and OLD-LEN supposed
> to mean?

That depends on whether the changes are insertions or deletions.  I
thought it was quite clear from the documentation, but if not, we can
spell that out better as well.

> > I had my share of hacking that code [insdel.c] and its callers.  It's
> > impossible to convey everything I learned while doing that in a
> > discussion such as this one.  I say more about this below.  If that's
> > still not enough, you will have to trust me on that.
> 
> OK.  In exchange, I'm asking you to trust me when I say that the sort of
> change you're expecting in CC Mode is impractical - months of work,
> rather than weeks.

I trust you.  I just hope you will find a simpler way of doing that,
after some thought and maybe more discussions.

> > Just don't assume that there will be a c-before-change call for each
> > c-after-change call, that's all.
> 
> What you seem to be saying is that before-change-functions isn't very
> useful.  Since lots of packages use it, you seem to be saying that these
> packages are broken, just as CC Mode is broken.

No, I don't think before-change-functions are useless, and I don't
know about any breakage of packages that use that hook.

> > Specifically, in the case in point, I believe that the "unbalanced"
> > call to c-after-change clearly tells you that text has been deleted,
> > so I think it should be enough for you to do the job of the "missing"
> > c-before-change call as well.
> 
> No.  CC Mode also needs to know what was there before the change.  It
> cannot guess.  I think it's fair to ask you to suggest a way for CC Mode
> to get that deleted buffer section, somehow.

The functions that delete text do call after-change-functions, and
those functions do get this information.  Or maybe I don't understand
what is it that you are asking.

> Here is a list of lisp files which use before-change-functions.
> Currently all of them are broken due to the unreliability of
> before-change-functions:

It depends how they are using this hook.

> Please tell me how CC Mode can get full details of any buffer change,
> then I'll be (reasonably) happy.

You will have to tell much more about your needs, for me to begin
being helpful on that level.  Right now, all I can say is that
after-change-functions provide that information, AFAIK.

> > But before-change-functions _are_ being called in the scenario of the
> > bugs we are discussing.  They just aren't balanced with calls to
> > after-change-functions, that's all.
> 
> No.  In the scenario of bug #24094 there was no call to
> before-change-functions whatsoever.

They were on my system, I saw in GDB signal_before_change being called
from that place in insert-file-contents.  I wonder how come you didn't.

> >       insert_from_buffer (XBUFFER (conversion_buffer),
> > 			  same_at_start_charpos, inserted_chars, 0);
> 
> > the before-change-functions are called from insert_from_buffer.  So
> > it's not like the replacement would be missed by the
> > before-change-functions.
> 
> It was missed in the recipe for bug #24094.

How can that be?  insert_from_buffer unconditionally calls
insert_from_buffer_1, which unconditionally calls
prepare_to_modify_buffer, which in turns calls signal_before_change.

> For an atomic user action (namely C-x C-f test.c++ followed by
> typing "yes" to the prompt to reload the file) there was no
> invocation of before-change-functions and one of
> after-change-functions.  This is the bug I am complaining of.

Then I guess we are not talking about the same situation, and some
more digging is in order.  I did reproduce this using the recipe in
those bugs.

> > > In the general case because information gets lost at a change; things
> > > like where the beginning of a statement was, where deleted template
> > > delimiters used to be, what the bounds of a macro or string used to be
> > > (before backslashes were deleted).
> 
> > I'm sure CC Mode is well equipped to recalculate all of these if
> > needed.  It does that when a file is first visited.
> 
> An empty buffer is a fully defined state.  A buffer immediately following
> a deletion is in an undefined state, which requires to know what was
> deleted to make it defined again.

You mean, you cannot just throw away the entire info and recalculate
it anew?  IOW, pretend that you know nothing about this buffer, as if
you have just visited its file?

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 1 Aug 2016 17:15:52 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> How about the following idea: we create a special purpose undo list
> separate from the existing one.  Every change is recorded on this list,
> regardless of whether or not main undo is disabled.  After each
> invocation of after-change-functions the list is emptied.
> 
>     (defmacro with-change-undone (&rest forms) ...)
> 
> An after-change-function can invoke the macro `with-change-undone' which
> would undo the changes, execute the &rest forms, then redo the changes.
> 
> Surely this would satisfy both of us, no?

Sounds gross, but if this is the only way to solve this problem, and
you can live with it, so can I.

I do suggest to wait with conclusions for a while, perhaps a better
idea will come up.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 01, 2016 at 10:39:03PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 1 Aug 2016 17:15:52 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > How about the following idea: we create a special purpose undo list
> > separate from the existing one.  Every change is recorded on this list,
> > regardless of whether or not main undo is disabled.  After each
> > invocation of after-change-functions the list is emptied.

> >     (defmacro with-change-undone (&rest forms) ...)

> > An after-change-function can invoke the macro `with-change-undone' which
> > would undo the changes, execute the &rest forms, then redo the changes.

> > Surely this would satisfy both of us, no?

> Sounds gross, but if this is the only way to solve this problem, and
> you can live with it, so can I.

> I do suggest to wait with conclusions for a while, perhaps a better
> idea will come up.

Slightly less gross, perhaps:
(i) Abandon the use of before-change-functions.
(ii) Enable undo in pre-command-hook, if necessary.
(iii) At each after-change-functions call:
  a - Analyze buffer-undo-list down as far as (or pos (next-nil)), and
    revert the insertion or deletion which as been made, taking
    particular care over coalesced multiple insertions/deletions.
  b - (setq pos (current-position-in-undo-list))
  c - Perform what used to be before-change actions.
  d - restore the insertion/deletion.
  e - Perform the normal after-change functions.
(iv) Disable undo in post-command-hook, if appropriate.

This is less gross in that it uses the standard undo list rather than
creating a new special purpose one.  It would also be usable in existing
Emacsen.  But I still say it would be preferable to have a function
`old-string' callable from after-change-functions rather than having to
mess around with the undo list in (ii), (iii)a, and (iv).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Richard.

On Mon, Aug 01, 2016 at 12:38:42PM -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > There are several functions in .../src/insdel.c which have a parameter
>   > `prepare'.  The meaning of `prepare' is "direct the function to call
>   > prepare_to_modify_buffer" (YUCK!).

> Why do you say "yuck" to that?

Hey, don't misquote me!  I said "YUCK!", not "yuck".  :-)

It's my emotional reaction to a dodgy coding practice.  We have two
functions, one of which assumes far too much about the inner workings of
the other.  This "convolution" (as Richard Copley described it) is
liable to lead to all sorts of tangles and awkwardnesses and things
generally being very difficult to maintain.

> I'm not saying it can't have problems, or that you can't
> think of some change for the better.  However, to merit "yuck",
> it would have to be grossly and conspicuously bad.

I think the particular instance IS grossly and conspicuously bad.  The
resulting code is so difficult to maintain that (i) our
before-change-functions/after-change-functions mechanism is broken, in
that the b-c-f hook is only called for some changes, not all changes;
(ii) Eli Z. has forcefully declined my offer to fix this, on the grounds
that the pertinent code (in .../src/insdel.c) is so fragile that any
attempt to fix it will inevitably introduce hidden bugs elsewhere.

This `prepare' parameter is central to the difficulties of fixing/not
being able to fix the b-c-f/a-c-f mechanism.

> -- 
> Dr Richard Stallman
> President, Free Software Foundation (gnu.org, fsf.org)
> Internet Hall-of-Famer (internethalloffame.org)
> Skype: No way! See stallman.org/skype.html.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Richard Copley]

On 2 August 2016 at 11:15, Alan Mackenzie <acm@muc.de> wrote:

> It's my emotional reaction to a dodgy coding practice.  We have two
> functions, one of which assumes far too much about the inner workings of
> the other.  This "convolution" (as Richard Copley described it) is
> liable to lead to all sorts of tangles and awkwardnesses and things
> generally being very difficult to maintain.

:) I think that was Eli.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> And in such calls, what are the parameters BEG, END, and OLD-LEN supposed
> to mean?

Same as ever: that compared to the last time the hook was run, nothing
was changed before BEG, nothing as changed after END and that the chunk
of text between those two had length OLD-LEN last time the hook was run.

> An empty buffer is a fully defined state.  A buffer immediately following
> a deletion is in an undefined state, which requires to know what was
> deleted to make it defined again.

If you need to know something about the deleted text, you can always
keep track of that info in things like text-properties (or buffer-local
vars).  E.g.:

>> > As a concrete example of this, suppose we have this in a buffer:
>> > 
>> >     #define foo() first line \
>> >                   second line \
>> > 	          third line \
>> > 	          fourth line
>> > 
>> > , and the change consists of deleting the backslash on L2.

You can handle this case with nothing more than an a-c-f and something
like a cc-mode-multiline property applied to the whole CPP construct:
in a-c-f you look at the cc-mode-multiline property at BEG and at END
and you expand the "area to recompute" according to that property.

That's the strategy used by font-lock.

Another strategy is the one used by syntax.el which is to simply
consider everything after BEG as being in need of review (actually,
font-lock also uses this strategy via jit-lock-context).


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 1 Aug 2016 20:52:23 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> (i) Abandon the use of before-change-functions.
> (ii) Enable undo in pre-command-hook, if necessary.
> (iii) At each after-change-functions call:
>   a - Analyze buffer-undo-list down as far as (or pos (next-nil)), and
>     revert the insertion or deletion which as been made, taking
>     particular care over coalesced multiple insertions/deletions.
>   b - (setq pos (current-position-in-undo-list))
>   c - Perform what used to be before-change actions.
>   d - restore the insertion/deletion.
>   e - Perform the normal after-change functions.
> (iv) Disable undo in post-command-hook, if appropriate.

What I don't get is why do you need to look at the deleted text.  The
after-change-functions called after deletion tells you which buffer
positions were removed, so any data CC Mode holds in its caches about
those positions should be discarded.  Why do you need to actually look
at the removed text itself?  What am I missing?

Regarding the proposal itself, undoing arbitrary changes might be
expensive, both CPU- and memory-wise.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> Slightly less gross, perhaps:
> (i) Abandon the use of before-change-functions.
> (ii) Enable undo in pre-command-hook, if necessary.
> (iii) At each after-change-functions call:
>   a - Analyze buffer-undo-list down as far as (or pos (next-nil)), and
>     revert the insertion or deletion which as been made, taking
>     particular care over coalesced multiple insertions/deletions.
>   b - (setq pos (current-position-in-undo-list))
>   c - Perform what used to be before-change actions.
>   d - restore the insertion/deletion.
>   e - Perform the normal after-change functions.
> (iv) Disable undo in post-command-hook, if appropriate.

Alan.  I urge you to take a step back and think about how to solve your
problem without looking at the details of each change.

I am 100% sure, based on my experience with lots of major modes
(including cc-mode), that the result will be more robust, simpler, and
at least as efficient.


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 2 Aug 2016 10:15:49 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> 
> I think the particular instance IS grossly and conspicuously bad.  The
> resulting code is so difficult to maintain that (i) our
> before-change-functions/after-change-functions mechanism is broken, in
> that the b-c-f hook is only called for some changes, not all changes;
> (ii) Eli Z. has forcefully declined my offer to fix this, on the grounds
> that the pertinent code (in .../src/insdel.c) is so fragile that any
> attempt to fix it will inevitably introduce hidden bugs elsewhere.

I'm sorry, but the above doesn't match my views and opinions on this
matter, so I would like to express them as clearly as possible:

 (i)  I disagree that the modification-hooks mechanism is broken.  It
      works as designed; if you read the code, you see that very
      clearly.  (Alan disagrees with the design, but that's another
      matter.)
 (ii) I never said the code which implements the support for these
      hooks is "so fragile that any attempt to fix it will inevitably
      introduce hidden bugs".  The code in insdel.c is rock-solid,
      and I feel confident when I make changes there (as indeed I did
      several times during the last year).  What I fear is not the
      fragility of insdel.c, it's the unintended consequences of
      changing the aspects of it and of its callers that are clearly
      against the original design.  Doing that for introducing some
      important new feature would be justified, but doing that because
      CC Mode made incorrect assumptions about how this mechanism
      works is IMO wrong.

> This `prepare' parameter is central to the difficulties of fixing/not
> being able to fix the b-c-f/a-c-f mechanism.

I don't think the mechanism needs fixing.  The intent of the 'prepare'
parameter is quite clear, and is consistently followed by all the
callers of these functions.

What we need is find the best way of fixing CC Mode based on this or
some other mechanism.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 05:44:34PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 1 Aug 2016 20:52:23 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > (i) Abandon the use of before-change-functions.
> > (ii) Enable undo in pre-command-hook, if necessary.
> > (iii) At each after-change-functions call:
> >   a - Analyze buffer-undo-list down as far as (or pos (next-nil)), and
> >     revert the insertion or deletion which as been made, taking
> >     particular care over coalesced multiple insertions/deletions.
> >   b - (setq pos (current-position-in-undo-list))
> >   c - Perform what used to be before-change actions.
> >   d - restore the insertion/deletion.
> >   e - Perform the normal after-change functions.
> > (iv) Disable undo in post-command-hook, if appropriate.

> What I don't get is why do you need to look at the deleted text.  The
> after-change-functions called after deletion tells you which buffer
> positions were removed, so any data CC Mode holds in its caches about
> those positions should be discarded.  Why do you need to actually look
> at the removed text itself?  What am I missing?

Things are cached in text properties on the text itself.  These
properties can only be examined before the text is deleted.

A specific instance of needing to see the deleted text is in the
handling of C++ raw strings.  Such a thing looks like this:

    "foo(<arbitrary characters, including ", \n, "bar) ....>)foo"

.  The "foo" can be any (sensible) sequence of 0-16 characters.  The
contents of the string can be literally anything _except_ the
terminating delimiter ")foo\""; for example it could include both
"\"bar(" and ")bar\"".

Now supposing the buffer is changed by deleting the opening delimiter
"\"foo(".  Lots of text properties in the (formally) raw string need to
be removed.  For example, each "\"" in the raw string has a syntax-table
text property on it to suppress its "string" syntax, and there are text
properties on what used to be the closing delimiter.  For this we need
to know where the string ended, i.e. we need to know the "foo" from the
opening delimiter so that we can search forward for ")foo\"".  This
involves looking at the buffer before it is changed.

> Regarding the proposal itself, undoing arbitrary changes might be
> expensive, both CPU- and memory-wise.

It might.  But what are the alternatives?

Anyhow, most buffer changes are going to be single character insertions
or deletions for which the extra processing time shouldn't be
noticeable.  For something like a large C-w, a little extra time
hopefully won't be too stressfull - undoing that C-w won't involve CC
Mode's syntactic analysis, since all the deleted text properties are
stored in the undo list.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Richard.

On Tue, Aug 02, 2016 at 11:37:09AM +0100, Richard Copley wrote:
> On 2 August 2016 at 11:15, Alan Mackenzie <acm@muc.de> wrote:

> > It's my emotional reaction to a dodgy coding practice.  We have two
> > functions, one of which assumes far too much about the inner workings of
> > the other.  This "convolution" (as Richard Copley described it) is
> > liable to lead to all sorts of tangles and awkwardnesses and things
> > generally being very difficult to maintain.

> :) I think that was Eli.

Actually, it was Noam.  Apologies for my confusion.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 2 Aug 2016 16:09:14 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > What I don't get is why do you need to look at the deleted text.  The
> > after-change-functions called after deletion tells you which buffer
> > positions were removed, so any data CC Mode holds in its caches about
> > those positions should be discarded.  Why do you need to actually look
> > at the removed text itself?  What am I missing?
> 
> Things are cached in text properties on the text itself.  These
> properties can only be examined before the text is deleted.

OK, so you must also have code that computes and places those
properties, right?  What triggers that code?  E.g., when a file is
visited, what invokes that code, and how much of the buffer it
processes when invoked?  If it doesn't process the entire buffer in
one go, how is it invoked to process the rest of the buffer?

> > Regarding the proposal itself, undoing arbitrary changes might be
> > expensive, both CPU- and memory-wise.
> 
> It might.  But what are the alternatives?

I don't know yet.  That's why I asked the questions above.

Thanks.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 05:57:18PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 2 Aug 2016 10:15:49 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org

> > I think the particular instance IS grossly and conspicuously bad.  The
> > resulting code is so difficult to maintain that (i) our
> > before-change-functions/after-change-functions mechanism is broken, in
> > that the b-c-f hook is only called for some changes, not all changes;
> > (ii) Eli Z. has forcefully declined my offer to fix this, on the grounds
> > that the pertinent code (in .../src/insdel.c) is so fragile that any
> > attempt to fix it will inevitably introduce hidden bugs elsewhere.

> I'm sorry, but the above doesn't match my views and opinions on this
> matter, so I would like to express them as clearly as possible:

>  (i)  I disagree that the modification-hooks mechanism is broken.  It
>       works as designed; if you read the code, you see that very
>       clearly.  (Alan disagrees with the design, but that's another
>       matter.)

I thought we'd already agreed that no rational design would omit the
before-change-hooks call from an arbitrary subset of changes.  I have
read the code, and what I see is not "works as designed" but "failure
consequently to adhere to the design".  b-c-f is used infrequently
enough that it took CC Mode to expose the failure.

>  (ii) I never said the code which implements the support for these
>       hooks is "so fragile that any attempt to fix it will inevitably
>       introduce hidden bugs".  The code in insdel.c is rock-solid,
>       and I feel confident when I make changes there (as indeed I did
>       several times during the last year).  What I fear is not the
>       fragility of insdel.c, it's the unintended consequences of
>       changing the aspects of it and of its callers that are clearly
>       against the original design.

To me it's perfectly clear the original design was, for each buffer
modification there is exactly one invocation of b-c-f and one of a-c-f.
(That's leaving aside precisely what qualifies as a "buffer
modification" and what doesn't.)

The fact that we're too scared to make the simple changes to adhere to
the design (as documented in the Elisp manual) is precisely what
"fragile code" means.  And "call b-c-f for all changes" should be a
simple change.

>       Doing that for introducing some important new feature would be
>       justified, but doing that because CC Mode made incorrect
>       assumptions about how this mechanism works is IMO wrong.

CC Mode didn't make incorrect assumptions - it followed rigourously the
feature as documented in the Elisp manual.  The implication of your last
paragraph is that before using any core feature, a Lisp hacker should
first examine its implementation to make sure the documentation is
correct.  This is absurd.

Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
files which use before-change-functions, and some of these uses are
going to assume it works as documented, just as CC Mode did.  Sporadic
failures are going to occur in some of these other places, due to those
hook functions sometimes not being called.

> > This `prepare' parameter is central to the difficulties of fixing/not
> > being able to fix the b-c-f/a-c-f mechanism.

> I don't think the mechanism needs fixing.

We'll have to carry on disagreeing about this.  For what it's worth, I
accept that it's not going to get fixed.

> The intent of the 'prepare' parameter is quite clear, and is
> consistently followed by all the callers of these functions.

I don't find the intent clear, since prepare_to_modify_buffer is a
mishmash of miscellaneous things, rather than a "do one thing and do it
well".  

If this `prepare' mechanism hadn't been used, but instead the functions
simply called where they were needed, then adding in the b-c-f calls
could have been trivial.

> What we need is find the best way of fixing CC Mode based on this or
> some other mechanism.

Yes.  CC Mode and the other 13 files.el too.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 2 Aug 2016 16:55:45 +0000
> Cc: rms@gnu.org, ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> >  (i)  I disagree that the modification-hooks mechanism is broken.  It
> >       works as designed; if you read the code, you see that very
> >       clearly.  (Alan disagrees with the design, but that's another
> >       matter.)
> 
> I thought we'd already agreed that no rational design would omit the
> before-change-hooks call from an arbitrary subset of changes.

No, I don't think there was such an agreement.  The set of changes for
which before-change-functions are not called is not arbitrary.

> To me it's perfectly clear the original design was, for each buffer
> modification there is exactly one invocation of b-c-f and one of a-c-f.

If such an original design existed in the distant future, it was long
ago (as in 20 years or more) replaced with what we basically see
today, where every change always triggers a call to
after-change-functions, but only some (the majority, but not all)
trigger the call to before-change-functions.

> (That's leaving aside precisely what qualifies as a "buffer
> modification" and what doesn't.)

I don't think it can be left aside, if you want to understand the
rationale.

> The fact that we're too scared to make the simple changes to adhere to
> the design (as documented in the Elisp manual) is precisely what
> "fragile code" means.  And "call b-c-f for all changes" should be a
> simple change.

We are not scared.  We (I) simply don't want to do that, at least not
yet.  When non-trivial changes were needed in insdel.c, they were done
without fear, including lately by yours truly.

> >       Doing that for introducing some important new feature would be
> >       justified, but doing that because CC Mode made incorrect
> >       assumptions about how this mechanism works is IMO wrong.
> 
> CC Mode didn't make incorrect assumptions - it followed rigourously the
> feature as documented in the Elisp manual.

The ELisp manual doesn't promise that these hooks will be called in
balanced pairs.

> The implication of your last paragraph is that before using any core
> feature, a Lisp hacker should first examine its implementation to
> make sure the documentation is correct.  This is absurd.

No, that's life.  What do you think I did before embarking on writing
the bidirectional display?  A lot of code reading, a lot of
discussions with Gerd Moellmann, more code reading, etc.  All that was
needed even though the display engine has much more detailed overview
comments than almost any other part of Emacs.

> Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
> files which use before-change-functions, and some of these uses are
> going to assume it works as documented, just as CC Mode did.  Sporadic
> failures are going to occur in some of these other places, due to those
> hook functions sometimes not being called.

I will believe that when I see specific bug reports about those other
packages.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> Things are cached in text properties on the text itself.

I doubt you only put them on the actual text.

>     "foo(<arbitrary characters, including ", \n, "bar) ....>)foo"

E.g. I doubt you put "foo" on the "foo" text.

> Now supposing the buffer is changed by deleting the opening delimiter
> "\"foo(".  Lots of text properties in the (formally) raw string need to
> be removed.

No, that doesn't have to be the case.  CC-mode does it this way
currently because it decided to add properties on various chars within
that string, but that's not the only choice.  E.g. you could put
a "string fence" property on the first char of the string and refrain
from putting a "string fence" property until the end of the string.
Most other major modes follow this strategy.

> This involves looking at the buffer before it is changed.

Even if you have properties that you need to remove, you don't actually
need to know the text that was there before.  All you need is any
position past the end of the used-to-be-string.  Then you just have to
refresh all properties until that position.  A trivial choice is
point-max but if you want to try and optimize this you can add some
text-property (such as cc-mode-multiline) over the whole string, which
makes it easy to find the end, even when some pats have been deleted.
That's also a common approach in many major modes.

Of course, there are plenty of alternatives if you like to do it
differently.  E.g. you can annotate every Nth char in the buffer with
a text property indicating its "state".  So you won't get the exact end
of the string, but some position slightly past it.  Or you can keep that
information is a side-table holding markers.  Or ...


        Stefan

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 02 Aug 2016 20:17:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> > Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
> > files which use before-change-functions, and some of these uses are
> > going to assume it works as documented, just as CC Mode did.  Sporadic
> > failures are going to occur in some of these other places, due to those
> > hook functions sometimes not being called.
> 
> I will believe that when I see specific bug reports about those other
> packages.

Btw, I'm slowly but surely arriving to the conclusion that the
problems we are discussing can only happen when insert-file-contents
is called with VISIT and REPLACE non-nil, i.e. when reverting a
buffer.  Do we have any evidence to the contrary?  If we do, can
someone show or point to such contradicting evidence?

If my conclusion is correct, then we should probably focus on this
particular use case and look for a solution for it, as opposed to
trying to solve some more general problem that seems not to exist.  It
might be much easier and simpler.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 08:17:35PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 2 Aug 2016 16:55:45 +0000
> > Cc: rms@gnu.org, ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > >  (i)  I disagree that the modification-hooks mechanism is broken.  It
> > >       works as designed; if you read the code, you see that very
> > >       clearly.  (Alan disagrees with the design, but that's another
> > >       matter.)

> > I thought we'd already agreed that no rational design would omit the
> > before-change-hooks call from an arbitrary subset of changes.

> No, I don't think there was such an agreement.  The set of changes for
> which before-change-functions are not called is not arbitrary.

Would you please characterize these circumstances in a way useful for
people considering the use of before-change-functions.  If you like, I
can then modify Elisp's "Change Hooks" with that information.

> > To me it's perfectly clear the original design was, for each buffer
> > modification there is exactly one invocation of b-c-f and one of a-c-f.

> If such an original design existed in the distant past, it was long
> ago (as in 20 years or more) replaced with what we basically see
> today, where every change always triggers a call to
> after-change-functions, but only some (the majority, but not all)
> trigger the call to before-change-functions.

From the point of view of design, what is the benefit in not calling
b-c-f for that subset of circumstances?

> > (That's leaving aside precisely what qualifies as a "buffer
> > modification" and what doesn't.)

> I don't think it can be left aside, if you want to understand the
> rationale.

I do want to.  Currently, I'm not terribly concerned about exactly
whether some change qualifies as a buffer modification or not.  Just that
when it does, and only half of the change hooks are called, I'd like to
know why.

> > The fact that we're too scared to make the simple changes to adhere to
> > the design (as documented in the Elisp manual) is precisely what
> > "fragile code" means.  And "call b-c-f for all changes" should be a
> > simple change.

> We are not scared.  We (I) simply don't want to do that, at least not
> yet.  When non-trivial changes were needed in insdel.c, they were done
> without fear, including lately by yours truly.

OK.

> > >       Doing that for introducing some important new feature would be
> > >       justified, but doing that because CC Mode made incorrect
> > >       assumptions about how this mechanism works is IMO wrong.

> > CC Mode didn't make incorrect assumptions - it followed rigourously the
> > feature as documented in the Elisp manual.

> The ELisp manual doesn't promise that these hooks will be called in
> balanced pairs.

It does.  before-change-functions is described as:

    .... a list of functions to call before any buffer modification.

after-change-functions is likewise described as:

    .... a list of functions to call after any buffer modification.

.  It may not explicitly say they are paired, but it says each hook is
called after every modification.  This naturally pairs them up.

As a native English speaker, the only alternative interpretation I see
(which is absurd) is that Emacs will call each of these hooks once, after
any buffer modification Emacs choses.  I see no way out of the conclusion
that both of these hooks are to be called after each and every buffer
modification.

> > The implication of your last paragraph is that before using any core
> > feature, a Lisp hacker should first examine its implementation to
> > make sure the documentation is correct.  This is absurd.

> No, that's life.  What do you think I did before embarking on writing
> the bidirectional display?  A lot of code reading, a lot of
> discussions with Gerd Moellmann, more code reading, etc.  All that was
> needed even though the display engine has much more detailed overview
> comments than almost any other part of Emacs.

bidi is a somewhat special case.  Somebody considering the use of the
change hooks should be able to rely on their documentation, without
having to read and understand fileio.c and insdel.c.  

> > Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
> > files which use before-change-functions, and some of these uses are
> > going to assume it works as documented, just as CC Mode did.  Sporadic
> > failures are going to occur in some of these other places, due to those
> > hook functions sometimes not being called.

> I will believe that when I see specific bug reports about those other
> packages.

You won't see them, and neither will I, because they'll be sporadic bugs
with wierd consequences, maybe the odd corrupted buffer once in a blue
moon.  Any that make it to a bug report will end up being closed as not
reproducible.  Other times, something will just fail to work, the user will
gently curse Emacs and try again.  We can just pretend those other files
are OK, I suppose.  A lot of them probably actually will be.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 2 Aug 2016 19:00:24 +0000
> Cc: rms@gnu.org, ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I thought we'd already agreed that no rational design would omit the
> > > before-change-hooks call from an arbitrary subset of changes.
> 
> > No, I don't think there was such an agreement.  The set of changes for
> > which before-change-functions are not called is not arbitrary.
> 
> Would you please characterize these circumstances in a way useful for
> people considering the use of before-change-functions.  If you like, I
> can then modify Elisp's "Change Hooks" with that information.

Let's leave this for later, and focus on solving the immediate problem
at hand.

> > If such an original design existed in the distant past, it was long
> > ago (as in 20 years or more) replaced with what we basically see
> > today, where every change always triggers a call to
> > after-change-functions, but only some (the majority, but not all)
> > trigger the call to before-change-functions.
> 
> >From the point of view of design, what is the benefit in not calling
> b-c-f for that subset of circumstances?

I'm not yet 100% sure, but I'm getting there.  In any case, the logic
is very clear, although its rationale I only half-understand for now.

> > The ELisp manual doesn't promise that these hooks will be called in
> > balanced pairs.
> 
> It does.  before-change-functions is described as:
> 
>     .... a list of functions to call before any buffer modification.
> 
> after-change-functions is likewise described as:
> 
>     .... a list of functions to call after any buffer modification.
> 
> .  It may not explicitly say they are paired, but it says each hook is
> called after every modification.  This naturally pairs them up.

Like I said, sticking to such a strict interpretation would cause our
code to be unnecessarily much too complex.  The modification hooks are
supposed to tell you that changes are about to be, or have been, made
in a region, but that doesn't necessarily mean you will have a single
call of before- and after- hook for each of the partial changes.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 09:30:37PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 02 Aug 2016 20:17:35 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org

> > > Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
> > > files which use before-change-functions, and some of these uses are
> > > going to assume it works as documented, just as CC Mode did.  Sporadic
> > > failures are going to occur in some of these other places, due to those
> > > hook functions sometimes not being called.

> > I will believe that when I see specific bug reports about those other
> > packages.

> Btw, I'm slowly but surely arriving to the conclusion that the
> problems we are discussing can only happen when insert-file-contents
> is called with VISIT and REPLACE non-nil, i.e. when reverting a
> buffer.  Do we have any evidence to the contrary?

As long as C-x C-f on a file changed outside of Emacs is included in
"reverting", then no.

> If we do, can someone show or point to such contradicting evidence?

You haven't said how you reached this conclusion.

I've grepped for calls of the five functions (insert_1_both,
replace_range, del_range_\(1\|byte\|both\)) which have `prepare'
parameters, and noted where the argument is false.  I see 15
occurrences.  They are in callproc.c, coding.c, editfns.c, fileio.c,
fns.c, insdel.c, print.c, and xdisp.c.

Can you persuade me that it wouldn't be a good use of my time to look at
each of these 15 occurrences?  :-)

> If my conclusion is correct, then we should probably focus on this
> particular use case and look for a solution for it, as opposed to
> trying to solve some more general problem that seems not to exist.  It
> might be much easier and simpler.

I hope you're right.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 2 Aug 2016 19:38:35 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Btw, I'm slowly but surely arriving to the conclusion that the
> > problems we are discussing can only happen when insert-file-contents
> > is called with VISIT and REPLACE non-nil, i.e. when reverting a
> > buffer.  Do we have any evidence to the contrary?
> 
> As long as C-x C-f on a file changed outside of Emacs is included in
> "reverting", then no.

You will see in files.el that C-x C-f on such a file calls
revert-buffer.

> > If we do, can someone show or point to such contradicting evidence?
> 
> You haven't said how you reached this conclusion.

By reading the code, of course.

> I've grepped for calls of the five functions (insert_1_both,
> replace_range, del_range_\(1\|byte\|both\)) which have `prepare'
> parameters, and noted where the argument is false.  I see 15
> occurrences.  They are in callproc.c, coding.c, editfns.c, fileio.c,
> fns.c, insdel.c, print.c, and xdisp.c.
> 
> Can you persuade me that it wouldn't be a good use of my time to look at
> each of these 15 occurrences?  :-)

You can if you want to, of course.  The above is my conclusion from
looking at all those places: the only one that counts is in
insert-file-contents.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Stefan.

On Tue, Aug 02, 2016 at 01:24:14PM -0400, Stefan Monnier wrote:
> > Things are cached in text properties on the text itself.

> I doubt you only put them on the actual text.

> >     "foo(<arbitrary characters, including ", \n, "bar) ....>)foo"

> E.g. I doubt you put "foo" on the "foo" text.

The text "foo" in "\"foo(" implicitly contains the information "the
matching closing delimiter is at the text ")foo\"" further down the
buffer".

> > Now supposing the buffer is changed by deleting the opening delimiter
> > "\"foo(".  Lots of text properties in the (formally) raw string need to
> > be removed.

> No, that doesn't have to be the case.  CC-mode does it this way
> currently because it decided to add properties on various chars within
> that string, but that's not the only choice.  E.g. you could put
> a "string fence" property on the first char of the string and refrain
> from putting a "string fence" property until the end of the string.
> Most other major modes follow this strategy.

Maybe it could, but that's not the point.  The point here is that after
deleting "\"foo(" (or part of it), the closing delimiter must be found,
even if all that is going to be done is to remove a string fence
property from it.  How will one find this closing delimiter without
knowing that the identifier in it is "foo"?  Indeed, how will one know
to find it without even knowing that a raw string delimiter has been
removed?

> > This involves looking at the buffer before it is changed.

> Even if you have properties that you need to remove, you don't actually
> need to know the text that was there before.

Again, how does one know that a raw string opening delimiter has been
removed?

> All you need is any position past the end of the used-to-be-string.
> Then you just have to refresh all properties until that position.  A
> trivial choice is point-max but if you want to try and optimize this
> you can add some text-property (such as cc-mode-multiline) over the
> whole string, which makes it easy to find the end, even when some pats
> have been deleted.  That's also a common approach in many major modes.

> Of course, there are plenty of alternatives if you like to do it
> differently.  E.g. you can annotate every Nth char in the buffer with
> a text property indicating its "state".  So you won't get the exact end
> of the string, but some position slightly past it.  Or you can keep that
> information is a side-table holding markers.  Or ...

Or one could note the state before a change, and see how the change
changes it.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> Maybe it could, but that's not the point.  The point here is that after
> deleting "\"foo(" (or part of it), the closing delimiter must be found,
> even if all that is going to be done is to remove a string fence
> property from it.  How will one find this closing delimiter without
> knowing that the identifier in it is "foo"?  Indeed, how will one know
> to find it without even knowing that a raw string delimiter has been
> removed?

Because the text-property on the remaining text says "I'm in the middle
of a raw-string" and it says so until the end of that raw-string, hence
telling you where was the end.

> Or one could note the state before a change, and see how the change
> changes it.

Experience shows that this approach is a lot more complex, especially if
you take into account that before- and after- change-functions calls aren't
guaranteed to come in pairs.

In any case the question was not "can the CC-mode way be made to work"
but "is it the only way".  If it was the only way, it would make sense
to try and support it better.  But it's far from the only way, and
I have enough experience in those matters to even claim that it's far
from being the best way.



        Stefan

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 10:25:45PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 2 Aug 2016 19:00:24 +0000
> > Cc: rms@gnu.org, ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > I thought we'd already agreed that no rational design would omit the
> > > > before-change-hooks call from an arbitrary subset of changes.

> > > No, I don't think there was such an agreement.  The set of changes for
> > > which before-change-functions are not called is not arbitrary.

> > Would you please characterize these circumstances in a way useful for
> > people considering the use of before-change-functions.  If you like, I
> > can then modify Elisp's "Change Hooks" with that information.

> Let's leave this for later, and focus on solving the immediate problem
> at hand.

You say the set of changes for which b-c-f isn't called isn't arbitrary.
That means it follows some sort of scheme.  Could a major mode, in an
after-change function use this scheme to determine whether
before-change-functions was called, for some particular change?  (This
won't be needed if my other idea (below) is workable.)

> > > If such an original design existed in the distant past, it was long
> > > ago (as in 20 years or more) replaced with what we basically see
> > > today, where every change always triggers a call to
> > > after-change-functions, but only some (the majority, but not all)
> > > trigger the call to before-change-functions.

> > >From the point of view of design, what is the benefit in not calling
> > b-c-f for that subset of circumstances?

> I'm not yet 100% sure, but I'm getting there.  In any case, the logic
> is very clear, although its rationale I only half-understand for now.

OK.

> > > The ELisp manual doesn't promise that these hooks will be called in
> > > balanced pairs.

> > It does.  before-change-functions is described as:

> >     .... a list of functions to call before any buffer modification.

> > after-change-functions is likewise described as:

> >     .... a list of functions to call after any buffer modification.

> > .  It may not explicitly say they are paired, but it says each hook is
> > called after every modification.  This naturally pairs them up.

> Like I said, sticking to such a strict interpretation would cause our
> code to be unnecessarily much too complex.  The modification hooks are
> supposed to tell you that changes are about to be, or have been, made
> in a region, but that doesn't necessarily mean you will have a single
> call of before- and after- hook for each of the partial changes.

OK, things are as they are.

I currently believe that most changes trigger before-change-functions but
EVERY CHANGE triggers after-change-functions.  Can you confirm that this
is the case?  If it isn't, could we fix Emacs so that it is?

If it is indeed the case, then a solution to the problem would be to
check that before- and after-c-f are called alternately.  Any time
after-c-f is called twice in a row, we could reverse the change just made
(using the undo list), invoke before-c-f, redo the change, then let
after-c-f continue to completion.  If necessary, we can temporarily
enable undo in a pre-command-hook function.  Given how rarely b-c-f isn't
called, this would have a negligible effect on performance and store
usage.

This would be a better solution than reversing and redoing every change.
It will also work for older Emacsen, and will be relatively easy to
implement.  It will be much easier to implement than a solution which
doesn't use before-change-functions at all.

What do you think about this?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Stefan.

On Sun, Aug 07, 2016 at 11:09:52AM -0400, Stefan Monnier wrote:
> > Maybe it could, but that's not the point.  The point here is that after
> > deleting "\"foo(" (or part of it), the closing delimiter must be found,
> > even if all that is going to be done is to remove a string fence
> > property from it.  How will one find this closing delimiter without
> > knowing that the identifier in it is "foo"?  Indeed, how will one know
> > to find it without even knowing that a raw string delimiter has been
> > removed?

> Because the text-property on the remaining text says "I'm in the middle
> of a raw-string" and it says so until the end of that raw-string, hence
> telling you where was the end.

OK.  Extrapolating that suggestion, you're suggesting that all the
different types of information about the buffer which are currently
determined in c-before-change should be cached in text properties, or
perhaps in special purpose Lisp structures, and retrieved and used in
c-after-change.

For some types of information this would be easy (such as putting text
properties over C++ raw strings), for others it would be difficult (such
as maintaining the integrity of the c-parse-state cache (yes, _that_
thing)).

> > Or one could note the state before a change, and see how the change
> > changes it.

> Experience shows that this approach is a lot more complex, especially if
> you take into account that before- and after- change-functions calls aren't
> guaranteed to come in pairs.

I now have an idea (see my recent post to Eli) to _make_ these hook
calls properly paired.  This should be reliable and easy to implement.

You seem to be arguing that using all the information about a change is
more complicated than ignoring some of it.  I can't see how that can be
the case.  If one is missing part of the information, it is surely going
to be rather complicated to work around that lack.

> In any case the question was not "can the CC-mode way be made to work"
> but "is it the only way".  If it was the only way, it would make sense
> to try and support it better.  But it's far from the only way, and
> I have enough experience in those matters to even claim that it's far
> from being the best way.

"It's far from the only way" is what I have been saying for some while
about your way of handling, for example, syntax-table text properties.

Have you ever tried doing things the "CC Mode way", i.e. using
information from before-change-functions in an essential fashion?  If
so, please tell me about it, and what it was that brought you to giving
up on it.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> For some types of information this would be easy (such as putting text
> properties over C++ raw strings), for others it would be difficult (such
> as maintaining the integrity of the c-parse-state cache (yes, _that_
> thing)).

For this one, you don't need to know what changed either.  You just
restart from some point before BEG and compare the resulting state at
some point after END (after potentially adjusting the old state's
positions based on LENGTH).

> I now have an idea (see my recent post to Eli) to _make_ these hook
> calls properly paired.  This should be reliable and easy to implement.

I personally can't see any way you can make them properly paired in 100%
of the cases (including the case where one of the hooks itself makes
a buffer modification, for example).

> You seem to be arguing that using all the information about a change is
> more complicated than ignoring some of it.

Oh, yes.  And my attempt at using syntax-propertize in CC-mode comforts
me in this belief (even tho, I obviously only did the easy part).

> I can't see how that can be the case.  If one is missing part of the
> information, it is surely going to be rather complicated to work
> around that lack.

As Eli pointed out several times already, your code *has* to handle the
case where you have no prior knowledge, because that's what happens when
you open the file (or when you insert a new chunk of text).

So actually trying to use the information about "what was there before"
is extra effort.

> Have you ever tried doing things the "CC Mode way", i.e. using
> information from before-change-functions in an essential fashion?

No, because I know it's based on an invalid assumption.

> If so, please tell me about it, and what it was that brought you to
> giving up on it.

But as mentioned above, I've since had a look at what it takes to go
from "the CC-mode way" to "the usual way", and it mostly involves
removing (somewhat duplicate) code.


        Stefan

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2016 at 09:30:37PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 02 Aug 2016 20:17:35 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org

> > > Anyhow, it's not just CC Mode.  As already discussed, there are 13 other
> > > files which use before-change-functions, and some of these uses are
> > > going to assume it works as documented, just as CC Mode did.  Sporadic
> > > failures are going to occur in some of these other places, due to those
> > > hook functions sometimes not being called.
 
> > I will believe that when I see specific bug reports about those other
> > packages.

> Btw, I'm slowly but surely arriving to the conclusion that the
> problems we are discussing can only happen when insert-file-contents
> is called with VISIT and REPLACE non-nil, i.e. when reverting a
> buffer.  Do we have any evidence to the contrary?  If we do, can
> someone show or point to such contradicting evidence?

> If my conclusion is correct, then we should probably focus on this
> particular use case and look for a solution for it, as opposed to
> trying to solve some more general problem that seems not to exist.  It
> might be much easier and simpler.

Using buffer-undo-list temporarily to reverse a change and run
c-before-change seems to work well when the undo list is enabled.  When
it isn't enabled, I run into problems which don't look like being solved
easily.

If you're right in supposing that only insert-file-contents with those
particular parameters can cause the omission of the
before-change-functions call, then a solution would be for CC Mode to
advise those commands which call insert-f-c this way.  That advice would
cause the command to test and clear a flag set by CC Mode, and if it was
non-nil, to repeat the command.

Repeating find-file and revert-buffer in these circumstances appears to
work.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Sun, 7 Aug 2016 21:16:27 +0000
> Cc: rms@gnu.org, ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> You say the set of changes for which b-c-f isn't called isn't arbitrary.
> That means it follows some sort of scheme.  Could a major mode, in an
> after-change function use this scheme to determine whether
> before-change-functions was called, for some particular change?

Not easily, not without some bookkeeping.

> I currently believe that most changes trigger before-change-functions but
> EVERY CHANGE triggers after-change-functions.  Can you confirm that this
> is the case?

Yes.

> If it is indeed the case, then a solution to the problem would be to
> check that before- and after-c-f are called alternately.  Any time
> after-c-f is called twice in a row, we could reverse the change just made
> (using the undo list), invoke before-c-f, redo the change, then let
> after-c-f continue to completion.  If necessary, we can temporarily
> enable undo in a pre-command-hook function.  Given how rarely b-c-f isn't
> called, this would have a negligible effect on performance and store
> usage.
> 
> This would be a better solution than reversing and redoing every change.
> It will also work for older Emacsen, and will be relatively easy to
> implement.  It will be much easier to implement than a solution which
> doesn't use before-change-functions at all.
> 
> What do you think about this?

I'm extremely uneasy about using undo for such purposes.  You have
already discovered one reason why.  In general, applications are free
to manipulate the undo records as they see fit, so you cannot expect
that to work reliably.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 8 Aug 2016 14:36:14 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> If you're right in supposing that only insert-file-contents with those
> particular parameters can cause the omission of the
> before-change-functions call, then a solution would be for CC Mode to
> advise those commands which call insert-f-c this way.

I thought our policy was to leave advising to users, and never do
that in core.

I asked a few questions a couple of days ago, with the purpose of
coming up with idea for how to solve this issue in a more acceptable
way.  You never answered them.  Could you please humor me?

If nothing better comes up, I'd prefer to do something special in
insert-file-contents, specifically for CC Mode, rather than going to
the extremes you describe above and in the other messages.

Thanks.

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 08, 2016 at 06:42:20PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 8 Aug 2016 14:36:14 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > If you're right in supposing that only insert-file-contents with those
> > particular parameters can cause the omission of the
> > before-change-functions call, then a solution would be for CC Mode to
> > advise those commands which call insert-f-c this way.

> I thought our policy was to leave advising to users, and never do
> that in core.

I suppose so.  In that case, I can do the advising in standalone CC
Mode, but we hard code a solution in savannah master.

> I asked a few questions a couple of days ago, with the purpose of
> coming up with idea for how to solve this issue in a more acceptable
> way.  You never answered them.  Could you please humor me?

Sorry about that.  I think you mean the following.  Otherwise, would you
please clarify.

#########################################################################

>> Things are cached in text properties on the text itself.  These
>> properties can only be examined before the text is deleted.

> OK, so you must also have code that computes and places those
> properties, right?

Yes.

> What triggers that code?

before/after-change-functions.  For example, in C++ Mode, if a "<"
template delimiter is about is in a region about to be deleted, and the
matching ">" is after that region, the pertinent function called from
c-before-change (c-before-change-check-<>-operators) removes the
syntax-table properties from each of them.

> E.g., when a file is visited, what invokes that code, and how much of
> the buffer it processes when invoked?

On a newly visited file, the entire buffer is scanned by (the components
of) c-before-change and c-after-change, setting the text properties.

> If it doesn't process the entire buffer in one go, how is it invoked
> to process the rest of the buffer?

It does process the entire buffer in one go.

#########################################################################

> If nothing better comes up, I'd prefer to do something special in
> insert-file-contents, specifically for CC Mode, rather than going to
> the extremes you describe above and in the other messages.

What I'm looking at at the moment is calling revert-buffer a second time
if CC Mode signals a missing before-change-functions the first time.
After the test scenario, M-x revert-buffer works, so it might work if
called from the code instead.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 8 Aug 2016 16:54:49 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > What triggers that code?
> 
> before/after-change-functions.  For example, in C++ Mode, if a "<"
> template delimiter is about is in a region about to be deleted, and the
> matching ">" is after that region, the pertinent function called from
> c-before-change (c-before-change-check-<>-operators) removes the
> syntax-table properties from each of them.
> 
> > E.g., when a file is visited, what invokes that code, and how much of
> > the buffer it processes when invoked?
> 
> On a newly visited file, the entire buffer is scanned by (the components
> of) c-before-change and c-after-change, setting the text properties.

So, if worse comes to worst, you could trigger such a complete scan
after revert-buffer, e.g. in a specialized revert-buffer-function.

Do you have a way of completely forgetting everything about a certain
region of buffer text, and rescanning that region anew?  If so, would
it work to do this in c-after-change, since its arguments tell you
both the old and the new buffer positions where the changes were made?
If this works, it might be less expensive then rescanning the entire
buffer.

> What I'm looking at at the moment is calling revert-buffer a second time
> if CC Mode signals a missing before-change-functions the first time.
> After the test scenario, M-x revert-buffer works, so it might work if
> called from the code instead.

I don't understand how this could work reliably: it has the same
problems as the original recipe.  Am I missing something?

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 08, 2016 at 08:17:42PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 8 Aug 2016 16:54:49 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > What triggers that code?

> > before/after-change-functions.  For example, in C++ Mode, if a "<"
> > template delimiter is about is in a region about to be deleted, and the
> > matching ">" is after that region, the pertinent function called from
> > c-before-change (c-before-change-check-<>-operators) removes the
> > syntax-table properties from each of them.

> > > E.g., when a file is visited, what invokes that code, and how much of
> > > the buffer it processes when invoked?

> > On a newly visited file, the entire buffer is scanned by (the components
> > of) c-before-change and c-after-change, setting the text properties.

> So, if worse comes to worst, you could trigger such a complete scan
> after revert-buffer, e.g. in a specialized revert-buffer-function.

Why would that be bad?  We detect the bad condition by two consecutive
after-c-f calls, check that revert-buffer is on the call stack, and
twiddle `beg' `end' `old-len' (the parameters to after-change functions)
to indicate the entire buffer.  Then let c-after-change carry on.  This
feels too easy.  :-)  What am I missing?

> Do you have a way of completely forgetting everything about a certain
> region of buffer text, and rescanning that region anew?

This is tricky, because, for example, template delimiters outside that
region are associated with deleted (or unmarked) delimiters inside the
region.

> If so, would it work to do this in c-after-change, since its arguments
> tell you both the old and the new buffer positions where the changes
> were made?  If this works, it might be less expensive then rescanning
> the entire buffer.

I don't think this would work.  beg, end, and old-len only say where
things were changed, not what syntax-table'd characters have been
deleted.  Anyway, what's so bad about scanning the entire buffer after
having reverted it?

> > What I'm looking at at the moment is calling revert-buffer a second time
> > if CC Mode signals a missing before-change-functions the first time.
> > After the test scenario, M-x revert-buffer works, so it might work if
> > called from the code instead.

> I don't understand how this could work reliably: it has the same
> problems as the original recipe.  Am I missing something?

There is nothing like buffer-undo-list which can be corrupted by other
programs.  Anyhow, I've implemented it, and it works well on the test
case from bug #24094.  Here is the embryonic patch (I haven't reindented
code, so as to keep the patch short, and there's message output).  It
works, for the moment, by putting "around" advice around revert-buffer,
and invoking ad-do-it a second time if the first time didn't work:




diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index 04d2ed6..4db71b6 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -492,10 +492,13 @@ c-maybe-stale-found-type
 (defvar c-just-done-before-change nil)
 (make-variable-buffer-local 'c-just-done-before-change)
 ;; This variable is set to t by `c-before-change' and to nil by
-;; `c-after-change'.  It is used to detect a spurious invocation of
-;; `before-change-functions' directly following on from a correct one.  This
-;; happens in some Emacsen, for example when `basic-save-buffer' does (insert
-;; ?\n) when `require-final-newline' is non-nil.
+;; `c-after-change'.  It is used for two purposes: (i) to detect a spurious
+;; invocation of `before-change-functions' directly following on from a
+;; correct one.  This happens in some Emacsen, for example when
+;; `basic-save-buffer' does (insert ?\n) when `require-final-newline' is
+;; non-nil; (ii) to detect when Emacs fails to invoke
+;; `before-change-functions'.  This can happend when reverting a buffer - see
+;; bug #24094.
 
 (defun c-basic-common-init (mode default-style)
   "Do the necessary initialization for the syntax handling routines
@@ -643,8 +646,9 @@ c-basic-common-init
   (add-hook 'after-change-functions 'c-after-change nil t)
   (when (boundp 'font-lock-extend-after-change-region-function)
     (set (make-local-variable 'font-lock-extend-after-change-region-function)
-         'c-extend-after-change-region))) ; Currently (2009-05) used by all
+         'c-extend-after-change-region)) ; Currently (2009-05) used by all
                           ; languages with #define (C, C++,; ObjC), and by AWK.
+  (add-hook 'pre-command-hook 'c-pre-command-hook t))
 
 (defun c-setup-doc-comment-style ()
   "Initialize the variables that depend on the value of `c-doc-comment-style'."
@@ -1225,6 +1229,22 @@ c-before-change
     ;; newly inserted parens would foul up the invalidation algorithm.
     (c-invalidate-state-cache beg)))
 
+(defvar c-missed-before-change nil)
+
+(defun c-pre-command-hook ()
+  (setq c-missed-before-change nil))
+
+(defadvice revert-buffer (around c-advise-revert-buffer activate)
+  ad-do-it
+  (when (and c-buffer-is-cc-mode
+	     c-missed-before-change)
+    (message "Repeating revert-buffer.")
+    (setq c-missed-before-change nil)
+    ad-do-it
+    (message "Repeated revert-buffer %s"
+	     (if c-missed-before-change "failed" "succeeded"))
+    ))
+
 (defvar c-in-after-change-fontification nil)
 (make-variable-buffer-local 'c-in-after-change-fontification)
 ;; A flag to prevent region expanding stuff being done twice for after-change
@@ -1244,6 +1264,10 @@ c-after-change
   ;; This calls the language variable c-before-font-lock-functions, if non nil.
   ;; This typically sets `syntax-table' properties.
 
+  ;; Rarely, Emacs will fail to call `before-change-functions'.  See bug
+  ;; #24094.  In such cases, do it "by hand".
+  (if (not c-just-done-before-change)
+      (setq c-missed-before-change t)
   ;; (c-new-BEG c-new-END) will be the region to fontify.  It may become
   ;; larger than (beg end).
   (setq c-new-END (- (+ c-new-END (- end beg)) old-len))
@@ -1291,7 +1315,7 @@ c-after-change
 	  (save-excursion
 	    (mapc (lambda (fn)
 		    (funcall fn beg end old-len))
-		  c-before-font-lock-functions)))))))
+		  c-before-font-lock-functions))))))))
 
 (defun c-fl-decl-start (pos)
   ;; If the beginning of the line containing POS is in the middle of a "local"
@@ -1483,6 +1507,8 @@ c-extend-after-change-region
   ;; Of the seven CC Mode languages, currently (2009-05) only C, C++, Objc
   ;; (the languages with #define) and AWK Mode make non-null use of this
   ;; function.
+  (if c-missed-before-change
+      (cons beg end)
   (when (eq font-lock-support-mode 'jit-lock-mode)
     (save-restriction
       (widen)
@@ -1491,7 +1517,7 @@ c-extend-after-change-region
 	    (put-text-property c-new-BEG beg 'fontified nil))
 	(if (> c-new-END end)
 	    (put-text-property end c-new-END 'fontified nil)))))
-  (cons c-new-BEG c-new-END))
+  (cons c-new-BEG c-new-END)))
 
 ;; Emacs < 22 and XEmacs
 (defmacro c-advise-fl-for-region (function)


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Mon, 8 Aug 2016 18:42:23 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > So, if worse comes to worst, you could trigger such a complete scan
> > after revert-buffer, e.g. in a specialized revert-buffer-function.
> 
> Why would that be bad?

Not bad, just potentially expensive for large buffers.  But if that's
"good enough", I don't really mind.

However, the patch you show below doesn't implement this, does it?

Re: Unbalanced change hooks (part 2) [PATCH]

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 08, 2016 at 10:04:45PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 8 Aug 2016 18:42:23 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > So, if worse comes to worst, you could trigger such a complete scan
> > > after revert-buffer, e.g. in a specialized revert-buffer-function.

> > Why would that be bad?

> Not bad, just potentially expensive for large buffers.  But if that's
> "good enough", I don't really mind.

> However, the patch you show below doesn't implement this, does it?

No, but the following does.  It looks kind of simple, given the problems
we've been through.  Maybe this one is the solution:



diff -r 7a969360b796 cc-mode.el
--- a/cc-mode.el	Fri Aug 05 08:13:50 2016 +0000
+++ b/cc-mode.el	Mon Aug 08 19:46:40 2016 +0000
@@ -475,10 +475,13 @@
 (defvar c-just-done-before-change nil)
 (make-variable-buffer-local 'c-just-done-before-change)
 ;; This variable is set to t by `c-before-change' and to nil by
-;; `c-after-change'.  It is used to detect a spurious invocation of
-;; `before-change-functions' directly following on from a correct one.  This
-;; happens in some Emacsen, for example when `basic-save-buffer' does (insert
-;; ?\n) when `require-final-newline' is non-nil.
+;; `c-after-change'.  It is used for two purposes: (i) to detect a spurious
+;; invocation of `before-change-functions' directly following on from a
+;; correct one.  This happens in some Emacsen, for example when
+;; `basic-save-buffer' does (insert ?\n) when `require-final-newline' is
+;; non-nil; (ii) to detect when Emacs fails to invoke
+;; `before-change-functions'.  This can happend when reverting a buffer - see
+;; bug #24094.
 
 (defun c-basic-common-init (mode default-style)
   "Do the necessary initialization for the syntax handling routines
@@ -1220,6 +1223,18 @@
   ;; This calls the language variable c-before-font-lock-functions, if non nil.
   ;; This typically sets `syntax-table' properties.
 
+  ;; We can sometimes get two consecutive calls to `after-change-functions'
+  ;; without an intervening call to `before-change-functions' when reverting
+  ;; the buffer (see bug #24094).  Whatever the cause assume that the entire
+  ;; buffer has changed.
+  (when (not c-just-done-before-change)
+    (save-restriction
+      (widen)
+      (c-before-change (point-min) (point-max))
+      (setq beg (point-min)
+	    end (point-max)
+	    old-len (- end beg))))
+
   ;; (c-new-BEG c-new-END) will be the region to fontify.  It may become
   ;; larger than (beg end).
   (setq c-new-END (- (+ c-new-END (- end beg)) old-len))


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> If it is indeed the case, then a solution to the problem would be to
> check that before- and after-c-f are called alternately.  Any time
> after-c-f is called twice in a row, we could reverse the change just made
> (using the undo list), invoke before-c-f, redo the change, then let
> after-c-f continue to completion.  If necessary, we can temporarily

Sounds like piling up workarounds to make up for a mistake in the
underlying design.


        Stefan

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Stefan

On Mon, Aug 08, 2016 at 03:56:24PM -0400, Stefan Monnier wrote:
> > If it is indeed the case, then a solution to the problem would be to
> > check that before- and after-c-f are called alternately.  Any time
> > after-c-f is called twice in a row, we could reverse the change just made
> > (using the undo list), invoke before-c-f, redo the change, then let
> > after-c-f continue to completion.  If necessary, we can temporarily

> Sounds like piling up workarounds to make up for a mistake in the
> underlying design.

I couldn't agree more!

Anyway, we've now got what is surely a satisfactory solution: in the
event of a missing call to before-change-functions, assume that the
entirety of the buffer has changed (which is not far from the truth,
given that it happens on a `revert-buffer') and run
c-{before,after}-change on this basis.  See the patch in my latest post
to Eli.

This switches off some of the optimisations intended for revert-buffer,
but these seem no longer quite so pressing, given that reverting
xdisp.c, even with my latest patch, takes less than 0.2s on my 7 y.o.
machine; 20 years ago, things would have been different.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2)

[Alan Mackenzie]

Hello, Eli.

On Sun, Jul 31, 2016 at 06:03:19PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 31 Jul 2016 12:16:42 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: Óscar Fuentes <ofv@wanadoo.es>,
> > 	Richard Copley <rcopley@gmail.com>

> > I propose that the call to signal_before_change should be removed from
> > prepare_to_modify_buffer, and that it should be called explicitly from
> > the places it is needed.  This would allow bug #240[79]4 to be fixed.

> > Comments?

> Some.

> First, I don't agree with your conclusion that calls to
> before-change-functions and after-change-functions must always be
> balanced.  For starters, they aren't, and never have been.

I don't think this is true.  A data point is that in XEmacs's insdel.c
(which is quite a bit simpler than Emacs's) the calls to b-c-f and a-c-f
are balanced.  Bug #24049 doesn't trigger on XEmacs.  I think it's
likely this balance existed in Emacs until some time after the Lucid
Emacs fork.

My guess is that originally these hooks were _intended_ to be balanced
(as they still are, at least to a very great extent, on XEmacs).  At
some stage (probably in the late 1990s/early 2000s) optimisations were
made in insdel.c and elsewhere, and the strict pairing of b-c-f and
a-c-f got lost in the wash.  At that time, very little, if anything,
used b-c-f, hence there was nothing to signal alarm bells during the
testing of the optimisations.

As I said, that's my guess.  I've tried searching the emacs-devel
archive for early discussions on the topic, but not found anything.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [PATCH]

[Eli Zaretskii]

> Date: Mon, 8 Aug 2016 19:54:59 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Eli.
> 
> On Mon, Aug 08, 2016 at 10:04:45PM +0300, Eli Zaretskii wrote:
> > > Date: Mon, 8 Aug 2016 18:42:23 +0000
> > > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>
> 
> > > > So, if worse comes to worst, you could trigger such a complete scan
> > > > after revert-buffer, e.g. in a specialized revert-buffer-function.
> 
> > > Why would that be bad?
> 
> > Not bad, just potentially expensive for large buffers.  But if that's
> > "good enough", I don't really mind.
> 
> > However, the patch you show below doesn't implement this, does it?
> 
> No, but the following does.  It looks kind of simple, given the problems
> we've been through.  Maybe this one is the solution:

LGTM, thanks.  Feel free to install, and close this issue.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> Date: Tue, 9 Aug 2016 15:01:32 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> My guess is that originally these hooks were _intended_ to be balanced
> (as they still are, at least to a very great extent, on XEmacs).  At
> some stage (probably in the late 1990s/early 2000s) optimisations were
> made in insdel.c and elsewhere, and the strict pairing of b-c-f and
> a-c-f got lost in the wash.

No, the 'prepare' argument was added in 1994 (Emacs 19.2x), and
starting from that change, the hooks were never balanced.  You can
clearly see that in the Git history output.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 09, 2016 at 06:08:14PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 8 Aug 2016 19:54:59 +0000
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > > However, the patch you show below doesn't implement this, does it?

> > No, but the following does.  It looks kind of simple, given the problems
> > we've been through.  Maybe this one is the solution:

> LGTM, thanks.  Feel free to install, and close this issue.

DONE.

However there's still the inaccuracy in the Elisp documentation for
before-change-functions.  In particular there's the bit in "Change Hooks"
that says:

    This variable holds a list of functions to call before _ANY_ buffer
    modification.

, which is grossly misleading, at best.

Should I open another bug report for this, or are you just going to be
fixing it anyway (or giving me the info so that I can fix it)?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Tue, 9 Aug 2016 16:38:14 +0000
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > No, but the following does.  It looks kind of simple, given the problems
> > > we've been through.  Maybe this one is the solution:
> 
> > LGTM, thanks.  Feel free to install, and close this issue.
> 
> DONE.

Thanks.

> However there's still the inaccuracy in the Elisp documentation for
> before-change-functions.  In particular there's the bit in "Change Hooks"
> that says:
> 
>     This variable holds a list of functions to call before _ANY_ buffer
>     modification.
> 
> , which is grossly misleading, at best.
> 
> Should I open another bug report for this, or are you just going to be
> fixing it anyway (or giving me the info so that I can fix it)?

I will fix it.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>     This variable holds a list of functions to call before _ANY_ buffer
>     modification.

I think this description is correct.  If there's a modification where we
forget to run the hook, please report it as a bug.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Tue, 09 Aug 2016 19:42:23 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> > However there's still the inaccuracy in the Elisp documentation for
> > before-change-functions.  In particular there's the bit in "Change Hooks"
> > that says:
> > 
> >     This variable holds a list of functions to call before _ANY_ buffer
> >     modification.
> > 
> > , which is grossly misleading, at best.
> > 
> > Should I open another bug report for this, or are you just going to be
> > fixing it anyway (or giving me the info so that I can fix it)?
> 
> I will fix it.

Done (on the emacs-25 branch).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 09 Aug 2016 13:14:56 -0400
> 
> >     This variable holds a list of functions to call before _ANY_ buffer
> >     modification.
> 
> I think this description is correct.  If there's a modification where we
> forget to run the hook, please report it as a bug.

I think this is just an argument about the definition of
"modification".  When a modification is done piecemeal, not every
partial modification calls the before hook.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 09, 2016 at 09:13:43PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 09 Aug 2016 19:42:23 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: ofv@wanadoo.es, rcopley@gmail.com, rms@gnu.org, emacs-devel@gnu.org

> > > However there's still the inaccuracy in the Elisp documentation for
> > > before-change-functions.  In particular there's the bit in "Change Hooks"
> > > that says:

> > >     This variable holds a list of functions to call before _ANY_ buffer
> > >     modification.

> > > , which is grossly misleading, at best.

> > > Should I open another bug report for this, or are you just going to be
> > > fixing it anyway (or giving me the info so that I can fix it)?

> > I will fix it.

> Done (on the emacs-25 branch).

You've changed "any" to "most", which although technically not incorrect,
is lacking in usefulness.  In particular, it gives a user of
before-change-functions no way to predict which modifications will call
it and which ones won't.

I think "most" is a bit misleading - it suggests a proportion of,
perhaps, 60% - 90%, whereas I think the figure in real usage is well over
99%.  "Almost all", I think, would be a better description than "most",
but still inadequate.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> >     This variable holds a list of functions to call before _ANY_ buffer
>> >     modification.
>> I think this description is correct.  If there's a modification where we
>> forget to run the hook, please report it as a bug.
> I think this is just an argument about the definition of
> "modification".  When a modification is done piecemeal, not every
> partial modification calls the before hook.

It's OK if we call it once for the whole region and then modify various
chunks of the region one by one, yes.  In that case, it's still called
before those modification.

Basically, It think we should guarantee that if buffer position N is
changed, then N is between the BEG and END of the latest call to b-c-f
(with END potentially adjusted to account for the other modifications
that took place in the mean time).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 09 Aug 2016 15:09:02 -0400
> 
> It's OK if we call it once for the whole region and then modify various
> chunks of the region one by one, yes.  In that case, it's still called
> before those modification.
> 
> Basically, It think we should guarantee that if buffer position N is
> changed, then N is between the BEG and END of the latest call to b-c-f
> (with END potentially adjusted to account for the other modifications
> that took place in the mean time).

AFAICT, this is what happens, indeed, except that the call to
before-change-functions in some cases does not precede the first
modification of the series.  IOW, by the time the hook is called, some
modifications were already done.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> AFAICT, this is what happens, indeed,

Then the old text was right: it's called before ANY modification.

> except that the call to before-change-functions in some cases does not
> precede the first modification of the series.  IOW, by the time the
> hook is called, some modifications were already done.

Sounds like a bug.

In any case, my point is that the doc should still say "before any
modification" because that's really what the code *should* do.  We could
add a blurb in the doc saying that the before and after hooks may not be
properly paired (neither in number of calls nor in the specific value of
BEG/END), but we should still claim that they're both called for any and
all modifications (modulo inhibit-modification-hooks, obviously).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Stefan and Eli.

On Wed, Aug 10, 2016 at 10:56:59AM -0400, Stefan Monnier wrote:
> > AFAICT, this is what happens, indeed,

> Then the old text was right: it's called before ANY modification.

> > except that the call to before-change-functions in some cases does not
> > precede the first modification of the series.  IOW, by the time the
> > hook is called, some modifications were already done.

> Sounds like a bug.

I concur.

> In any case, my point is that the doc should still say "before any
> modification" because that's really what the code *should* do.  We could
> add a blurb in the doc saying that the before and after hooks may not be
> properly paired (neither in number of calls nor in the specific value of
> BEG/END),

You mean we should add commentary saying that BEG and END _usually_
bound the region about to be deleted, but sometimes they don't.  And
another comment saying that BEG, END, and OLD-LEN are only _usually_ set
to what they're currently documented as, the rest of the time they're
random numbers vaguely related to a change region.

That doesn't sound much better to me.

If we're going to treat the code (as opposed to the documentation) as a
bug, let's fix it completely, with rigorously paired b-c-f and a-c-f,
rather than half-heartedly.

Otherwise, let's document fully the circumstances in which
before-change-functions doesn't get called (which I don't as yet
understand), so that Lisp hackers can use that information in their
programs.

> but we should still claim that they're both called for any and
> all modifications (modulo inhibit-modification-hooks, obviously).

Given that the situation has been in place for several decades, I don't
see any urgency to fix it in 25.1.  But let's fix it properly, whatever
we decide the bug is.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> In any case, my point is that the doc should still say "before any
>> modification" because that's really what the code *should* do.  We could
>> add a blurb in the doc saying that the before and after hooks may not be
>> properly paired (neither in number of calls nor in the specific value of
>> BEG/END),
> You mean we should add commentary saying that BEG and END _usually_
> bound the region about to be deleted, but sometimes they don't.

No.  They should always bound the region.  But sometimes the before and
after bounds are different (typically, they're tighter in a-c-f because
when the b-c-f is run we don't know yet exactly which parts will be
modified, so we give pessimistic bounds).

E.g. I think it would be perfectly acceptable to have a revert-buffer
with b-c-f called once on point-min/point-max and then a-c-f called
a bunch of times on those few places where the buffer actually needed to
be modified.

>> but we should still claim that they're both called for any and
>> all modifications (modulo inhibit-modification-hooks, obviously).
> Given that the situation has been in place for several decades, I don't
> see any urgency to fix it in 25.1.

Agreed.  But the doc should still say what the "spec" should be, so it's
clear what is a bug.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 10 Aug 2016 10:56:59 -0400
> 
> > AFAICT, this is what happens, indeed,
> 
> Then the old text was right: it's called before ANY modification.

Convince Alan, and I will change it back.

> > except that the call to before-change-functions in some cases does not
> > precede the first modification of the series.  IOW, by the time the
> > hook is called, some modifications were already done.
> 
> Sounds like a bug.

I'm not convinced it's a bug.

> In any case, my point is that the doc should still say "before any
> modification" because that's really what the code *should* do.  We could
> add a blurb in the doc saying that the before and after hooks may not be
> properly paired (neither in number of calls nor in the specific value of
> BEG/END), but we should still claim that they're both called for any and
> all modifications

Which is inaccurate when modifications are done in several separate
parts, and you already agreed it's okay to call the hooks only once.
So "for any and all modifications" is bound to draw fire from people
who take that at face value.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> > AFAICT, this is what happens, indeed,
>> Then the old text was right: it's called before ANY modification.
> Convince Alan, and I will change it back.
>> > except that the call to before-change-functions in some cases does not
>> > precede the first modification of the series.  IOW, by the time the
>> > hook is called, some modifications were already done.
>> Sounds like a bug.
> I'm not convinced it's a bug.

before-change-functions should be called *before* a
change occurs.  If there is a call to b-c-f before the first
modification, and then another call before the second modification in
that series, then it's OK (assuming the BEG/END values are sufficiently
large in each call to cover the changes that occur before the next call).

But if a change occurs without having been "announced" by an earlier
b-c-f then it's a bug.

>> In any case, my point is that the doc should still say "before any
>> modification" because that's really what the code *should* do.  We could
>> add a blurb in the doc saying that the before and after hooks may not be
>> properly paired (neither in number of calls nor in the specific value of
>> BEG/END), but we should still claim that they're both called for any and
>> all modifications
> Which is inaccurate when modifications are done in several separate
> parts, and you already agreed it's okay to call the hooks only once.
> So "for any and all modifications" is bound to draw fire from people
> who take that at face value.

They're free to misunderstand it, I'm fine with it.  But saying it's
called for "most" implies that there are changes where b-c-f is simply
not called at all, which would break uses such as syntax-ppss's.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Wed, Aug 10, 2016 at 07:03:04PM +0300, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Wed, 10 Aug 2016 10:56:59 -0400

> > > AFAICT, this is what happens, indeed,
 
> > Then the old text was right: it's called before ANY modification.

> Convince Alan, and I will change it back.

For what it's worth, I don't like the vagueness of the doc as it
currently is, and I wouldn't be against changing it back _FOR 25.1_
We've got a bug, but it's not urgent enough to have to be fixed for
25.1.

> > > except that the call to before-change-functions in some cases does not
> > > precede the first modification of the series.  IOW, by the time the
> > > hook is called, some modifications were already done.

> > Sounds like a bug.

> I'm not convinced it's a bug.

> > In any case, my point is that the doc should still say "before any
> > modification" because that's really what the code *should* do.  We could
> > add a blurb in the doc saying that the before and after hooks may not be
> > properly paired (neither in number of calls nor in the specific value of
> > BEG/END), but we should still claim that they're both called for any and
> > all modifications

> Which is inaccurate when modifications are done in several separate
> parts, and you already agreed it's okay to call the hooks only once.
> So "for any and all modifications" is bound to draw fire from people
> who take that at face value.

Some while ago, I think you said that there is a systematic basis for
when before-change-functions doesn't get called.  Is that still your
view?  If so, let's document that system.  If not, then let's fix the
code so that it is systematic.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Wed, 10 Aug 2016 16:18:21 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Then the old text was right: it's called before ANY modification.
> 
> > Convince Alan, and I will change it back.
> 
> For what it's worth, I don't like the vagueness of the doc as it
> currently is, and I wouldn't be against changing it back _FOR 25.1_
> We've got a bug, but it's not urgent enough to have to be fixed for
> 25.1.

I made further adjustments to the text.  It is now closer to what it
was before, but I don't feel it's wise for it to be as definitive as
it was originally, as someone will surely interpret that too literally
and file another bug report, or design a feature based on incorrect
assumptions.

> Some while ago, I think you said that there is a systematic basis for
> when before-change-functions doesn't get called.  Is that still your
> view?

Yes.

> If so, let's document that system.

It's too complicated to describe that.  We would need to describe some
internal aspects of the implementation in great detail.  I don't see
that as worth our while.

Thanks.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Wed, Aug 10, 2016 at 07:54:36PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 10 Aug 2016 16:18:21 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Then the old text was right: it's called before ANY modification.

[ .... ]

> > Some while ago, I think you said that there is a systematic basis for
> > when before-change-functions doesn't get called.  Is that still your
> > view?

> Yes.

> > If so, let's document that system.

> It's too complicated to describe that.  We would need to describe some
> internal aspects of the implementation in great detail.  I don't see
> that as worth our while.

I can't see the difference between your last paragraph and saying it's
totally arbitrary.  If the only way to categorise it is to say "look at
the source code" that's arbitrary.

It can not be a good thing for the call to before-change-functions to be
missed out arbitrarily.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Wed, 10 Aug 2016 17:49:30 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > It's too complicated to describe that.  We would need to describe some
> > internal aspects of the implementation in great detail.  I don't see
> > that as worth our while.
> 
> I can't see the difference between your last paragraph and saying it's
> totally arbitrary.  If the only way to categorise it is to say "look at
> the source code" that's arbitrary.
> 
> It can not be a good thing for the call to before-change-functions to be
> missed out arbitrarily.

It's not arbitrary.  But describing when we don't call the
before-change hooks would first need to describe the various parts of
some buffer modifications, because otherwise you'd be unable to
understand which part does and which doesn't call them.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Wed, Aug 10, 2016 at 07:54:36PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 10 Aug 2016 16:18:21 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Then the old text was right: it's called before ANY modification.

> > > Convince Alan, and I will change it back.

> > For what it's worth, I don't like the vagueness of the doc as it
> > currently is, and I wouldn't be against changing it back _FOR 25.1_
> > We've got a bug, but it's not urgent enough to have to be fixed for
> > 25.1.

> I made further adjustments to the text.  It is now closer to what it
> was before, but I don't feel it's wise for it to be as definitive as
> it was originally, as someone will surely interpret that too literally
> and file another bug report, or design a feature based on incorrect
> assumptions.

#########################################################################

  Do @emph{not} expect the before-change hooks and the after-change
  hooks be called in balanced pairs around each buffer change.  Also
  don't expect the before-change hooks to be called for every chunk of
  text Emacs is about to delete.  These hooks are provided on the
  assumption that Lisp programs will use either before- or the
  after-change hooks, but not both, and the boundaries of the region
  where the changes happen might include more than just the actual
  changed text, or even lump together several changes done piecemeal.

#########################################################################

I don't like that text at all.  For what it's worth, I find it
objectionable in the extreme.  It's not even true.  One can (indeed,
often MUST) expect b-c-f and a-c-f to work in pairs, and it is only on
the rarest of occasions that they do not - it's taken ~10 years for this
bug to show up.

Editing text involves deletions and insertions.  Deletions can be known
only in before-change-functions, insertions only in
after-change-functions.  What you are proposing is partitioning Elisp
programs into those that know only deletions, and those that know only
insertions.  Any program using syntax-ppss would by that principle be
debarred from using a-c-f, since s-ppss uses b-c-f.  This is absurd.

If you insist on this strand of documentation (which is surely
controversial enough not to be allowed into Emacs 25.1 at this late
stage) then please provide a function called something like `old-text'
which an after-change function could use to recover deleted text.

CC Mode absolutely requires to know both deletions and insertions for
its proper functioning.  Nobody has yet demonstrated that it can work
properly knowing only insertions, despite Stefan loudly and repeatedly
proclaiming it can.

Must I look forward to a bug report from some busybody moaning that CC
Mode "illegitimately" uses both change hooks?

OK, I've complained enough, so I ought to suggest something positive.
Here is what I would put into the documentation in place of that
paragraph.  I think it is accurate and reasonable.

#########################################################################

  Normally the before-change-functions and the after-change-functions
  hooks are called in balanced pairs around each buffer change, enabling
  state to be determined in a before-change function and used in an
  after-change function.  However on rare occasions,
  before-change-functions is not called before a buffer change and
  after-change-functions is thus called twice in succession.  Your
  program should be ready to deal with such an occurrence.

#########################################################################

[ .... ]

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Wed, 10 Aug 2016 18:57:35 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> #########################################################################
> 
>   Do @emph{not} expect the before-change hooks and the after-change
>   hooks be called in balanced pairs around each buffer change.  Also
>   don't expect the before-change hooks to be called for every chunk of
>   text Emacs is about to delete.  These hooks are provided on the
>   assumption that Lisp programs will use either before- or the
>   after-change hooks, but not both, and the boundaries of the region
>   where the changes happen might include more than just the actual
>   changed text, or even lump together several changes done piecemeal.
> 
> #########################################################################
> 
> I don't like that text at all.  For what it's worth, I find it
> objectionable in the extreme.  It's not even true.

OK, I'm now officially done with discussing this with you.  I find
your ingratitude and downright rudeness extremely out of line.  I
regret I ever started talking about this.

This text stays.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>   Do @emph{not} expect the before-change hooks and the after-change
>   hooks be called in balanced pairs around each buffer change.  Also
>   don't expect the before-change hooks to be called for every chunk of
>   text Emacs is about to delete.  These hooks are provided on the
>   assumption that Lisp programs will use either before- or the
>   after-change hooks, but not both, and the boundaries of the region
>   where the changes happen might include more than just the actual
>   changed text, or even lump together several changes done piecemeal.

That looks OK, thank you.  It does make it sound like sometimes b-c-f
won't be called at all, whereas AFAIU it is *always* called before
a change, tho maybe "too early" (i.e. announcing several changes at the
same time), but I think it's good enough for now.

> One can (indeed, often MUST) expect b-c-f and a-c-f to work in pairs,

No.  I know you *want* to expect it, but that assumption has not been
true for many years and we don't want to impose it on ourselves.
Furthermore there is no evidence that you "MUST".
I gave various examples of ways you can live with only a-c-f.

> Editing text involves deletions and insertions.

It also involves replacements.

> Deletions can be known only in before-change-functions, insertions
> only in after-change-functions.

Not at all.  Deletions are definitely announced to a-c-f.

> CC Mode absolutely requires to know both deletions and insertions for
> its proper functioning.  Nobody has yet demonstrated that it can work
> properly knowing only insertions, despite Stefan loudly and repeatedly
> proclaiming it can.

All major modes using syntax-propertize demonstrate it.  C++ has its
share of syntactic quirks, but none that are fundamentally very
different from what we find in other languages such as Perl.

I also outlined in more detailed how to deal with a specific example you
seemed to think required absolutely to know what was there before
the deletion.

>   enabling state to be determined in a before-change function and used
>   in an after-change function.

That's exactly what I want to strongly discourage.  Based on my
experience with CC-mode, I'd even say that it's just a bad design, even
if the hooks were always called in pairs.  I don't know nor care about
how we ended up with such a design, and I understand that you might
prefer to try and live with it than to go back and rethink it, but it's
still a bad design (nor is it the only bad design we have in Emacs, of
course).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Stefan.

On Wed, Aug 10, 2016 at 03:50:06PM -0400, Stefan Monnier wrote:
> >   Do @emph{not} expect the before-change hooks and the after-change
> >   hooks be called in balanced pairs around each buffer change.  Also
> >   don't expect the before-change hooks to be called for every chunk of
> >   text Emacs is about to delete.  These hooks are provided on the
> >   assumption that Lisp programs will use either before- or the
> >   after-change hooks, but not both, and the boundaries of the region
> >   where the changes happen might include more than just the actual
> >   changed text, or even lump together several changes done piecemeal.

> That looks OK, thank you.  It does make it sound like sometimes b-c-f
> won't be called at all,

That is the current status, yes.  In the bug which gave rise to this
discussion, bug #24094, a certain change did not call b-c-f at all.

> whereas AFAIU it is *always* called before a change, tho maybe "too
> early" (i.e. announcing several changes at the same time), but I think
> it's good enough for now.

> > One can (indeed, often MUST) expect b-c-f and a-c-f to work in pairs,

> No.  I know you *want* to expect it, but that assumption has not been
> true for many years and we don't want to impose it on ourselves.

The Elisp manual, up until yesterday, documented implicitly that b-c-f
and a-c-f were paired.  It has also been true, up to 99.99..%, otherwise
CC Mode wouldn't work at all.

> Furthermore there is no evidence that you "MUST".
> I gave various examples of ways you can live with only a-c-f.

Vague hand-waving is what I remember.

> > Editing text involves deletions and insertions.

> It also involves replacements.

> > Deletions can be known only in before-change-functions, insertions
> > only in after-change-functions.

> Not at all.  Deletions are definitely announced to a-c-f.

You know full well what I mean.  By the time you get to a-c-f the
deletion as announced is a hollowed out husk, all its details except for
its position and size having been lost.

What you and Eli are asserting is that one NEVER IN ANY CIRCUMSTANCES
needs to know the full details of a buffer modification, and no program
now or in the future ever will.  The burden of proof has not been met.
With the recent change in documentation, Emacs 25.1 has become a less
powerful programming system, one in which these details cannot be
determined.

> > CC Mode absolutely requires to know both deletions and insertions for
> > its proper functioning.  Nobody has yet demonstrated that it can work
> > properly knowing only insertions, despite Stefan loudly and repeatedly
> > proclaiming it can.

> All major modes using syntax-propertize demonstrate it.  C++ has its
> share of syntactic quirks, but none that are fundamentally very
> different from what we find in other languages such as Perl.

If you want to try and construct a CC Mode which disregards deletions,
you are welcome.  I won't stand in your way.  But I will try and break
it.

[ .... ]

> >   enabling state to be determined in a before-change function and used
> >   in an after-change function.

> That's exactly what I want to strongly discourage.  Based on my
> experience with CC-mode, I'd even say that it's just a bad design, even
> if the hooks were always called in pairs.  I don't know nor care about
> how we ended up with such a design, ....

You really ought to: those who are ignorant of history are doomed to
repeat it.

> .... and I understand that you might prefer to try and live with it
> than to go back and rethink it, but it's still a bad design (nor is it
> the only bad design we have in Emacs, of course).

You've been libelling my design for longer than I can remember.  It's
different from how you'd try to do it, that's all.  It's a good design,
and it works well.  It's a straightforward and obvious distillation of
requirements and Emacs's documented features.  You recently admitted
you'd never tried to build anything that way, so you're just saying "my
way is better".

As I say, if you really think your way is better, hack it, and we can
compare.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> That is the current status, yes.  In the bug which gave rise to this
> discussion, bug #24094, a certain change did not call b-c-f at all.

Right, but that's a bug, not a misfeature we want to document.

> The Elisp manual, up until yesterday, documented implicitly that b-c-f
> and a-c-f were paired.

No: you read it as documenting this implicitly, but that's a misunderstanding.

>> Furthermore there is no evidence that you "MUST".
>> I gave various examples of ways you can live with only a-c-f.
> Vague hand-waving is what I remember.

That's because you don't want to think about it seriously: Of course
it's not in the form of a patch to CC-mode that passes all your tests
plus various performance assessments, no.  But those "vague hand-waving"
refer to techniques used in actual code in various major modes facing
similar problems.

> You know full well what I mean.  By the time you get to a-c-f the
> deletion as announced is a hollowed out husk, all its details except for
> its position and size having been lost.

As mentioned in my so-called "hand-waving", any details you may need can
be stored somewhere for use.

But there's no reason your code could *require* this data for correct
behavior, since your code also has to handle the more general case where
all the text after BEG has been replaced by something else which may or
may not include similar contents and may or may not already include some
text-properties like `syntax-table'.

> What you and Eli are asserting is that one NEVER IN ANY CIRCUMSTANCES
> needs to know the full details of a buffer modification, and no program
> now or in the future ever will.

Exactly.  And that's because the code has to handle the previously
mentioned general case anyway.

> With the recent change in documentation, Emacs 25.1 has become a less
> powerful programming system, one in which these details cannot
> be determined.

Nope.  The documentation is just more honest than before.

> If you want to try and construct a CC Mode which disregards deletions,
> you are welcome.  I won't stand in your way.  But I will try and break it.

I expect no less.

>> .... and I understand that you might prefer to try and live with it
>> than to go back and rethink it, but it's still a bad design (nor is it
>> the only bad design we have in Emacs, of course).
> You've been libelling my design for longer than I can remember.
> It's different from how you'd try to do it, that's all.

No, that's not all.  It's based on an invalid assumption about b-c-f and
a-c-f, and even if that assumption was true I contend that it leads to
more complex code.

It probably has some performance advantages in some cases, of course.
By I doubt it has a performance advantage "on average".

> It's a straightforward and obvious distillation of requirements and
> Emacs's documented features.

No idea what you mean by that.

> You recently admitted you'd never tried to build anything that way, so
> you're just saying "my way is better".

I also tried to replace in CC-mode "your way" with mine and found mine
to be simpler.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 10 Aug 2016 12:11:21 -0400
> 
> >> > except that the call to before-change-functions in some cases does not
> >> > precede the first modification of the series.  IOW, by the time the
> >> > hook is called, some modifications were already done.
> >> Sounds like a bug.
> > I'm not convinced it's a bug.
> 
> before-change-functions should be called *before* a
> change occurs.  If there is a call to b-c-f before the first
> modification, and then another call before the second modification in
> that series, then it's OK (assuming the BEG/END values are sufficiently
> large in each call to cover the changes that occur before the next call).
> 
> But if a change occurs without having been "announced" by an earlier
> b-c-f then it's a bug.

I've reviewed all the places where we may not call the before- and the
after-change hooks.  They are just a few.

One questionable place is insert-file-contents: it deliberately never
calls the before-change-functions when it deletes portions of the
text, only when it inserts text.  I don't really understand the
reasons behind your insisting on before-change-functions being called
before any changes in this case, but maybe a simple way out would be
to announce at the beginning of the function that the entire range
between point-min and point-max is about to be changed, when REPLACE
is non-nil.

Another place where before-change-functions is not called, but
after-change-functions are is set-buffer-multibyte.  Moreover, if the
function is called with a nil argument, we don't call
after-change-functions as well.  Is that supposed to be handled as a
buffer change or not?

In the low levels of encoding and decoding routines, we sometimes call
the hooks and sometimes don't, depending on what is the source and the
destination of the en/decoding, and whether there are pre-write or
pre-write conversions.  But the buffer in question is a temporary
conversion buffer, so do we care?

In message_dolog, which is called by any code which logs a message in
*Messages*, we never call the before-change-functions for the
*Messages* buffer, but we do call the after-change-functions for it.
Problem or not?

That's all I found.

> >> In any case, my point is that the doc should still say "before any
> >> modification" because that's really what the code *should* do.  We could
> >> add a blurb in the doc saying that the before and after hooks may not be
> >> properly paired (neither in number of calls nor in the specific value of
> >> BEG/END), but we should still claim that they're both called for any and
> >> all modifications
> > Which is inaccurate when modifications are done in several separate
> > parts, and you already agreed it's okay to call the hooks only once.
> > So "for any and all modifications" is bound to draw fire from people
> > who take that at face value.
> 
> They're free to misunderstand it, I'm fine with it.

Then what is the value of such a misleading documentation?

The current text says:

  This variable holds a list of functions to call when Emacs is about
  to modify a buffer.

It intentionally refrains from saying "any and all modifications", and
instead tries to be more vague.  Do you still think this is not good
enough?  Because if you do, I don't see how we can say anything more
accurate and yet avoid misleading the reader.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> One questionable place is insert-file-contents: it deliberately never
> calls the before-change-functions when it deletes portions of the
> text, only when it inserts text.

So, if insert-file-contents only deletes a portion of text, it doesn't
call before-change-functions at all?  That'd be a bug, yes.

> I don't really understand the reasons behind your insisting on
> before-change-functions being called before any changes in this case,

For example, because syntax-ppss uses before-change-functions to notice
when the cache needs to be flushed, and that is necessary for any
change, including deletions, and whether the deletion is performed by 
insert-file-contents or something else is mostly irrelevant.

> but maybe a simple way out would be to announce at the beginning of
> the function that the entire range between point-min and point-max is
> about to be changed, when REPLACE is non-nil.

That'd be pessimistic, but yes, it's an easy way to fix the bug, and
it'd probably be good enough.

> Another place where before-change-functions is not called, but
> after-change-functions are is set-buffer-multibyte.  Moreover, if the
> function is called with a nil argument, we don't call
> after-change-functions as well.  Is that supposed to be handled as a
> buffer change or not?

Good question.  I guess it would make sense to call b-c-f on the
whole buffer.  This said, I think we should strongly discourage use of
set-buffer-multibyte in non-empty buffers, so it's not terribly
important: in the "normal" case set-buffer-multibyte should not perform
any buffer modification anyway.

> In the low levels of encoding and decoding routines, we sometimes call
> the hooks and sometimes don't, depending on what is the source and the
> destination of the en/decoding, and whether there are pre-write or
> pre-write conversions.  But the buffer in question is a temporary
> conversion buffer, so do we care?

The temp-conversion-buffer is an internal implementation detail.
I'd rather we never call a/b-c-f in those, but it's not terribly important.

> In message_dolog, which is called by any code which logs a message in
> *Messages*, we never call the before-change-functions for the
> *Messages* buffer, but we do call the after-change-functions for it.
> Problem or not?

The *Messages* buffer is sufficiently special that I'm not worried.

> That's all I found.

Great, thank you very much.

> It intentionally refrains from saying "any and all modifications", and
> instead tries to be more vague.  Do you still think this is not good
> enough?

I think it's good enough.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Thu, Aug 18, 2016 at 05:26:03PM +0300, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 10 Aug 2016 12:11:21 -0400

> > >> > except that the call to before-change-functions in some cases does not
> > >> > precede the first modification of the series.  IOW, by the time the
> > >> > hook is called, some modifications were already done.
> > >> Sounds like a bug.
> > > I'm not convinced it's a bug.

> > before-change-functions should be called *before* a
> > change occurs.  If there is a call to b-c-f before the first
> > modification, and then another call before the second modification in
> > that series, then it's OK (assuming the BEG/END values are sufficiently
> > large in each call to cover the changes that occur before the next call).

> > But if a change occurs without having been "announced" by an earlier
> > b-c-f then it's a bug.

> I've reviewed all the places where we may not call the before- and the
> after-change hooks.  They are just a few.

Thanks!

> One questionable place is insert-file-contents: it deliberately never
> calls the before-change-functions when it deletes portions of the
> text, only when it inserts text.  I don't really understand the
> reasons behind your insisting on before-change-functions being called
> before any changes in this case, but maybe a simple way out would be
> to announce at the beginning of the function that the entire range
> between point-min and point-max is about to be changed, when REPLACE
> is non-nil.

That might create difficulties for CC Mode.  CC Mode now recognises when
b-c-f isn't called at all, but isn't, as far as I've thought it through,
prepared to deal with b-c-f with inaccurate arguments.

> Another place where before-change-functions is not called, but
> after-change-functions are is set-buffer-multibyte.  Moreover, if the
> function is called with a nil argument, we don't call
> after-change-functions as well.  Is that supposed to be handled as a
> buffer change or not?

Possibly not - only the lowest levels of code would notice any
difference.

> In the low levels of encoding and decoding routines, we sometimes call
> the hooks and sometimes don't, depending on what is the source and the
> destination of the en/decoding, and whether there are pre-write or
> pre-write conversions.  But the buffer in question is a temporary
> conversion buffer, so do we care?

> In message_dolog, which is called by any code which logs a message in
> *Messages*, we never call the before-change-functions for the
> *Messages* buffer, but we do call the after-change-functions for it.
> Problem or not?

"Change Hooks" in the Elisp manual states that b-c-f and a-c-f are not
called for the *Messages* buffer.  So I would say calling a-c-f here is a
bug.

> That's all I found.

> > >> In any case, my point is that the doc should still say "before any
> > >> modification" because that's really what the code *should* do.  We could
> > >> add a blurb in the doc saying that the before and after hooks may not be
> > >> properly paired (neither in number of calls nor in the specific value of
> > >> BEG/END), but we should still claim that they're both called for any and
> > >> all modifications
> > > Which is inaccurate when modifications are done in several separate
> > > parts, and you already agreed it's okay to call the hooks only once.
> > > So "for any and all modifications" is bound to draw fire from people
> > > who take that at face value.

> > They're free to misunderstand it, I'm fine with it.

> Then what is the value of such a misleading documentation?

> The current text says:

>   This variable holds a list of functions to call when Emacs is about
>   to modify a buffer.

Somebody wanting to use b-c-f and reading that text is likely to conclude
that b-c-f is called for _every_ change, just as I did ~10 years ago.

> It intentionally refrains from saying "any and all modifications", and
> instead tries to be more vague.  Do you still think this is not good
> enough?  Because if you do, I don't see how we can say anything more
> accurate and yet avoid misleading the reader.

How can accurate documentation mislead?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Fri, 19 Aug 2016 08:45:26 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > One questionable place is insert-file-contents: it deliberately never
> > calls the before-change-functions when it deletes portions of the
> > text, only when it inserts text.  I don't really understand the
> > reasons behind your insisting on before-change-functions being called
> > before any changes in this case, but maybe a simple way out would be
> > to announce at the beginning of the function that the entire range
> > between point-min and point-max is about to be changed, when REPLACE
> > is non-nil.
> 
> That might create difficulties for CC Mode.  CC Mode now recognises when
> b-c-f isn't called at all, but isn't, as far as I've thought it through,
> prepared to deal with b-c-f with inaccurate arguments.

The arguments are not inaccurate: what insert-file-contents does in
that case is really replace the text between point-min and point-max
with what's in the file on disk.  That it includes internal
optimizations which actually try to make minimal changes is just that
-- an optimization.  Modes that hook the changes cannot depend on
internal optimizations like that.

More generally, since CC mode already knows how to recalculate
everything when a buffer is reverted, this should no longer be an
issue.

> > In message_dolog, which is called by any code which logs a message in
> > *Messages*, we never call the before-change-functions for the
> > *Messages* buffer, but we do call the after-change-functions for it.
> > Problem or not?
> 
> "Change Hooks" in the Elisp manual states that b-c-f and a-c-f are not
> called for the *Messages* buffer.  So I would say calling a-c-f here is a
> bug.

You are reading the manual too literally.

> >   This variable holds a list of functions to call when Emacs is about
> >   to modify a buffer.
> 
> Somebody wanting to use b-c-f and reading that text is likely to conclude
> that b-c-f is called for _every_ change, just as I did ~10 years ago.

Then they will be wrong, because the text doesn't say so.

> > It intentionally refrains from saying "any and all modifications", and
> > instead tries to be more vague.  Do you still think this is not good
> > enough?  Because if you do, I don't see how we can say anything more
> > accurate and yet avoid misleading the reader.
> 
> How can accurate documentation mislead?

By using terminology that is not rigorously defined.  Like "buffer
modification", for example.  You yourself say that
set-buffer-multibyte shouldn't call the modification hooks, but that
function surely does perform "buffer modifications" in some sense of
that term.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> This variable holds a list of functions to call when Emacs is about
>> to modify a buffer.
> Somebody wanting to use b-c-f and reading that text is likely to conclude
> that b-c-f is called for _every_ change, just as I did ~10 years ago.

That's a correct reading, and the code has been obeying this rule
"forever", except for a few bugs maybe, the only remaining one
apparently being the one in insert-file-contents.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> Somebody wanting to use b-c-f and reading that text is likely to conclude
>> that b-c-f is called for _every_ change, just as I did ~10 years ago.
> Then they will be wrong, because the text doesn't say so.

I think we're talking about different notions of "every change".
More specifically, I think I don't understand what Alan thinks "every
change" means, and I suspect that he doesn't understand what I mean by
that either.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> Date: Fri, 19 Aug 2016 05:21:04 -0400
> 
> >> Somebody wanting to use b-c-f and reading that text is likely to conclude
> >> that b-c-f is called for _every_ change, just as I did ~10 years ago.
> > Then they will be wrong, because the text doesn't say so.
> 
> I think we're talking about different notions of "every change".

Exactly.

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
> My interpretation of these two variables is that they are two hooks
> provided for different kinds of features, because hooking into both
> should not generally be required.  It should suffice for an
> application that wants to react to a change in a buffer to hook into
> after-change-functions only.


My apologies for jumping into the discussion so late, but I have been
travelling.

I make use of both b-c-f and a-c-f in my package called lentic. This
percolates changes in one buffer into another which is related but may
not be indentical to the first.

To enable this, I have to be able to convert between cognate positions
in the two buffers; that is the positions which are equivalent in the
two buffers. But I have to do this BEFORE the changes occur, because at
this point the two buffers are in a consistent state. This is not true
after the change has happened in one buffer and before it has been
percolated to the next.

Having b-c-f and a-c-f paired is not essential for this, but is (or
would be) greately helpful. It would also be good if the values passed
to b-c-f and a-c-f were consistent. Unfortunately this is not true
either (subst-char-in-region at least is guilty here, passing different
"end" positions in b-c-f and a-c-f).

I use the following logic to detect when b-c-f and a-c-f are "skewed" --
that is inconsistent with each other. "start" and "stop" are the
parameters of a-c-f and "length-before" is the value of the length
parameter of b-c-f called immediately before a-c-f.

 (or
                ;; previously this was not skewed if no region, but actually,
                ;; if there is no region we need to copy everything, we can
                ;; also do by declaring skew -- this is important for the
                ;; multi-lentic situation
                (not (or start stop length-before))
                ;; skews only occur in insertions which result in a positive
                ;; length-before. This also picks up no-insertion changes
                (and (< 0 length-before)
                     ;; = start stop means we have a deletion because
                     ;; there is no range after. Deletions seem to be
                     ;; safe.
                     (not (= start stop))))

This seems to work for me, but it's guess work. I do not understand
quite why it works, and there are (undoubtly) some cases where it does
not.

I realise that there may be practical difficulties in making the two
hooks consistent, although I think that this would be a laudable aim.
But failing this, it would be at least nice to know when the hooks are
not consistent; either in documentation, or better as a parameter, so I
could stop guessing.

The implications for lentic are significant; even though I can generally
avoid issues with the code above, the assumption that the whole buffer
is dirty generates a large amount of undo information. Fortuantely, it
happens only rarely, or this would simply prevent my approach from
working.

Which leads me to two conclusions:

1) being able to know when b-c-f and a-c-f are not paired or consistent
would be useful

2) decreasing the number of times these occurs would be useful, even if
it cannot be removed entirely.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/11/2016 09:43 AM, Stefan Monnier wrote:
>> That is the current status, yes.  In the bug which gave rise to this
>> discussion, bug #24094, a certain change did not call b-c-f at all.
>
> Right, but that's a bug, not a misfeature we want to document.
>
>> The Elisp manual, up until yesterday, documented implicitly that b-c-f
>> and a-c-f were paired.
>
> No: you read it as documenting this implicitly, but that's a misunderstanding.

Alan's interpretation is the text is the natural one, and it's the one I 
had as well. It's a technical specification, not some kind of allegory. 
It's meant to be taken literally. There is nothing in the old manual to 
hint at the more complex reality of the situation, and the additional 
text just confuses the issue.

Besides, you haven't fixed the docstring, which still suggests that 
b-c-f is called before every change. (And a "change" must be anything 
that results in a-c-f being called.)

Alan is absolutely correct that current Emacs behavior should count as a 
bug. The behavior Alan wants, which is also the behavior XEmacs 
provides, and the behavior the manual suggestions on plain reading, is 
also the most elegant and useful behavior for these hooks.

Eli's original argument for not balancing b-c-f with a-c-f is the 
delicacy of other Emacs code, but this specific change is almost 
certainly safe, considering that b-c-f is almost always balanced with 
a-c-f already. Alan's given plenty examples of legitimate pieces of code 
that can break if b-c-f is not called for a given a-c-f.

What pieces of code could conceivably break if Emacs balances 
currently-unbalanced a-c-f calls?

>
>>> Furthermore there is no evidence that you "MUST".
>>> I gave various examples of ways you can live with only a-c-f.
>> Vague hand-waving is what I remember.
>
> That's because you don't want to think about it seriously: Of course
> it's not in the form of a patch to CC-mode that passes all your tests
> plus various performance assessments, no.  But those "vague hand-waving"
> refer to techniques used in actual code in various major modes facing
> similar problems.
>
>> You know full well what I mean.  By the time you get to a-c-f the
>> deletion as announced is a hollowed out husk, all its details except for
>> its position and size having been lost.
>
> As mentioned in my so-called "hand-waving", any details you may need can
> be stored somewhere for use.

Is it really better to force that complexity (which, in the limit, 
amounts to full shadow buffers) onto mode authors instead of changing 
Emacs to balance b-c-f with a-c-f 100% of the time instead of 99%?

> But there's no reason your code could *require* this data for correct
> behavior, since your code also has to handle the more general case where
> all the text after BEG has been replaced by something else which may or
> may not include similar contents and may or may not already include some
> text-properties like `syntax-table'.
>
>> What you and Eli are asserting is that one NEVER IN ANY CIRCUMSTANCES
>> needs to know the full details of a buffer modification, and no program
>> now or in the future ever will.
>
> Exactly.  And that's because the code has to handle the previously
> mentioned general case anyway.
>
>> With the recent change in documentation, Emacs 25.1 has become a less
>> powerful programming system, one in which these details cannot
>> be determined.
>
> Nope.  The documentation is just more honest than before.

It's also self-contradictory: upon calling syntax-ppss, Emacs installs a 
before-change-function, which in a mode that perfectly legitimately has 
an after-change-function, violates the rule that we use one set of 
functions or another.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> Besides, you haven't fixed the docstring, which still suggests that b-c-f is
> called before every change. (And a "change" must be anything that results in
> a-c-f being called.)

It doesn't just suggest it, it says so, and rightly so.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/28/2016 08:01 AM, Stefan Monnier wrote:
>> Besides, you haven't fixed the docstring, which still suggests that b-c-f is
>> called before every change. (And a "change" must be anything that results in
>> a-c-f being called.)
>
> It doesn't just suggest it, it says so, and rightly so.

So will insdel.c or the docstring change?

FWIW, there *is* a certain elegance to the idea of the "forget 
everything after the first bit that changed" model of change tracking, 
but as long as b-c-f exists, it might as well work consistently.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> No: you read it as documenting this implicitly, but that's
>> a misunderstanding.
> Alan's interpretation is the text is the natural one, and it's the one I had
> as well.

If you say so.  I must say I don't know/understand what is his
interpretation, so if you can give details it would help.  E.g. give us
a concrete scenario that currently happens in Emacs and which you think
is incompatible with the description.

[ Tho, please note that we already know about 1 such scenario (the lack
  of b-c-f from insert-file-contents with `replace' non-nil and where
  the new content has no new text but has simply some part of the file
  deleted) which we all agree is a bug.  ]

> What pieces of code could conceivably break if Emacs balances
> currently-unbalanced a-c-f calls?

That's the wrong question.  A more interesting question could be: how
can we make those hooks always perfectly balanced even in the most twisted
cases (e.g. when a b/a-c-f call itself makes a change to the buffer),
and without undue performance penalty.  Another one would be: who would
write such a patch and then make sure that this patch doesn't introduce
other bugs?  On the sideline: the only known case which would benefit,
really, is that of Phillip Lord (because, as best I can see,
Alan's case would benefit much more from getting rid of this assumption
altogether (although Alan himself wouldn't benefit as much because he'd
first have to learn to appreciate this other way to deal with the
problem)).

> Is it really better to force that complexity (which, in the limit, amounts
> to full shadow buffers)

What complexity?  Compare the complexity of CC-mode with that of
cperl-mode or perl-mode.  I can guarantee you that it's actually
*simpler* to give up on the idea of perfectly matched b/a-c-f calls for
purposes of syntax highlighting and parsing.

> legitimately has an after-change-function, violates the rule that we use one
> set of functions or another.

There's no such rule.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>>> Besides, you haven't fixed the docstring, which still suggests that b-c-f is
>>> called before every change. (And a "change" must be anything that results in
>>> a-c-f being called.)
>> It doesn't just suggest it, it says so, and rightly so.
> So will insdel.c or the docstring change?

I don't know what you're referring to or how that relates to the quoted text.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/28/2016 03:44 PM, Stefan Monnier wrote:
>>>> Besides, you haven't fixed the docstring, which still suggests that b-c-f is
>>>> called before every change. (And a "change" must be anything that results in
>>>> a-c-f being called.)
>>> It doesn't just suggest it, it says so, and rightly so.
>> So will insdel.c or the docstring change?
>
> I don't know what you're referring to or how that relates to the quoted text.

Sorry --- let me try adding more context. According to the very long 
thread above, I think these are the things everyone agrees to be true:

1) b-c-f isn't currently balanced with a-c-f in the insdel.c implementation

2) b-c-f isn't currently guaranteed to be called at all before any given 
call to a-c-f (e.g., the C-x C-f case)

2) the documentation suggests that a-c-f and b-c-f are balanced

3) many people, Alan and me included, were under the impression that 
b-c-f and a-c-f were balanced and wrote code accordingly

AIUI, Eli's position is that the current behavior should not change at 
all, because making changes at this level is too risky. The 
documentation should change to reflect #1 and #2. Your position is that 
#2 is a bug, but #1 is not, and the documentation needs to change. My 
position, and Alan's, is that #1 and #2 are both bugs and the current 
documentation is fine.

The status quo is that the documentation misleads developers about what 
Emacs actually does. I don't think it's fair to suggest that both Alan 
and I both just misunderstood the documentation.

So what do you propose doing? Upthread, I explained why I think that 
fixing both #1 and #2 is pretty low-risk.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> 1) b-c-f isn't currently balanced with a-c-f in the insdel.c implementation

There's no guarantee that it be balanced, indeed.

> 2) b-c-f isn't currently guaranteed to be called at all before any given
> call to a-c-f (e.g., the C-x C-f case)

It is guaranteed, modulo bugs (the insert-file-contents (with replace) case).

> 2) the documentation suggests that a-c-f and b-c-f are balanced

I don't see where in the documentation it suggests it.

> 3) many people, Alan and me included, were under the impression that b-c-f
> and a-c-f were balanced and wrote code accordingly

Ah, so you too have code which makes this assumption.  Could you point
to it, explaining what it does and why it makes this assumption, so we
have a better picture of why it could be useful?

> AIUI, Eli's position is that the current behavior should not change at all,
> because making changes at this level is too risky.

I don't think he said "at all", but just that it's tricky to make such
changes, so they had better be backed by really good reasons.

> Your position is that #2 is a bug, but #1 is not, and the
> documentation needs to change.

Almost: I don't see the need to change the doc because I don't see where
it misleads.

> So what do you propose doing? Upthread, I explained why I think that fixing
> both #1 and #2 is pretty low-risk.

The problem is that changing the code w.r.t b/a-c-f might cause
*other* changes.  Clearly making them perfectly balanced and paired
should not *in itself* introduce any bugs: the bugs we're afraid of are
the accompanying unintended changes.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/28/2016 05:09 PM, Stefan Monnier wrote:
>> 1) b-c-f isn't currently balanced with a-c-f in the insdel.c implementation
>
> There's no guarantee that it be balanced, indeed.
>
>> 2) b-c-f isn't currently guaranteed to be called at all before any given
>> call to a-c-f (e.g., the C-x C-f case)
>
> It is guaranteed, modulo bugs (the insert-file-contents (with replace) case).

Sure. The current implementation provides neither property though.

>> 2) the documentation suggests that a-c-f and b-c-f are balanced
>
> I don't see where in the documentation it suggests it.

See below.

>> 3) many people, Alan and me included, were under the impression that b-c-f
>> and a-c-f were balanced and wrote code accordingly
>
> Ah, so you too have code which makes this assumption.  Could you point
> to it, explaining what it does and why it makes this assumption, so we
> have a better picture of why it could be useful?

Same as Alan's --- examining the buffer before the change to see which 
brace-delimited scopes need to be invalidated. It's not possible to do 
that in a-c-f because the text that was changed is unavailable in a-c-f. 
Emacs core also contains a lot of code that looks fragile under 
asymmetric b-c-f behavior: see tex-mode, org, and viper. (I'm not 
claiming that these modes are broken, but it's not obvious from the code 
that they aren't broken.)

Yes, it's always possible, with enough shadowing of buffer contents, to 
rely on a-c-f alone. I think. But since Emacs already *has* the b-c-f 
feature, since it's pretty widely used, and since it *almost* works as 
expected today, it's be best to just change the behavior instead of 
sending a "gotcha! The manual is correct only under close reading!" 
message to everyone who uses b-c-f.

>> AIUI, Eli's position is that the current behavior should not change at all,
>> because making changes at this level is too risky.
>
> I don't think he said "at all", but just that it's tricky to make such
> changes, so they had better be backed by really good reasons.
>
>> Your position is that #2 is a bug, but #1 is not, and the
>> documentation needs to change.
> Please trust me that the documentation misleads.
> Almost: I don't see the need to change the doc because I don't see where
> it misleads.

b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented 
as "List of functions to call before each text change." a-c-f is 
documented as "List of functions to call after each text change." The 
elisp manual documentation is similarly symmetric. This symmetry 
produces an expectation that the behavior is symmetric, and this 
expectation is reinforced by how the observed behavior is almost always 
symmetric in practice. Symmetric behavior here is also what intuitively 
makes sense.

Please believe me when I say that the current documentation misleads.

I mean, imagine you were writing a home automation system and saw 
before-doorbell-functions and after-doorbell-functions hooks, read the 
documentation, and saw that the former is a list of functions to be 
called "before the doorbell chime rings" and the latter is a list of 
functions to be called "after the doorbell chime rings".

Imagine that when you tested these hooks, you saw that the b-d-f were 
indeed run before you heard the chime and a-d-f functions after. What on 
earth would make you suspect that this behavior wouldn't be the case in 
some circumstances? Wouldn't you treat a failure to run b-d-f if the 
doorbell butter were pressed forcefully as a bug? Wouldn't you think it 
strange if pressing the doorbell button quickly five times in a row 
after 9pm on Tuesdays produced five chimes, five calls to a-d-f, but 
only one call to b-d-f?

Sure, I guess you could argue that current Emacs behavior is consistent 
with the manual, but it's not what anyone would reasonably expect, and 
the current behavior is surprising even to people who have been writing 
elisp for a long time. It's like that joke about the mathematician who 
sees only half of one black sheep in Scotland.

A feature that almost works as expected --- except in certain weird 
cases where it corrupts your state --- is at best an attractive nuisance.

>> So what do you propose doing? Upthread, I explained why I think that fixing
>> both #1 and #2 is pretty low-risk.
>
> The problem is that changing the code w.r.t b/a-c-f might cause
> *other* changes.  Clearly making them perfectly balanced and paired
> should not *in itself* introduce any bugs: the bugs we're afraid of are
> the accompanying unintended changes.

I was under the impression that Eli's concern was with the effect of the 
b-c-f change itself, not with bugs that might be introduced by the 
change. I'm confident that with enough review, the core code could be 
changed to make b-c-f and a-c-f symmetric without causing weird bugs 
elsewhere. The necessary refactoring will probably make the logic 
cleaner as well.

Of course there's a risk that changing b-c-f will itself produce weird 
side effects, but I have a hard time seeing how any code could actually 
depend on the current surprising behavior.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> It is guaranteed, modulo bugs (the insert-file-contents (with
>> replace) case).
> Sure. The current implementation provides neither property though.

That's OK.  We're talking about the doc and the intended behavior.
Bugs are part of life.

>> Ah, so you too have code which makes this assumption.  Could you point
>> to it, explaining what it does and why it makes this assumption, so we
>> have a better picture of why it could be useful?
> Same as Alan's --- examining the buffer before the change to see which
> brace-delimited scopes need to be invalidated.

I need more details.

> Emacs core also contains a lot of code that looks fragile under
> asymmetric b-c-f behavior: see tex-mode, org, and viper. (I'm not
> claiming that these modes are broken, but it's not obvious from the
> code that they aren't broken.)

tex-mode's b-c-f doesn't care about when or how any a-f-c gets run or if
it even get run at all.  It just sets some text-properties lazily
(because those properties are only needed when and where a change
happens, so they can be set via b-c-f rather than setting them eagerly
over the whole buffer right from the start).

No idea about org, as for Viper I'm not completely sure what it's trying
to do, but the code looks fiddly (with or without b/a-c-f's asymmetry)
so I'd expect breakage in corner cases anyway.

>> Almost: I don't see the need to change the doc because I don't see where
>> it misleads.
> b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented as
> "List of functions to call before each text change." a-c-f is documented as
> "List of functions to call after each text change." The elisp manual
> documentation is similarly symmetric. This symmetry produces an expectation
> that the behavior is symmetric, and this expectation is reinforced by how
> the observed behavior is almost always symmetric in practice. Symmetric
> behavior here is also what intuitively makes sense.
>
> Please believe me when I say that the current documentation misleads.

OK, so you agree that the documentation does not say they are
properly paired.  You just extrapolate it.

I agree there's such a risk that the reader extrapolates in the way you
described, which is why the current documentation states explicitly that
they're not always paired:

      Do @emph{not} expect the before-change hooks and the after-change
    hooks be called in balanced pairs around each buffer change.  Also

I think it's pretty clear.

> A feature that almost works as expected --- except in certain weird cases
> where it corrupts your state --- is at best an attractive nuisance.

I don't claim that the lack of symmetry is a feature.  But fixing it is
not easy.

Furthermore, the cases I've seen of code designs which wants this
behavior tend to be brittle and overly complicated, compared to
alternative designs which don't rely on this behavior.

I guess a way to "fix" it would be to add a random fudge factor to the
b-c-f calls, plus add random dummy calls to b-c-f, and sprinkle the
lisp/*.el code with combine-after-change-calls.  This way, the "properly
paired up" case will be much more exceptional.

> I was under the impression that Eli's concern was with the effect of
> the b-c-f change itself, not with bugs that might be introduced by
> the change.

I can't speak for Eli, but at least that's not my worry.

> I'm confident that with enough review, the core code could be changed
> to make b-c-f and a-c-f symmetric without causing weird
> bugs elsewhere.

Sure.  But I see more benefits in changing the few places that
mistakenly rely on this pairing.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Sun, 28 Aug 2016 20:18:32 -0700
> 
>     Please trust me that the documentation misleads.

You are welcome to suggest more accurate wording that describes the
current implementation.

> b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented as "List of functions to call before each text change." a-c-f is documented as "List of functions to call after each text change." The elisp manual documentation is similarly symmetric. This symmetry produces an expectation that the behavior is symmetric, and this expectation is reinforced by how the observed behavior is almost always symmetric in practice. Symmetric behavior here is also what intuitively makes sense.

This is a naïve interpretation of what a "change" means and entails.
In reality, some changes are done with a single call to an insdel
function, while others need multiple calls that delete and insert text
piecemeal.  Thus the need to call the hooks before and after each
insdel call only sometimes.

Your bell analogy has the same issue: internally, the "chime" is in
fact a complex array of transitions and changes, not an atomic effect.
By treating it as an atomic unit, you actually uphold the view taken
by the current implementation, which attempts to minimize the number
of hook calls per change.

> Sure, I guess you could argue that current Emacs behavior is consistent with the manual, but it's not what anyone would reasonably expect, and the current behavior is surprising even to people who have been writing elisp for a long time.

Surprising or not, the existing implementation is in use for many
years, and until now no complaints were presented about it.  And even
now we have a single complaint from a single use case of a single
package (which meanwhile fixed the problem without any core changes).
Which is ample evidence that the existing implementation does a good
job after all.

> I'm confident that with enough review, the core code could be changed to make b-c-f and a-c-f symmetric without causing weird bugs elsewhere. The necessary refactoring will probably make the logic cleaner as well.
> 
> Of course there's a risk that changing b-c-f will itself produce weird side effects, but I have a hard time seeing how any code could actually depend on the current surprising behavior.

That's exactly the nonchalant attitude towards changes in core that
explains why, after 30-odd years of development, which should have
given us an asymptotically more and more stable Emacs, we still
succeed quite regularly to break core code and basic features, while
"confidently" relying on our mythical abilities to refactor correctly
without any serious testing coverage.  Never again.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 29 Aug 2016 09:00:40 -0400
> 
> > I was under the impression that Eli's concern was with the effect of
> > the b-c-f change itself, not with bugs that might be introduced by
> > the change.
> 
> I can't speak for Eli, but at least that's not my worry.

I actually don't understand the difference: how is "the effect of the
change itself" different from "bugs that might be introduced" by it.
Sounds like the latter is a subset of the former?

> > I'm confident that with enough review, the core code could be changed
> > to make b-c-f and a-c-f symmetric without causing weird
> > bugs elsewhere.
> 
> Sure.  But I see more benefits in changing the few places that
> mistakenly rely on this pairing.

Agreed.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 06:00 AM, Stefan Monnier wrote:
> Furthermore, the cases I've seen of code designs which wants this
> behavior tend to be brittle and overly complicated, compared to
> alternative designs which don't rely on this behavior.
>
> I guess a way to "fix" it would be to add a random fudge factor to the
> b-c-f calls, plus add random dummy calls to b-c-f, and sprinkle the
> lisp/*.el code with combine-after-change-calls.  This way, the "properly
> paired up" case will be much more exceptional.

You can't be serious.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 07:50 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Sun, 28 Aug 2016 20:18:32 -0700
>>
>>     Please trust me that the documentation misleads.
>
> You are welcome to suggest more accurate wording that describes the
> current implementation.

Just mark b-c-f deprecated.

>> b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented as "List of functions to call before each text change." a-c-f is documented as "List of functions to call after each text change." The elisp manual documentation is similarly symmetric. This symmetry produces an expectation that the behavior is symmetric, and this expectation is reinforced by how the observed behavior is almost always symmetric in practice. Symmetric behavior here is also what intuitively makes sense.
>
> This is a naïve interpretation of what a "change" means and entails.

It doesn't matter what a "change" means and entails. Whatever a "change" 
is, b-c-f and a-c-f use the same word for it, and I've already explained 
why it's natural to suppose symmetry between these hooks. In order for 
b-c-f and a-c-f to be asymmetric, the definition of the word "change" 
needs to somehow change.

Do you really not understand why many people would find the current 
behavior surprising? You have at least *three* people independently 
annoyed by it: Alan, Phillip Lord, and me.

>> Sure, I guess you could argue that current Emacs behavior is consistent with the manual, but it's not what anyone would reasonably expect, and the current behavior is surprising even to people who have been writing elisp for a long time.
>
> Surprising or not, the existing implementation is in use for many
> years, and until now no complaints were presented about it.  And even
> now we have a single complaint from a single use case of a single
> package (which meanwhile fixed the problem without any core changes).
> Which is ample evidence that the existing implementation does a good
> job after all.

The current implementation is buggy. Even Stefan agrees that it 
currently contains bona fide bugs (the file visit case). These bugs 
produce difficult-to-diagnose user bugs reports. that have generated 
real bug reports. The current implementation does not serve the needs of 
its users.

I mean, you could just declare the current implementation correct and 
point to workarounds as evidence of the fact --- but that's like 
declaring a drain unclogged by definition and pointing to the plunger as 
evidence.

>> I'm confident that with enough review, the core code could be changed to make b-c-f and a-c-f symmetric without causing weird bugs elsewhere. The necessary refactoring will probably make the logic cleaner as well.
>>
>> Of course there's a risk that changing b-c-f will itself produce weird side effects, but I have a hard time seeing how any code could actually depend on the current surprising behavior.
>
> That's exactly the nonchalant attitude towards changes in core that
> explains why, after 30-odd years of development, which should have
> given us an asymptotically more and more stable Emacs, we still
> succeed quite regularly to break core code and basic features, while
> "confidently" relying on our mythical abilities to refactor correctly
> without any serious testing coverage.

The modern software engineering approach to preventing regressions is 
test automation. Insisting on adding a test as part of the change is 
reasonable. Forbidding all change, no matter how carefully done, is not.

 >  Never again.

The surest way to make sure a computer doesn't do anything wrong is to 
turn it off.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> I guess a way to "fix" it would be to add a random fudge factor to the
>> b-c-f calls, plus add random dummy calls to b-c-f, and sprinkle the
>> lisp/*.el code with combine-after-change-calls.  This way, the "properly
>> paired up" case will be much more exceptional.
> You can't be serious.

If it can finally get us rid of people who insist they want exactly
paired/matched/symmetric b/a-c-f, then I'd willing to do that, yes.

AFAICT the result would be simpler, cleaner, and more robust code, since
instead of trying to think of the various different ways a buffer can
change here or there, the coder would be forced to step back and think
how to handle the general case.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> > I was under the impression that Eli's concern was with the effect of
>> > the b-c-f change itself, not with bugs that might be introduced by
>> > the change.
>> I can't speak for Eli, but at least that's not my worry.
> I actually don't understand the difference: how is "the effect of the
> change itself" different from "bugs that might be introduced" by it.
> Sounds like the latter is a subset of the former?

The request is to change the way b-c-f and a-c-f are run such that
they're always exactly paired.  Any patch which implements such a change
could have any number of other impacts.  So we can divide the impacts of such
a patch into 2 parts:
- the part which changes b-c-f and a-c-f to be exactly paired.
- the rest.

I think the first part should be 100% safe, because any code which works
correctly with the current looser behavior of b/a-c-f should also work
correctly with the tighter behavior.  Of course, there might be code out
there which relies on the details of the current behavior b/a-c-f, but
I'm not worried about that at all.

"The rest" is the more likely source of worry.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 29 Aug 2016 08:30:21 -0700
> 
> >> b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented as "List of functions to call before each text change." a-c-f is documented as "List of functions to call after each text change." The elisp manual documentation is similarly symmetric. This symmetry produces an expectation that the behavior is symmetric, and this expectation is reinforced by how the observed behavior is almost always symmetric in practice. Symmetric behavior here is also what intuitively makes sense.
> >
> > This is a naïve interpretation of what a "change" means and entails.
> 
> It doesn't matter what a "change" means and entails. Whatever a "change" 
> is, b-c-f and a-c-f use the same word for it, and I've already explained 
> why it's natural to suppose symmetry between these hooks. In order for 
> b-c-f and a-c-f to be asymmetric, the definition of the word "change" 
> needs to somehow change.

Of course, you have changed the subject.

> Do you really not understand why many people would find the current 
> behavior surprising? You have at least *three* people independently 
> annoyed by it: Alan, Phillip Lord, and me.

It doesn't matter what I understand or not, because that's not the
issue at hand.  We are talking about code that runs virtually
unchanged for many years.  Making significant changes in it needs a
good reason.  When such good reasons emerge, we can discuss whether
they justify the risks.  For now, the reasons presented do not.

> The current implementation is buggy.

Every piece of software is buggy.  That doesn't mean it's useless or
must be refactored the heck out of.

> (the file visit case). These bugs produce difficult-to-diagnose user
> bugs reports. that have generated real bug reports.

Those bugs were fixed.

> I mean, you could just declare the current implementation correct and 
> point to workarounds as evidence of the fact --- but that's like 
> declaring a drain unclogged by definition and pointing to the plunger as 
> evidence.

I'm not interested in an argument about correctness.  That's not what
this is about.  You are changing the subject again.

> > That's exactly the nonchalant attitude towards changes in core that
> > explains why, after 30-odd years of development, which should have
> > given us an asymptotically more and more stable Emacs, we still
> > succeed quite regularly to break core code and basic features, while
> > "confidently" relying on our mythical abilities to refactor correctly
> > without any serious testing coverage.
> 
> The modern software engineering approach to preventing regressions is 
> test automation. Insisting on adding a test as part of the change is 
> reasonable. Forbidding all change, no matter how carefully done, is not.

No one have forbidden all change, and no one in their right minds
will.  Do you really not understand what the above is about?

>  >  Never again.
> 
> The surest way to make sure a computer doesn't do anything wrong is to 
> turn it off.

If you aren't interested in it doing anything useful, sure.  But I am,
so these simple solutions are wrong for me (and for all the rest of
Emacs users).

Of course, if those absurd "arguments" are the only ones you have
left, I guess this is EOD.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Mon, 29 Aug 2016 11:50:13 -0400
> 
> "The rest" is the more likely source of worry.

Yes.  And that is what I understood "the effect of the change" is
about.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 09:20 AM, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 29 Aug 2016 08:30:21 -0700
>>
>>>> b-c-f and a-c-f are symmetric in name and signature. b-c-f is documented as "List of functions to call before each text change." a-c-f is documented as "List of functions to call after each text change." The elisp manual documentation is similarly symmetric. This symmetry produces an expectation that the behavior is symmetric, and this expectation is reinforced by how the observed behavior is almost always symmetric in practice. Symmetric behavior here is also what intuitively makes sense.
>>>
>>> This is a naïve interpretation of what a "change" means and entails.
>>
>> It doesn't matter what a "change" means and entails. Whatever a "change"
>> is, b-c-f and a-c-f use the same word for it, and I've already explained
>> why it's natural to suppose symmetry between these hooks. In order for
>> b-c-f and a-c-f to be asymmetric, the definition of the word "change"
>> needs to somehow change.
>
> Of course, you have changed the subject.
>
>> Do you really not understand why many people would find the current
>> behavior surprising? You have at least *three* people independently
>> annoyed by it: Alan, Phillip Lord, and me.
>
> It doesn't matter what I understand or not, because that's not the
> issue at hand.  We are talking about code that runs virtually
> unchanged for many years.  Making significant changes in it needs a
> good reason.  When such good reasons emerge, we can discuss whether
> they justify the risks.  For now, the reasons presented do not.

What criteria are you using to determine whether a bug is sufficiently 
serious to fix? What would convince you that a change in this behavior 
is warranted?

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 29 Aug 2016 09:26:38 -0700
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > We are talking about code that runs virtually
> > unchanged for many years.  Making significant changes in it needs a
> > good reason.  When such good reasons emerge, we can discuss whether
> > they justify the risks.  For now, the reasons presented do not.
> 
> What criteria are you using to determine whether a bug is sufficiently 
> serious to fix? What would convince you that a change in this behavior 
> is warranted?

I described up-thread what would constitute a good enough reason for
me to consider such changes for admission.  Here's the list again:

  . a bug that affects (i.e. breaks) the core code itself (e.g., see
    bug#5131 fixed in 00b6647 as a recent example related to insdel.c)
  . a problem that affects several Lisp packages for which there's no
    reasonably practical workaround/fix as part of the package itself
  . refactoring done as part of introducing a significant new feature

Please note that not every change/bugfix is required to pass such
scrutiny, only changes in code that is very central to Emacs
operation.  I think manipulation of buffer text, display engine, basic
file I/O, and encoding/decoding stuff are such areas.  Changes that
affect some aspects of more local, specialized behavior are normally
less risky.  IOW, there's still a judgment call needed in each case,
so that the above is applicable to as few changes as possible.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 10:01 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 29 Aug 2016 09:26:38 -0700
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>
>>> We are talking about code that runs virtually
>>> unchanged for many years.  Making significant changes in it needs a
>>> good reason.  When such good reasons emerge, we can discuss whether
>>> they justify the risks.  For now, the reasons presented do not.
>>
>> What criteria are you using to determine whether a bug is sufficiently
>> serious to fix? What would convince you that a change in this behavior
>> is warranted?
>
> I described up-thread what would constitute a good enough reason for
> me to consider such changes for admission.  Here's the list again:
>
>   . a bug that affects (i.e. breaks) the core code itself (e.g., see
>     bug#5131 fixed in 00b6647 as a recent example related to insdel.c)
>   . a problem that affects several Lisp packages for which there's no
>     reasonably practical workaround/fix as part of the package itself
>   . refactoring done as part of introducing a significant new feature

We have two separate bugs here. In order of severity:

1) Something b-c-f is not called _at_ _all_ before certain changes
2) Sometimes b-c-f is called once, then a-c-f is called multiple times, 
because internally, changes are split into several chunks, each of which 
gets its own a-c-f call

I can understand #2 being wontfix, but can you reconsider fixing #1? #1 
breaks the entire b-c-f model --- "hey, I'm about to modify the buffer, 
so throw away caches" ---- and can lead to anything with a cache flush 
in b-c-f (like syntax-ppss) not properly discarding out-of-date buffer 
information.

>
> Please note that not every change/bugfix is required to pass such
> scrutiny, only changes in code that is very central to Emacs
> operation.  I think manipulation of buffer text, display engine, basic
> file I/O, and encoding/decoding stuff are such areas.  Changes that
> affect some aspects of more local, specialized behavior are normally
> less risky.  IOW, there's still a judgment call needed in each case,
> so that the above is applicable to as few changes as possible.
>
>

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 29 Aug 2016 10:48:31 -0700
> 
> 1) Something b-c-f is not called _at_ _all_ before certain changes
> 2) Sometimes b-c-f is called once, then a-c-f is called multiple times, 
> because internally, changes are split into several chunks, each of which 
> gets its own a-c-f call
> 
> I can understand #2 being wontfix, but can you reconsider fixing #1?

There's only one such case, and the only package that's affected
(CC Mode) already worked around that case.

So, while I agreed that #1 should probably be fixed, and even
suggested how to do that in the least risky way, actually doing that
is not a priority, IMO, not until we have a very grave problem caused
by it.

> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
> buffer, so throw away caches" ---- and can lead to anything with a
> cache flush in b-c-f (like syntax-ppss) not properly discarding
> out-of-date buffer information.

That single case of #1 is revert-buffer, which conceptually throws
away the entire buffer and replaces it with what's on disk.  That it
actually keeps portions of the buffer is an optimization, but the
concept still stands.  So I don't see how it breaks the entire model,
at least not in practice.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Mon, 29 Aug 2016 21:04:42 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> That single case of #1 is revert-buffer

Btw: not every call to revert-buffer hits this case.  Quite the
contrary: almost none of them do.  IOW, the situation where the
before-change hook is not called at all is very rare even if you only
consider the uses of revert-buffer.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> So, while I agreed that #1 should probably be fixed, and even
> suggested how to do that in the least risky way, actually doing that
> is not a priority, IMO, not until we have a very grave problem caused
> by it.

FWIW, I think it's important to fix the insert-file-contents case.
It breaks syntax-ppss, for example, although for various reasons of how
syntax-ppss is implemented and used, this happens to rarely show up.
[ So, yes, it's not top priority, but it should still be fixed.  ]

>> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
>> buffer, so throw away caches" ---- and can lead to anything with a
>> cache flush in b-c-f (like syntax-ppss) not properly discarding
>> out-of-date buffer information.
> That single case of #1 is revert-buffer, which conceptually throws
> away the entire buffer and replaces it with what's on disk.  That it
> actually keeps portions of the buffer is an optimization, but the
> concept still stands.  So I don't see how it breaks the entire model,
> at least not in practice.

The optimization is beside the point: not calling b-c-f in some corner
case breaks the entire model because a user such as syntax-ppss relies
on b-c-f to know when to flush its cache, so if you don't call it when
the buffer is modified, the cache ends up stale.


        Stefan

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>  (or
>                 ;; previously this was not skewed if no region, but actually,
>                 ;; if there is no region we need to copy everything, we can
>                 ;; also do by declaring skew -- this is important for the
>                 ;; multi-lentic situation
>                 (not (or start stop length-before))
>                 ;; skews only occur in insertions which result in a positive
>                 ;; length-before. This also picks up no-insertion changes
>                 (and (< 0 length-before)
>                      ;; = start stop means we have a deletion because
>                      ;; there is no range after. Deletions seem to be
>                      ;; safe.
>                      (not (= start stop))))

I don't understand what this is doing.  It seems to be looking for
indirect clues rather than checking directly for the properties on which
the code relies.

Why not do something like:

    (defun lentic--b-c-f (start end)
      (setq-local lentic--b-c-pos (cons start end))
      ...)

    (defun lentic--a-c-f (start end length)
      (if (prog1 (not (and (eq start (car lentic--b-c-pos))
                           (eq (+ start length) (cdr lentic--b-c-pos))))
            (setq lentic--b-c-pos nil))
          ;; This a-c-f is not properly paired with the previous b-c-f.
          <do-the-skewed-thing>
        ;; Proper pairing
        <do-the-non-skew-thing>))

If the performance issue is serious enough, you can of course try to "do
it right" with something like:

    (defun lentic--b-c-f (start end)
      (setq-local lentic--b-c-data
                  (cons start (buffer-substring start end)))
      ...)

    (defun lentic--a-c-f (start end length)
      ;; Get the before-change content of start..end.
      (let ((old-text (substring (cdr lentic--b-c-data)
                                 (- start (car lentic--b-c-data))
                                 (- end length (car lentic--b-c-data)))))
        ;; Update lentic--b-c-data for possible subsequent other a-c-f calls.
        (setcdr lentic--b-c-data
                (concat (substring (cdr lentic--b-c-data)
                                   0 (- start (car lentic--b-c-data)))
                        (buffer-substring start end)
                        (substring (cdr lentic--b-c-data)
                                   (- end length (car lentic--b-c-data)))))
      <now-do-what-we-have-to-do>))

But it can admittedly be cumbersome since the old-text is now in a string
rather than being inside the buffer.

> 1) being able to know when b-c-f and a-c-f are not paired or consistent
> would be useful

Does the above lentic--b-c-pos give the needed info?

> 2) decreasing the number of times these occurs would be useful, even if
> it cannot be removed entirely.

Note that in the subst-char-in-region you could "make it pairup"
yourself by hand: if you have (< (+ start length) (cdr lentic--b-c-pos)),
then you can just

    (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
      (cl-incf length diff)
      (cl-incf end diff))

such that (eq (+ start length) (cdr lentic--b-c-pos)).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 05:25 PM, Stefan Monnier wrote:
>> So, while I agreed that #1 should probably be fixed, and even
>> suggested how to do that in the least risky way, actually doing that
>> is not a priority, IMO, not until we have a very grave problem caused
>> by it.
>
> FWIW, I think it's important to fix the insert-file-contents case.
> It breaks syntax-ppss, for example, although for various reasons of how
> syntax-ppss is implemented and used, this happens to rarely show up.
> [ So, yes, it's not top priority, but it should still be fixed.  ]

Agreed.

>
>>> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
>>> buffer, so throw away caches" ---- and can lead to anything with a
>>> cache flush in b-c-f (like syntax-ppss) not properly discarding
>>> out-of-date buffer information.
>> That single case of #1 is revert-buffer, which conceptually throws
>> away the entire buffer and replaces it with what's on disk.  That it
>> actually keeps portions of the buffer is an optimization, but the
>> concept still stands.  So I don't see how it breaks the entire model,
>> at least not in practice.
>
> The optimization is beside the point: not calling b-c-f in some corner
> case breaks the entire model because a user such as syntax-ppss relies
> on b-c-f to know when to flush its cache, so if you don't call it when
> the buffer is modified, the cache ends up stale.
>

Yes. That's what I meant. If you don't do this, b-c-f becomes 
practically useless.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 11:16 AM, Eli Zaretskii wrote:
>> Date: Mon, 29 Aug 2016 21:04:42 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>
>> That single case of #1 is revert-buffer
>
> Btw: not every call to revert-buffer hits this case.  Quite the
> contrary: almost none of them do.  IOW, the situation where the
> before-change hook is not called at all is very rare even if you only
> consider the uses of revert-buffer.

I think we fundamentally disagree about how rarity affects bug important:

You: "This bug is rare. Cool. We don't have to worry about it much."
Me: "This bug is rare. That's awful. That means that people won't hit 
this bug until it causes some horrible, non-reproducible problem."

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
> Date: Mon, 29 Aug 2016 20:25:06 -0400
> 
> >> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
> >> buffer, so throw away caches" ---- and can lead to anything with a
> >> cache flush in b-c-f (like syntax-ppss) not properly discarding
> >> out-of-date buffer information.
> > That single case of #1 is revert-buffer, which conceptually throws
> > away the entire buffer and replaces it with what's on disk.  That it
> > actually keeps portions of the buffer is an optimization, but the
> > concept still stands.  So I don't see how it breaks the entire model,
> > at least not in practice.
> 
> The optimization is beside the point: not calling b-c-f in some corner
> case breaks the entire model because a user such as syntax-ppss relies
> on b-c-f to know when to flush its cache, so if you don't call it when
> the buffer is modified, the cache ends up stale.

I'm saying that flushing the entire cache in that case is not a
problem, it's what needs to be done anyway.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 29 Aug 2016 19:27:37 -0700
> 
> I think we fundamentally disagree about how rarity affects bug important:

We've seen such fundamental disagreements before, so it's nothing new.

> You: "This bug is rare. Cool. We don't have to worry about it much."
> Me: "This bug is rare. That's awful. That means that people won't hit 
> this bug until it causes some horrible, non-reproducible problem."

When the fix is risky and touches code that affects almost everything
in Emacs, my method is much less likely to break important
functionality for the benefit of rare cases that can be worked around
or solved elsewhere.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/29/2016 07:38 PM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
>> Date: Mon, 29 Aug 2016 20:25:06 -0400
>>
>>>> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
>>>> buffer, so throw away caches" ---- and can lead to anything with a
>>>> cache flush in b-c-f (like syntax-ppss) not properly discarding
>>>> out-of-date buffer information.
>>> That single case of #1 is revert-buffer, which conceptually throws
>>> away the entire buffer and replaces it with what's on disk.  That it
>>> actually keeps portions of the buffer is an optimization, but the
>>> concept still stands.  So I don't see how it breaks the entire model,
>>> at least not in practice.
>>
>> The optimization is beside the point: not calling b-c-f in some corner
>> case breaks the entire model because a user such as syntax-ppss relies
>> on b-c-f to know when to flush its cache, so if you don't call it when
>> the buffer is modified, the cache ends up stale.
>
> I'm saying that flushing the entire cache in that case is not a
> problem, it's what needs to be done anyway.

How is a mode supposed to know that it's supposed to flush its cache?

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> The optimization is beside the point: not calling b-c-f in some corner
>> case breaks the entire model because a user such as syntax-ppss relies
>> on b-c-f to know when to flush its cache, so if you don't call it when
>> the buffer is modified, the cache ends up stale.
> I'm saying that flushing the entire cache in that case is not a
> problem, it's what needs to be done anyway.

The problem is that b-c-f is not called at all, so syntax-ppss doesn't
know that it should flush (any part of) its cache.
Calling b-c-f on point-min...point-max would be perfectly fine, indeed.
But currently b-c-f is not called at all.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Daniel Colascione <dancol@dancol.org> writes:

>> As mentioned in my so-called "hand-waving", any details you may need can
>> be stored somewhere for use.
>
> Is it really better to force that complexity (which, in the limit, amounts to
> full shadow buffers) onto mode authors instead of changing Emacs to balance
> b-c-f with a-c-f 100% of the time instead of 99%?


I think to really avoid the complexity for mode authors, b-c-f and a-c-f
need to not only balance, but also to be consistent. At the moment, they
are not that either -- they can signal different locations for a given
change.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> I think to really avoid the complexity for mode authors, b-c-f and a-c-f
> need to not only balance, but also to be consistent. At the moment, they
> are not that either -- they can signal different locations for a given
> change.

I think "balanced + a guarantee that a-c-f covers a subset of b-c-f" is
sufficient, since you can then trivially extend a-c-f's subset to the
corresponding "identical" locations if you need it.

AFAIK we do have the "a-c-f covers a subset of b-c-f" property,
right now (modulo bugs, of course).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> I guess a way to "fix" it would be to add a random fudge factor to the
>>> b-c-f calls, plus add random dummy calls to b-c-f, and sprinkle the
>>> lisp/*.el code with combine-after-change-calls.  This way, the "properly
>>> paired up" case will be much more exceptional.
>> You can't be serious.
>
> If it can finally get us rid of people who insist they want exactly
> paired/matched/symmetric b/a-c-f, then I'd willing to do that, yes.
>
> AFAICT the result would be simpler, cleaner, and more robust code, since
> instead of trying to think of the various different ways a buffer can
> change here or there, the coder would be forced to step back and think
> how to handle the general case.


I use these functions for my package lentic, and I do handle the general
case.

In fact, I started off by handling the general case -- this involves
using a-c-f only and just ignoring the parameters. It works, of course,
but it is inefficient enough that it limits the usability to small
buffers.

So, I made use of the information on a-c-f and b-c-f. This increases the
code complexity significantly, but it reduces the load massively. That
b-c-f and a-c-f are not always balanced or consistent, unfortunately,
works directly against this aim.

So having them balanced and consistent would be nice. I agree that in
fixing this, bad things might happens. But surely that is a risk of
fiddling with any core part of Emacs. One response would be to not to
it. Another response would be to incrementally increase the number of
test cases Emacs has to enable this (and other) changes in the core to
be made with more confidence.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Sure, I guess you could argue that current Emacs behavior is consistent with
>> the manual, but it's not what anyone would reasonably expect, and the
>> current behavior is surprising even to people who have been writing elisp
>> for a long time.
>
> Surprising or not, the existing implementation is in use for many
> years, and until now no complaints were presented about it.

Actually, it caused me significant issues several years ago, and I would
have prefered that the hooks were balanced. I'd rather fix things than
complain, but you can consider this a complaint if you will.

> And even now we have a single complaint from a single use case of a
> single package (which meanwhile fixed the problem without any core
> changes). Which is ample evidence that the existing implementation
> does a good job after all.

No, I don't think it is. It might be evidence that people know to seek
alternative solutions instead of using b-c-f and a-c-f.


>> I'm confident that with enough review, the core code could be changed to
>> make b-c-f and a-c-f symmetric without causing weird bugs elsewhere. The
>> necessary refactoring will probably make the logic cleaner as well.
>> 
>> Of course there's a risk that changing b-c-f will itself produce weird side
>> effects, but I have a hard time seeing how any code could actually depend on
>> the current surprising behavior.
>
> That's exactly the nonchalant attitude towards changes in core that
> explains why, after 30-odd years of development, which should have
> given us an asymptotically more and more stable Emacs, we still
> succeed quite regularly to break core code and basic features, while
> "confidently" relying on our mythical abilities to refactor correctly
> without any serious testing coverage.  Never again.

Can I deliberately misinterpret this as saying that you think that the
changes would be fine so long as we add lots of tests at the same time?

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Daniel Colascione <dancol@dancol.org> writes:

> On 08/29/2016 07:38 PM, Eli Zaretskii wrote:
>> I'm saying that flushing the entire cache in that case is not a
>> problem, it's what needs to be done anyway.
>
> How is a mode supposed to know that it's supposed to flush its cache?


I don't know that either and it would be lovely to know when.

I've posted my heuristic earlier, which works for me but comes with no
guarantees at all.


https://lists.gnu.org/archive/html/emacs-devel/2016-08/msg00381.html

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  (or
>>                 ;; previously this was not skewed if no region, but actually,
>>                 ;; if there is no region we need to copy everything, we can
>>                 ;; also do by declaring skew -- this is important for the
>>                 ;; multi-lentic situation
>>                 (not (or start stop length-before))
>>                 ;; skews only occur in insertions which result in a positive
>>                 ;; length-before. This also picks up no-insertion changes
>>                 (and (< 0 length-before)
>>                      ;; = start stop means we have a deletion because
>>                      ;; there is no range after. Deletions seem to be
>>                      ;; safe.
>>                      (not (= start stop))))
>
> I don't understand what this is doing.


That's a shame. I don't understand what it is doing either, and I was
hoping that you might do.


> It seems to be looking for indirect clues rather than checking
> directly for the properties on which the code relies.

Perhaps. At the moment, the only thing function that I know definately
causes me problems is subst-char-in-region.

> Why not do something like:
>
>     (defun lentic--b-c-f (start end)
>       (setq-local lentic--b-c-pos (cons start end))
>       ...)
>
>     (defun lentic--a-c-f (start end length)
>       (if (prog1 (not (and (eq start (car lentic--b-c-pos))
>                            (eq (+ start length) (cdr lentic--b-c-pos))))
>             (setq lentic--b-c-pos nil))
>           ;; This a-c-f is not properly paired with the previous b-c-f.
>           <do-the-skewed-thing>
>         ;; Proper pairing
>         <do-the-non-skew-thing>))

I will try and test this to see.


> If the performance issue is serious enough, you can of course try to "do
> it right" with something like:
>
>     (defun lentic--b-c-f (start end)
>       (setq-local lentic--b-c-data
>                   (cons start (buffer-substring start end)))
>       ...)
>
>     (defun lentic--a-c-f (start end length)
>       ;; Get the before-change content of start..end.
>       (let ((old-text (substring (cdr lentic--b-c-data)
>                                  (- start (car lentic--b-c-data))
>                                  (- end length (car lentic--b-c-data)))))
>         ;; Update lentic--b-c-data for possible subsequent other a-c-f calls.
>         (setcdr lentic--b-c-data
>                 (concat (substring (cdr lentic--b-c-data)
>                                    0 (- start (car lentic--b-c-data)))
>                         (buffer-substring start end)
>                         (substring (cdr lentic--b-c-data)
>                                    (- end length (car lentic--b-c-data)))))
>       <now-do-what-we-have-to-do>))
>
> But it can admittedly be cumbersome since the old-text is now in a string
> rather than being inside the buffer.

Because, I am not using the before and after positions just to recognise
the string that has changed. I am have to calculate the equivalent
positions in the other "lentic" buffer; and this can only be done before
the change, since the two buffers are them in a consistent state.

In the case that they are skewed, I have to throw all my knowledge away,
since the converted positions are not unreliable. So I copy the whole
buffer, and apply the general (unoptimized) transformation.

>> 1) being able to know when b-c-f and a-c-f are not paired or consistent
>> would be useful
>
> Does the above lentic--b-c-pos give the needed info?
>
>> 2) decreasing the number of times these occurs would be useful, even if
>> it cannot be removed entirely.
>
> Note that in the subst-char-in-region you could "make it pairup"
> yourself by hand: if you have (< (+ start length) (cdr lentic--b-c-pos)),
> then you can just
>
>     (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
>       (cl-incf length diff)
>       (cl-incf end diff))
>
> such that (eq (+ start length) (cdr lentic--b-c-pos)).


So, this works because subst-char-in-region is guaranteed not to change
the size of the region right? But, how do I know that
subst-char-in-region has been called? I discovered this problem first,
when fill-paragraph broke lentic. It just so happens that
subst-char-in-region was the culprit; it wasn't the command though.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 29 Aug 2016 19:54:52 -0700
> 
> On 08/29/2016 07:38 PM, Eli Zaretskii wrote:
> >> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> >> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
> >> Date: Mon, 29 Aug 2016 20:25:06 -0400
> >>
> >>>> #1 breaks the entire b-c-f model --- "hey, I'm about to modify the
> >>>> buffer, so throw away caches" ---- and can lead to anything with a
> >>>> cache flush in b-c-f (like syntax-ppss) not properly discarding
> >>>> out-of-date buffer information.
> >>> That single case of #1 is revert-buffer, which conceptually throws
> >>> away the entire buffer and replaces it with what's on disk.  That it
> >>> actually keeps portions of the buffer is an optimization, but the
> >>> concept still stands.  So I don't see how it breaks the entire model,
> >>> at least not in practice.
> >>
> >> The optimization is beside the point: not calling b-c-f in some corner
> >> case breaks the entire model because a user such as syntax-ppss relies
> >> on b-c-f to know when to flush its cache, so if you don't call it when
> >> the buffer is modified, the cache ends up stale.
> >
> > I'm saying that flushing the entire cache in that case is not a
> > problem, it's what needs to be done anyway.
> 
> How is a mode supposed to know that it's supposed to flush its cache?

I think I misunderstood you, sorry.  I thought you were saying that in
the absence of the before-change call a mode will decide to flush its
caches, and that you've considered such flushing a problem of some
sort.

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>> But it can admittedly be cumbersome since the old-text is now in a string
>> rather than being inside the buffer.
> Because, I am not using the before and after positions just to recognise
> the string that has changed.

I was not suggesting to use those position to recognize the string:
I was suggesting to use the suggested code to get the old-string so that
you can then do what you need on the "before change" state.

> I am have to calculate the equivalent positions in the other "lentic"
> buffer; and this can only be done before the change, since the two
> buffers are them in a consistent state.

The idea is that my suggested code gives you the needed info.
More specifically, the other buffer is (well, should be) in a state
consistent with "the current buffer where start..end is replaced (back)
with `old-text'".

So that should give you the info needed to update things without doing
a "complete flush".  As mentioned, it may be inconvenient to use it
(depending on details of how the rest of the code is implemented) since
the old-text is now in a string rather than being right there in
the buffer.

>> Note that in the subst-char-in-region you could "make it pairup"
>> yourself by hand: if you have (< (+ start length) (cdr lentic--b-c-pos)),
>> then you can just
>> 
>>     (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
>>     (cl-incf length diff)
>>     (cl-incf end diff))
>> 
>> such that (eq (+ start length) (cdr lentic--b-c-pos)).
> So, this works because subst-char-in-region is guaranteed not to change
> the size of the region right?

No.  It works because a-c-f's arguments say "nothing was changed before
START and after END", so it's always safe (conservative) to move START
towards BOB or move END towards EOB (of course, when updating END you
also need to update LENGTH correspondingly).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Tue, 30 Aug 2016 14:30:13 +0100
> Cc: Alan Mackenzie <acm@muc.de>, Stefan Monnier <monnier@IRO.UMontreal.CA>,
> 	emacs-devel@gnu.org
> 
> I think to really avoid the complexity for mode authors, b-c-f and a-c-f
> need to not only balance, but also to be consistent. At the moment, they
> are not that either -- they can signal different locations for a given
> change.

You are setting the bar impossibly high by expecting that.
Implementing what you want would need significant changes in how the
insdel functions are implemented and used.  In particular, some
complex changes will probably have to do things twice: once just to
figure out which changes are needed, the other time to actually do
that while calling the hooks in the way you want.  That's because in
some situations we discover the required changes as we go, and have no
idea in advance what we will find.

Read the code where these functions are called, and you will see what
I mean.

insert-file-contents as called by revert-buffer is a case in point: we
don't know which parts of the old text will need to be deleted until
we do the job by comparing the buffer with the file on disk; we delete
each portion as we discover it.

To keep this issue in its proper perspective, we have other hooks that
are similarly "inconsistent".  For example, window-scroll-functions
can be called by the display engine any number of times during a
single redisplay cycle, because the display engine many times plays
"catch and loose", trying something, then falling back to some backup
plan if plan A failed; but by the time it understands it needs to give
up on plane A, it already have called the hooks.

So I submit that this "inconsistency" is actually an inherent trait of
the Emacs hooks, at least some of them, and at this point the only way
to avoid that is completely rewrite the corresponding part(s) of the
internals.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Tue, 30 Aug 2016 15:07:27 +0100
> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
> 
> So having them balanced and consistent would be nice. I agree that in
> fixing this, bad things might happens. But surely that is a risk of
> fiddling with any core part of Emacs. One response would be to not to
> it. Another response would be to incrementally increase the number of
> test cases Emacs has to enable this (and other) changes in the core to
> be made with more confidence.

My response is that we should not do it until we have decent test
coverage of the insdel functionality.  I simply don't believe in our
ability to avoid breakage in this core functionality unless we have
good test coverage.  Been there, done that, have bruises to prove it.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> You are setting the bar impossibly high by expecting that.
> Implementing what you want would need significant changes in how the
> insdel functions are implemented and used.  In particular, some
> complex changes will probably have to do things twice: once just to
> figure out which changes are needed, the other time to actually do
> that while calling the hooks in the way you want.  That's because in
> some situations we discover the required changes as we go, and have no
> idea in advance what we will find.

Actually, we do have some idea beforehand, as evidenced by the fact that
we call b-c-f with valid data.  It's just that after the fact we have
a more precise knowledge of what has actually been changed, so a-c-f can
give more precise bounds.

But as mentioned in some other message, the a-c-f function can itself
extend its arguments to match those of the b-c-f if it needs them to
be matched.

We could also change the C code to perform this "extension" of the
region passed to a-c-f so that the b-c-f region and the a-c-f region
match, but that would make the region passed to a-c-f less precise, so
those users of a-c-f which don't care about the pairing (the majority of
the users) would be penalized (tho they'd still work correctly: just
suboptimally) since there's no way for them to recover the
tighter region.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Daniel Colascione <dancol@dancol.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 30 Aug 2016 15:12:13 +0100
> 
> > Surprising or not, the existing implementation is in use for many
> > years, and until now no complaints were presented about it.
> 
> Actually, it caused me significant issues several years ago, and I would
> have prefered that the hooks were balanced.

Me too, believe it or not.  But this is not about our preferences.
This is about the tough choices we must make when faced with the
current situation, where this code was working for many years, we have
a very small number of people who even understand the intricacies of
the implementation, and users certainly don't expect to see Emacs 26
have bugs in basic text handling in buffers.

IOW, we need to find the lesser evil.

> > And even now we have a single complaint from a single use case of a
> > single package (which meanwhile fixed the problem without any core
> > changes). Which is ample evidence that the existing implementation
> > does a good job after all.
> 
> No, I don't think it is. It might be evidence that people know to seek
> alternative solutions instead of using b-c-f and a-c-f.

Fine with me.  I indeed prefer that in the current situation any bugs
this could cause be fixed or worked around in the packages that are
hit by them.  As long as these problems are rare and far between, that
is better than risking to break insdel and its callers, which could be
virtually anywhere.

If someone refactors insdel to introduce some significant new feature,
then instability can be justified.  But otherwise, it sounds nuts to
me to risk that, given the rarity of the problems and the ability of
the packages to work around them.  Especially, as we have no way
whatsoever of verifying the changes by tests.

> > That's exactly the nonchalant attitude towards changes in core that
> > explains why, after 30-odd years of development, which should have
> > given us an asymptotically more and more stable Emacs, we still
> > succeed quite regularly to break core code and basic features, while
> > "confidently" relying on our mythical abilities to refactor correctly
> > without any serious testing coverage.  Never again.
> 
> Can I deliberately misinterpret this as saying that you think that the
> changes would be fine so long as we add lots of tests at the same time?

Not "at the same time", but "as a prerequisite" for any serious
consideration of such changes.  IOW, I want to see the tests first,
decide whether I like their degree of coverage, and only then I will
be prepared to _talk_ about any changes in this area.  And the first
step in that talk will have to be someone presenting a rather complete
design of the changes, including an analysis of the current callers
and how each one (or each class) of them will work under the new
design.

Re: Unbalanced change hooks (part 2)

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Tue, 30 Aug 2016 15:34:19 +0100
> Cc: emacs-devel@gnu.org
> 
> So, this works because subst-char-in-region is guaranteed not to change
> the size of the region right?

Caution: in characters, but not necessarily in bytes!

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 08:51 AM, Eli Zaretskii wrote:
>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Tue, 30 Aug 2016 15:07:27 +0100
>> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
>>
>> So having them balanced and consistent would be nice. I agree that in
>> fixing this, bad things might happens. But surely that is a risk of
>> fiddling with any core part of Emacs. One response would be to not to
>> it. Another response would be to incrementally increase the number of
>> test cases Emacs has to enable this (and other) changes in the core to
>> be made with more confidence.
>
> My response is that we should not do it until we have decent test
> coverage of the insdel functionality.  I simply don't believe in our
> ability to avoid breakage in this core functionality unless we have
> good test coverage.  Been there, done that, have bruises to prove it.

More tests in general would be nice. I have no objection to requiring 
test coverage for insdel before making major changes; my objection is to 
a blanket ban on changes even *with* test coverage. I was the under the 
impression that your preference was for the latter model, but IIUC, you 
actually hold the former position.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 08:47 AM, Eli Zaretskii wrote:
>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Tue, 30 Aug 2016 14:30:13 +0100
>> Cc: Alan Mackenzie <acm@muc.de>, Stefan Monnier <monnier@IRO.UMontreal.CA>,
>> 	emacs-devel@gnu.org
>>
>> I think to really avoid the complexity for mode authors, b-c-f and a-c-f
>> need to not only balance, but also to be consistent. At the moment, they
>> are not that either -- they can signal different locations for a given
>> change.
>
> You are setting the bar impossibly high by expecting that.

To be fair, according to Alan, XEmacs meets this impossibly high bar. 
(And its insdel is considerably simpler.)

In today's world, do we really *need* the optimizations that complicate 
change region tracking? Might these things just be just as unnecessary 
as the old code that used to special-case self-insert-command in the 
event loop?

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 09:22:21 -0700
> 
> > My response is that we should not do it until we have decent test
> > coverage of the insdel functionality.  I simply don't believe in our
> > ability to avoid breakage in this core functionality unless we have
> > good test coverage.  Been there, done that, have bruises to prove it.
> 
> More tests in general would be nice. I have no objection to requiring 
> test coverage for insdel before making major changes; my objection is to 
> a blanket ban on changes even *with* test coverage. I was the under the 
> impression that your preference was for the latter model, but IIUC, you 
> actually hold the former position.

I'm simply disillusioned wrt to possibility of any significant
progress in having tests in these areas.  Almost all of our tests are
in application areas, while the functions in src/ remain almost
completely uncovered.  I have no reason to believe this will change
any time soon (but I will be happy to be mistaken, of course).

That is why I could have sent ambiguous messages in this regard.

(And it's not only insdel that will be affected, because
signal_before_change and signal_after_change also affect the various
caches we maintain for a buffer, its markers, text properties and
overlays, etc.  All of which will have to be covered by tests, before
we can consider serious changes in this area.)

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: acm@muc.de, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 09:23:51 -0700
> 
> > You are setting the bar impossibly high by expecting that.
> 
> To be fair, according to Alan, XEmacs meets this impossibly high bar. 

I disagree with Alan (having looked at XEmacs' sources).

> (And its insdel is considerably simpler.)

That wasn't my impression.  Beyond some stuff that is missing simply
because XEmacs doesn't support it, the code is the same, except that
they also have a separate "multi-change" hook called around changes
that are done piecemeal, in addition to the hooks we have.  IOW, it's
a different model, and not necessarily a solution that we are looking
for.

Of course, I didn't read their sources too seriously, so maybe I
missed something.

> In today's world, do we really *need* the optimizations that complicate 
> change region tracking? Might these things just be just as unnecessary 
> as the old code that used to special-case self-insert-command in the 
> event loop?

I'm not sure which optimizations are you talking about, so it's hard
to answer that question.  I can tell that decoding a large file when
we visit it takes a significant amount of time, even with today's
machines.  So at least some optimizations we have, like reading into
the gap, are probably still justified.

Of course, removing/simplifying optimizations in these internals would
also need good test coverage, to make sure we don't introduce bugs
while at that ;-)

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 30 Aug 2016 12:01:22 -0400
> 
> > You are setting the bar impossibly high by expecting that.
> > Implementing what you want would need significant changes in how the
> > insdel functions are implemented and used.  In particular, some
> > complex changes will probably have to do things twice: once just to
> > figure out which changes are needed, the other time to actually do
> > that while calling the hooks in the way you want.  That's because in
> > some situations we discover the required changes as we go, and have no
> > idea in advance what we will find.
> 
> Actually, we do have some idea beforehand, as evidenced by the fact that
> we call b-c-f with valid data.  It's just that after the fact we have
> a more precise knowledge of what has actually been changed, so a-c-f can
> give more precise bounds.

I was under the impression that Phillip wanted the data be accurate,
not just valid.  Because if validity is the only requirement, we could
always call the before-change hook with the limits of the entire
buffer, and be done.  Clearly, that would not be very useful, to say
the least.

> We could also change the C code to perform this "extension" of the
> region passed to a-c-f so that the b-c-f region and the a-c-f region
> match, but that would make the region passed to a-c-f less precise, so
> those users of a-c-f which don't care about the pairing (the majority of
> the users) would be penalized (tho they'd still work correctly: just
> suboptimally) since there's no way for them to recover the
> tighter region.

Right.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Mon, Aug 29, 2016 at 05:50:11PM +0300, Eli Zaretskii wrote:
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Sun, 28 Aug 2016 20:18:32 -0700

> >     Please trust me that the documentation misleads.

> You are welcome to suggest more accurate wording that describes the
> current implementation.

I think it's only fair to point out that I did precisly this almost
three weeks ago (on 2016-08-10) and the welcome my efforts got was
somewhat less than wholeheartedly warm.

The current documentation misleads in asserting that b-c-f and a-c-f
cannot be used together in balanced pairs.  They most assuredly can,
with the exception of a tiny number of rarely occurring cases.  If this
were not true, CC Mode would not work at all.

Using b-c-f and a-c-f together is an essential technique - there are
things that cannot be done without it, or at least not in any reasonable
manner.  In this sense, maintaining a duplicate shadow buffer, or
scanning from BOB at every buffer change, or maintaining a separate
record of all text properties on a buffer, don't fall into the category
of reasonable, from my point of view.

Eliminating this technique from Emacs, whether by "polluting" the source
to stop it working altogether (as suggested by Stefan), or by forbidding
it in the documentation, has the effect of making Emacs a less powerful
programming system.  I don't think this is what any of us should want.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 10:12 AM, Alan Mackenzie wrote:
> Hello, Eli.
>
> On Mon, Aug 29, 2016 at 05:50:11PM +0300, Eli Zaretskii wrote:
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Date: Sun, 28 Aug 2016 20:18:32 -0700
>
>>>     Please trust me that the documentation misleads.
>
>> You are welcome to suggest more accurate wording that describes the
>> current implementation.
>
> I think it's only fair to point out that I did precisly this almost
> three weeks ago (on 2016-08-10) and the welcome my efforts got was
> somewhat less than wholeheartedly warm.
>
> The current documentation misleads in asserting that b-c-f and a-c-f
> cannot be used together in balanced pairs.  They most assuredly can,
> with the exception of a tiny number of rarely occurring cases.  If this
> were not true, CC Mode would not work at all.
>
> Using b-c-f and a-c-f together is an essential technique - there are
> things that cannot be done without it, or at least not in any reasonable
> manner.  In this sense, maintaining a duplicate shadow buffer, or
> scanning from BOB at every buffer change, or maintaining a separate
> record of all text properties on a buffer, don't fall into the category
> of reasonable, from my point of view.
>
> Eliminating this technique from Emacs, whether by "polluting" the source
> to stop it working altogether (as suggested by Stefan), or by forbidding
> it in the documentation, has the effect of making Emacs a less powerful
> programming system.  I don't think this is what any of us should want.
>
> [ .... ]

How about this? I think this revised text captures the intent of the 
current implementation and preserves the ability to use b-c-f and a-c-f 
together. (The current revert-buffer behavior is just a bug and is 
contrary to my proposed documentation.)

diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi
index 213eec9..6511da8 100644
--- a/doc/lispref/text.texi
+++ b/doc/lispref/text.texi
@@ -4798,6 +4798,13 @@ Change Hooks
  has been changed is always the current buffer when the function is
  called.

+The region given to each of these functions is a conservative
+approximation of the region about to changed.  After running the
+before-change-functions, Emacs will make zero or more fine-grained
+buffer changes and run after-change-functions for each.  Do not expect
+before-change-functions and after-change-functions to be called in
+balanced pairs.
+
  The length of the old text is the difference between the buffer
  positions before and after that text as it was before the change.  As
  for the changed text, its length is simply the difference between the
@@ -4809,14 +4816,14 @@ Change Hooks
  as changes in buffers created by Emacs internally for certain jobs,
  that should not be visible to Lisp programs.

-  Do @emph{not} expect the before-change hooks and the after-change
-hooks be called in balanced pairs around each buffer change.  Also
-don't expect the before-change hooks to be called for every chunk of
-text Emacs is about to delete.  These hooks are provided on the
-assumption that Lisp programs will use either before- or the
-after-change hooks, but not both, and the boundaries of the region
-where the changes happen might include more than just the actual
-changed text, or even lump together several changes done piecemeal.
+   Do @emph{not} expect the before-change hooks and the after-change
+hooks be called in balanced pairs around each buffer change.
+The before-change-functions region is a conservative bound on the zero
+or more fine-grained changes to follow.  Emacs informs user code about
+the actual changes to the buffer through calls to
+after-change-functions; these fine-grained changes will always fall
+inside the broad change region Emacs describes by calling
+before-change-functions.

  @defmac combine-after-change-calls body@dots{}
  The macro executes @var{body} normally, but arranges to call the
diff --git a/src/buffer.c b/src/buffer.c
index 24c997f..60e448f 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -6027,6 +6027,13 @@ the beginning and end of the range of old text to 
be changed.
  \(For an insertion, the beginning and end are at the same place.)
  No information is given about the length of the text after the change.

+The region given to each of these functions is a conservative
+approximation of the region about to changed.  After running the
+before-change-functions, Emacs will make zero or more fine-grained
+buffer changes and run `after-change-functions' for each.  Do not
+expect `before-change-functions' and `after-change-functions' to be
+called in balanced pairs.
+
  Buffer changes made while executing the `before-change-functions'
  don't call any before-change or after-change functions.
  That's because `inhibit-modification-hooks' is temporarily set non-nil.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Tue, 30 Aug 2016 17:12:22 +0000
> Cc: Daniel Colascione <dancol@dancol.org>, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > You are welcome to suggest more accurate wording that describes the
> > current implementation.
> 
> I think it's only fair to point out that I did precisly this almost
> three weeks ago (on 2016-08-10) and the welcome my efforts got was
> somewhat less than wholeheartedly warm.

Your attempt didn't describe the current implementation.  It described
what you thought it should be.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 30, 2016 at 08:27:48PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 30 Aug 2016 17:12:22 +0000
> > Cc: Daniel Colascione <dancol@dancol.org>, monnier@iro.umontreal.ca,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > You are welcome to suggest more accurate wording that describes the
> > > current implementation.

> > I think it's only fair to point out that I did precisly this almost
> > three weeks ago (on 2016-08-10) and the welcome my efforts got was
> > somewhat less than wholeheartedly warm.

> Your attempt didn't describe the current implementation.  It described
> what you thought it should be.

The text in question (intended for the Elisp manual) was:

  Normally the before-change-functions and the after-change-functions
  hooks are called in balanced pairs around each buffer change, enabling
  state to be determined in a before-change function and used in an
  after-change function.  However on rare occasions,
  before-change-functions is not called before a buffer change and
  after-change-functions is thus called twice in succession.  Your
  program should be ready to deal with such an occurrence.

The crucial part of that is the first sentence.  It is true.

The second sentence is true in essence, though it possibly needs to be
made more accurate to deal with other cases.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 10:27:45 -0700
> 
> +The region given to each of these functions is a conservative
> +approximation of the region about to changed.  After running the
> +before-change-functions, Emacs will make zero or more fine-grained
> +buffer changes and run after-change-functions for each.  Do not expect
> +before-change-functions and after-change-functions to be called in
> +balanced pairs.

The last sentence here is repeated afterwards, for no good reason.
(Also, the markup is missing, but that's just an aside.)

> -  Do @emph{not} expect the before-change hooks and the after-change
> -hooks be called in balanced pairs around each buffer change.  Also
> -don't expect the before-change hooks to be called for every chunk of
> -text Emacs is about to delete.  These hooks are provided on the
> -assumption that Lisp programs will use either before- or the
> -after-change hooks, but not both, and the boundaries of the region
> -where the changes happen might include more than just the actual
> -changed text, or even lump together several changes done piecemeal.
> +   Do @emph{not} expect the before-change hooks and the after-change
> +hooks be called in balanced pairs around each buffer change.
> +The before-change-functions region is a conservative bound on the zero
> +or more fine-grained changes to follow.  Emacs informs user code about
> +the actual changes to the buffer through calls to
> +after-change-functions; these fine-grained changes will always fall
> +inside the broad change region Emacs describes by calling
> +before-change-functions.

You removed the part about text deletion, which is not specific to
revert-buffer, so that information is now lost.  I don't want to lose
it.

Other than that, I don't see how your text is more accurate, it's just
a different wording dancing around the same issues trying to side-step
them by replacing one vague description by another.

If all you want is to remove this part:

  These hooks are provided on the assumption that Lisp programs will
  use either before- or the after-change hooks, but not both

then I don't necessarily mind, although I do believe it is true, and
the readers should be aware of that.

Thanks.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 30 Aug 2016 10:27:45 -0700
>>
>> +The region given to each of these functions is a conservative
>> +approximation of the region about to changed.  After running the
>> +before-change-functions, Emacs will make zero or more fine-grained
>> +buffer changes and run after-change-functions for each.  Do not expect
>> +before-change-functions and after-change-functions to be called in
>> +balanced pairs.
>
> The last sentence here is repeated afterwards, for no good reason.
> (Also, the markup is missing, but that's just an aside.)

I figured it was a good idea to highlight this fact directly in the 
variable documentation blob. I can add a "see below" link.

>
>> -  Do @emph{not} expect the before-change hooks and the after-change
>> -hooks be called in balanced pairs around each buffer change.  Also
>> -don't expect the before-change hooks to be called for every chunk of
>> -text Emacs is about to delete.  These hooks are provided on the
>> -assumption that Lisp programs will use either before- or the
>> -after-change hooks, but not both, and the boundaries of the region
>> -where the changes happen might include more than just the actual
>> -changed text, or even lump together several changes done piecemeal.
>> +   Do @emph{not} expect the before-change hooks and the after-change
>> +hooks be called in balanced pairs around each buffer change.
>> +The before-change-functions region is a conservative bound on the zero
>> +or more fine-grained changes to follow.  Emacs informs user code about
>> +the actual changes to the buffer through calls to
>> +after-change-functions; these fine-grained changes will always fall
>> +inside the broad change region Emacs describes by calling
>> +before-change-functions.
>
> You removed the part about text deletion, which is not specific to
> revert-buffer, so that information is now lost.  I don't want to lose
> it.

The text deletion part is a real and serious bug. As Stefan points out, 
it makes it impossible to use b-c-f to invalidate caches.

>
> Other than that, I don't see how your text is more accurate, it's just
> a different wording dancing around the same issues trying to side-step
> them by replacing one vague description by another.

My proposed description highlights how the b-c-f region contains the 
a-c-f regions. I understand that you believe that the existing 
documentation communicates this fact, but I strongly disagree. The 
relationship between the b-c-f region and the a-c-f regions needs to be 
spelled out explicitly.


>
> If all you want is to remove this part:
>
>   These hooks are provided on the assumption that Lisp programs will
>   use either before- or the after-change hooks, but not both
>
> then I don't necessarily mind, although I do believe it is true, and
> the readers should be aware of that.

I strongly disagree. b-c-f is a perfectly good way to invalidate caches.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Tue, 30 Aug 2016 17:42:24 +0000
> Cc: dancol@dancol.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Your attempt didn't describe the current implementation.  It described
> > what you thought it should be.
> 
> The text in question (intended for the Elisp manual) was:
> 
>   Normally the before-change-functions and the after-change-functions
>   hooks are called in balanced pairs around each buffer change, enabling
>   state to be determined in a before-change function and used in an
>   after-change function.  However on rare occasions,
>   before-change-functions is not called before a buffer change and
>   after-change-functions is thus called twice in succession.  Your
>   program should be ready to deal with such an occurrence.
> 
> The crucial part of that is the first sentence.  It is true.

We use "normally" in the manual to mean "always, unless changed by
customizations".  So the above use of that word is extremely confusing
and thus a bad idea.

And the part about "balanced pairs" is simply false, if "balanced" is
to be interpreted in its everyday's meaning.

> The second sentence is true in essence, though it possibly needs to be
> made more accurate to deal with other cases.

The current text is even more "true in essence", though.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> > You are setting the bar impossibly high by expecting that.
>> > Implementing what you want would need significant changes in how the
>> > insdel functions are implemented and used.  In particular, some
>> > complex changes will probably have to do things twice: once just to
>> > figure out which changes are needed, the other time to actually do
>> > that while calling the hooks in the way you want.  That's because in
>> > some situations we discover the required changes as we go, and have no
>> > idea in advance what we will find.
>> Actually, we do have some idea beforehand, as evidenced by the fact that
>> we call b-c-f with valid data.  It's just that after the fact we have
>> a more precise knowledge of what has actually been changed, so a-c-f can
>> give more precise bounds.
> I was under the impression that Phillip wanted the data be accurate,
> not just valid.

More precise (i.e. tighter bounds) is better, but valid is indispensable
(I used "valid" to mean that every change is covered by the bounds, even
though those bounds may be too lose.  I consider "invalid" to be a plain
bug, as in insert-file-contents, for example).

In Phillip's case he'd prefer to have suboptimal data than mispaired
data since in the mispaired case he just falls back to "redo
everything", which is equivalent to calling b/a-c-f with
point-min/point-max.  So the message you quoted just pointed out that as
long as b/a-c-f are paired (actually, you just need the guarantee that
to each b-c-f corresponds at most one a-c-f), you can manually make
those pairs "proper" (i.e. with identical bounds) inside your own a-c-f.

So I'm pretty sure he's more concerned about "proper pairing" than about
"utmost tightness of the bounds".

> Because if validity is the only requirement, we could always call the
> before-change hook with the limits of the entire buffer, and be done.
> Clearly, that would not be very useful, to say the least.

Indeed.  But in the case of subst-chars-in-region, what we pass to b-c-f
is suboptimal but better than point-min/max, so there's a useful middle
point between the difficult "tightest bound" and the degenerate "just
pass point-min/max".


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: acm@muc.de, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 10:46:44 -0700
> 
> >> -  Do @emph{not} expect the before-change hooks and the after-change
> >> -hooks be called in balanced pairs around each buffer change.  Also
> >> -don't expect the before-change hooks to be called for every chunk of
> >> -text Emacs is about to delete.  These hooks are provided on the
> >> -assumption that Lisp programs will use either before- or the
> >> -after-change hooks, but not both, and the boundaries of the region
> >> -where the changes happen might include more than just the actual
> >> -changed text, or even lump together several changes done piecemeal.
> >> +   Do @emph{not} expect the before-change hooks and the after-change
> >> +hooks be called in balanced pairs around each buffer change.
> >> +The before-change-functions region is a conservative bound on the zero
> >> +or more fine-grained changes to follow.  Emacs informs user code about
> >> +the actual changes to the buffer through calls to
> >> +after-change-functions; these fine-grained changes will always fall
> >> +inside the broad change region Emacs describes by calling
> >> +before-change-functions.
> >
> > You removed the part about text deletion, which is not specific to
> > revert-buffer, so that information is now lost.  I don't want to lose
> > it.
> 
> The text deletion part is a real and serious bug. As Stefan points out, 
> it makes it impossible to use b-c-f to invalidate caches.

You misunderstand what Stefan says.  He says not calling the
before-change hook _at_all_ is a bug.  Not calling it for every chunk
of deleted text is not necessarily a bug, if there's a previous less
fine-grained call to the hook.  And that's what the text above
conveys: that note every chunk to be deleted will have its own call to
a hook.

> > Other than that, I don't see how your text is more accurate, it's just
> > a different wording dancing around the same issues trying to side-step
> > them by replacing one vague description by another.
> 
> My proposed description highlights how the b-c-f region contains the 
> a-c-f regions. I understand that you believe that the existing 
> documentation communicates this fact, but I strongly disagree. The 
> relationship between the b-c-f region and the a-c-f regions needs to be 
> spelled out explicitly.

They cannot be spelled out explicitly without going into a lot more
internal details that are inappropriate for the Lisp-level manual.

> > If all you want is to remove this part:
> >
> >   These hooks are provided on the assumption that Lisp programs will
> >   use either before- or the after-change hooks, but not both
> >
> > then I don't necessarily mind, although I do believe it is true, and
> > the readers should be aware of that.
> 
> I strongly disagree. b-c-f is a perfectly good way to invalidate caches.

So the readers need to know they cannot rely on that.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> +The region given to each of these functions is a conservative
> +approximation of the region about to changed.  After running the
> +before-change-functions, Emacs will make zero or more fine-grained
> +buffer changes and run after-change-functions for each.  Do not expect
> +before-change-functions and after-change-functions to be called in
> +balanced pairs.

Looks OK to me.

> -  Do @emph{not} expect the before-change hooks and the after-change
> -hooks be called in balanced pairs around each buffer change.  Also
> -don't expect the before-change hooks to be called for every chunk of
> -text Emacs is about to delete.  These hooks are provided on the
> -assumption that Lisp programs will use either before- or the
> -after-change hooks, but not both, and the boundaries of the region
> -where the changes happen might include more than just the actual
> -changed text, or even lump together several changes done piecemeal.
> +   Do @emph{not} expect the before-change hooks and the after-change
> +hooks be called in balanced pairs around each buffer change.
> +The before-change-functions region is a conservative bound on the zero
> +or more fine-grained changes to follow.  Emacs informs user code about
> +the actual changes to the buffer through calls to
> +after-change-functions; these fine-grained changes will always fall
> +inside the broad change region Emacs describes by calling
> +before-change-functions.

Repeats the previous, but otherwise looks fine to me.
thanks,


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Daniel.

On Tue, Aug 30, 2016 at 10:46:44AM -0700, Daniel Colascione wrote:
> On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
> >> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> >> From: Daniel Colascione <dancol@dancol.org>
> >> Date: Tue, 30 Aug 2016 10:27:45 -0700

> >> +The region given to each of these functions is a conservative
> >> +approximation of the region about to changed.  After running the
> >> +before-change-functions, Emacs will make zero or more fine-grained
> >> +buffer changes and run after-change-functions for each.  Do not expect
> >> +before-change-functions and after-change-functions to be called in
> >> +balanced pairs.

> > The last sentence here is repeated afterwards, for no good reason.
> > (Also, the markup is missing, but that's just an aside.)

> I figured it was a good idea to highlight this fact directly in the 
> variable documentation blob. I can add a "see below" link.

Why are you advocating this?  It is not true.  You _can_ expect b-c-f and
a-c-f to be balanced in all but, perhaps, one occurrence per million.  It
happens so seldom that in practice, one can assume that b-c-f and a-c-f
match completely[*].  You are describing the exception as though it were the
typical case.

[*] provided the exceptions are handled somehow.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 11:00 AM, Eli Zaretskii wrote:
>> Cc: acm@muc.de, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 30 Aug 2016 10:46:44 -0700
>>
>>>> -  Do @emph{not} expect the before-change hooks and the after-change
>>>> -hooks be called in balanced pairs around each buffer change.  Also
>>>> -don't expect the before-change hooks to be called for every chunk of
>>>> -text Emacs is about to delete.  These hooks are provided on the
>>>> -assumption that Lisp programs will use either before- or the
>>>> -after-change hooks, but not both, and the boundaries of the region
>>>> -where the changes happen might include more than just the actual
>>>> -changed text, or even lump together several changes done piecemeal.
>>>> +   Do @emph{not} expect the before-change hooks and the after-change
>>>> +hooks be called in balanced pairs around each buffer change.
>>>> +The before-change-functions region is a conservative bound on the zero
>>>> +or more fine-grained changes to follow.  Emacs informs user code about
>>>> +the actual changes to the buffer through calls to
>>>> +after-change-functions; these fine-grained changes will always fall
>>>> +inside the broad change region Emacs describes by calling
>>>> +before-change-functions.
>>>
>>> You removed the part about text deletion, which is not specific to
>>> revert-buffer, so that information is now lost.  I don't want to lose
>>> it.
>>
>> The text deletion part is a real and serious bug. As Stefan points out,
>> it makes it impossible to use b-c-f to invalidate caches.
>
> You misunderstand what Stefan says.  He says not calling the
> before-change hook _at_all_ is a bug.  Not calling it for every chunk
> of deleted text is not necessarily a bug, if there's a previous less
> fine-grained call to the hook.  And that's what the text above
> conveys: that note every chunk to be deleted will have its own call to
> a hook.

So we're in agreement? True or false: b-c-f ought to be a conservative 
bound on subsequent a-c-f calls.

>
>>> Other than that, I don't see how your text is more accurate, it's just
>>> a different wording dancing around the same issues trying to side-step
>>> them by replacing one vague description by another.
>>
>> My proposed description highlights how the b-c-f region contains the
>> a-c-f regions. I understand that you believe that the existing
>> documentation communicates this fact, but I strongly disagree. The
>> relationship between the b-c-f region and the a-c-f regions needs to be
>> spelled out explicitly.
>
> They cannot be spelled out explicitly without going into a lot more
> internal details that are inappropriate for the Lisp-level manual.

Not the case at all. That's the point of saying b-c-f is a conservative 
bound. The words "conservative bound" free you from having to describe 
the precise ways the b-c-f region and the a-c-f region can differ.

>
>>> If all you want is to remove this part:
>>>
>>>   These hooks are provided on the assumption that Lisp programs will
>>>   use either before- or the after-change hooks, but not both
>>>
>>> then I don't necessarily mind, although I do believe it is true, and
>>> the readers should be aware of that.
>>
>> I strongly disagree. b-c-f is a perfectly good way to invalidate caches.
>
> So the readers need to know they cannot rely on that.

Why shouldn't they be able to rely on that? What *should* they use to 
invalidate caches? Your position is not very clear to me.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 11:01 AM, Alan Mackenzie wrote:
> Hello, Daniel.
>
> On Tue, Aug 30, 2016 at 10:46:44AM -0700, Daniel Colascione wrote:
>> On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
>>>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>>> From: Daniel Colascione <dancol@dancol.org>
>>>> Date: Tue, 30 Aug 2016 10:27:45 -0700
>
>>>> +The region given to each of these functions is a conservative
>>>> +approximation of the region about to changed.  After running the
>>>> +before-change-functions, Emacs will make zero or more fine-grained
>>>> +buffer changes and run after-change-functions for each.  Do not expect
>>>> +before-change-functions and after-change-functions to be called in
>>>> +balanced pairs.
>
>>> The last sentence here is repeated afterwards, for no good reason.
>>> (Also, the markup is missing, but that's just an aside.)
>
>> I figured it was a good idea to highlight this fact directly in the
>> variable documentation blob. I can add a "see below" link.
>
> Why are you advocating this?  It is not true.  You _can_ expect b-c-f and
> a-c-f to be balanced in all but, perhaps, one occurrence per million.  It
> happens so seldom that in practice, one can assume that b-c-f and a-c-f
> match completely[*].  You are describing the exception as though it were the
> typical case.
>
> [*] provided the exceptions are handled somehow.
>
> [ .... ]
>

I'd vastly prefer the calls to be balanced. It looks like it will be 
difficult to make them that way, so describing in the manual behavior 
that preserves most of the goodness and spells out the workarounds mode 
authors need to use seems like the least awful approach.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>   Normally the before-change-functions and the after-change-functions
>   hooks are called in balanced pairs around each buffer change,

I could agree with that.

>   enabling state to be determined in a before-change function and used in an
>   after-change function.

But this encourages reliance on this pairing, even though experience
shows that it's better to try not to rely on it (both in the sense that
I personally believe that it results in better designs, and in the
sense that such designs will break down in the non-"normal" case).

So I'm strongly opposed to such statements.

>   However on rare occasions, before-change-functions is not called
>   before a buffer change and

That would be documenting bugs as being normal behavior.

>   after-change-functions is thus called twice in succession.  Your
>   program should be ready to deal with such an occurrence.

Two successive calls of a-c-f are indeed acceptable and code should be
ready to deal with such occurrence, but only because a previous b-c-f
call covered both changes, not because b-c-f failed to be called.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 30, 2016 at 08:53:48PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 30 Aug 2016 17:42:24 +0000
> > Cc: dancol@dancol.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Your attempt didn't describe the current implementation.  It described
> > > what you thought it should be.

> > The text in question (intended for the Elisp manual) was:

> >   Normally the before-change-functions and the after-change-functions
> >   hooks are called in balanced pairs around each buffer change, enabling
> >   state to be determined in a before-change function and used in an
> >   after-change function.  However on rare occasions,
> >   before-change-functions is not called before a buffer change and
> >   after-change-functions is thus called twice in succession.  Your
> >   program should be ready to deal with such an occurrence.
 
> > The crucial part of that is the first sentence.  It is true.

> We use "normally" in the manual to mean "always, unless changed by
> customizations".  So the above use of that word is extremely confusing
> and thus a bad idea.

  The before-change-functions and after-change-functions hooks are
  almost always called in balanced pairs around each buffer change,
  enabling state to be determined in a before-change-function and used
  in an after-change-function.  However, on rare occasions, ....

> And the part about "balanced pairs" is simply false, if "balanced" is
> to be interpreted in its everyday's meaning.

I don't understand what you mean.  By balanced, I essentially mean that
(i) b-c-f and a-c-f alternate,
(ii) b-c-f's beg is the same as a-c-f's beg,
(iii) b-c-f's (- end beg) is equal to a-c-f's old-len.

In that sense b/a-c-f are balanced in the overwhelming majority of
cases.


> > The second sentence is true in essence, though it possibly needs to be
> > made more accurate to deal with other cases.

> The current text is even more "true in essence", though.

No it's not.  It describes the exceptional and rare case as though it
were common.  It implies that using b/a-c-f together is not a
practicable programming technique, whereas the opposite is true.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On 08/30/2016 11:06 AM, Daniel Colascione wrote:
> On 08/30/2016 11:01 AM, Alan Mackenzie wrote:
>> Hello, Daniel.
>>
>> On Tue, Aug 30, 2016 at 10:46:44AM -0700, Daniel Colascione wrote:
>>> On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
>>>>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>>>> From: Daniel Colascione <dancol@dancol.org>
>>>>> Date: Tue, 30 Aug 2016 10:27:45 -0700
>>
>>>>> +The region given to each of these functions is a conservative
>>>>> +approximation of the region about to changed.  After running the
>>>>> +before-change-functions, Emacs will make zero or more fine-grained
>>>>> +buffer changes and run after-change-functions for each.  Do not
>>>>> expect
>>>>> +before-change-functions and after-change-functions to be called in
>>>>> +balanced pairs.
>>
>>>> The last sentence here is repeated afterwards, for no good reason.
>>>> (Also, the markup is missing, but that's just an aside.)
>>
>>> I figured it was a good idea to highlight this fact directly in the
>>> variable documentation blob. I can add a "see below" link.
>>
>> Why are you advocating this?  It is not true.

It's not true. If it's true except for one time in a million, it's not 
true. Programs need to be written for the world we inhabit, not the one 
we want. Relying on behavior that's usually but not always the case is 
antithetical to software robustness.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> -  Do @emph{not} expect the before-change hooks and the after-change
>> -hooks be called in balanced pairs around each buffer change.  Also
>> -don't expect the before-change hooks to be called for every chunk of
>> -text Emacs is about to delete.  These hooks are provided on the
>> -assumption that Lisp programs will use either before- or the
>> -after-change hooks, but not both, and the boundaries of the region
>> -where the changes happen might include more than just the actual
>> -changed text, or even lump together several changes done piecemeal.
>> +   Do @emph{not} expect the before-change hooks and the after-change
>> +hooks be called in balanced pairs around each buffer change.
>> +The before-change-functions region is a conservative bound on the zero
>> +or more fine-grained changes to follow.  Emacs informs user code about
>> +the actual changes to the buffer through calls to
>> +after-change-functions; these fine-grained changes will always fall
>> +inside the broad change region Emacs describes by calling
>> +before-change-functions.

> You removed the part about text deletion, which is not specific to
> revert-buffer, so that information is now lost.  I don't want to
> lose it.

I do want to lose it, because this failure to call b-c-f is a bug, not
something part of the intended behavior.

I could live with something like "There are still some known bugs in
which b-c-f fails to be called", tho.

> Other than that, I don't see how your text is more accurate, it's just
> a different wording dancing around the same issues trying to side-step
> them by replacing one vague description by another.

They do seem mostly equivalent, so if it can satisfy one of the
complainant I'm happy to accept the change ;-)


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> You misunderstand what Stefan says.  He says not calling the
> before-change hook _at_all_ is a bug.  Not calling it for every chunk
> of deleted text is not necessarily a bug, if there's a previous less
> fine-grained call to the hook.  And that's what the text above
> conveys: that note every chunk to be deleted will have its own call to
> a hook.

FWIW, when I read

    hooks be called in balanced pairs around each buffer change.  Also
    don't expect the before-change hooks to be called for every chunk of
    text Emacs is about to delete.  These hooks are provided on the

I do understand this to mean "b-c-f is just unreliable" and more
specifically it does sound to me like it refers to the known
insert-file-contents bug.  If that was not your intention, then consider
this as evidence that a rewording could be beneficial.

>> My proposed description highlights how the b-c-f region contains the 
>> a-c-f regions. I understand that you believe that the existing 
>> documentation communicates this fact, but I strongly disagree. The 
>> relationship between the b-c-f region and the a-c-f regions needs to be 
>> spelled out explicitly.
> They cannot be spelled out explicitly without going into a lot more
> internal details that are inappropriate for the Lisp-level manual.

He considers his text to spell it in enough detail.  So unless you
disagree with the text itself, I think his text is an improvement: you
both agree with the validity and level of precision of the description
in his new text, which is not the case with the current text.

>> I strongly disagree. b-c-f is a perfectly good way to invalidate caches.
> So the readers need to know they cannot rely on that.

syntax-ppss relies on it, so we had better make sure we can rely on
it ;-0


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Daniel.

On Tue, Aug 30, 2016 at 11:17:48AM -0700, Daniel Colascione wrote:
> On 08/30/2016 11:06 AM, Daniel Colascione wrote:
> > On 08/30/2016 11:01 AM, Alan Mackenzie wrote:

> >> On Tue, Aug 30, 2016 at 10:46:44AM -0700, Daniel Colascione wrote:
> >>> On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
> >>>>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> >>>>> From: Daniel Colascione <dancol@dancol.org>
> >>>>> Date: Tue, 30 Aug 2016 10:27:45 -0700

> >>>>> +The region given to each of these functions is a conservative
> >>>>> +approximation of the region about to changed.  After running the
> >>>>> +before-change-functions, Emacs will make zero or more fine-grained
> >>>>> +buffer changes and run after-change-functions for each.  Do not
> >>>>> expect
> >>>>> +before-change-functions and after-change-functions to be called in
> >>>>> +balanced pairs.

> >>>> The last sentence here is repeated afterwards, for no good reason.
> >>>> (Also, the markup is missing, but that's just an aside.)

> >>> I figured it was a good idea to highlight this fact directly in the
> >>> variable documentation blob. I can add a "see below" link.

> >> Why are you advocating this?  It is not true.

> It's not true. If it's true except for one time in a million, it's not 
> true.

That's straying off-topic into philosophy.

> Programs need to be written for the world we inhabit, not the one 
> we want. Relying on behavior that's usually but not always the case is 
> antithetical to software robustness.

Coming back to the concrete, your proposed way of expressing things
would prevent future hackers from using b-c-f and a-c-f together the way
that we do.  Is that what you want?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On August 30, 2016 11:30:09 AM PDT, Alan Mackenzie <acm@muc.de> wrote:
>Hello, Daniel.
>
>On Tue, Aug 30, 2016 at 11:17:48AM -0700, Daniel Colascione wrote:
>> On 08/30/2016 11:06 AM, Daniel Colascione wrote:
>> > On 08/30/2016 11:01 AM, Alan Mackenzie wrote:
>
>> >> On Tue, Aug 30, 2016 at 10:46:44AM -0700, Daniel Colascione wrote:
>> >>> On 08/30/2016 10:42 AM, Eli Zaretskii wrote:
>> >>>>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> >>>>> From: Daniel Colascione <dancol@dancol.org>
>> >>>>> Date: Tue, 30 Aug 2016 10:27:45 -0700
>
>> >>>>> +The region given to each of these functions is a conservative
>> >>>>> +approximation of the region about to changed.  After running
>the
>> >>>>> +before-change-functions, Emacs will make zero or more
>fine-grained
>> >>>>> +buffer changes and run after-change-functions for each.  Do
>not
>> >>>>> expect
>> >>>>> +before-change-functions and after-change-functions to be
>called in
>> >>>>> +balanced pairs.
>
>> >>>> The last sentence here is repeated afterwards, for no good
>reason.
>> >>>> (Also, the markup is missing, but that's just an aside.)
>
>> >>> I figured it was a good idea to highlight this fact directly in
>the
>> >>> variable documentation blob. I can add a "see below" link.
>
>> >> Why are you advocating this?  It is not true.
>
>> It's not true. If it's true except for one time in a million, it's
>not 
>> true.
>
>That's straying off-topic into philosophy.

Just defining terms

>
>> Programs need to be written for the world we inhabit, not the one 
>> we want. Relying on behavior that's usually but not always the case
>is 
>> antithetical to software robustness.
>
>Coming back to the concrete, your proposed way of expressing things
>would prevent future hackers from using b-c-f and a-c-f together the
>way
>that we do.  Is that what you want?

It doesn't prevent them per second: it just forces them to be more careful. I'd prefer that these hooks work the way we both want, but if that's not the case, I'd rather not mislead developers. It's not the documentation that prevents this use, but the Emacs core

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> Coming back to the concrete, your proposed way of expressing things
> would prevent future hackers from using b-c-f and a-c-f together the way
> that we do.  Is that what you want?

Not at all.  It just tells them what could happen in the general case,
so they can properly detect the general case and know what to do when
it happens.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: acm@muc.de, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 11:04:07 -0700
> 
> > You misunderstand what Stefan says.  He says not calling the
> > before-change hook _at_all_ is a bug.  Not calling it for every chunk
> > of deleted text is not necessarily a bug, if there's a previous less
> > fine-grained call to the hook.  And that's what the text above
> > conveys: that note every chunk to be deleted will have its own call to
> > a hook.
> 
> So we're in agreement? True or false: b-c-f ought to be a conservative 
> bound on subsequent a-c-f calls.

Of course we are in agreement, about the essence.  Your text saus the
same as mine, except that I don't find "conservative bound" to be more
helpful than what I wrote, quite the contrary.

> >> I strongly disagree. b-c-f is a perfectly good way to invalidate caches.
> >
> > So the readers need to know they cannot rely on that.
> 
> Why shouldn't they be able to rely on that? What *should* they use to 
> invalidate caches? Your position is not very clear to me.

Stefan described the alternatives up-thread.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Daniel.

On Tue, Aug 30, 2016 at 11:32:42AM -0700, Daniel Colascione wrote:


> On August 30, 2016 11:30:09 AM PDT, Alan Mackenzie <acm@muc.de> wrote:

[ .... ]

> >> Programs need to be written for the world we inhabit, not the one 
> >> we want. Relying on behavior that's usually but not always the case
> >> is antithetical to software robustness.

We build error handling into software.  Coping with non-matching b-c-f
and a-c-f calls is simply error handling.

> >Coming back to the concrete, your proposed way of expressing things
> >would prevent future hackers from using b-c-f and a-c-f together the
> >way that we do.  Is that what you want?

> It doesn't prevent them per second: it just forces them to be more careful.

Your wording would prevent such use, so clearly does it express that
b-c-f and a-c-f frequently don't match.

> I'd prefer that these hooks work the way we both want, but if that's
> not the case, I'd rather not mislead developers. It's not the
> documentation that prevents this use, but the Emacs core

CC Mode works.  The Emacs core does not prevent it working.  Nor does it
prevent your or Phillip's software from working.

You evaded the question, so I'll put it to you again: do you want to
prevent future hackers from  using b-c-f and a-c-f together the way we
do?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Date: Tue, 30 Aug 2016 18:16:38 +0000
> Cc: dancol@dancol.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > And the part about "balanced pairs" is simply false, if "balanced" is
> > to be interpreted in its everyday's meaning.
> 
> I don't understand what you mean.  By balanced, I essentially mean that
> (i) b-c-f and a-c-f alternate,
> (ii) b-c-f's beg is the same as a-c-f's beg,
> (iii) b-c-f's (- end beg) is equal to a-c-f's old-len.

That's just it: the above happens only in "simple" changes, like a
simple deletion or insertion.  Any more complex changes don't do it
"balanced" according to your definition.

Look, do we have to reiterate this endless discussion that leads
nowhere?  I wasted all my free time today reading and replying to
messages about this, something that I thought was closed long ago.
Important patches that were ready to be applied and pushed are still
waiting because I didn't have time for them.

I suggest that we leave it alone -- again.

Thanks.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On August 30, 2016 11:47:49 AM PDT, Alan Mackenzie <acm@muc.de> wrote:
>Hello, Daniel.
>
>On Tue, Aug 30, 2016 at 11:32:42AM -0700, Daniel Colascione wrote:
>
>
>> On August 30, 2016 11:30:09 AM PDT, Alan Mackenzie <acm@muc.de>
>wrote:
>
>[ .... ]
>
>> >> Programs need to be written for the world we inhabit, not the one 
>> >> we want. Relying on behavior that's usually but not always the
>case
>> >> is antithetical to software robustness.
>
>We build error handling into software.  Coping with non-matching b-c-f
>and a-c-f calls is simply error handling.
>
>> >Coming back to the concrete, your proposed way of expressing things
>> >would prevent future hackers from using b-c-f and a-c-f together the
>> >way that we do.  Is that what you want?
>
>> It doesn't prevent them per second: it just forces them to be more
>careful.
>
>Your wording would prevent such use, so clearly does it express that
>b-c-f and a-c-f frequently don't match.
>
>> I'd prefer that these hooks work the way we both want, but if that's
>> not the case, I'd rather not mislead developers. It's not the
>> documentation that prevents this use, but the Emacs core
>
>CC Mode works.  The Emacs core does not prevent it working.  Nor does
>it
>prevent your or Phillip's software from working.
>
>You evaded the question, so I'll put it to you again: do you want to
>prevent future hackers from  using b-c-f and a-c-f together the way we
>do?

Yes, because the approach we use today is broken and there's little hope in fixing it. The language in the documentation doesn't change what Emacs does. I don't understand what's so hard to see. You want to lead developers into writing code against an Emacs that doesn't exist

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Daniel Colascione]

On August 30, 2016 11:46:24 AM PDT, Eli Zaretskii <eliz@gnu.org> wrote:
>> Cc: acm@muc.de, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 30 Aug 2016 11:04:07 -0700
>> 
>> > You misunderstand what Stefan says.  He says not calling the
>> > before-change hook _at_all_ is a bug.  Not calling it for every
>chunk
>> > of deleted text is not necessarily a bug, if there's a previous
>less
>> > fine-grained call to the hook.  And that's what the text above
>> > conveys: that note every chunk to be deleted will have its own call
>to
>> > a hook.
>> 
>> So we're in agreement? True or false: b-c-f ought to be a
>conservative 
>> bound on subsequent a-c-f calls.
>
>Of course we are in agreement, about the essence.  Your text saus the
>same as mine, except that I don't find "conservative bound" to be more
>helpful than what I wrote, quite the contrary.

Your text is silent on the relationship between the regions given to b-c-f and the ones given to b-c-f. I don't understand what's so hard to see. My proposed change tightens the contract between the Emacs core and developers and does it in a way that doesn't impose additional requirements on the current implementation.


>
>> >> I strongly disagree. b-c-f is a perfectly good way to invalidate
>caches.
>> >
>> > So the readers need to know they cannot rely on that.
>> 
>> Why shouldn't they be able to rely on that? What *should* they use to
>
>> invalidate caches? Your position is not very clear to me.
>
>Stefan described the alternatives up-thread.


Those alternatives are all incredibly awkward

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 30, 2016 at 09:51:20PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 30 Aug 2016 18:16:38 +0000
> > Cc: dancol@dancol.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> Look, do we have to reiterate this endless discussion that leads
> nowhere?  I wasted all my free time today reading and replying to
> messages about this, something that I thought was closed long ago.
> Important patches that were ready to be applied and pushed are still
> waiting because I didn't have time for them.

> I suggest that we leave it alone -- again.

I think these matters are important.  Very important.  Critically
important even, because they set an upper bound on Emacs's future
capabilities.

But I think we've both said everything that's to be said, so OK, let's
leave it alone.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Daniel.

On Tue, Aug 30, 2016 at 11:55:41AM -0700, Daniel Colascione wrote:


> On August 30, 2016 11:47:49 AM PDT, Alan Mackenzie <acm@muc.de> wrote:

> >Do you want to prevent future hackers from  using b-c-f and a-c-f
> >together the way we do?

> Yes, because the approach we use today is broken and there's little
> hope in fixing it.

There is software that cannot be written in any reasonable fashion
without the technique under discussion.  I suspect, even if I don't know
for sure, that CC Mode comes into that category.  Any lisp program which
requires full details of a change cannot be reasonably written without the
technique.

Documenting, falsely, that it doesn't work restricts the set of what can
be written for Emacs.

> The language in the documentation doesn't change what Emacs does. I
> don't understand what's so hard to see. You want to lead developers
> into writing code against an Emacs that doesn't exist.

No, I quite clearly and accurately described the Emacs that does exist:
The technique works almost always, but you have to detect and handle
exceptions carefully, something that can be easily done.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 30 Aug 2016 11:58:04 -0700
> CC: acm@muc.de,monnier@iro.umontreal.ca,emacs-devel@gnu.org
> 
> Your text is silent on the relationship between the regions given to b-c-f and the ones given to b-c-f.

It's not silent.  It tells the reader specifically what assumptions
are invalid.  I think that is much more useful in practice than
talking about conservative bounds, something that is more abstract and
therefore more easily misinterpreted/misunderstood.

> My proposed change tightens the contract between the Emacs core and developers and does it in a way that doesn't impose additional requirements on the current implementation.

With the current imperfect implementation, I don't see how that is an
advantage in practice.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> There is software that cannot be written in any reasonable fashion
> without the technique under discussion.  I suspect, even if I don't know
> for sure, that CC Mode comes into that category.

It's trivial to demonstrate that it doesn't:
- consider any change between START and END to the buffer.
- you can always decompose it into:
  (a) delete everything from START to point-max.
  (b) insert at START (now point-max) the rest of the text.
- clearly the above deletions and insertions *could* happen, so CC-mode
  has to handle them and since all changes could be decomposed this way,
  CC-mode does not *need* to handle any other case.
- to handle (a) you don't need to know anything from b-c-f.
- to handle (b) you don't need to know anything from b-c-f.

CQFD

> No, I quite clearly and accurately described the Emacs that does exist:
> The technique works almost always, but you have to detect and handle
> exceptions carefully, something that can be easily done.

Daniel's description gives you the added info necessary to know how to
detect that situation and what you might want to do about it.

FWIW, I like Daniel's wording.  It is about as precise as we can make it
without imposing undue burden (IOW I think Emacs's implementation can
reasonably (and should) obey that description) and I think it gives the
right intuition.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Alan Mackenzie]

Hello, Stefan.

On Tue, Aug 30, 2016 at 04:34:46PM -0400, Stefan Monnier wrote:
> > There is software that cannot be written in any reasonable fashion
> > without the technique under discussion.  I suspect, even if I don't know
> > for sure, that CC Mode comes into that category.

> It's trivial to demonstrate that it doesn't:
> - consider any change between START and END to the buffer.
> - you can always decompose it into:
>   (a) delete everything from START to point-max.
>   (b) insert at START (now point-max) the rest of the text.
> - clearly the above deletions and insertions *could* happen, so CC-mode
>   has to handle them and since all changes could be decomposed this way,
>   CC-mode does not *need* to handle any other case.
> - to handle (a) you don't need to know anything from b-c-f.
> - to handle (b) you don't need to know anything from b-c-f.

That's garbage.  Pure garbage.  I said above in any _REASONABLE_
fashion, specifically excluding, in the bit you snipped, scanning from
BOB on every single change.

Even you would start complaining if CC Mode slowed to the slowness it
would have on implementing anything like you're suggesting.

> CQFD

????

> > No, I quite clearly and accurately described the Emacs that does exist:
> > The technique works almost always, but you have to detect and handle
> > exceptions carefully, something that can be easily done.

> Daniel's description gives you the added info necessary to know how to
> detect that situation and what you might want to do about it.

No, Daniel's description, also Eli's description, by any reasonable
reading, completely exclude the use of the technique, using words
like "expect", but using it to mean something different from its
everyday meaning.  I expect b-c-f and a-c-f to match in the same way I
expect to cross the road without being knocked down by a lorry.  It's
the normal event, though not 100% guaranteed.

Nowhere do Daniel or Eli document the reality, that b-c-f and a-c-f
match very close to 100% of the time.  The elisp manual places an upper
limit on what hackers can do with Emacs.  That limit is now lower than
it was before 25.1.  I find that a very sad state of affairs.

> FWIW, I like Daniel's wording.  It is about as precise as we can make it
> without imposing undue burden (IOW I think Emacs's implementation can
> reasonably (and should) obey that description) and I think it gives the
> right intuition.

I disagree with you totally.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

>> It's trivial to demonstrate that it doesn't:
>> - consider any change between START and END to the buffer.
>> - you can always decompose it into:
>> (a) delete everything from START to point-max.
>> (b) insert at START (now point-max) the rest of the text.
>> - clearly the above deletions and insertions *could* happen, so CC-mode
>> has to handle them and since all changes could be decomposed this way,
>> CC-mode does not *need* to handle any other case.
>> - to handle (a) you don't need to know anything from b-c-f.
>> - to handle (b) you don't need to know anything from b-c-f.

> That's garbage.  Pure garbage.  I said above in any _REASONABLE_
> fashion, specifically excluding, in the bit you snipped, scanning from
> BOB on every single change.

[ You said "written in any reasonable fashion", so I took it you
  referred to the complexity of the code-writing part, not the runtime
  complexity of the algorithm.  As a matter of fact the above
  suggestion makes the code's writing *more* reasonable since you have
  to handle fewer distinct cases.  ]

Nothing in the above requires scanning from BOB.
And it's reasonable enough (in terms of performance) for all other known
major modes, BTW.

> Even you would start complaining if CC Mode slowed to the slowness it
> would have on implementing anything like you're suggesting.

As a matter fact, my local CC-mode works just the way I describe above
(via syntax-propertize), and I haven't noticed any slowness so far.

>> Daniel's description gives you the added info necessary to know how to
>> detect that situation and what you might want to do about it.
> No, Daniel's description, also Eli's description, by any reasonable
> reading, completely exclude the use of the technique,

Daniel's wording (and Eli as well, tho maybe slightly less crisply) say
precisely what might come up and does not exclude the technique you use.

You could argue that it lacks an argument of what is *frequent*, which
might influence the design choice, but it in no way restrict the
*possible* designs.

> Nowhere do Daniel or Eli document the reality, that b-c-f and a-c-f
> match very close to 100% of the time.

I think it's OK.  As you and Daniel have argued, it's "natural" for the
reader to presume that b-c-f and a-c-f will always come in perfect pairs
(and that's also what they will quickly find out when testing or
debugging their code), so there's no need to reinforce this natural
assumption.  Instead what the doc needs to focus on is the fact that
this presumption is not always valid and clarify *how* it's invalid.

> The elisp manual places an upper limit on what hackers can do with
> Emacs.  That limit is now lower than it was before 25.1.  I find that
> a very sad state of affairs.

Your CC-mode design is actually correct w.r.t Daniel's wording.
Its good performance is not guaranteed by this wording since it handles
the non-paired case suboptimally but we rarely (if ever) document
performance-related specification anyway, so most code's performance
design is based on experiments rather than on reading the doc.

> I disagree with you totally.

Wonderful!  On the upside, you might like to take a look at the closely
related bug#24340 where you might actually agree with me for once ;-)


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Stefan Monnier]

> Of course we are in agreement, about the essence.  Your text saus the
> same as mine, except that I don't find "conservative bound" to be more
> helpful than what I wrote, quite the contrary.

Let's take a step back:

We have 4 people in this thread who have trouble agreeing about a tiny
spot of the doc but really want to see some closure.  Current doc's
score:

   Alan: -1
   Daniel: -1
   Eli: +1
   Stef: 0

Where 0 means I can live with it.  Daniel's doc's score:

   Alan: -1
   Daniel: +1
   Stef: +1
   Eli: 0

[ I based the last "0" on the above paragraph where you seem to say that
  you prefer your text, but not super strongly, IOW you can live with it.  ]

For these reason, I think we should go with Daniel's wording, which only
irks Alan and hence rubs fewer people the wrong way.  Another advantage
is that this should close this thread (I think we all know at this point
that Alan won't ever agree with what you and I would agree to, which is
why the discussion had died before Daniel came along).

Please?


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniel Colascione <dancol@dancol.org>,  acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 30 Aug 2016 17:46:49 -0400
> 
> For these reason, I think we should go with Daniel's wording, which only
> irks Alan and hence rubs fewer people the wrong way.  Another advantage
> is that this should close this thread (I think we all know at this point
> that Alan won't ever agree with what you and I would agree to, which is
> why the discussion had died before Daniel came along).
> 
> Please?

No, sorry.  My opinion on this weighs more.

And please let's leave this alone, for as long as the code is as it
is.

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> But it can admittedly be cumbersome since the old-text is now in a string
>>> rather than being inside the buffer.
>> Because, I am not using the before and after positions just to recognise
>> the string that has changed.
>
> I was not suggesting to use those position to recognize the string:
> I was suggesting to use the suggested code to get the old-string so that
> you can then do what you need on the "before change" state.
>
>> I am have to calculate the equivalent positions in the other "lentic"
>> buffer; and this can only be done before the change, since the two
>> buffers are them in a consistent state.

Oh dear, my English seems to be terrible in this paragraph.

>
> The idea is that my suggested code gives you the needed info.
> More specifically, the other buffer is (well, should be) in a state
> consistent with "the current buffer where start..end is replaced (back)
> with `old-text'".

I think it does not, I am afraid, because the "end" position of b-c-f is
not reliably correct.

Consider this buffer:

One
Two

which is converted into this buffer

;; One
;; Two

To calulate that the "T" are cognate locations, I count the number of
lines, and then count backward from the end of the line. This works, so
long as the buffers are also cognate.

Now, say we have a deletion in buffer one starting at pos 3 ("e") to
pos 5 ("T"). Pos 3 is fine -- it's equivalent to pos 6 in the commented
buffer. But pos 5 is difficult. I have to calculate it's cognate
position before the deletion, since the number of lines change.

Which is what I do. All of this breaks if the end location on b-c-f is
wrong wrt to a-c-f. Counting the number of lines does not work on a-c-f
because the uncommented buffer has already changed.


> So that should give you the info needed to update things without doing
> a "complete flush".  As mentioned, it may be inconvenient to use it
> (depending on details of how the rest of the code is implemented) since
> the old-text is now in a string rather than being right there in
> the buffer.
>
>>> Note that in the subst-char-in-region you could "make it pairup"
>>> yourself by hand: if you have (< (+ start length) (cdr lentic--b-c-pos)),
>>> then you can just
>>> 
>>>     (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
>>>     (cl-incf length diff)
>>>     (cl-incf end diff))
>>> 
>>> such that (eq (+ start length) (cdr lentic--b-c-pos)).
>> So, this works because subst-char-in-region is guaranteed not to change
>> the size of the region right?
>
> No.  It works because a-c-f's arguments say "nothing was changed before
> START and after END", so it's always safe (conservative) to move START
> towards BOB or move END towards EOB (of course, when updating END you
> also need to update LENGTH correspondingly).


Ah, yes. But, unfortunately, I cannot calculate the location of the END
position in the cognate buffer. If subst-char-in-region did just what
you are suggesting I do (i.e. signally the maximal extent on a-c-f, as
it does on b-c-f), then there would be no problem.

I would love to change subst-char-in-region to achieve this. Making
a-c-f signal the maximal extent is the trivial change; the alternative
of having b-c-f know the real extent of the change is harder, but also
could be done.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I think to really avoid the complexity for mode authors, b-c-f and a-c-f
>> need to not only balance, but also to be consistent. At the moment, they
>> are not that either -- they can signal different locations for a given
>> change.
>
> I think "balanced + a guarantee that a-c-f covers a subset of b-c-f" is
> sufficient, since you can then trivially extend a-c-f's subset to the
> corresponding "identical" locations if you need it.
>
> AFAIK we do have the "a-c-f covers a subset of b-c-f" property,
> right now (modulo bugs, of course).

To my mind, it's not enough. Even if a-c-f could signal the minimal
change AND the positions reported by b-c-f that would work for me.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Actually, we do have some idea beforehand, as evidenced by the fact that
>> we call b-c-f with valid data.  It's just that after the fact we have
>> a more precise knowledge of what has actually been changed, so a-c-f can
>> give more precise bounds.
>
> I was under the impression that Phillip wanted the data be accurate,
> not just valid.  Because if validity is the only requirement, we could
> always call the before-change hook with the limits of the entire
> buffer, and be done.  Clearly, that would not be very useful, to say
> the least.

I want it to be as accurate as possible given the constraint of
validity. If it is invalid this breaks my use case (except that I detect
this and have an unoptimized fall back plan). If it is inaccurate (in
the sense of wider than necessary), this makes my code potentially less
efficient.

>> We could also change the C code to perform this "extension" of the
>> region passed to a-c-f so that the b-c-f region and the a-c-f region
>> match, but that would make the region passed to a-c-f less precise, so
>> those users of a-c-f which don't care about the pairing (the majority of
>> the users) would be penalized (tho they'd still work correctly: just
>> suboptimally) since there's no way for them to recover the
>> tighter region.
>
> Right.

The sad thing is that, certainly for subst-char-in-region, both of these
pieces of information are available to the code that calls a-c-f. But I
can see no clean way of signalling this information: adding the data to
a-c-f would change it's interface, and storing it in global state is
just ugly.


Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I was under the impression that Phillip wanted the data be accurate,
>> not just valid.
>
> More precise (i.e. tighter bounds) is better, but valid is indispensable
> (I used "valid" to mean that every change is covered by the bounds, even
> though those bounds may be too lose.  I consider "invalid" to be a plain
> bug, as in insert-file-contents, for example).
>
> So I'm pretty sure he's more concerned about "proper pairing" than about
> "utmost tightness of the bounds".

This is exactly correct. I think in my last email I was using "valid" to
mean consistent between b-c-f and a-c-f; sorry for confused terminology.

>
>> Because if validity is the only requirement, we could always call the
>> before-change hook with the limits of the entire buffer, and be done.
>> Clearly, that would not be very useful, to say the least.
>
> Indeed.  But in the case of subst-chars-in-region, what we pass to b-c-f
> is suboptimal but better than point-min/max, so there's a useful middle
> point between the difficult "tightest bound" and the degenerate "just
> pass point-min/max".

I think you mean "painful middle point" rather than "useful"!

Well, it's been on my list of things to do for a while. I will have
another look at subst-char-in-region, to see if I can work out whether
having b-c-f accurately signal the start is possible.

Phil

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Can I deliberately misinterpret this as saying that you think that the
>> changes would be fine so long as we add lots of tests at the same time?
>
> Not "at the same time", but "as a prerequisite" for any serious
> consideration of such changes.  IOW, I want to see the tests first,
> decide whether I like their degree of coverage, and only then I will
> be prepared to _talk_ about any changes in this area.  And the first
> step in that talk will have to be someone presenting a rather complete
> design of the changes, including an analysis of the current callers
> and how each one (or each class) of them will work under the new
> design.


Actually, for me, any individual change that removes inconsistency
between the b-c-f and a-c-f locations is a positive thing. I already
have a fall back plan, but the fewer times this is called, the more
efficient my code gets.

Phil

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

>> The idea is that my suggested code gives you the needed info.
>> More specifically, the other buffer is (well, should be) in a state
>> consistent with "the current buffer where start..end is replaced (back)
>> with `old-text'".
> I think it does not, I am afraid, because the "end" position of b-c-f is
> not reliably correct.

The `end' position of b-c-f is correct (barring bugs), but is indeed
different from the `end' position of a-c-f (the difference is provided
by `length').

All I'm saying is that with the code I provided you could do (in a-c-f)
what Alan considered doing:

    (goto-char start)
    (let ((new-text (delete-and-extract-region start end)))
      (insert old-text)
      (let ((old-end (+ start (length old-text))))
        (run-my-b-c-f-code start old-end)
        (goto-char start)
        (delete-region start old-end)
        (insert new-text)))

Doing it this way would be really annoying, so it's better if your code
is able to work directly with `old-text' without having to insert it
into the buffer, but at least all the needed info is available.

> Ah, yes. But, unfortunately, I cannot calculate the location of the END
> position in the cognate buffer. If subst-char-in-region did just what
> you are suggesting I do (i.e. signally the maximal extent on a-c-f, as
> it does on b-c-f), then there would be no problem.

The code I provided does just that (except it does it inside a-c-f but
the result is *exactly* the same as if subst-char-in-region were to do it
for you).


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: dancol@dancol.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 31 Aug 2016 12:20:21 +0100
> 
> Actually, for me, any individual change that removes inconsistency
> between the b-c-f and a-c-f locations is a positive thing. I already
> have a fall back plan, but the fewer times this is called, the more
> efficient my code gets.

Having read through the code for quite some time, I simply don't
believe this is possible.  You could fix one marginal use case (they
all are marginal), but by doing that break another.  There are too
many layers of patchwork.

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Note that in the subst-char-in-region you could "make it pairup"
>>> yourself by hand: if you have (< (+ start length) (cdr lentic--b-c-pos)),
>>> then you can just
>>> 
>>>     (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
>>>     (cl-incf length diff)
>>>     (cl-incf end diff))
>>> 
>>> such that (eq (+ start length) (cdr lentic--b-c-pos)).
>> So, this works because subst-char-in-region is guaranteed not to change
>> the size of the region right?
>
> No.  It works because a-c-f's arguments say "nothing was changed before
> START and after END", so it's always safe (conservative) to move START
> towards BOB or move END towards EOB (of course, when updating END you
> also need to update LENGTH correspondingly).

Okay, yes, I understand now, and yes, this should be conservative, in
the sense that it works for all changes, I guess. We are making the
assumption here that "start" is always consistent between b-c-f and
a-c-f, and that this is never maximal in one case and minimal in the
other?

Phil

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The idea is that my suggested code gives you the needed info.
>>> More specifically, the other buffer is (well, should be) in a state
>>> consistent with "the current buffer where start..end is replaced (back)
>>> with `old-text'".
>> I think it does not, I am afraid, because the "end" position of b-c-f is
>> not reliably correct.
>
> The `end' position of b-c-f is correct (barring bugs), but is indeed
> different from the `end' position of a-c-f (the difference is provided
> by `length').
>
> All I'm saying is that with the code I provided you could do (in a-c-f)
> what Alan considered doing:
>
>     (goto-char start)
>     (let ((new-text (delete-and-extract-region start end)))
>       (insert old-text)
>       (let ((old-end (+ start (length old-text))))
>         (run-my-b-c-f-code start old-end)
>         (goto-char start)
>         (delete-region start old-end)
>         (insert new-text)))
>
> Doing it this way would be really annoying, so it's better if your code
> is able to work directly with `old-text' without having to insert it
> into the buffer, but at least all the needed info is available.
>
>> Ah, yes. But, unfortunately, I cannot calculate the location of the END
>> position in the cognate buffer. If subst-char-in-region did just what
>> you are suggesting I do (i.e. signally the maximal extent on a-c-f, as
>> it does on b-c-f), then there would be no problem.
>
> The code I provided does just that (except it does it inside a-c-f but
> the result is *exactly* the same as if subst-char-in-region were to do it
> for you).


So, on a-c-f revert the change, do what we are currently doing on b-c-f,
then reapply the change and run what we are currently running on a-c-f.

Yes, that does seem a possibility albeit a fairly hideous one. It's
really not something I'd want to be doing after every command.

I'll test out expanding the region with the code you suggested earlier.

  (let ((diff (- (cdr lentic--b-c-pos) (+ start length))))
     (cl-incf length diff)
     (cl-incf end diff))

and see if this works.

Phil

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> Okay, yes, I understand now, and yes, this should be conservative, in
> the sense that it works for all changes, I guess. We are making the
> assumption here that "start" is always consistent between b-c-f and
> a-c-f,

No, we don't make such an assumption either.  The assumption we make is
that the region passed to a-c-f is within the region passed to b-c-f
(i.e. is tighter).


        Stefan

Re: Unbalanced change hooks (part 2)

[Stefan Monnier]

> So, on a-c-f revert the change, do what we are currently doing on b-c-f,
> then reapply the change and run what we are currently running on a-c-f.

Yup.

> Yes, that does seem a possibility albeit a fairly hideous one.
> It's really not something I'd want to be doing after every command.

Yes, hideous sounds about right.  OTOH if your code can work straight
from the old-text string without too much gymnastics, it's not that bad.

> I'll test out expanding the region with the code you suggested earlier.

Yes, that's probably a much better option.


        Stefan

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 30 Aug 2016 14:27:29 -0400
> 
> > You misunderstand what Stefan says.  He says not calling the
> > before-change hook _at_all_ is a bug.  Not calling it for every chunk
> > of deleted text is not necessarily a bug, if there's a previous less
> > fine-grained call to the hook.  And that's what the text above
> > conveys: that note every chunk to be deleted will have its own call to
> > a hook.
> 
> FWIW, when I read
> 
>     hooks be called in balanced pairs around each buffer change.  Also
>     don't expect the before-change hooks to be called for every chunk of
>     text Emacs is about to delete.  These hooks are provided on the
> 
> I do understand this to mean "b-c-f is just unreliable" and more
> specifically it does sound to me like it refers to the known
> insert-file-contents bug.  If that was not your intention, then consider
> this as evidence that a rewording could be beneficial.

I was not referring to this text, I was referring to our discussion.

> He considers his text to spell it in enough detail.  So unless you
> disagree with the text itself, I think his text is an improvement: you
> both agree with the validity and level of precision of the description
> in his new text, which is not the case with the current text.

I don't think it's an improvement.  It replaces specific practical
advice with abstract principle whose relation to practice might not be
clear to some.

Really, Stefan, after all these years, I'd expect you to trust me a
bit more on documentation issues.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Davis Herring]

> This is a naïve interpretation of what a "change" means and entails.
> In reality, some changes are done with a single call to an insdel
> function, while others need multiple calls that delete and insert text
> piecemeal.  Thus the need to call the hooks before and after each
> insdel call only sometimes.

Indeed, no one knows what we mean by "change".  To make progress, we 
must introduce a formal definition for it, in particular in conjunction 
with the word "before" -- preferably a definition that the current 
implementation satisfies but that allows useful reasoning.

As has been said, some of the proposed documentation that disclaims 
"pairing" produces a definition that is satisfied but is uselessly vague 
-- allowing the implementation to simply call b-c-f once with 
(point-min) and (point-max) when the buffer is created, and never again. 
  (Changes beyond the initial (point-max) are simply insertions into 
that interval!)

However, we intuitively know that b-c-f is supposed to be called 
"immediately before" a change; what does that mean?  Certainly it is not 
a statement about the real passage of time or function calls.  It has to 
be based on an ordering of relevant events -- b-c-f calls, a-c-f calls, 
and the changes themselves.  The "properly paired" ideal defines 
"immediately before" as "with no change intervening" (and similarly for 
a-c-f), disallowing the implementation from reporting what Dan calls 
"fine-grained" changes only via a-c-f.

We can offer a weaker (but still useful) formal guarantee that the 
current implementation does satisfy phrased instead in terms of 
_absence_ of change: a call to a-c-f "publishes" an interval of text 
where changes have been completed and which shall not be altered until 
after a subsequent b-c-f call that "unlocks" it.  (These intervals act 
like rear-advancing overlays: a zero-length interval must be unlocked to 
allow an insertion, and an a-c-f call that covers an unlocked interval 
entirely destroys it without any zero-length remainder.)  Lack of 
alteration is easier to define than change (because it is a state 
function, not a path function), and it is what all caching clients rely 
on for the validity of their caches.

There remains a gap in the information provided by this guarantee (and 
the current implementation): when no a-c-f region corresponds exactly to 
a preceding b-c-f call, clients are left to wonder whether more changes 
may yet occur.  At the least, we must publish all text before redisplay 
(to prevent the user from observing text not yet processed by a-c-f), 
but we can leave redisplay out of it by having each function which 
currently makes "unpaired" calls do one of several things when it exits:

1. Make an additional call to a-c-f that covers the whole region (which, 
in this interpretation, is not a violation because a-c-f is a promise 
not to modify in the future rather than a statement about modifications 
in the past).  This destroys the utility of the smaller a-c-f regions, 
of course.

2. Call some new hook to indicate that the remaining unlocked text has 
been republished in its original state.  (Note that it does not matter 
if the text was actually changed and then changed back, so long as all 
text properties and markers were restored and no other hooks were called.)

3. Make another call to b-c-f and a-c-f with, say, (point-max) 
(point-max).  We would then document that each call to b-c-f (for a 
buffer) indicates that any text not covered by a-c-f since the last 
b-c-f is republished unaltered.  This fits the current implementation 
and the intuitive idea that b-c-f is called before "each" change because 
it separates changes into epochs demarcated by b-c-f.

I therefore propose approach #3 as the minimal change to the current 
implementation that provides a behavioral guarantee worth documenting. 
Moreover, the demarcation of changes into epochs allows interested 
clients to work with the "paired" behavior by deferring a-c-f operations 
until the next b-c-f or precisely matching a-c-f.  (If all functions 
follow #3, then either the first a-c-f matches or the sequence will be 
terminated by a b-c-f.)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: Unbalanced change hooks (part 2) [Documentation fix still remaining]

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Davis Herring <herring@lanl.gov>
> Date: Thu, 1 Sep 2016 15:25:01 -0600
> 
> > This is a naïve interpretation of what a "change" means and entails.
> > In reality, some changes are done with a single call to an insdel
> > function, while others need multiple calls that delete and insert text
> > piecemeal.  Thus the need to call the hooks before and after each
> > insdel call only sometimes.
> 
> Indeed, no one knows what we mean by "change".  To make progress, we 
> must introduce a formal definition for it

I envision that to be a very hard job, given the complexity of some
changes and the fact that at least some of them are not considered
"changes" for this purpose (see past discussions for the details).

Moreover, doing so was never the Emacs way.  Emacs always tries to do
things the naturally expected way, so that these issues don't arise.
If we need to go into such fine details and argue about formal
definitions of a "buffer change", that is a clear sign that something
is wrong and needs to be fixed, in a way that makes the "natural"
definition be implicitly correct, because the effect is as expected by
Lisp programmers.

Therefore, I think that spending any effort on making the manual text
more formally correct will be a wasted effort.  As long as the
implementation remains as it is now, the problem with its accurate
description in the manual will persist, and no small patches to the
code will resolve it completely.

> I therefore propose approach #3 as the minimal change to the current 
> implementation that provides a behavioral guarantee worth documenting. 

I don't think this issue can be attacked using only abstract
high-level considerations and resolved by solutions that are logically
complete, but may not be helpful in practice.

Instead, the use cases we want to support, both wrt the implementation
of complex changes we have in Emacs and wrt the users of these hooks,
should be collected and analyzed.  Then the best approach that
accommodates these needs (including perhaps any future needs we can
envision) should be designed and implemented.  My gut feeling is that
the correct model which will emerge from this will be very different
from what we have now.

Thanks.

Re: Unbalanced change hooks (part 2)

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Okay, yes, I understand now, and yes, this should be conservative, in
>> the sense that it works for all changes, I guess. We are making the
>> assumption here that "start" is always consistent between b-c-f and
>> a-c-f,
>
> No, we don't make such an assumption either.  The assumption we make is
> that the region passed to a-c-f is within the region passed to b-c-f
> (i.e. is tighter).


Let me rephrase. Can I make the assumption that start is consistent? If
not I need to check to see whether they are inconsistent.

I'm testing the code out at the moment. My test cases work anyway, but
it will take more use in practice until I am confident.

Phil