bug#21819: 24.5; Overlay modification function called with wrong arguments

bug#21819: 24.5; Overlay modification function called with wrong arguments

[Yutian Li | 李雨田]

Hi.

This happens on both Mac and GNU/Linux with latest release of Emacs. I
also tried the master branch of `git://git.sv.gnu.org/emacs.git` and
it still exists.

The discussion that led to the discovery of the bug is
[here](https://github.com/capitaomorte/yasnippet/issues/607). There is
a stack trace on the OP. Some guesses in the bottom. There is also
instructions to reproduce this.

The direct cause is the modification hook registered with the overlay
is called with apparently wrong arguments. The hook function is
registered with an overlay in the C++ buffer, but called with `beg`
and `end` referring to positions probably in the `*Messages*` buffer.
As the stack trace shows, Flycheck called `message` to display a
message indicating syntax errors, and `message` somehow invokes the
modification hook function, which was defined in C++ buffer but called
now in `*Messages*` buffer.

I traced into the Emacs source file `src/buffer.c`.  There is a
function `report_overlay_modification`. It *should* (I guess) be
called with `after` set to `false` and `true` alternatively. But on
the erroneous invocation, it is called with `after` set to `true`
twice consecutively. The first call `_beg` and `_end` are normal small
integer (inside C++ buffer). The second call they are huge and must
refer to the `*Messages*` buffer. Inside the call, it uses a global
array to store overlay modification hook functions. Now it's called
twice consecutively with `after` set to `true`, it must have reused
old hook functions, hence the call to our function with correct
overlay but wrong cursor. Because on a call with `after` set to
`false`, it will "clear" the hook functions.

Best regards,
Yutian LI | 李雨田

Mobile: +1 (650) 798-9092
E-mail: hotpxless@gmail.com

Computer Science Department
Stanford University

bug#21819: 24.5; Overlay modification function called with wrong arguments

[Eli Zaretskii]

> Date: Tue, 3 Nov 2015 00:24:41 -0800
> From: Yutian Li | 李雨田 <hotpxless@gmail.com>
> 
> This happens on both Mac and GNU/Linux with latest release of Emacs. I
> also tried the master branch of `git://git.sv.gnu.org/emacs.git` and
> it still exists.
> 
> The discussion that led to the discovery of the bug is
> [here](https://github.com/capitaomorte/yasnippet/issues/607). There is
> a stack trace on the OP. Some guesses in the bottom. There is also
> instructions to reproduce this.

Thanks.

I've read that discussion.  Unfortunately, the instruction for
reproducing the problem are not clear to me, I'd appreciate if someone
could post them here in their entirety, ideally a single Lisp file
that needs to be 'load'ed into "emacs -Q" after installing the minimal
number of add-on packages that are strictly necessary for reproducing
the problem.

> The direct cause is the modification hook registered with the overlay
> is called with apparently wrong arguments. The hook function is
> registered with an overlay in the C++ buffer, but called with `beg`
> and `end` referring to positions probably in the `*Messages*` buffer.
> As the stack trace shows, Flycheck called `message` to display a
> message indicating syntax errors, and `message` somehow invokes the
> modification hook function, which was defined in C++ buffer but called
> now in `*Messages*` buffer.

It is entirely not clear to me how come a call to 'message' caused an
overlay modification hook, probably set up by yasnippet, to be
invoked.  This is crucial information in order to investigate the
problem, but AFAICS it's missing from the discussion you cite and from
your report.

> I traced into the Emacs source file `src/buffer.c`.  There is a
> function `report_overlay_modification`. It *should* (I guess) be
> called with `after` set to `false` and `true` alternatively. But on
> the erroneous invocation, it is called with `after` set to `true`
> twice consecutively. The first call `_beg` and `_end` are normal small
> integer (inside C++ buffer). The second call they are huge and must
> refer to the `*Messages*` buffer. Inside the call, it uses a global
> array to store overlay modification hook functions. Now it's called
> twice consecutively with `after` set to `true`, it must have reused
> old hook functions, hence the call to our function with correct
> overlay but wrong cursor. Because on a call with `after` set to
> `false`, it will "clear" the hook functions.

Thanks for looking into this, but without the information showing how
we wind up in 'report_overlay_modification' when some Lisp code called
'message', it is impossible to follow your reasoning.  E.g., it is
quite possible that the Lisp code in question sets up the modification
hook incorrectly, or maybe the modification hook should check the
value of the current buffer to avoid the problem.

In general, 'report_overlay_modification' is called immediately before
a change to a buffer that has overlays, and immediately after a
change.  The value of the AFTER argument depends on which one is it.
So just given the fact you see 2 invocations with AFTER set to true
does not yet mean there's a problem in Emacs core, it could be a
legitimate call due to the fact that *Messages* gets changed when
'message' is called.

Thanks.

bug#21819: 24.5; Overlay modification function called with wrong arguments

[Yutian Li | 李雨田]

On Tue, Nov 3, 2015 at 7:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 3 Nov 2015 00:24:41 -0800
>> From: Yutian Li | 李雨田 <hotpxless@gmail.com>
>>
>> This happens on both Mac and GNU/Linux with latest release of Emacs. I
>> also tried the master branch of `git://git.sv.gnu.org/emacs.git` and
>> it still exists.
>>
>> The discussion that led to the discovery of the bug is
>> [here](https://github.com/capitaomorte/yasnippet/issues/607). There is
>> a stack trace on the OP. Some guesses in the bottom. There is also
>> instructions to reproduce this.
>
> Thanks.
>
> I've read that discussion.  Unfortunately, the instruction for
> reproducing the problem are not clear to me, I'd appreciate if someone
> could post them here in their entirety, ideally a single Lisp file
> that needs to be 'load'ed into "emacs -Q" after installing the minimal
> number of add-on packages that are strictly necessary for reproducing
> the problem.
>

Yes. The init file is
[here](https://gist.github.com/hotpxl/cfad63d5e88ed164bc76). It will
automatically install dependencies. You have to manually `M-x
irony-install-server` to install Irony.

Then you could open up any C++ file. Define a function with a long
name, and some parameters. Then when you try to type the name of the
function, you will get completion candidates. If you choose the
candidate, Irony will invoke YASnippet to lay out a template for the
parameters, so you could tab through the parameters. If you then
quickly fill the parameters and tab through, you will see the error.

>> The direct cause is the modification hook registered with the overlay
>> is called with apparently wrong arguments. The hook function is
>> registered with an overlay in the C++ buffer, but called with `beg`
>> and `end` referring to positions probably in the `*Messages*` buffer.
>> As the stack trace shows, Flycheck called `message` to display a
>> message indicating syntax errors, and `message` somehow invokes the
>> modification hook function, which was defined in C++ buffer but called
>> now in `*Messages*` buffer.
>
> It is entirely not clear to me how come a call to 'message' caused an
> overlay modification hook, probably set up by yasnippet, to be
> invoked.  This is crucial information in order to investigate the
> problem, but AFAICS it's missing from the discussion you cite and from
> your report.
>

I don't think it *should* call the modification hook. I think this is
where the bug happens. It basically calls the modification hook
function with the overlay from C++ buffer, and positions from the
`*Messages*` buffer.

>> I traced into the Emacs source file `src/buffer.c`.  There is a
>> function `report_overlay_modification`. It *should* (I guess) be
>> called with `after` set to `false` and `true` alternatively. But on
>> the erroneous invocation, it is called with `after` set to `true`
>> twice consecutively. The first call `_beg` and `_end` are normal small
>> integer (inside C++ buffer). The second call they are huge and must
>> refer to the `*Messages*` buffer. Inside the call, it uses a global
>> array to store overlay modification hook functions. Now it's called
>> twice consecutively with `after` set to `true`, it must have reused
>> old hook functions, hence the call to our function with correct
>> overlay but wrong cursor. Because on a call with `after` set to
>> `false`, it will "clear" the hook functions.
>
> Thanks for looking into this, but without the information showing how
> we wind up in 'report_overlay_modification' when some Lisp code called
> 'message', it is impossible to follow your reasoning.  E.g., it is
> quite possible that the Lisp code in question sets up the modification
> hook incorrectly, or maybe the modification hook should check the
> value of the current buffer to avoid the problem.
>
> In general, 'report_overlay_modification' is called immediately before
> a change to a buffer that has overlays, and immediately after a
> change.  The value of the AFTER argument depends on which one is it.
> So just given the fact you see 2 invocations with AFTER set to true
> does not yet mean there's a problem in Emacs core, it could be a
> legitimate call due to the fact that *Messages* gets changed when
> 'message' is called.
>
> Thanks.