bug#7117: 23.2.2 mangles terminal escape sequences

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

  Hi all,

Emacs-23.2 occasionally loses parts of terminal escape sequences, 
confusing xt-mouse.el and dumping garbage text in the user's buffer.

Steps to reproduce:

   1. ssh to a machine with emacs-23 installed (I suspect slower
      networks would expose this more, but the bug bites me even over
      intranet)
   2. compile tee-input.c (see below) into a shared library (on solaris:
      `cc -g -G -xcode=pic13 -ldl -hlibtee-input.so tee-input.c -g -o
      libtee-input.so')
   3. invoke `LD_PRELOAD=libtee-input.so emacs -nw -Q 2>input.txt'
   4. M-x xterm-mouse-mode
   5. flick the mouse scroll wheel hard, so it generates many ticks in
      quick succession (note the garbage that results)
   6. M-x show-lossage (note the mangled escape sequences)
   7. C-x C-c
   8. Examine input.txt (note the intact escape sequences)

Doing the above with emacs-22.3.1 always works fine, but emacs-23.2.2 
mangles one or more escape sequences (input.txt is not corrupted). I 
suspect this and bug #6920 are symptoms of the same problem.

The number of characters missing ranges from 1 (only ESC missing) to all 
but one (only [ present); I don't think I've yet seen an entire sequence 
disappear. The harder the flick the more numerous the mangled escape 
sequences. It's not an OS issue because I can reproduce it on three very 
different machines (32-bit cygwin, 64-bit redhat, 32-bit solaris). It's 
also not a problem with the terminal because the intercepted input is 
correct. Finally, it's not a problem with xt-mouse.el because 
show-lossage is incorrect.

To give one example I generated using the above steps, emacs *scratch* 
buffer shows:
`b[[M`b[

The output of show-lossage is (note the orphaned ` b [, and later the 
orphaned [, and finally the orphaned M ` b [):
ESC [ > 7 7 ; 9 0 0 ; 0 c ESC x x t e r m - m o u s
e TAB RET ESC [ M ` b [ ` b [ ESC [ M ` b [ ESC [ M
` b [ ESC [ M ` b [ ESC [ M ` b [ ESC [ M ` b [ [ M
` b [ ESC [ M ` b [ ESC O P l

The relevant snippet of input.txt is:
;;;;;ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESC[M`b[ESCOPl

Thoughts?
Ryan

===== tee-input.c ====
#include <dlfcn.h>
#include <unistd.h>
typedef ssize_t (*read_func)(int fildes, void *buf, size_t nbyte);
ssize_t read(int fildes, void *buf, size_t nbyte) {
     static read_func old_read = 0;
     if(!old_read)
         old_read = (read_func) dlsym(RTLD_NEXT, "read");
     ssize_t rval = old_read(fildes, buf, nbyte);
     if(rval > 0)
         write(2, buf, rval);
     return rval;
}
=================


[-- Attachment #2: Type: text/html, Size: 3430 bytes --]

bug#7117: 23.2.2 mangles terminal escape sequences

[Stefan Monnier]

> Steps to reproduce:

>   1. ssh to a machine with emacs-23 installed (I suspect slower
>      networks would expose this more, but the bug bites me even over
>      intranet)
>   2. compile tee-input.c (see below) into a shared library (on solaris:
>      `cc -g -G -xcode=pic13 -ldl -hlibtee-input.so tee-input.c -g -o
>      libtee-input.so')
>   3. invoke `LD_PRELOAD=libtee-input.so emacs -nw -Q 2>input.txt'
>   4. M-x xterm-mouse-mode
>   5. flick the mouse scroll wheel hard, so it generates many ticks in
>      quick succession (note the garbage that results)
>   6. M-x show-lossage (note the mangled escape sequences)
>   7. C-x C-c
>   8. Examine input.txt (note the intact escape sequences)

Does (set-keyboard-coding-system 'binary) circumvent the problem?


        Stefan

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

  On 9/27/2010 10:52 PM, Stefan Monnier wrote:
>> Steps to reproduce:
>>    1. ssh to a machine with emacs-23 installed (I suspect slower
>>       networks would expose this more, but the bug bites me even over
>>       intranet)
>>    2. compile tee-input.c (see below) into a shared library (on solaris:
>>       `cc -g -G -xcode=pic13 -ldl -hlibtee-input.so tee-input.c -g -o
>>       libtee-input.so')
>>    3. invoke `LD_PRELOAD=libtee-input.so emacs -nw -Q 2>input.txt'
>>    4. M-x xterm-mouse-mode
>>    5. flick the mouse scroll wheel hard, so it generates many ticks in
>>       quick succession (note the garbage that results)
>>    6. M-x show-lossage (note the mangled escape sequences)
>>    7. C-x C-c
>>    8. Examine input.txt (note the intact escape sequences)
> Does (set-keyboard-coding-system 'binary) circumvent the problem?
No. In fact, the solaris machine which I ran the example on defaults to 
`no-conversion' (an alias of binary iirc) for some reason. Sorry, I 
forgot to mention before.

This matches my expectations, since bug #6920 arises before any coding 
system touches the input.

Ryan

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

  On 9/28/2010 6:53 AM, Ryan Johnson wrote:
>  On 9/27/2010 10:52 PM, Stefan Monnier wrote:
>>> Steps to reproduce:
>>>    1. ssh to a machine with emacs-23 installed (I suspect slower
>>>       networks would expose this more, but the bug bites me even over
>>>       intranet)
>>>    2. compile tee-input.c (see below) into a shared library (on 
>>> solaris:
>>>       `cc -g -G -xcode=pic13 -ldl -hlibtee-input.so tee-input.c -g -o
>>>       libtee-input.so')
>>>    3. invoke `LD_PRELOAD=libtee-input.so emacs -nw -Q 2>input.txt'
>>>    4. M-x xterm-mouse-mode
>>>    5. flick the mouse scroll wheel hard, so it generates many ticks in
>>>       quick succession (note the garbage that results)
>>>    6. M-x show-lossage (note the mangled escape sequences)
>>>    7. C-x C-c
>>>    8. Examine input.txt (note the intact escape sequences)
>> Does (set-keyboard-coding-system 'binary) circumvent the problem?
> No. In fact, the solaris machine which I ran the example on defaults 
> to `no-conversion' (an alias of binary iirc) for some reason. Sorry, I 
> forgot to mention before.
>
> This matches my expectations, since bug #6920 arises before any coding 
> system touches the input.
Any hints on where the problem might lurk? I poked around a little but 
without luck -- the functions that power read-char are many hundreds of 
lines long and seem to weave between lisp and C at regular intervals. 
I'm willing to go source diving but it's a bit daunting to wade into the 
C code again without a starting reference.

Thanks,
Ryan

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

  On 9/30/2010 1:37 PM, Ryan Johnson wrote:
>  On 9/28/2010 6:53 AM, Ryan Johnson wrote:
>>  On 9/27/2010 10:52 PM, Stefan Monnier wrote:
>>>> Steps to reproduce:
>>>>    1. ssh to a machine with emacs-23 installed (I suspect slower
>>>>       networks would expose this more, but the bug bites me even over
>>>>       intranet)
>>>>    2. compile tee-input.c (see below) into a shared library (on 
>>>> solaris:
>>>>       `cc -g -G -xcode=pic13 -ldl -hlibtee-input.so tee-input.c -g -o
>>>>       libtee-input.so')
>>>>    3. invoke `LD_PRELOAD=libtee-input.so emacs -nw -Q 2>input.txt'
>>>>    4. M-x xterm-mouse-mode
>>>>    5. flick the mouse scroll wheel hard, so it generates many ticks in
>>>>       quick succession (note the garbage that results)
>>>>    6. M-x show-lossage (note the mangled escape sequences)
>>>>    7. C-x C-c
>>>>    8. Examine input.txt (note the intact escape sequences)
>>> Does (set-keyboard-coding-system 'binary) circumvent the problem?
>> No. In fact, the solaris machine which I ran the example on defaults 
>> to `no-conversion' (an alias of binary iirc) for some reason. Sorry, 
>> I forgot to mention before.
>>
>> This matches my expectations, since bug #6920 arises before any 
>> coding system touches the input.
> Any hints on where the problem might lurk? I poked around a little but 
> without luck -- the functions that power read-char are many hundreds 
> of lines long and seem to weave between lisp and C at regular 
> intervals. I'm willing to go source diving but it's a bit daunting to 
> wade into the C code again without a starting reference.
OK, I bit the bullet and fired up a debugger. I've narrowed down the 
problem some.

In keyboard.c, all keyboard input is buffered in kbd_buffer, which is a 
circular buffer with tail kbd_fetch_ptr. In all cases, when a control 
sequence gets mangled it's because kbd_fetch_ptr was incremented by more 
than one slot between iterations of the command loop (often by several 
dozen). However, it turns out that incrementing by more than one slot 
happens quite often and only sometimes exhibits the bug.

More digging shows that multiple increments are due to Fdiscard_input 
being called with the following stack trace:
=>[1] Fdiscard_input(), line 10947 in "keyboard.c"
   [2] cmd_error_internal(data = -1059860928, context = 0xffbfee40 ""), 
line 1301 in "keyboard.c"
   [3] cmd_error(data = -1059860928), line 1234 in "keyboard.c"
   [4] internal_condition_case(bfun = 0x111578 = &command_loop_1(), 
handlers = 1077839640, hfun = 0x109b8c = &cmd_error(register Lisp_Object 
data)), line 1480 in "eval.c"
   [5] command_loop_2(), line 1360 in "keyboard.c"
   [6] internal_catch(tag = 1077836792, func = 0x10283c = 
&command_loop_2(), arg = 1077782856), line 1226 in "eval.c"
   [7] command_loop(), line 1339 in "keyboard.c"
   [8] recursive_edit_1(), line 954 in "keyboard.c"
   [9] Frecursive_edit(), line 1016 in "keyboard.c"
   [10] main(argc = 3, argv = 0xffbff374), line 1833 in "emacs.c"

At this point the symptom is easy enough to understand: sometimes 
discard-input gets unlucky and chops the front off of a partly-arrived 
escape sequence.

The real question is, why would the command loop encounter errors and 
need to discard input at all?
   [1] _longjmp()
=>[2] unwind_to_catch()
   [3] Fsignal()
   [4] xsignal()
   [5] xsignal0()
   [6] window_scroll()
   [7] scroll_command()
   [8] Fscroll_down()
   [9] Ffuncall()
   [10] Feval()
   [11] internal_lisp_condition_case()
   [12] Fbyte_code()
   [13] funcall_lambda()
   [14] Ffuncall()
   [15] Fapply()
   [16] apply1()
   [17] Fcall_interactively()
   [18] Ffuncall()
   [19] call3()
   [20] Fcommand_execute()
   [21] command_loop_1()
   [22] internal_condition_case()
   [23] command_loop_2()
   [24] internal_catch()
   [25] command_loop()
   [26] recursive_edit_1()
   [27] Frecursive_edit()
   [28] main()

 From the above, it seems that scrolling past beginning or end of buffer 
triggers an error, which I guess is somewhat justifiable. However, on a 
very slow connection this morning mouse-clicking between or even within 
buffers caused problems. Unfortunately, setting debug-on-error doesn't 
seem to catch errors generated from C code.

I could try to break on longjmp over another slow connection, but with 
the number of times it gets called for other reasons this would be slow 
going...

Ideas?
Ryan

bug#7117: 23.2.2 mangles terminal escape sequences

[Eli Zaretskii]

> From: Ryan Johnson <ryanjohn@ece.cmu.edu>
> Cc: 7117@debbugs.gnu.org
> 
> At this point the symptom is easy enough to understand: sometimes 
> discard-input gets unlucky and chops the front off of a partly-arrived 
> escape sequence.
> 
> The real question is, why would the command loop encounter errors and 
> need to discard input at all?
>    [1] _longjmp()
> =>[2] unwind_to_catch()
>    [3] Fsignal()
>    [4] xsignal()
>    [5] xsignal0()
>    [6] window_scroll()

The only calls to xsignal0 I see in window_scroll (or, rather, in its
subroutines) is to signal Qbeginning_of_buffer or Qend_of_buffer.

> From the above, it seems that scrolling past beginning or end of buffer 
> triggers an error, which I guess is somewhat justifiable.

Indeed.  Perhaps we need some infrastructure to ignore errors in this
case (I assume `ignore-errors' won't help).  Or maybe we should allow
not to discard input when we signal an error.  Or maybe discard-input
should be smarter, and not discard partial escape sequences?

> However, on a 
> very slow connection this morning mouse-clicking between or even within 
> buffers caused problems. Unfortunately, setting debug-on-error doesn't 
> seem to catch errors generated from C code.
> 
> I could try to break on longjmp over another slow connection, but with 
> the number of times it gets called for other reasons this would be slow 
> going...
> 
> Ideas?

You could put a breakpoint in Fsignal, and see if the problems with
mouse-clicking within buffers is also caused by signaling an error.

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

  On 9/30/2010 4:59 PM, Eli Zaretskii wrote:
>> From: Ryan Johnson<ryanjohn@ece.cmu.edu>
>> Cc: 7117@debbugs.gnu.org
>>
>> At this point the symptom is easy enough to understand: sometimes
>> discard-input gets unlucky and chops the front off of a partly-arrived
>> escape sequence.
>>
>> The real question is, why would the command loop encounter errors and
>> need to discard input at all?
>>     [1] _longjmp()
>> =>[2] unwind_to_catch()
>>     [3] Fsignal()
>>     [4] xsignal()
>>     [5] xsignal0()
>>     [6] window_scroll()
> The only calls to xsignal0 I see in window_scroll (or, rather, in its
> subroutines) is to signal Qbeginning_of_buffer or Qend_of_buffer.
>
>>  From the above, it seems that scrolling past beginning or end of buffer
>> triggers an error, which I guess is somewhat justifiable.
> Indeed.  Perhaps we need some infrastructure to ignore errors in this
> case (I assume `ignore-errors' won't help).  Or maybe we should allow
> not to discard input when we signal an error.  Or maybe discard-input
> should be smarter, and not discard partial escape sequences?
One thing I don't get is, I've been using emacs over painfully slow ssh 
connections for literally years -- sometimes slow enough that keystrokes 
take visible time to echo. This was never really an issue before. In 
retrospect, it did pop up occasionally all along, but it was infrequent 
enough to hide in the shadow of xterm mouse escape sequences confusing 
emacs for x>95. Now it interferes with normal work. Is there some change 
that would have done this? I'm using the same bad internet connection as 
when I had emacs-22...

>> However, on a
>> very slow connection this morning mouse-clicking between or even within
>> buffers caused problems. Unfortunately, setting debug-on-error doesn't
>> seem to catch errors generated from C code.
>>
>> I could try to break on longjmp over another slow connection, but with
>> the number of times it gets called for other reasons this would be slow
>> going...
>>
>> Ideas?
> You could put a breakpoint in Fsignal, and see if the problems with
> mouse-clicking within buffers is also caused by signaling an error.
I'll try that next time it becomes annoying (it's hit-and-miss...)

Ryan

bug#7117: 23.2.2 mangles terminal escape sequences

[Eli Zaretskii]

> Date: Thu, 30 Sep 2010 17:14:16 +0200
> From: Ryan Johnson <ryanjohn@ece.cmu.edu>
> CC: monnier@iro.umontreal.ca, 7117@debbugs.gnu.org
> 
> >>  From the above, it seems that scrolling past beginning or end of buffer
> >> triggers an error, which I guess is somewhat justifiable.
> > Indeed.  Perhaps we need some infrastructure to ignore errors in this
> > case (I assume `ignore-errors' won't help).  Or maybe we should allow
> > not to discard input when we signal an error.  Or maybe discard-input
> > should be smarter, and not discard partial escape sequences?
> One thing I don't get is, I've been using emacs over painfully slow ssh 
> connections for literally years -- sometimes slow enough that keystrokes 
> take visible time to echo. This was never really an issue before.

What would you expect to see, that would cause you think it was "an
issue"?  When Emacs is keyboard-driven, typing text and scrolling
commands seldom happen in such a quick succession that hitting end of
buffer while scrolling would discard text you typed meanwhile.  And
even if it did, how would you know for sure you actually typed it?

bug#7117: 23.2.2 mangles terminal escape sequences

[Ryan Johnson]

  On 9/30/2010 5:44 PM, Eli Zaretskii wrote:
>> Date: Thu, 30 Sep 2010 17:14:16 +0200
>> From: Ryan Johnson<ryanjohn@ece.cmu.edu>
>> CC: monnier@iro.umontreal.ca, 7117@debbugs.gnu.org
>>
>>>>    From the above, it seems that scrolling past beginning or end of buffer
>>>> triggers an error, which I guess is somewhat justifiable.
>>> Indeed.  Perhaps we need some infrastructure to ignore errors in this
>>> case (I assume `ignore-errors' won't help).  Or maybe we should allow
>>> not to discard input when we signal an error.  Or maybe discard-input
>>> should be smarter, and not discard partial escape sequences?
>> One thing I don't get is, I've been using emacs over painfully slow ssh
>> connections for literally years -- sometimes slow enough that keystrokes
>> take visible time to echo. This was never really an issue before.
> What would you expect to see, that would cause you think it was "an
> issue"?  When Emacs is keyboard-driven, typing text and scrolling
> commands seldom happen in such a quick succession that hitting end of
> buffer while scrolling would discard text you typed meanwhile.  And
> even if it did, how would you know for sure you actually typed it?
It's not just keyboard driven. It's a mouse-enabled terminal and the 
mouse click/wheel actions cause the issues (each one generates a burst 
of 6-12 characters). As you say, I don't think I've ever seen this 
happen with keyboard input; pasting commands from a remote X clipboard 
might have a similar effect, but I've never tried.

In the past, scrolling once or twice past buffer begin/end wasn't 
guaranteed to dump garbage. It would just beep/flash most of the time, 
and the slow network connection didn't seem to make it worse. Now, if 
I'm not on intranet, virtually every scroll past-end will dump garbage 
somewhere.

Also, the problem I hit this morning was not related to fast 
typing/clicking over a slow connection. I was doing an ediff-files 
session, and about every other time I clicked on a buffer to edit a 
conflict, emacs would reward me with a mangled escape code... sometimes 
where point had been and sometimes where it ended up. I wasn't typing 
fast or clicking madly -- it was more like "a n n n n b n n <stare a 
while> <click> <grumble> <clean up mess>"

The latter problem went away once I got to work and had a fast network; 
and the former happens much less frequently here (unless I give the 
wheel a really good spin, but that's understandable).

Ryan

bug#7117: 23.2.2 mangles terminal escape sequences

[Lars Ingebrigtsen]

Ryan Johnson <ryanjohn@ece.cmu.edu> writes:

> Emacs-23.2 occasionally loses parts of terminal escape sequences, confusing
> xt-mouse.el and dumping garbage text in the user's buffer. 

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

There's been many changes in the terminal escape sequence handling since
this was reported over a decade ago -- are you still seeing these issues
in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#7117: 23.2.2 mangles terminal escape sequences

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> There's been many changes in the terminal escape sequence handling since
> this was reported over a decade ago -- are you still seeing these issues
> in recent Emacs versions?

More information was requested, but no response was given within a
month, so I'm closing this bug report.  If the problem still exists,
please respond to this email and we'll reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no