bug#17497: 24.4.50; TTY menu glitches

bug#17497: 24.4.50; TTY menu glitches

[Dmitry Antipov]

[-- Attachment #1: Type: text/plain, Size: 3367 bytes --]

Now I'm seeing two TTY menu glitches:

0) Double-"selected" item (actually selected item on this screenshot
    is "New Window Below").

1) Out-of-menu "--" draw combined with incorrect help text in echo area.

$TERM is rxvt-unicode (version 9.20), if that matters.

Dmitry

In GNU Emacs 24.4.50.3 (x86_64-unknown-linux-gnu)
  of 2014-05-15 on localhost.localdomain
Repository revision: 117110 dmantipov@yandex.ru-20140515100645-4wktg3eo5f0wpbob
System Description:     Fedora release 20 (Heisenbug)

Configured using:
  `configure --prefix=/not/exists --enable-gcc-warnings
  --enable-check-lisp-object-type --enable-checking --without-x
  --without-all --disable-acl 'CFLAGS=-O0 -g3''

Configured features:

Important settings:
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   electric-indent-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC
[ B ESC [ B ESC [ A ESC [ A ESC [ A ESC [ B ESC [ B
ESC [ B ESC [ B ESC [ A ESC [ B ESC [ B ESC [ B ESC
[ A ESC [ A ESC [ A ESC [ A ESC [ B ESC [ A ESC [ A
ESC [ A ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B C-g
ESC [ B ESC [ B ESC x e m a c s DEL DEL DEL DEL DEL
DEL r e p o TAB r TAB RET

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
End of buffer [2 times]
delete-backward-char: Text is read-only
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail regexp-opt rfc2047
rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils time-date
tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process multi-tty emacs)

Memory information:
((conses 16 73001 4484)
  (symbols 48 16870 0)
  (miscs 40 33 117)
  (strings 32 10242 5816)
  (string-bytes 1 279721)
  (vectors 16 7679)
  (vector-slots 8 331222 25610)
  (floats 8 60 583)
  (intervals 56 203 4)
  (buffers 960 12)
  (heap 1024 23973 1892))

[-- Attachment #2: glitch0.png --]
[-- Type: image/png, Size: 8736 bytes --]

[-- Attachment #3: glitch1.png --]
[-- Type: image/png, Size: 10572 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Thu, 15 May 2014 16:26:25 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> 
> Now I'm seeing two TTY menu glitches:
> 
> 0) Double-"selected" item (actually selected item on this screenshot
>     is "New Window Below").
> 
> 1) Out-of-menu "--" draw combined with incorrect help text in echo area.
> 
> $TERM is rxvt-unicode (version 9.20), if that matters.

Any hope of a reproducible recipe?  I don't see this on the systems to
which I have access.

FWIW, this looks like a deja-vu of the phenomenon described in
http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00546.html.
As concluded (after a long discussion) in
http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00667.html,
we never actually understood why this happens.  Maybe try to play with
the size of your TTY window, and see if that changes anything.  Or
switch to xterm instead of rxvt.

Also, you could look into a termscript file to see whether Emacs
indeed wrote 2 lines with the red background in the first snapshot, or
wrote the "--" string at incorrect coordinates in the second.  The
discussion mentioned above concluded that perfectly correct terminal
commands caused strange unexplained effects on the screen.

> System Description:     Fedora release 20 (Heisenbug)
                                             ^^^^^^^^^
On second thought, a system with that description is supposed to do
this, no?

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]


Using "xfce4-terminal 0.6.3 (Xfce 4.10)", I did

   emacs-24.3.91 -Q -nw
   M-x menu-bar-open RET

then simply held down the down array, and very quickly got it to look
like the first attached image. By holding down the down arrow for a bit,
then the up arrow for a bit, then the down arrow, etc, I got the second
image. It's not 100% reproducible, but seemed to happen fairly often.
I don't expect to hold down the arrow keys in normal usage. :)


[-- Attachment #2: 1.png --]
[-- Type: image/png, Size: 33411 bytes --]

[-- Attachment #3: 2.png --]
[-- Type: image/png, Size: 36289 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Glenn Morris wrote:

> then simply held down the down array, and very quickly got it to look

s/array/arrow

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]


And with the same strategy under "XTerm(303)", I quickly got the attached.
I don't normally use xterm, though, and it must be misconfigured
somehow, because pressing "M-x" produces LATIN SMALL LETTER O WITH
STROKE... :)


[-- Attachment #2: 3.png --]
[-- Type: image/png, Size: 7528 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Dmitry Antipov <dmantipov@yandex.ru>,  17497@debbugs.gnu.org
> Date: Fri, 16 May 2014 02:53:59 -0400
> 
> And with the same strategy under "XTerm(303)", I quickly got the attached.
> I don't normally use xterm, though, and it must be misconfigured
> somehow, because pressing "M-x" produces LATIN SMALL LETTER O WITH
> STROKE... :)

I have no idea what causes this.  I cannot reproduce this, neither on
Windows, nor when running Emacs on GNU/Linux via PuTTY (which emulates
xterm).

Does this problem go away if you enlarge the terminal window such that
the entire File and Tools menus can be displayed without overlaying
the mode line?  If the problems disappear then, perhaps we could solve
them by limiting the number of displayed menu items some more.

If enlarging the window doesn't help, then I don't know what to do,
unless someone shows me a recipe I can use to reproduce the problem on
some machine where I can debug them.  Failing that, perhaps some
terminfo expert could analyze the termscript and show what we do
wrongly.  Last time I looked at termscript produced in a similar case,
it looked entirely legitimate, i.e. at no time did Emacs send a
terminal command to write to the portions of display where a
screenshot showed incorrectly displayed text, and every time a
red-background menu item was incorrectly left behind, there was a
clear command in the termscript that told the terminal to redraw that
very item with the normal (blue) background.  IOW, it looked like for
some reason the terminal was not obeying the commands it was sent.
Doers anyone know why would this happen?

It could also be some timing problem, specific to X, in which case
someone will have to explain how to avoid that, because I have no
idea.

Does this happen if you use the menu "reasonably", i.e. without
leaning on arrow keys?  Does it happen if you use C-p instead of the
down arrow?

I'm at a loss here...

bug#17497: 24.4.50; TTY menu glitches

[Dmitry Antipov]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

On 05/16/2014 10:53 AM, Glenn Morris wrote:

> And with the same strategy under "XTerm(303)", I quickly got the attached.

I found this issue XTerm-compatible too.

Just for the record, TTY menus are even more broken with Eterm.

Dmitry


[-- Attachment #2: eterm.png --]
[-- Type: image/png, Size: 52631 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Fri, 16 May 2014 13:42:54 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: 17497@debbugs.gnu.org
> 
> Just for the record, TTY menus are even more broken with Eterm.

How is this more broken?  All I see is the same problem with artifacts
left behind where they shouldn't be.  Like in your original
screenshots.  The number of artifacts is not important; even one of
them is a sign of some problem.

Do you see in the termscript any commands to write these artifacts?
E.g., the red-background "Close" was presumably the selected menu item
at some previous time; do you see a command to overwrite that with the
blue-background "Close" (or something else) in insert mode?

Is there any difference in what you see if you do that in a buffer
which is large enough to fill the entire window with text, like
xdisp.c in its first portion, where there's a large commentary?  IOW,
do these problems depend on what was on the screen before the menu was
dropped down?

The TTY menus work by overwriting portions of the glyph matrix with
the text derived from the menu.  The screen is updated by the normal
Emacs code, which was not touched at all.  So I don't understand why
these artifacts appear when menus are displayed, but not with normal
buffer text display...

Could this be a buffering issue?  Maybe adding some fflush calls will
make a difference?  (Not that I understand how buffering could change
the final result.)

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Fri, 16 May 2014 13:42:54 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: 17497@debbugs.gnu.org
> 
> Just for the record, TTY menus are even more broken with Eterm.

Does the change below help in any way?

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2014-04-18 08:35:09 +0000
+++ src/xdisp.c	2014-05-16 10:35:55 +0000
@@ -21307,6 +21307,8 @@ display_tty_menu_item (const char *item_
 		    width, 0, FRAME_COLS (f) - 1, -1);
 
   row->used[TEXT_AREA] = max (saved_used, row->used[TEXT_AREA]);
+  if (row->used[TEXT_AREA] > 0)
+    row->displays_text_p = 1;
   row->truncated_on_right_p = saved_truncated;
   row->hash = row_hash (row);
   row->full_width_p = saved_width;

bug#17497: 24.4.50; TTY menu glitches

[Dmitry Antipov]

On 05/16/2014 02:46 PM, Eli Zaretskii wrote:

> Does the change below help in any way?

Unfortunately no, AFAICS.

Dmitry

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Fri, 16 May 2014 18:59:58 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: 17497@debbugs.gnu.org
> 
> On 05/16/2014 02:46 PM, Eli Zaretskii wrote:
> 
> > Does the change below help in any way?
> 
> Unfortunately no, AFAICS.

If you record in a termscript while doing what it takes to reproduce
the problem, and the dump that termscript to the screen, do you see
the same artifacts as you saw in Emacs?

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Eli Zaretskii wrote:

> Does this problem go away if you enlarge the terminal window such that
> the entire File and Tools menus can be displayed without overlaying
> the mode line? 

I maximized the xterm before starting Emacs, and still very quickly got
into the state where there is a duplicate "--" off to the right of the
real menu (same as my "1.png" from earlier). It does seem less common
with a bigger terminal though.

> Does this happen if you use the menu "reasonably", i.e. without
> leaning on arrow keys? 

So far no, IME.

> Does it happen if you use C-p instead of the down arrow?

Leaning on C-p produces glitches, yes. Here's a fun one:


[-- Attachment #2: 4.png --]
[-- Type: image/png, Size: 7038 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Fri, 16 May 2014 11:47:27 -0400
> 
> > Does this happen if you use the menu "reasonably", i.e. without
> > leaning on arrow keys? 
> 
> So far no, IME.

Which means what? that the terminal somehow "optimizes out" some of
the commands it receives when they arrive at a high rate??

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Fri, 16 May 2014 11:47:27 -0400
> 
> > Does this problem go away if you enlarge the terminal window such that
> > the entire File and Tools menus can be displayed without overlaying
> > the mode line? 
> 
> I maximized the xterm before starting Emacs, and still very quickly got
> into the state where there is a duplicate "--" off to the right of the
> real menu (same as my "1.png" from earlier). It does seem less common
> with a bigger terminal though.
> 
> > Does this happen if you use the menu "reasonably", i.e. without
> > leaning on arrow keys? 
> 
> So far no, IME.
> 
> > Does it happen if you use C-p instead of the down arrow?
> 
> Leaning on C-p produces glitches, yes. Here's a fun one:

Could both of you please record the terminal commands issued while you
reproduce the problem in a termscript, and then replay that termscript
to the same terminal (outside Emacs) with the attached shell script?
I'm interested to see whether the problem is reproduced by sending the
same commands at a low rate.

If sleeping for 2 seconds doesn't reproduce the problem, maybe try
decreasing the sleep time until it does.

TIA


[-- Attachment #2: script --]
[-- Type: application/octet-stream, Size: 179 bytes --]

#! /bin/sh
# Invoke as "script FILE"
l=`wc -l $1 | awk '{ print $1 }'`
i=1
while (expr $i "<=" $l > /dev/null); do
  sed -n -e `echo $i`p $1 && sleep 2 && i=`expr $i "+" 1`;
done

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

Ping!  Could you please do what I asked below?  Thanks.

> Date: Sat, 17 May 2014 12:56:04 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmantipov@yandex.ru, 17497@debbugs.gnu.org
> 
> Could both of you please record the terminal commands issued while you
> reproduce the problem in a termscript, and then replay that termscript
> to the same terminal (outside Emacs) with the attached shell script?
> I'm interested to see whether the problem is reproduced by sending the
> same commands at a low rate.
> 
> If sleeping for 2 seconds doesn't reproduce the problem, maybe try
> decreasing the sleep time until it does.
> 
> TIA

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]


Replaying the termscript at full speed does not produce the same
glitches, if that makes any sense.

See attached image and assoicated termscript.

BTW, your script reduces to:

while read line; do
  echo "$line"
  sleep 2
done < "$1"


[-- Attachment #2: 1.png --]
[-- Type: image/png, Size: 34661 bytes --]

[-- Attachment #3: tscript3.xz --]
[-- Type: application/octet-stream, Size: 1644 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Andreas Schwab]

Glenn Morris <rgm@gnu.org> writes:

> Replaying the termscript at full speed does not produce the same
> glitches, if that makes any sense.

Then it can only be a bug in the terminal emulator.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Andreas Schwab wrote:

> Glenn Morris <rgm@gnu.org> writes:
>
>> Replaying the termscript at full speed does not produce the same
>> glitches, if that makes any sense.
>
> Then it can only be a bug in the terminal emulator.

Then it's common to several of them, as has already been said.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Thu, 22 May 2014 01:44:05 -0400
> 
> Replaying the termscript at full speed does not produce the same
> glitches, if that makes any sense.

That's weird.  Earlier you said that if you don't lean on the arrow
keys, but press them at normal typing speed, the problem doesn't
happen as well, is that right?  If so, perhaps you could produce a
termscript for the same sequence of keypresses, just at that "normal"
speed, so we could compare them.  (If one of the termscripts has fewer
keypresses than the other, it doesn't matter, as long as you press the
same keys.)

> BTW, your script reduces to:
> 
> while read line; do
>   echo "$line"
>   sleep 2
> done < "$1"

Thanks.

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> That's weird.  Earlier you said that if you don't lean on the arrow
> keys, but press them at normal typing speed, the problem doesn't
> happen as well, is that right? 

So far I have not seen it to occur without me leaning on keys, but it
could be that it only happens on say 1/100 key presses, as opposed to
when the key rate is high. I don't use text-mode menus normally.

> If so, perhaps you could produce a termscript for the same sequence of
> keypresses, just at that "normal" speed, so we could compare them. (If
> one of the termscripts has fewer keypresses than the other, it doesn't
> matter, as long as you press the same keys.)

You mean I can replace "lean on the down key, lean on the up key, lean
on the down key" with just "down, up, down"?
If so, I'll give it a go later on.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Thu, 22 May 2014 19:19:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmantipov@yandex.ru, 17497@debbugs.gnu.org
> 
> > From: Glenn Morris <rgm@gnu.org>
> > Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> > Date: Thu, 22 May 2014 01:44:05 -0400
> > 
> > Replaying the termscript at full speed does not produce the same
> > glitches, if that makes any sense.
> 
> That's weird.

Wait, maybe I misunderstood you.  Did you mean that replaying at full
speed (i.e. without the 'sleep' line, I presume?) makes the glitches
go away, or did you mean there are still glitches, but they look
different on the screen?

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Thu, 22 May 2014 12:26:47 -0400
> 
> > If so, perhaps you could produce a termscript for the same sequence of
> > keypresses, just at that "normal" speed, so we could compare them. (If
> > one of the termscripts has fewer keypresses than the other, it doesn't
> > matter, as long as you press the same keys.)
> 
> You mean I can replace "lean on the down key, lean on the up key, lean
> on the down key" with just "down, up, down"?

Actually, something like 20 times down, then 20 times up would be
better, I think.

Also, if you revert revision 117033 on emacs-24 branch (or try with a
binary built before Apr 29, if you keep them), does the problem still
exist?

TIA

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> Wait, maybe I misunderstood you.  Did you mean that replaying at full
> speed (i.e. without the 'sleep' line, I presume?) makes the glitches
> go away, or did you mean there are still glitches, but they look
> different on the screen?

With no sleep, there are no glitches (that I could see).
I certainly did not end up with that stray "--" off to the right.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Thu, 22 May 2014 12:54:26 -0400
> 
> With no sleep, there are no glitches (that I could see).
> I certainly did not end up with that stray "--" off to the right.

IOW, sending commands at fast rate by leaning on the keys does produce
the glitches, but replaying the same commands at fast rate doesn't.
Curiouser and curiouser...

bug#17497: 24.4.50; TTY menu glitches

[Andreas Schwab]

Glenn Morris <rgm@gnu.org> writes:

> Andreas Schwab wrote:
>
>> Glenn Morris <rgm@gnu.org> writes:
>>
>>> Replaying the termscript at full speed does not produce the same
>>> glitches, if that makes any sense.
>>
>> Then it can only be a bug in the terminal emulator.
>
> Then it's common to several of them, as has already been said.

That's very well possible, since they are often just forks of each
other.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Andreas Schwab wrote:

> That's very well possible, since they are often just forks of each
> other.

OK, so far people have seen such issues with

xterm 303
rxvt-unicode 9.20
xfce4-terminal 0.6.3
eterm
whatever martin was using in
   http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00462.html

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 22 May 2014 13:29:40 -0400
> Cc: dmantipov@yandex.ru, 17497@debbugs.gnu.org
> 
> whatever martin was using in
>    http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00462.html

Not sure that was the same problem, as it disappeared when replayed
with a script like the one you used.  It also disappeared in xterm on
the same machine, and when the size of the terminal window was
enlarged.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Thu, 22 May 2014 19:46:34 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmantipov@yandex.ru, 17497@debbugs.gnu.org
> 
> > From: Glenn Morris <rgm@gnu.org>
> > Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> > Date: Thu, 22 May 2014 12:26:47 -0400
> > 
> > > If so, perhaps you could produce a termscript for the same sequence of
> > > keypresses, just at that "normal" speed, so we could compare them. (If
> > > one of the termscripts has fewer keypresses than the other, it doesn't
> > > matter, as long as you press the same keys.)
> > 
> > You mean I can replace "lean on the down key, lean on the up key, lean
> > on the down key" with just "down, up, down"?
> 
> Actually, something like 20 times down, then 20 times up would be
> better, I think.
> 
> Also, if you revert revision 117033 on emacs-24 branch (or try with a
> binary built before Apr 29, if you keep them), does the problem still
> exist?

Ping!  Any more info on this?

TIA

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

Eli Zaretskii wrote:

>> Also, if you revert revision 117033 on emacs-24 branch (or try with a
>> binary built before Apr 29, if you keep them), does the problem still
>> exist?

Reverting 117033: still glitchy.

> > > If so, perhaps you could produce a termscript for the same sequence of
> > > keypresses, just at that "normal" speed, so we could compare them. (If
> > > one of the termscripts has fewer keypresses than the other, it doesn't
> > > matter, as long as you press the same keys.)
> > 
> > You mean I can replace "lean on the down key, lean on the up key, lean
> > on the down key" with just "down, up, down"?
> 
> Actually, something like 20 times down, then 20 times up would be
> better, I think.

OK, so in trying to do this, I have noticed it glitching when I press
the keys at normal speed. So I presume it's not useful to send you that
comparison after all.

See attached image and associated typescript.
In this case I think I just opened the menu-bar and pressed the down
arrow three times at normal speed.

FWIW, I could not make it happen on a RHEL 6.5 system at all.
But on a Debian testing system, it happens with multiple terminal emulators.

(I suspect this is going to be one of those things you need to be able
to reproduce to fix...)


[-- Attachment #2: 2.png --]
[-- Type: image/png, Size: 34593 bytes --]

[-- Attachment #3: t2.txt.xz --]
[-- Type: application/octet-stream, Size: 836 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Fri, 30 May 2014 22:22:29 -0400
> 
> >> Also, if you revert revision 117033 on emacs-24 branch (or try with a
> >> binary built before Apr 29, if you keep them), does the problem still
> >> exist?
> 
> Reverting 117033: still glitchy.

OK, one suspect down.

> OK, so in trying to do this, I have noticed it glitching when I press
> the keys at normal speed. So I presume it's not useful to send you that
> comparison after all.

No, it's not useful.  But I wonder why earlier you thought that this
didn't happen at normal speed.  Is it more rare at normal speed?

> See attached image and associated typescript.
> In this case I think I just opened the menu-bar and pressed the down
> arrow three times at normal speed.

When I replay that typescript on fencepost.gnu.org (logging into it
via PuTTY, which emulates xterm), I see no glitches at all, FWIW.

Interestingly, there's a cursor shown in the image to the left of
"Visit New File" menu item; it shouldn't be there.

Moreover, if I login to fencepost, set my terminal's height to be 24
lines, like in your screenshot, and then record the termscript with
the same 3 keystrokes as you did, I get an identical script!  So Emacs
behaves identically on these 2 systems, it's something in the terminal
emulators and/or the libraries they use that causes the differences in
behavior.

> FWIW, I could not make it happen on a RHEL 6.5 system at all.
> But on a Debian testing system, it happens with multiple terminal emulators.
> 
> (I suspect this is going to be one of those things you need to be able
> to reproduce to fix...)

Can you reproduce the problem when you log into that Debian testing
system remotely, via some xterm emulator?  If so, I'd appreciate an
SSH login to that system, and a directory with an Emacs tree to play
in.  Thanks.

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> Can you reproduce the problem when you log into that Debian testing
> system remotely, via some xterm emulator?  If so, I'd appreciate an
> SSH login to that system, and a directory with an Emacs tree to play
> in.  Thanks.

Sorry, that is my laptop, and I don't have incoming ssh enabled on it,
and as a general rule don't want to. Nothing against you personally! :)

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

One more idea: can you try decreasing the number 900 in this snippet
from dispnew.c:

	  if (FRAME_TERMCAP_P (f))
	    {
	      /* Flush out every so many lines.
		 Also flush out if likely to have more than 1k buffered
		 otherwise.   I'm told that some telnet connections get
		 really screwed by more than 1k output at once.  */
	      FILE *display_output = FRAME_TTY (f)->output;
	      if (display_output)
		{
		  ptrdiff_t outq = __fpending (display_output);
		  if (outq > 900
		      || (outq > 20 && ((i - 1) % preempt_count == 0)))
		    fflush (display_output);
		}
	    }

Or maybe even make display_output line-buffered.  I wonder if that has
any effect on the problem.

TIA

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

[I asked Thomas for his expert opinion on this strange issue, and he
graciously agreed.]

> Date: Sun, 01 Jun 2014 11:26:57 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: dickey@his.com
> 
> On Sun, Jun 01, 2014 at 06:07:33PM +0300, Eli Zaretskii wrote:
> > > Date: Sat, 31 May 2014 16:09:47 -0400
> > > From: Thomas Dickey <dickey@his.com>
> > > Cc: dickey@his.com
> > > 
> > > The discussion sounds like a common problem: terminals send cursor- and
> > > function-keys generally as a sequence of bytes, requiring an application
> > > to wait for each sequence to complete.  If each keystroke causes the
> > > application to do something that writes a lot of text to the screen (such
> > > as scrolling), it's possible to have some of those sequences timeout.
> > > Once that happens, the application doesn't see cursor-keys, it sees the
> > > individual bytes - which can be interpreted differently (including echoing
> > > and further messing up the display).
> > > 
> > > While I'm aware that Emacs doesn't use ncurses for screen optimization,
> > > which is the same issue implied in the description of ESCDELAY in the
> > > ncurses manpage, which begins
> > > 
> > >        ESCDELAY
> > >             Specifies the total time, in milliseconds, for  which  ncurses
> > >             will  await  a  character sequence, e.g., a function key.  The
> > >             default value, 1000 milliseconds, is  enough  for  most  uses.
> > >             However, it is made a variable to accommodate unusual applica‐
> > >             tions.
> > > 
> > >             The most common instance where you may  wish  to  change  this
> > >             value  is to work with slow hosts, e.g., running on a network.
> > >             If the host cannot read characters  rapidly  enough,  it  will
> > >             have  the  same effect as if the terminal did not send charac‐
> > >             ters rapidly enough.  The library will still see a timeout.
> > 
> > Thanks.
> > 
> > However, what puzzles me is that the code which implements text-mode
> > menus doesn't change at all how Emacs handles display on a TTY.  We
> > just overwrite portions of the display with the menu text in some
> > internal data structure, then hand out that structure to the code
> > which updates the screen, as if the buffer text has changed.  So how
> > come we never saw similar problems with normal Emacs display on a TTY,
> > e.g. when the user presses PgDn to scroll the text?
> 
> Scrolling one line at a time versus scrolling the whole page differs
> most noticeably if the application doesn't (or can't) use a terminal's
> scrolling features.  If it doesn't scroll, then the whole screen has to
> be repainted for each update.

Emacs optimizes redrawing by comparing the previous and the desired
display, and then only repainting the changed parts.

None of that changes when the menu is displayed, we just overwrite
portions of the desired display data structure, then call the normal
redrawing routines.  That's why it puzzles me why the problems are
being reported only for the menus.

> Repainting the screen after scrolling one line is going to be much slower
> than repainting the screen after scrolling one page.

Yes, I understand.  But repainting the menu when the user presses,
e.g., the down arrow, is no different: Emacs just repaints the
previously-selected menu item in the normal color, then repaints the
one below it in the "selected" color.  (Actually, since the menu item
is usually shorter than the terminal width, only part of each of those
2 lines is redrawn.)

> > Btw, is it OK to CC the Emacs bug tracker, so that this discussion is
> > recorded there?
> 
> yes

Thanks.  The 17497@debbugs.gnu.org address above will achieve that.

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On Sun, Jun 01, 2014 at 07:25:29PM +0300, Eli Zaretskii wrote:
> Emacs optimizes redrawing by comparing the previous and the desired
> display, and then only repainting the changed parts.

That's the intent - but
> 
> None of that changes when the menu is displayed, we just overwrite
> portions of the desired display data structure, then call the normal
> redrawing routines.  That's why it puzzles me why the problems are
> being reported only for the menus.

I downloaded tscript3.xz from message #50, and am cat'ing it (slowly)
to a 24x80 xterm.  It very clearly is not repainting just changed portions.
At the very beginning of the script, the screen updates just a couple of
lines (old/new) as the cursor moves across menu items.  But as soon as
the cursor goes past the menu items which are first shown, the program
is repainting the entire menu.  Thereafter, even after it has returned
to the original view, there are occasional repaints of the entire menu,
as well as cursor movement to/from the "Close" entry.

Most of the file, however, consists of repainting the menu.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

On Sun, Jun 01, 2014 at 01:12:44PM -0400, Thomas Dickey wrote:
> Most of the file, however, consists of repainting the menu.

By the way, since the menu covers only half the screen, there aren't
any general-purpose scrolling optimizations that would help.  (xterm
does support left/right margins, which would be interesting to explore
in this area).  What ncurses does when it's getting behind is to drop
updates - the typeahead feature:

       The  curses  library does ``line-breakout optimization'' by looking for
       typeahead periodically while updating the screen.  If input  is  found,
       and  it is coming from a tty, the current update is postponed until re-
       fresh or doupdate is called again.  This allows faster response to com-
       mands  typed  in  advance.   Normally, the input FILE pointer passed to
       newterm, or stdin in the case that initscr was used, will be used to do
       this typeahead checking.  The typeahead routine specifies that the file
       descriptor fd is to be used to check for typeahead instead.  If  fd  is
       -1, then no typeahead checking is done.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Sun, 01 Jun 2014 13:12:44 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: dickey@his.com, 17497@debbugs.gnu.org
> 
> I downloaded tscript3.xz from message #50, and am cat'ing it (slowly)
> to a 24x80 xterm.  It very clearly is not repainting just changed portions.
> At the very beginning of the script, the screen updates just a couple of
> lines (old/new) as the cursor moves across menu items.  But as soon as
> the cursor goes past the menu items which are first shown, the program
> is repainting the entire menu.  Thereafter, even after it has returned
> to the original view, there are occasional repaints of the entire menu,
> as well as cursor movement to/from the "Close" entry.

I will look into this, thanks.  I think I actually force full redraws
under some conditions.  I will see if they are necessary.

But even these full repaints shouldn't be causing any artifacts as
shown in the screenshots, should they?  Emacs sometimes redraws a full
screen (e.g., when you switch a frame on a TTY), but I never heard any
complaints about the results.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Sun, 01 Jun 2014 13:18:17 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 17497@debbugs.gnu.org
> 
> What ncurses does when it's getting behind is to drop updates - the
> typeahead feature:
> 
>        The  curses  library does ``line-breakout optimization'' by looking for
>        typeahead periodically while updating the screen.  If input  is  found,
>        and  it is coming from a tty, the current update is postponed until re-
>        fresh or doupdate is called again.  This allows faster response to com-
>        mands  typed  in  advance.   Normally, the input FILE pointer passed to
>        newterm, or stdin in the case that initscr was used, will be used to do
>        this typeahead checking.  The typeahead routine specifies that the file
>        descriptor fd is to be used to check for typeahead instead.  If  fd  is
>        -1, then no typeahead checking is done.

So buffering output more aggressively could help, is that what you are
saying?

We currently fflush the stream every 900 bytes and also every 10
screen lines or so.  Does that sound reasonable?

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

On Sun, Jun 01, 2014 at 09:45:07PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 01 Jun 2014 13:18:17 -0400
> > From: Thomas Dickey <dickey@his.com>
> > Cc: Eli Zaretskii <eliz@gnu.org>, 17497@debbugs.gnu.org
> > 
> > What ncurses does when it's getting behind is to drop updates - the
> > typeahead feature:
> > 
> >        The  curses  library does ``line-breakout optimization'' by looking for
> >        typeahead periodically while updating the screen.  If input  is  found,
> >        and  it is coming from a tty, the current update is postponed until re-
> >        fresh or doupdate is called again.  This allows faster response to com-
> >        mands  typed  in  advance.   Normally, the input FILE pointer passed to
> >        newterm, or stdin in the case that initscr was used, will be used to do
> >        this typeahead checking.  The typeahead routine specifies that the file
> >        descriptor fd is to be used to check for typeahead instead.  If  fd  is
> >        -1, then no typeahead checking is done.
> 
> So buffering output more aggressively could help, is that what you are
> saying?
> 
> We currently fflush the stream every 900 bytes and also every 10
> screen lines or so.  Does that sound reasonable?

I don't think that will be enough: the output stream simply is not fast
enough to keep up.  The typeahead feature is crude, but works.  A more
elegant approach (perhaps complicated) would be to keep track of the
changes since the beginning of a repaint, and store _those_ into a
buffer which could be discarded if it is not written before the next
input character is detected.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Sun, 01 Jun 2014 15:46:00 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: dickey@his.com, 17497@debbugs.gnu.org
> 
> > So buffering output more aggressively could help, is that what you are
> > saying?
> > 
> > We currently fflush the stream every 900 bytes and also every 10
> > screen lines or so.  Does that sound reasonable?
> 
> I don't think that will be enough: the output stream simply is not fast
> enough to keep up.

That'd be strange, since it evidently does succeed to keep up when we
redraw the display "normally", i.e. not due to dropping down or
updating a menu.

Anyway, I took another look at all the screenshots posted in this bug
report, and my conclusion is that all of the artifacts seem to be
caused by 2 root causes:

  . Incorrect position of the cursor when a menu item is redrawn.

    This explains why some menu items are "duplicated" elsewhere in
    the menu, and also why items that needed to be redrawn with blue
    background (i.e. in the "non-selected" face) are left with the red
    background of the "selected" face.

    In all of these cases, it looks like the cursor was positioned at
    the EOB of the buffer (*scratch*) displayed beneath the menu.
    This is the "normal" cursor position, as Emacs perceives it, and
    that's where redisplay leaves the cursor after updating the frame
    display.  We override that in the menu-display code, by sending a
    cursor motion command after the frame is completely displayed; it
    looks like in at least some of the cases this cursor motion
    command was not obeyed.

  . The insert mode is not turned off before some string is written to
    the display.

    This explains why we see menu items to the right of the menu: they
    were "pushed" by writing some other text to their left, while in
    insert mode.  There are similar problems in some of the
    screenshots with the help-echo displayed in the echo area.

    Alternatively, this type of artifacts can also be explained by
    incorrect cursor position in the horizontal axis.  To decide which
    explanation is correct, I'd need to see the artifacts when the
    underlying window is full of some buffer text, not almost empty as
    in *scratch*.  Glenn, could I persuade you to try that and show
    the screenshots?

Does the above ring any bells?

Please also note that Emacs tries to be "clever" about cursor motion
on a TTY: it chooses out of several possible methods of moving the
cursor, comparing their "costs" (see cm.c:cmgoto for details).  So
it's possible that different cursor movements emit different commands
to the terminal driver, and somehow trigger these strange effects.

Finally, one difference between the "normal" screen update and the one
we use to display and update TTY menus is that the latter seems to
cause significantly more cursor motion.  One possible way to cut down
on that is to set show-help-function to nil.  Glenn, can I ask you to
try that and see if it helps in any way?

TIA

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

Eli Zaretskii wrote:

>     explanation is correct, I'd need to see the artifacts when the
>     underlying window is full of some buffer text, not almost empty as
>     in *scratch*.  Glenn, could I persuade you to try that and show
>     the screenshots?

See 1.png.

> cause significantly more cursor motion.  One possible way to cut down
> on that is to set show-help-function to nil.  Glenn, can I ask you to
> try that and see if it helps in any way?

It does not help.
See 2.png. It leaves my screen like 3.png after closing the menu.


[-- Attachment #2: 1.png --]
[-- Type: image/png, Size: 56644 bytes --]

[-- Attachment #3: 2.png --]
[-- Type: image/png, Size: 55513 bytes --]

[-- Attachment #4: 3.png --]
[-- Type: image/png, Size: 34252 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Thomas Dickey <dickey@his.com>,  17497@debbugs.gnu.org
> Date: Mon, 02 Jun 2014 12:14:05 -0400
> 
> >     explanation is correct, I'd need to see the artifacts when the
> >     underlying window is full of some buffer text, not almost empty as
> >     in *scratch*.  Glenn, could I persuade you to try that and show
> >     the screenshots?
> 
> See 1.png.

Thanks.  Inconclusive.

> > cause significantly more cursor motion.  One possible way to cut down
> > on that is to set show-help-function to nil.  Glenn, can I ask you to
> > try that and see if it helps in any way?
> 
> It does not help.
> See 2.png. It leaves my screen like 3.png after closing the menu.

Not sure what that means, except that the terminal scrolled for some
reason.

Btw, did you use menu-bar-open (as opposed to F10) in all your
experiments till now, or just this time?

Thanks.

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> Btw, did you use menu-bar-open (as opposed to F10) in all your
> experiments till now, or just this time?

I generally (but not I think always) used menu-bar-open.
Some terminal emulators use F10 to open their own menus, and I could not
be bothered to figure out how to disable that. :)

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dickey@his.com,  17497@debbugs.gnu.org
> Date: Mon, 02 Jun 2014 12:46:53 -0400
> 
> I generally (but not I think always) used menu-bar-open.

Just FYI: doing so exposes a subtle bug, due to the fact that "M-x"
adds an item to the menu bar, which confuses the menu code if you use
horizontal arrows or C-f/C-b.

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

>> I generally (but not I think always) used menu-bar-open.
>
> Just FYI: doing so exposes a subtle bug, due to the fact that "M-x"
> adds an item to the menu bar, which confuses the menu code if you use
> horizontal arrows or C-f/C-b.

OK, well I did say that I used M-x menu-bar-open in my first post to
this report (which isn't even my bug report BTW... :) )

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497#11

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> One more idea: can you try decreasing the number 900 in this snippet
> from dispnew.c:

I set it to 450: still glitchy.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dmantipov@yandex.ru,  17497@debbugs.gnu.org
> Date: Tue, 03 Jun 2014 00:51:50 -0400
> 
> Eli Zaretskii wrote:
> 
> > One more idea: can you try decreasing the number 900 in this snippet
> > from dispnew.c:
> 
> I set it to 450: still glitchy.

Thanks for trying.

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

Hi Thomas,

>> Most of the file, however, consists of repainting the menu.
> By the way, since the menu covers only half the screen, there aren't
> any general-purpose scrolling optimizations that would help.  (xterm
> does support left/right margins, which would be interesting to explore
> in this area).  What ncurses does when it's getting behind is to drop
> updates - the typeahead feature:

Currently, we're not concerned about optimization, just about tracking
down a display glitch.  The tricky part of this glitch is that if we
record&replay Emacs's output, the "replay" does not suffer from the
same glitch.

So apparently the terminal emulator behaves differently in the "live"
case than in the "replay" case for some reason.  We tried to replay at
different speeds to see if it was related to timing, but to no avail.

To me, the next logical explanation is that the terminal emulator's
behavior is influenced y the relative timing of *input* and output.

For that reasons, your ncurses explanation is very interesting, yet
I can't imagine how it can be directly related since our "record&replay"
is done "after" ncurses, directly in the stream between Emacs's process
and the terminal emulator.

Do you have some other insight that could explain why the terminal
emulator would react differently in the "live" case than in the
"replay" case?


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 2190 bytes --]

On Tue, Jun 03, 2014 at 09:43:04AM -0400, Stefan Monnier wrote:
> Hi Thomas,
> 
> >> Most of the file, however, consists of repainting the menu.
> > By the way, since the menu covers only half the screen, there aren't
> > any general-purpose scrolling optimizations that would help.  (xterm
> > does support left/right margins, which would be interesting to explore
> > in this area).  What ncurses does when it's getting behind is to drop
> > updates - the typeahead feature:
> 
> Currently, we're not concerned about optimization, just about tracking
> down a display glitch.  The tricky part of this glitch is that if we
> record&replay Emacs's output, the "replay" does not suffer from the
> same glitch.

sure - but I gave my best answer at the outset.  The behavior which
you are describing won't show up by doing replay's, because it would
occur when you have input (from the keyboard) interfering with the
output.
 
> So apparently the terminal emulator behaves differently in the "live"
> case than in the "replay" case for some reason.  We tried to replay at
> different speeds to see if it was related to timing, but to no avail.
> 
> To me, the next logical explanation is that the terminal emulator's
> behavior is influenced y the relative timing of *input* and output.

:-)
 
> For that reasons, your ncurses explanation is very interesting, yet
> I can't imagine how it can be directly related since our "record&replay"
> is done "after" ncurses, directly in the stream between Emacs's process
> and the terminal emulator.
> 
> Do you have some other insight that could explain why the terminal
> emulator would react differently in the "live" case than in the
> "replay" case?

repeating: if the cursor-key sequence of characters is misinterpreted
(for example due to timeouts), then fragments of the sequences will
echo as unexpected characters.

Incidentally, if an *output* splits up an escape sequence across
buffers in fflush's, then that can also open up holes in timing.
But I think that's less of concern...

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Tue, 03 Jun 2014 14:47:49 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: Thomas Dickey <dickey@his.com>, 17497@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> 
> > Currently, we're not concerned about optimization, just about tracking
> > down a display glitch.  The tricky part of this glitch is that if we
> > record&replay Emacs's output, the "replay" does not suffer from the
> > same glitch.
> 
> sure - but I gave my best answer at the outset.  The behavior which
> you are describing won't show up by doing replay's, because it would
> occur when you have input (from the keyboard) interfering with the
> output.
>  
> > So apparently the terminal emulator behaves differently in the "live"
> > case than in the "replay" case for some reason.  We tried to replay at
> > different speeds to see if it was related to timing, but to no avail.
> > 
> > To me, the next logical explanation is that the terminal emulator's
> > behavior is influenced y the relative timing of *input* and output.
> 
> :-)
>  
> > For that reasons, your ncurses explanation is very interesting, yet
> > I can't imagine how it can be directly related since our "record&replay"
> > is done "after" ncurses, directly in the stream between Emacs's process
> > and the terminal emulator.
> > 
> > Do you have some other insight that could explain why the terminal
> > emulator would react differently in the "live" case than in the
> > "replay" case?
> 
> repeating: if the cursor-key sequence of characters is misinterpreted
> (for example due to timeouts), then fragments of the sequences will
> echo as unexpected characters.
> 
> Incidentally, if an *output* splits up an escape sequence across
> buffers in fflush's, then that can also open up holes in timing.
> But I think that's less of concern...

All this, including input interfering with output, happens during
normal Emacs redisplay, but we have never heard any complaints about
corrupted display like the ones we get with the menus.

The Emacs code which outputs commands and text to the terminal does
not know whether a menu has been dropped or not, it just compares the
previous display with the desired one, and sends commands to update
the regions of the screen that has been changed.

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]

On Wed, Jun 04, 2014 at 12:07:24AM +0300, Eli Zaretskii wrote:
> > Date: Tue, 03 Jun 2014 14:47:49 -0400
> > From: Thomas Dickey <dickey@his.com>
> > Cc: Thomas Dickey <dickey@his.com>, 17497@debbugs.gnu.org,
> >  Eli Zaretskii <eliz@gnu.org>
> > 
> > > Currently, we're not concerned about optimization, just about tracking
> > > down a display glitch.  The tricky part of this glitch is that if we
> > > record&replay Emacs's output, the "replay" does not suffer from the
> > > same glitch.
> > 
> > sure - but I gave my best answer at the outset.  The behavior which
> > you are describing won't show up by doing replay's, because it would
> > occur when you have input (from the keyboard) interfering with the
> > output.
> >  
> > > So apparently the terminal emulator behaves differently in the "live"
> > > case than in the "replay" case for some reason.  We tried to replay at
> > > different speeds to see if it was related to timing, but to no avail.
> > > 
> > > To me, the next logical explanation is that the terminal emulator's
> > > behavior is influenced y the relative timing of *input* and output.
> > 
> > :-)
> >  
> > > For that reasons, your ncurses explanation is very interesting, yet
> > > I can't imagine how it can be directly related since our "record&replay"
> > > is done "after" ncurses, directly in the stream between Emacs's process
> > > and the terminal emulator.
> > > 
> > > Do you have some other insight that could explain why the terminal
> > > emulator would react differently in the "live" case than in the
> > > "replay" case?
> > 
> > repeating: if the cursor-key sequence of characters is misinterpreted
> > (for example due to timeouts), then fragments of the sequences will
> > echo as unexpected characters.
> > 
> > Incidentally, if an *output* splits up an escape sequence across
> > buffers in fflush's, then that can also open up holes in timing.
> > But I think that's less of concern...
> 
> All this, including input interfering with output, happens during
> normal Emacs redisplay, but we have never heard any complaints about
> corrupted display like the ones we get with the menus.

The menus cover _half_ of the screen, defeating any attempt to speed up
scrolling by using the terminal's indexing/scrolling features.  In the
view without menus, the scrolled text (unless you're considering scrolling
a pane of a vertically split window), is full-width, and works with
the terminal's scrolling features.

(If I were debugging something of the sort we're discussing, I'd log
the input-stream in terms of what the application is seeing, to look
for broken cursor-up/down sequences).
 
> The Emacs code which outputs commands and text to the terminal does
> not know whether a menu has been dropped or not, it just compares the
> previous display with the desired one, and sends commands to update
> the regions of the screen that has been changed.

sure - we're talking about characters.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

> repeating: if the cursor-key sequence of characters is misinterpreted
> (for example due to timeouts), then fragments of the sequences will
> echo as unexpected characters.

I do not understand what that means.  By "cursor-key sequence of
characters" do you mean the escape sequence sent by the terminal to
Emacs or the cursor-motion escape sequences sent from Emacs to the
text terminal?
What timeouts are you referring to?  Are you saying that the terminal
emulator has internal timeouts?  If so, can you describe when and how
they come into play?

It seems you're saying that the escape sequences Emacs sends to the
terminal need to be within a single "packet", because if such an escape
sequence is split among several "fflush" the terminal emulator won't
wait indefinitely for the second chunk, and instead will timeout after
a fairly short wait and consider the escape sequence as "unexpected
characters".


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Tue, 03 Jun 2014 18:21:55 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: dickey@his.com, monnier@iro.umontreal.ca, 17497@debbugs.gnu.org
> 
> > All this, including input interfering with output, happens during
> > normal Emacs redisplay, but we have never heard any complaints about
> > corrupted display like the ones we get with the menus.
> 
> The menus cover _half_ of the screen, defeating any attempt to speed up
> scrolling by using the terminal's indexing/scrolling features.  In the
> view without menus, the scrolled text (unless you're considering scrolling
> a pane of a vertically split window), is full-width, and works with
> the terminal's scrolling features.

You can easily make the "normal" display do the same: just divide the
window in 2 halves by typing "C-x 3", then scroll one of them with
C-v or the arrow keys.  If you actually lean on the scrolling key, you
will make sure output and input are intermingled.

If this situation were inherently conducive to display bugs, we would
have heard about it long time ago.

So I think there's something about the screen update due to menus that
triggers these problems.  I just have run out of ideas about what that
factor(s) could be, and I don't have access to any system where this
happens to debug that.

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On Tue, Jun 03, 2014 at 11:03:08PM -0400, Stefan Monnier wrote:
> > repeating: if the cursor-key sequence of characters is misinterpreted
> > (for example due to timeouts), then fragments of the sequences will
> > echo as unexpected characters.
> 
> I do not understand what that means.  By "cursor-key sequence of
> characters" do you mean the escape sequence sent by the terminal to
> Emacs or the cursor-motion escape sequences sent from Emacs to the
> text terminal?

From the terminal to Emacs.

Cursor up typically is

	ESC [ A

three characters.  Curses applications behave differently if you type those
three characters, one-by-one, because of timing.

> What timeouts are you referring to?  Are you saying that the terminal
> emulator has internal timeouts?  If so, can you describe when and how
> they come into play?

Typically the _application_ (Emacs) waits for completion of a control sequence.
 
> It seems you're saying that the escape sequences Emacs sends to the
> terminal need to be within a single "packet", because if such an escape
> sequence is split among several "fflush" the terminal emulator won't
> wait indefinitely for the second chunk, and instead will timeout after
> a fairly short wait and consider the escape sequence as "unexpected
> characters".

That's part of it.  Usually the input (from the terminal to the application)
is more important.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Wed, Jun 04, 2014 at 04:31:02AM -0400, Thomas Dickey wrote:
> On Tue, Jun 03, 2014 at 11:03:08PM -0400, Stefan Monnier wrote:
> > > repeating: if the cursor-key sequence of characters is misinterpreted
> > > (for example due to timeouts), then fragments of the sequences will
> > > echo as unexpected characters.
> > 
> > I do not understand what that means.  By "cursor-key sequence of
> > characters" do you mean the escape sequence sent by the terminal to
> > Emacs or the cursor-motion escape sequences sent from Emacs to the
> > text terminal?
> 
> From the terminal to Emacs.
> 
> Cursor up typically is
> 
> 	ESC [ A

...normal mode - application mode is still three:
  	ESC O A

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

On the hunch that the problems discussed here are caused by excess
movement of the cursor, I've made changes in revision 117203 on the
emacs-24 branch to avoid some of them.  Please try that.

If the problems are still there, one last idea I have is to ifdef away
the code in tty_menu_activate that invokes the help-echo display:

      /* Display the help-echo message for the currently-selected menu
	 item.  */
      if ((menu_help_message || prev_menu_help_message)
	  && menu_help_message != prev_menu_help_message)
	{
	  help_callback (menu_help_message,
			 menu_help_paneno, menu_help_itemno);
	  /* Move the cursor to the beginning of the current menu
	     item, so that screen readers and other accessibility aids
	     know where the active region is.  */
	  cursor_to (sf, row, col);
	  prev_menu_help_message = menu_help_message;
	}

This code invokes another update of the frame, in addition to the one
we already did when we called tty_menu_display immediately before
that.  It also moves the cursor (which is positioned by the frame
update to where point is in the selected window).  Maybe avoiding that
will avoid the problems.

If even that doesn't help, then try to ifdef away the calls to
tty_hide_cursor made by tty_menu_activate.

These two last ideas are the only differences I know of between a
"normal" redisplay and redisplay due to update of a menu.  If even
they don't make any difference, I guess this bug will have to remain
open until someone debugs it on a system where the problem can be
reproduced.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

Here's a simple recipe to generate screen updates very similar to the
ones caused by a menu update.  Do they cause similar artifacts on the
offending terminal emulators?

 emacs -Q -nw

 Insert and evaluate this function:
 (defun my-help (old new)
  ""
  (message "We are at %s" (point)))

 C-x 3

 C-x C-f src/xdisp.c RET (any sufficiently large file will do)

 M-x hl-line-mode RET

 M-: (progn
      (beginning-of-buffer)
      (dotimes (i 40)
       (put-text-property
        (line-beginning-position)
        (line-end-position)
        'point-entered
        'my-help)
       (forward-line))) RET

Now move cursor down and up through the first 40 lines of the buffer:
you should see the background of the current line change as you move,
and also help-echo-like messages in the echo area for each non-empty
line in the buffer that follows an empty line.

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

>> > repeating: if the cursor-key sequence of characters is misinterpreted
>> > (for example due to timeouts), then fragments of the sequences will
>> > echo as unexpected characters.
>> I do not understand what that means.  By "cursor-key sequence of
>> characters" do you mean the escape sequence sent by the terminal to
>> Emacs or the cursor-motion escape sequences sent from Emacs to the
>> text terminal?
> From the terminal to Emacs.
> Cursor up typically is
> 	ESC [ A
> three characters.

But that's a completely different problem from the one
we're investigating.  The problem is about how the terminal emulator
reacts to Emacs's output, not how Emacs reacts to the terminal
emulator's output.

> Curses applications behave differently if you type those
> three characters, one-by-one, because of timing.

We try pretty hard to avoid using such timeouts in Emacs, so by and
large this doesn't apply to Emacs at all.

> That's part of it.  Usually the input (from the terminal to the application)
> is more important.

But the present bug-report is about the other case.
To repeat the problem:
- a particular run of Emacs sends bytes <typescript> to the terminal
  emulator, and the result doesn't look right.
- yet when we replay the same <typescript> directly to the terminal, the
  result looks correct.
Note also that (other than the display), the state of Emacs at the end
of the test is correct (and forcing a redisplay indeed fixes the
display), so the problem is not in Emacs misinterpreting input because
of timing issues.


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

Eli Zaretskii wrote:

> Here's a simple recipe to generate screen updates very similar to the
> ones caused by a menu update.  Do they cause similar artifacts on the
> offending terminal emulators?

Not for me, no.

bug#17497: 24.4.50; TTY menu glitches

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 248 bytes --]

Eli Zaretskii wrote:

> On the hunch that the problems discussed here are caused by excess
> movement of the cursor, I've made changes in revision 117203 on the
> emacs-24 branch to avoid some of them.  Please try that.

Sorry, still glitches. :(


[-- Attachment #2: 1.png --]
[-- Type: image/png, Size: 32194 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: dickey@his.com,  Stefan Monnier <monnier@iro.umontreal.ca>,  17497@debbugs.gnu.org
> Date: Wed, 04 Jun 2014 12:08:47 -0400
> 
> Eli Zaretskii wrote:
> 
> > Here's a simple recipe to generate screen updates very similar to the
> > ones caused by a menu update.  Do they cause similar artifacts on the
> > offending terminal emulators?
> 
> Not for me, no.

As expected.  Thanks for testing.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Thomas Dickey <dickey@his.com>,  monnier@iro.umontreal.ca,  17497@debbugs.gnu.org
> Date: Wed, 04 Jun 2014 12:09:26 -0400
> 
> > On the hunch that the problems discussed here are caused by excess
> > movement of the cursor, I've made changes in revision 117203 on the
> > emacs-24 branch to avoid some of them.  Please try that.
> 
> Sorry, still glitches. :(

Not surprisingly.

It still happens because these 3 lines:

  --
  Quit                  C-x C-c
 Save unsaved buffers, then exit

were written starting from cursor position at the EOB of *scratch*,
instead of way below, on the line below the "Delete Frame" item.  IOW,
some cursor motion commands emitted by Emacs are not obeyed for some
reason.

Thanks.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Thomas Dickey <dickey@his.com>,  monnier@iro.umontreal.ca,  17497@debbugs.gnu.org
> Date: Wed, 04 Jun 2014 12:09:26 -0400
> 
> Eli Zaretskii wrote:
> 
> > On the hunch that the problems discussed here are caused by excess
> > movement of the cursor, I've made changes in revision 117203 on the
> > emacs-24 branch to avoid some of them.  Please try that.
> 
> Sorry, still glitches. :(

If you put a breakpoint in update_frame and in update_frame_with_menu,
like shown below, how many calls do you see when you press an arrow
key (or C-n/C-p) that produces a glitch?  I expect 1 call to
update_frame_with_menu followed by one call to update_frame.  Here's a
GDB session on fencepost to show what I see:

  (gdb) break update_frame
  Breakpoint 1 at 0x41b605: file dispnew.c, line 3006.
  (gdb) commands
  Type commands for breakpoint(s) 3, one per line.
  End with a line saying just "end".
  >continue
  >end
  (gdb) break update_frame_with_menu
  Breakpoint 2 at 0x41b81b: file dispnew.c, line 3112.
  (gdb) commands
  Type commands for breakpoint(s) 4, one per line.
  End with a line saying just "end".
  >continue
  >end
  (gdb) c
  Continuing.

  Breakpoint 1, update_frame (f=0xd3e808, force_p=false, 
      inhibit_hairy_id_p=false) at dispnew.c:3006
  3006	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=-1, col=-1)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=0, col=0)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=1, col=0)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 1, update_frame (f=0xd3e808, force_p=true, inhibit_hairy_id_p=true)
      at dispnew.c:3006
  3006	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=2, col=0)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 1, update_frame (f=0xd3e808, force_p=true, inhibit_hairy_id_p=true)
      at dispnew.c:3006
  3006	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=3, col=0)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 1, update_frame (f=0xd3e808, force_p=true, inhibit_hairy_id_p=true)
      at dispnew.c:3006
  3006	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 2, update_frame_with_menu (f=0xd3e808, row=2, col=0)
      at dispnew.c:3112
  3112	  struct window *root_window = XWINDOW (f->root_window);

  Breakpoint 1, update_frame (f=0xd3e808, force_p=true, inhibit_hairy_id_p=true)
      at dispnew.c:3006
  3006	  struct window *root_window = XWINDOW (f->root_window);

Explanations:

 . first, a call to update_frame is because menu-bar-open calls
   'redisplay'

 . next, a call to update_frame_with_menu which comes from
   the beginning of tty_menu_activate, line 3230 of term.c

 . another call to update_frame_with_menu, to display the "File >"
   header of the File menu.

 . yet another call to update_frame_with_menu to display the first
   menu item (Visit New File) in red background

 . finally, a call to update_frame to display the help-echo for "Visit
   New File"

Thereafter, for each keypress of an arrow key, there's always 1 call
to update_frame_with_menu, to display a different menu item in the red
background (see the values of row and col arguments, which identify
the menu item), followed by a single call to update_frame to display
the help echo.

Do you see any calls to update_frame beyond what I described?  If so,
perhaps they are the reason the cursor gets back to the EOB of the
underlying buffer.

TIA

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Wed, Jun 04, 2014 at 09:06:11AM -0400, Stefan Monnier wrote:
> > Curses applications behave differently if you type those
> > three characters, one-by-one, because of timing.
> 
> We try pretty hard to avoid using such timeouts in Emacs, so by and
> large this doesn't apply to Emacs at all.

As it happens, that rule of thumb doesn't apply to terminals:

https://github.com/mirrors/emacs/blob/master/src/keyboard.c

(it's implemented there, at any rate)

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

>> > Curses applications behave differently if you type those
>> > three characters, one-by-one, because of timing.
>> We try pretty hard to avoid using such timeouts in Emacs, so by and
>> large this doesn't apply to Emacs at all.
> As it happens, that rule of thumb doesn't apply to terminals:
> https://github.com/mirrors/emacs/blob/master/src/keyboard.c
> (it's implemented there, at any rate)

Where do you see the timeouts?

Or rather, no, don't bother, because even there might be problem in how
we process the input escape sequences, these are unrelated to the
display glitches we see.  So let's focus on the display glitches.


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Wed, Jun 04, 2014 at 08:47:42PM -0400, Stefan Monnier wrote:
> >> > Curses applications behave differently if you type those
> >> > three characters, one-by-one, because of timing.
> >> We try pretty hard to avoid using such timeouts in Emacs, so by and
> >> large this doesn't apply to Emacs at all.
> > As it happens, that rule of thumb doesn't apply to terminals:
> > https://github.com/mirrors/emacs/blob/master/src/keyboard.c
> > (it's implemented there, at any rate)
> 
> Where do you see the timeouts?

start with
	static bool get_input_pending (int);

The code's waiting for a while to ensure that it gets all of the bytes in
a cursor-key/function-key/mouse-event sequence.
 
> Or rather, no, don't bother, because even there might be problem in how
> we process the input escape sequences, these are unrelated to the
> display glitches we see.  So let's focus on the display glitches.

You could cut the discussion short by making the check that I suggested:
logging the decoded character/special-key values to look for instances
where the decoding returns individual bytes.

-- 
Thomas E. Dickey <dickey@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

bug#17497: 24.4.50; TTY menu glitches

[Andreas Schwab]

Thomas Dickey <dickey@his.com> writes:

> start with
> 	static bool get_input_pending (int);
>
> The code's waiting for a while to ensure that it gets all of the bytes in
> a cursor-key/function-key/mouse-event sequence.

No, it doesn't.  It just reads whatever is pending, without paying
attention to key boundaries.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

> You could cut the discussion short by making the check that I suggested:
> logging the decoded character/special-key values to look for instances
> where the decoding returns individual bytes.

No we cut it short by noticing that Emacs's internal state is the one we
expect and the display is completely wrong, which is something that
cannot happen due to some weird input sequence.

Please believe us, we *really* know that the problem is not in Emacs's
misunderstanding/misdecoding of the input stream (yes, it does have
bugs there, of course, but they're unrelated).

To repeat: replaying the termscript results in a different display (not
only that, it results in the display we want)!
This can't be due to Emacs's input processing.


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Stefan Monnier]

> The code's waiting for a while to ensure that it gets all of the bytes in
> a cursor-key/function-key/mouse-event sequence.
 
No, it's not.  Try it:

    emacs -Q -nw
    then hit ESC O A on your keyboard.

Feel free to type those three keys real slow, or real fast.  It makes
little difference.


        Stefan

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> Date: Thu, 05 Jun 2014 04:21:51 -0400
> From: Thomas Dickey <dickey@his.com>
> Cc: Thomas Dickey <dickey@his.com>, 17497@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> 
> > Or rather, no, don't bother, because even there might be problem in how
> > we process the input escape sequences, these are unrelated to the
> > display glitches we see.  So let's focus on the display glitches.
> 
> You could cut the discussion short by making the check that I suggested:
> logging the decoded character/special-key values to look for instances
> where the decoding returns individual bytes.

I don't think this is the problem in this case.  The arrow keys are
clearly decoded correctly and obeyed, as we see the reaction to them,
which is to redraw certain portions of the screen.  It's not like an
arrow key we get from the keyboard is sent verbatim to the terminal;
rather, Emacs interprets that key as a command to change the
background of two screen lines, and then sends the related commands,
including cursor motion, to the terminal.  IOW, the cursor motion
commands sent to the terminal are not what we receive from the
keyboard, they are generated by Emacs using a non-trivial logic in
cmgoto (which could decide that it is better to send a single newline
character, if it needs to move down just one line, or move to the
upper-left corner of the screen, for example).

Moreover, using C-n and C-p, which are single bytes, doesn't make the
problem go away.

So some other factor is at work here.  Keyboard input is unrelated,
at least as far as Emacs's code is concerned.

bug#17497: 24.4.50; TTY menu glitches

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Thu, 05 Jun 2014 10:29:14 +0200
> Cc: 17497@debbugs.gnu.org
> 
> Thomas Dickey <dickey@his.com> writes:
> 
> > start with
> > 	static bool get_input_pending (int);
> >
> > The code's waiting for a while to ensure that it gets all of the bytes in
> > a cursor-key/function-key/mouse-event sequence.
> 
> No, it doesn't.  It just reads whatever is pending, without paying
> attention to key boundaries.

In addition, the function which reads keys isn't called during menu
updates, because we call update_frame_1 with its argument force_p
non-zero, so update_frame_1 doesn't even pay attention whether there's
input available.

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> Date: Wed, 18 Jun 2014 13:57:45 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> Cc: emacs-devel@gnu.org
> 
> On 06/18/2014 01:50 PM, Mario Lang wrote:
> 
> > Just tried the text-mode menu in emacs-24 branch, and it is rather
> > broken.  It is kind of hard for me to describe the bug in detail, but
> > scrolling through the menu items of the "Edit" menu gives me randomly
> > wrong aligned menu items.  "Undo" is not aligned with the rest of the
> > menu, but aligned to the left of the screen.  Quitting the menu gives me
> > left-over characters from these wrong-aligned menu items.
> >
> > Can someone experienced with the new text mode menu take a look at this
> > please?  I am seeing it on Linux in tty mode, with a "--without-x"
> > complied emacs.
> 
> Check screenshots from http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497,
> most probably you hit the same known issue.

Since I cannot reproduce this on any system I have access to, I'm
basically waiting for someone to step forward to either give me a
login on a system where this happens, or work closely with me on
debugging this on their machine.  The two last ideas I had and asked
to try didn't get any followup at all, so I guess no one is interested
enough to work on this, or let me do it on their machine.

If no one steps forward, I'm afraid this will remain "a known issue",
at least on some systems and with some terminal emulators.

(I've redirected this discussion to the bug address, btw.)

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Date: Wed, 18 Jun 2014 13:14:51 +0200
> 
> > Check screenshots from http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497,
> > most probably you hit the same known issue.
> 
> I can't see plain image file screenshots, but the (terse) description in
> the bug report sort of matches what I see, so likely this is the same
> issue.  Thanks for pointing this out.

Can you state your system name/version and the terminal emulator you
are using, for the record?  Past experience indicates that perhaps
this problem affects only some systems/emulators.

Thanks.

bug#17497: The text-mode menu looks very broken in emacs-24

[Alan Mackenzie]

Hi, Eli.

Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Mario Lang <mlang@delysid.org>
>> Date: Wed, 18 Jun 2014 13:14:51 +0200
 
>> > Check screenshots from http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497,
>> > most probably you hit the same known issue.
 
>> I can't see plain image file screenshots, but the (terse) description in
>> the bug report sort of matches what I see, so likely this is the same
>> issue.  Thanks for pointing this out.

> Can you state your system name/version and the terminal emulator you
> are using, for the record?  Past experience indicates that perhaps
> this problem affects only some systems/emulators.

I think this is the case.  I've started an up to date system from the
Emacs-24 branch, running on a GNU system on a Linux tty, with -Q.  I
configured the build with:

   ./configure --with-gif=no --with-tiff=no --with-gpm

.  The menus seem to work fine, except for one glitch.  When I click the
(GPM) mouse on a menu, the menu drops down beginning horizontally where
the mouse was, rather than where the menu label was.  I.e., I see this:

    File Ed Edit > ns Buffers Tools Lisp-Interaction Help
            Undo                    C-x u
            Cut                     C-w
    ...

, when I would rather expect to see this:

    File Edit > tions Buffers Tools Lisp-Interaction Help
         Undo                    C-x u
         Cut                     C-w
    ....

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> From: Alan Mackenzie <acm@muc.de>
> Date: Thu, 19 Jun 2014 14:11:41 +0000 (UTC)
> 
> .  The menus seem to work fine, except for one glitch.  When I click the
> (GPM) mouse on a menu, the menu drops down beginning horizontally where
> the mouse was, rather than where the menu label was.  I.e., I see this:
> 
>     File Ed Edit > ns Buffers Tools Lisp-Interaction Help
>             Undo                    C-x u
>             Cut                     C-w
>     ...
> 
> , when I would rather expect to see this:
> 
>     File Edit > tions Buffers Tools Lisp-Interaction Help
>          Undo                    C-x u
>          Cut                     C-w
>     ....

This is not a glitch, this is intended behavior when your text
terminal has a mouse.  Clicking anywhere inside "Edit" will drop down
the "Edit" menu starting at the click position.

Can you drop the menu with F10, try navigating through menus with
arrow keys, both horizontal and vertical, and report whether you see
any problems?  If you see any problems, I'd appreciate screenshots
showing them.

Whatever the results, I'd like you know what are your OS and terminal
emulator, please.

Thanks.

bug#17497: The text-mode menu looks very broken in emacs-24

[Alan Mackenzie]

Hi, Eli.

On Thu, Jun 19, 2014 at 06:07:08PM +0300, Eli Zaretskii wrote:
> > From: Alan Mackenzie <acm@muc.de>
> > Date: Thu, 19 Jun 2014 14:11:41 +0000 (UTC)

> > .  The menus seem to work fine, except for one glitch.  When I click the
> > (GPM) mouse on a menu, the menu drops down beginning horizontally where
> > the mouse was, rather than where the menu label was.  I.e., I see this:

> >     File Ed Edit > ns Buffers Tools Lisp-Interaction Help
> >             Undo                    C-x u
> >             Cut                     C-w
> >     ...

> > , when I would rather expect to see this:

> >     File Edit > tions Buffers Tools Lisp-Interaction Help
> >          Undo                    C-x u
> >          Cut                     C-w
> >     ....

> This is not a glitch, this is intended behavior when your text
> terminal has a mouse.  Clicking anywhere inside "Edit" will drop down
> the "Edit" menu starting at the click position.

Ah.  OK.  This seems at bit strange to me, but if it's intended
behaviour, then it's clearly not a glitch.

> Can you drop the menu with F10, try navigating through menus with
> arrow keys, both horizontal and vertical, and report whether you see
> any problems?  If you see any problems, I'd appreciate screenshots
> showing them.

I see no problems whatsoever.

> Whatever the results, I'd like you know what are your OS and terminal
> emulator, please.

Gentoo GNU/Linux, on a Linux virtual terminal (i.e. not in a terminal
emulator, not on X).

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#17497: The text-mode menu looks very broken in emacs-24

[Mario Lang]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mario Lang <mlang@delysid.org>
>> Date: Wed, 18 Jun 2014 13:14:51 +0200
>> 
>> > Check screenshots from http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497,
>> > most probably you hit the same known issue.
>> 
>> I can't see plain image file screenshots, but the (terse) description in
>> the bug report sort of matches what I see, so likely this is the same
>> issue.  Thanks for pointing this out.
>
> Can you state your system name/version and the terminal emulator you
> are using, for the record?  Past experience indicates that perhaps
> this problem affects only some systems/emulators.

I am running in GNU/Screen on GNU/Linux (kernel 3.14).

This bug only happens when scrolling down anything else than the first
toplevel entry, because the first toplevel menu is already
left-justified...  And it does not happen all the time.  In fact,
while it is reproducible, it doesnt reproduce the same way everytime I
try it.  Sometimes I need to go down 10 or 15 elements until one gets}
drawn in the wrong position, and sometimes I just go down two or three
items.  I am not sure, but it *feels* like the issue is easier to
reproduce if I hit down-arrow rapidly in succession.

This *feels* like memory corruption, as if the x coordinate is randomly
set to 0 in some situations.

-- 
CYa,
  ⡍⠁⠗⠊⠕

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Cc: 17497@debbugs.gnu.org
> Date: Sat, 21 Jun 2014 12:14:44 +0200
> 
> I am running in GNU/Screen on GNU/Linux (kernel 3.14).

Thanks.  Which brand of GNU/Linux is that?

> This bug only happens when scrolling down anything else than the first
> toplevel entry, because the first toplevel menu is already
> left-justified...  And it does not happen all the time.  In fact,
> while it is reproducible, it doesnt reproduce the same way everytime I
> try it.  Sometimes I need to go down 10 or 15 elements until one gets}
> drawn in the wrong position, and sometimes I just go down two or three
> items.  I am not sure, but it *feels* like the issue is easier to
> reproduce if I hit down-arrow rapidly in succession.
> 
> This *feels* like memory corruption, as if the x coordinate is randomly
> set to 0 in some situations.

I don't think it's memory corruption.  I suspect the problem is
related to the different ways cmgoto chooses to get to the specified
screen coordinates.  If someone on the affected systems could tweak
cmgoto such that only the direct move is used, i.e. this code:

      p = (dcm == tty->Wcm->cm_habs
           ? tgoto (dcm, row, col)
           : tgoto (dcm, col, row));
      emacs_tputs (tty, p, 1, cmputc);
      curY (tty) = row, curX (tty) = col;

then perhaps we could see if my suspicions are in the right direction.

Thanks.

bug#17497: The text-mode menu looks very broken in emacs-24

[Martin Trojer]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

> I don't think it's memory corruption.  I suspect the problem is
> related to the different ways cmgoto chooses to get to the specified
> screen coordinates.  If someone on the affected systems could tweak
> cmgoto such that only the direct move is used, i.e. this code:
>
>      p = (dcm == tty->Wcm->cm_habs
>           ? tgoto (dcm, row, col)
>           : tgoto (dcm, col, row));
>      emacs_tputs (tty, p, 1, cmputc);
>      curY (tty) = row, curX (tty) = col;
>
> then perhaps we could see if my suspicions are in the right direction.


Hi Eli,

I just applied this patch trying to solve a different but related
issue (https://www.virtualbox.org/ticket/13687).

I am happy to report that applying this patch completely solved the
redrawing issues I was experiencing!

One option for you to recreate this issue locally seems to be to
install virtual box, config a virtual machine with multiple cores, ssh
into it and rung emacs.

I've put my diff here
(https://github.com/martintrojer/emacs/commit/bdff1ff98d02f4307659c052d0b35a40a36f0706),
and I understand it can't be merged as-is hopefully this can help you
find a solution that go into mainline emacs.

[-- Attachment #2: Type: text/html, Size: 2307 bytes --]

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> Date: Wed, 18 Mar 2015 11:45:17 +0000
> From: Martin Trojer <martin.trojer@gmail.com>
> 
> > I don't think it's memory corruption.  I suspect the problem is
> > related to the different ways cmgoto chooses to get to the specified
> > screen coordinates.  If someone on the affected systems could tweak
> > cmgoto such that only the direct move is used, i.e. this code:
> >
> >      p = (dcm == tty->Wcm->cm_habs
> >           ? tgoto (dcm, row, col)
> >           : tgoto (dcm, col, row));
> >      emacs_tputs (tty, p, 1, cmputc);
> >      curY (tty) = row, curX (tty) = col;
> >
> > then perhaps we could see if my suspicions are in the right direction.
> 
> Hi Eli,
> I just applied this patch trying to solve a different but related issue (https://www.virtualbox.org/ticket/13687).
> I am happy to report that applying this patch completely solved the redrawing issues I was experiencing!

You don't quote the person who wrote the above, so I need to ask you
who wrote the code.  We need to know to whom to attribute it.

Also, I'd be happier if someone could explain why this is the the
right thing to do in that place.

Thanks.

bug#17497: The text-mode menu looks very broken in emacs-24

[Martin Trojer]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Hello,

I believe that snippet of code was from you :) I simply put that quoted
snipped in the cmgoto function as per your instructions. I'm sure this
isn't the 'right thing to do' since it completely bypasses the switch
statement that tries to do the cheapest cursor move.

For what I can tell from the long discussion this is very hairy bug to
observe and debug. But for some reason that I don't fully understand this
patch solves my re-draw issues.

Also, if you are interested in solving this issue properly, running emacs
inside a VM (in this case Virtualbox) seems to make re-draw issues a lot
more likely and thus perhaps easier to diagnose.

Cheers...

On Wed, Mar 18, 2015 at 4:40 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Wed, 18 Mar 2015 11:45:17 +0000
> > From: Martin Trojer <martin.trojer@gmail.com>
> >
> > > I don't think it's memory corruption.  I suspect the problem is
> > > related to the different ways cmgoto chooses to get to the specified
> > > screen coordinates.  If someone on the affected systems could tweak
> > > cmgoto such that only the direct move is used, i.e. this code:
> > >
> > >      p = (dcm == tty->Wcm->cm_habs
> > >           ? tgoto (dcm, row, col)
> > >           : tgoto (dcm, col, row));
> > >      emacs_tputs (tty, p, 1, cmputc);
> > >      curY (tty) = row, curX (tty) = col;
> > >
> > > then perhaps we could see if my suspicions are in the right direction.
> >
> > Hi Eli,
> > I just applied this patch trying to solve a different but related issue (
> https://www.virtualbox.org/ticket/13687).
> > I am happy to report that applying this patch completely solved the
> redrawing issues I was experiencing!
>
> You don't quote the person who wrote the above, so I need to ask you
> who wrote the code.  We need to know to whom to attribute it.
>
> Also, I'd be happier if someone could explain why this is the the
> right thing to do in that place.
>
> Thanks.
>

[-- Attachment #2: Type: text/html, Size: 2885 bytes --]

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> Date: Wed, 18 Mar 2015 16:51:48 +0000
> From: Martin Trojer <martin.trojer@gmail.com>
> 
> I believe that snippet of code was from you :) I simply put that quoted snipped
> in the cmgoto function as per your instructions. I'm sure this isn't the 'right
> thing to do' since it completely bypasses the switch statement that tries to do
> the cheapest cursor move. 

So what patch did you install, exactly, that fixed the problem?

bug#17497: The text-mode menu looks very broken in emacs-24

[Martin Trojer]

[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]

From bdff1ff98d02f4307659c052d0b35a40a36f0706 Mon Sep 17 00:00:00 2001
From: Martin Trojer <martin.trojer@gmail.com>
Date: Wed, 18 Mar 2015 11:44:02 +0000
Subject: [PATCH] emacs bug #17497 fix

---
 src/cm.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/cm.c b/src/cm.c
index 474f280..ed17447 100644
--- a/src/cm.c
+++ b/src/cm.c
@@ -371,6 +371,16 @@ cmgoto (struct tty_display_info *tty, int row, int col)
       dcm = tty->Wcm->cm_abs;
     }

+    /* only use direct moves */
+    cost = 0;
+    p = (dcm == tty->Wcm->cm_habs
+         ? tgoto (dcm, row, col)
+         : tgoto (dcm, col, row));
+    emacs_tputs (tty, p, 1, evalcost);
+    emacs_tputs (tty, p, 1, cmputc);
+    curY (tty) = row, curX (tty) = col;
+    return;
+
   /*
    * In the following comparison, the = in <= is because when the costs
    * are the same, it looks nicer (I think) to move directly there.


On Wed, Mar 18, 2015 at 5:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Wed, 18 Mar 2015 16:51:48 +0000
> > From: Martin Trojer <martin.trojer@gmail.com>
> >
> > I believe that snippet of code was from you :) I simply put that quoted
> snipped
> > in the cmgoto function as per your instructions. I'm sure this isn't the
> 'right
> > thing to do' since it completely bypasses the switch statement that
> tries to do
> > the cheapest cursor move.
>
> So what patch did you install, exactly, that fixed the problem?
>

[-- Attachment #2: Type: text/html, Size: 2019 bytes --]

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> Date: Wed, 18 Mar 2015 17:13:13 +0000
> From: Martin Trojer <martin.trojer@gmail.com>
> Cc: 17497@debbugs.gnu.org
> 
> From bdff1ff98d02f4307659c052d0b35a40a36f0706 Mon Sep 17 00:00:00 2001
> From: Martin Trojer <martin.trojer@gmail.com>
> Date: Wed, 18 Mar 2015 11:44:02 +0000
> Subject: [PATCH] emacs bug #17497 fix
> 
> ---
>  src/cm.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/src/cm.c b/src/cm.c
> index 474f280..ed17447 100644
> --- a/src/cm.c
> +++ b/src/cm.c
> @@ -371,6 +371,16 @@ cmgoto (struct tty_display_info *tty, int row, int col)
>        dcm = tty->Wcm->cm_abs;
>      }
>  
> +    /* only use direct moves */
> +    cost = 0;
> +    p = (dcm == tty->Wcm->cm_habs
> +         ? tgoto (dcm, row, col)
> +         : tgoto (dcm, col, row));
> +    emacs_tputs (tty, p, 1, evalcost);
> +    emacs_tputs (tty, p, 1, cmputc);
> +    curY (tty) = row, curX (tty) = col;
> +    return;
> +
>    /*
>     * In the following comparison, the = in <= is because when the costs
>     * are the same, it looks nicer (I think) to move directly there.

I now do see this problem on a GNU/Linux system to which I have
access, but applying this patch didn't solve the problem...

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> Date: Sat, 07 Oct 2017 12:38:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 17497@debbugs.gnu.org
> 
> > Date: Wed, 18 Mar 2015 17:13:13 +0000
> > From: Martin Trojer <martin.trojer@gmail.com>
> > Cc: 17497@debbugs.gnu.org
> > 
> > From bdff1ff98d02f4307659c052d0b35a40a36f0706 Mon Sep 17 00:00:00 2001
> > From: Martin Trojer <martin.trojer@gmail.com>
> > Date: Wed, 18 Mar 2015 11:44:02 +0000
> > Subject: [PATCH] emacs bug #17497 fix
> > 
> > ---
> >  src/cm.c | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> > 
> > diff --git a/src/cm.c b/src/cm.c
> > index 474f280..ed17447 100644
> > --- a/src/cm.c
> > +++ b/src/cm.c
> > @@ -371,6 +371,16 @@ cmgoto (struct tty_display_info *tty, int row, int col)
> >        dcm = tty->Wcm->cm_abs;
> >      }
> >  
> > +    /* only use direct moves */
> > +    cost = 0;
> > +    p = (dcm == tty->Wcm->cm_habs
> > +         ? tgoto (dcm, row, col)
> > +         : tgoto (dcm, col, row));
> > +    emacs_tputs (tty, p, 1, evalcost);
> > +    emacs_tputs (tty, p, 1, cmputc);
> > +    curY (tty) = row, curX (tty) = col;
> > +    return;
> > +
> >    /*
> >     * In the following comparison, the = in <= is because when the costs
> >     * are the same, it looks nicer (I think) to move directly there.
> 
> I now do see this problem on a GNU/Linux system to which I have
> access, but applying this patch didn't solve the problem...

Some debugging indicates that somehow the cursor is sometimes moved
behind our backs, when a frame with a TTY menu is being updated.  So I
installed a semi-kludgey workaround that seems to fix 99% of the
glitches.  Since I don't really understand why this change is needed,
and because of the 1% of glitches that still happen, I'm not closing
the bug, and will continue looking into this.

If someone knows which display ops could move the cursor without going
through cursor_to, please tell.

I hope the solution is not limited to the one system where I see this
problem.  People who experienced the problem are encouraged to try the
latest release branch and report the results here.

bug#17497: The text-mode menu looks very broken in emacs-24

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Some debugging indicates that somehow the cursor is sometimes moved
> behind our backs, when a frame with a TTY menu is being updated.  So I
> installed a semi-kludgey workaround that seems to fix 99% of the
> glitches.  Since I don't really understand why this change is needed,
> and because of the 1% of glitches that still happen, I'm not closing
> the bug, and will continue looking into this.

This was a year and a half ago.  I tried the recipe for reproducing the
bug for a couple of minutes, and I couldn't get it to glitch here, so is
it time to close this bug report?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17497: The text-mode menu looks very broken in emacs-24

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: martin.trojer@gmail.com,  17497@debbugs.gnu.org
> Date: Wed, 26 Jun 2019 01:32:52 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Some debugging indicates that somehow the cursor is sometimes moved
> > behind our backs, when a frame with a TTY menu is being updated.  So I
> > installed a semi-kludgey workaround that seems to fix 99% of the
> > glitches.  Since I don't really understand why this change is needed,
> > and because of the 1% of glitches that still happen, I'm not closing
> > the bug, and will continue looking into this.
> 
> This was a year and a half ago.  I tried the recipe for reproducing the
> bug for a couple of minutes, and I couldn't get it to glitch here, so is
> it time to close this bug report?

Yes, I think so.  Thanks.