bug#23819: 25.0.95; display botched badly in xterm window

bug#23819: 25.0.95; display botched badly in xterm window

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2034 bytes --]

This appears to be a relatively recent regression in the emacs-25 
branch. It may not be easy to bisect. I first noticed it a few days ago, 
and the display glitches are pretty bad.

I just now reproduced the problem by using ssh to log into a Fedora 23 
x86-64 system running emacs-25 (commit 
dc5e65b5deb2f5b67f6c3a06ae81c6b074bd4b56) from my laptop, which is 
running Ubuntu 12.04.5 gnome-terminal in a 37x80 window (TERM=xterm in 
the environment). I have seen the problem from recent Ubuntu clients as 
well.

I changed to the Emacs source directory and ran the command

src/emacs -nw -Q src/conf_post.h

I then typed:

C-s h a s _

The screen display was messed up at this point; see attached image. 
Notice that the minibuffer says "hhs_" (with a highlighted second "h") 
instead of the correct ("has_").

The problem is not easily reproducible. Often Emacs works. Sometimes it 
does not, and the screen keeps getting more and more corrupted as time 
goes on. Symptoms often differ.

I just now tried a similar recipe (without the '_'), and this time Emacs 
contained the following text at the start of the (now-modified) 
conf_post.h buffer:

1;3201;0chas/* conf_post.h --- configure.ac includes this via AH_BOTTOM

and view-lossage showed the following:
  C-s C-s C-s [isearch-forward]
  ESC [ > [nil]
  1 [self-insert-command]
  ; [c-electric-semi&comma]
  3 [self-insert-command]
  2 [self-insert-command]
  0 [self-insert-command]
  1 [self-insert-command]
  ; [c-electric-semi&comma]
  0 [self-insert-command]
  c [self-insert-command]
  h [self-insert-command]
  a [self-insert-command]
  s [self-insert-command]
  ESC x [execute-extended-command]
  v [self-insert-command]
  i [self-insert-command]
  e [self-insert-command]
  w [self-insert-command]
  - [self-insert-command]
  l [self-insert-command]
...


My guess is that there is something wrong with the initial handshake 
with the terminal, to find out its characteristics; if memory serves 
this is something we've fiddled with in emacs-25 reasonably recently.

[-- Attachment #2: Emacs-screenshot.png --]
[-- Type: image/png, Size: 130978 bytes --]

bug#23819: 25.0.95; display botched badly in xterm window

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 22 Jun 2016 04:01:34 +0200
> 
> This appears to be a relatively recent regression in the emacs-25 
> branch.

How recent?  Can you estimate when this started to happen to you?  The
relevant file (xterm.el) hasn't been touched in months, except by you,
for the paste feature.

> I just now tried a similar recipe (without the '_'), and this time Emacs 
> contained the following text at the start of the (now-modified) 
> conf_post.h buffer:
> 
> 1;3201;0chas/* conf_post.h --- configure.ac includes this via AH_BOTTOM
> 
> and view-lossage showed the following:
>   C-s C-s C-s [isearch-forward]
>   ESC [ > [nil]
>   1 [self-insert-command]
>   ; [c-electric-semi&comma]
>   3 [self-insert-command]
>   2 [self-insert-command]
>   0 [self-insert-command]
>   1 [self-insert-command]
>   ; [c-electric-semi&comma]
>   0 [self-insert-command]
>   c [self-insert-command]
>   h [self-insert-command]
>   a [self-insert-command]
>   s [self-insert-command]
>   ESC x [execute-extended-command]
>   v [self-insert-command]
>   i [self-insert-command]
>   e [self-insert-command]
>   w [self-insert-command]
>   - [self-insert-command]
>   l [self-insert-command]

The "ESC [ > 1;3201;0c" thingy is the response to query by
xterm--query, see xterm--version-handler.  Is it true that all these
problems happen right in the beginning of a session?  What happens if
you wait at least 2 sec after starting a session, without typing any
commands -- do these problems still happen?

> My guess is that there is something wrong with the initial handshake 
> with the terminal, to find out its characteristics; if memory serves 
> this is something we've fiddled with in emacs-25 reasonably recently.

Your guess is probably right, except that I don't see any fiddling in
the Git log, certainly not recently.  So if this indeed started
happening recently, there must be some other factor at work here.

bug#23819: 25.0.95; display botched badly in xterm window

[Paul Eggert]

On 06/22/2016 04:54 PM, Eli Zaretskii wrote:
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Wed, 22 Jun 2016 04:01:34 +0200
>>
>> This appears to be a relatively recent regression in the emacs-25
>> branch.
> How recent?  Can you estimate when this started to happen to you?  The
> relevant file (xterm.el) hasn't been touched in months, except by you,
> for the paste feature.

Sorry, my memory is inexact. I vaguely recall seeing a problem or two a 
couple of months ago. The problem has gotten worse recently, I expect 
partly because I am using lower-throughput and/or longer-latency ssh 
connections recently.

> The "ESC [ > 1;3201;0c" thingy is the response to query by 
> xterm--query, see xterm--version-handler. Is it true that all these 
> problems happen right in the beginning of a session? What happens if 
> you wait at least 2 sec after starting a session, without typing any 
> commands -- do these problems still happen?

I have observed the problem in later parts of a session. In particular, 
after a problem occurs and I type control-L to refresh the screen 
completely, I have observed the problem recur in the same Emacs session. 
Typically the problem occurs during interactive search, when substrings 
are being highlighted in the text and in the minibuffer. Typically I am 
searching code, where keywords and suchlike are also highlighted. I 
assume all this highlighting is being done in the background.

All this suggests that the issue is not limited to startup. Possibly 
there are two issues, one at startup and one later. For example, perhaps 
Emacs is deducing the incorrect terminal type during startup, or perhaps 
my (typically Ubuntu) clients are reporting an incorrect terminal type 
during startup.

> Your guess is probably right, except that I don't see any fiddling in 
> the Git log, certainly not recently. So if this indeed started 
> happening recently, there must be some other factor at work here. 

I agree.

bug#23819: 25.0.95; display botched badly in xterm window

[Eli Zaretskii]

> Cc: 23819@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 22 Jun 2016 20:55:57 +0200
> 
> > The "ESC [ > 1;3201;0c" thingy is the response to query by 
> > xterm--query, see xterm--version-handler. Is it true that all these 
> > problems happen right in the beginning of a session? What happens if 
> > you wait at least 2 sec after starting a session, without typing any 
> > commands -- do these problems still happen?
> 
> I have observed the problem in later parts of a session. In particular, 
> after a problem occurs and I type control-L to refresh the screen 
> completely, I have observed the problem recur in the same Emacs session. 
> Typically the problem occurs during interactive search, when substrings 
> are being highlighted in the text and in the minibuffer. Typically I am 
> searching code, where keywords and suchlike are also highlighted. I 
> assume all this highlighting is being done in the background.

When you say "the problem" here, do you mean the "ESC [ > ..."
response?  IOW, do you see this sequence in Emacs input stream in the
middle of a session?  That would be highly unusual, as AFAIK we only
query the terminal at the session beginning.

bug#23819: 25.0.95; display botched badly in xterm window

[Eli Zaretskii]

> Cc: 23819@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 22 Jun 2016 20:55:57 +0200
> 
> All this suggests that the issue is not limited to startup. Possibly 
> there are two issues, one at startup and one later. For example, perhaps 
> Emacs is deducing the incorrect terminal type during startup, or perhaps 
> my (typically Ubuntu) clients are reporting an incorrect terminal type 
> during startup.

I think this might happen if we deduce incorrect xterm version.  Can
you modify xterm.el such that it stores the value of 'version' in
xterm--version-handler in a variable?  Then you could report the
values of that variable when you see the problems and when you don't,
and maybe we will learn something that way.

bug#23819: 25.0.95; display botched badly in xterm window

[Paul Eggert]

On 06/22/2016 09:00 PM, Eli Zaretskii wrote:
>> Cc:23819@debbugs.gnu.org
>> >From: Paul Eggert<eggert@cs.ucla.edu>
>> >Date: Wed, 22 Jun 2016 20:55:57 +0200
>> >
>>> > >The "ESC [ > 1;3201;0c" thingy is the response to query by
>>> > >xterm--query, see xterm--version-handler. Is it true that all these
>>> > >problems happen right in the beginning of a session? What happens if
>>> > >you wait at least 2 sec after starting a session, without typing any
>>> > >commands -- do these problems still happen?
>> >
>> >I have observed the problem in later parts of a session. In particular,
>> >after a problem occurs and I type control-L to refresh the screen
>> >completely, I have observed the problem recur in the same Emacs session.
>> >Typically the problem occurs during interactive search, when substrings
>> >are being highlighted in the text and in the minibuffer. Typically I am
>> >searching code, where keywords and suchlike are also highlighted. I
>> >assume all this highlighting is being done in the background.
> When you say "the problem" here, do you mean the "ESC [ > ..."
> response?
No, I meant the problem that the display becomes garbled, typically 
during interactive search.

bug#23819: 25.0.95; display botched badly in xterm window

[Paul Eggert]

On 06/22/2016 09:04 PM, Eli Zaretskii wrote:
> Can you modify xterm.el such that it stores the value of 'version' in 
> xterm--version-handler in a variable? Then you could report the values 
> of that variable when you see the problems and when you don't, and 
> maybe we will learn something that way. 

The version string is "1;3201;0" and the version is therefore computed 
to be 200, when things work at startup. This corresponds to GNOME 
Terminal 3.4.1.1 on the client (Ubuntu 12.04.5 LTS). Right now I can't 
reproduce the problem where the version string is copied into the start 
of the first buffer (I assume this is because my Internet connection is 
good right now). However, after doing a lot of searches I can 
occasionally reproduce the problem where characters are misdisplayed, 
with 'emacs -Q src/conf_post.h'. I repeatedly type "C-s h a s M-<" with 
a few extra C-s's thrown in.

bug#23819: 25.0.95; display botched badly in xterm window

[Eli Zaretskii]

> Cc: 23819@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 23 Jun 2016 10:58:15 +0200
> 
>     Can you modify xterm.el such that it stores the value of 'version' in xterm--version-handler in a variable? Then you could report the values of that variable when you see the problems and when you don't, and maybe we will learn something that way. 
> 
> The version string is "1;3201;0" and the version is therefore computed to be 200, when things work at startup. This corresponds to GNOME Terminal 3.4.1.1 on the client (Ubuntu 12.04.5 LTS). Right now I can't reproduce the problem where the version string is copied into the start of the first buffer (I assume this is because my Internet connection is good right now). However, after doing a lot of searches I can occasionally reproduce the problem where characters are misdisplayed, with 'emacs -Q src/conf_post.h'. I repeatedly type "C-s h a s M-<" with a few extra C-s's thrown in.

Which probably means there are two separate problems.

Actually, what you describe now kinda rings a bell: we had similar
unexplained glitches when TTY menus were introduced.  See bug#17497;
starting with http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497#230
there are some thoughts about a particular code fragment in cm.c that
never got addressed; perhaps you can dig into this, as you evidently
have access to a system where these problems can be reproduced.

Also, in http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17497#236, you
will see a link to a Virtual Box related discussion where someone
claims Emacs fails to deliver a crucial cursor-motion command to the
TTY driver, for some reason.  At the time, I analyzed many termscript
files people sent from affected machines, and saw no missing commands,
but maybe I missed something.

If you can see the same menu-related glitches, perhaps they will
provide an easier way to reproduce the problem and try solving it.

OTOH, if what you see is the same problem as in bug#17497, then it's
by no means new.

Thanks.

bug#23819: 25.0.95; display botched badly in xterm window

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> OTOH, if what you see is the same problem as in bug#17497, then it's
> by no means new.

And bug#17497 is no longer reproducible, so I guess it's possible that
the glitch Paul was seeing is also gone.

Paul, are you still seeing these glitches in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23819: 25.0.95; display botched badly in xterm window

[Paul Eggert]

On 1/29/22 08:55, Lars Ingebrigtsen wrote:
> Paul, are you still seeing these glitches in recent Emacs versions?

No, I just now tried to reproduce the problem and could not.

bug#23819: 25.0.95; display botched badly in xterm window

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 1/29/22 08:55, Lars Ingebrigtsen wrote:
>> Paul, are you still seeing these glitches in recent Emacs versions?
>
> No, I just now tried to reproduce the problem and could not.

Thanks for checking; I'm closing this bug report, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no