";0c" in terminal with slow connections

";0c" in terminal with slow connections

[Vincent Lefevre]

I have the following problem with the recent snapshot emacs versions
on Debian (at least in the emacs-snapshot 20070824-1 Debian package).

When I connect by ssh via an ADSL connection, I get some characters
(see explanation below) in the Emacs buffer just after starting Emacs
(running in the current terminal, i.e. not with its graphic interface).
This is almost 100% reproducible under these conditions. But this bug
doesn't occur with fast ssh connections. I suppose that Emacs does some
flushing, but it probably occurs too early with an ADSL connection, due
to a higher latency.

Also note that the bug doesn't occur with Emacs22 on the same machine
(I first thought it could be a bug in the ncurses library, but this
doesn't seem so).

The characters that appear in the buffer are those coming from the
"Send Device Attributes". I analyzed the terminal logs and could see
that Emacs sent the following sequences:

  home cnorm cvvis \E[60d el rmkx cnorm rmcup op ^M smcup cvvis smkx
  \E[>0c \E>4;1m \E[60;1H

The problem comes from the \E[>0c. /usr/share/doc/xterm/ctlseqs.txt.gz
(under Debian) says:

  CSI > Ps c
          Send Device Attributes (Secondary DA)
            Ps = 0  or omitted -> request the terminal's identification
          code.  The response depends on the decTerminalID resource set-
          ting.  It should apply only to VT220 and up, but xterm extends
          this to VT100.
            -> CSI  > Pp ; Pv ; Pc c
          where Pp denotes the terminal type
            -> 0  (``VT100'')
            -> 1  (``VT220'')
          and Pv is the firmware version (for xterm, this was originally
          the XFree86 patch number, starting with 95).  In a DEC termi-
          nal, Pc indicates the ROM cartridge registration number and is
          always zero.

Note that when ssh isn't used, this sequence is still sent by Emacs,
but I think that some flushing of the input takes care of this. From
a shell, you can type

  printf "\e[>0c"

to see the effect: characters are sent to the input and appears in the
command line.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Richard Stallman]

    "Send Device Attributes". I analyzed the terminal logs and could see
    that Emacs sent the following sequences:

      home cnorm cvvis \E[60d el rmkx cnorm rmcup op ^M smcup cvvis smkx
      \E[>0c \E>4;1m \E[60;1H

Can you find where in the code of Emacs it sends \E[>0c?
A breakpoint at emacs_tputs should make it easy enough,
especially a conditional breakpoint.

Re: ";0c" in terminal with slow connections

[Stefan Monnier]

>     "Send Device Attributes". I analyzed the terminal logs and could see
>     that Emacs sent the following sequences:

>       home cnorm cvvis \E[60d el rmkx cnorm rmcup op ^M smcup cvvis smkx
>       \E[>0c \E>4;1m \E[60;1H

> Can you find where in the code of Emacs it sends \E[>0c?
> A breakpoint at emacs_tputs should make it easy enough,
> especially a conditional breakpoint.

Grep says it's in lisp/term/xterm.el


        Stefan

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

On 2007-09-04 15:49:39 -0400, Stefan Monnier wrote:
> > Can you find where in the code of Emacs it sends \E[>0c?
> > A breakpoint at emacs_tputs should make it easy enough,
> > especially a conditional breakpoint.
> 
> Grep says it's in lisp/term/xterm.el

Note that the bug also occurs with:

  emacs -nw -q --no-site-file --eval '(inhibit-splash-screen t)'

Is lisp/term/xterm.el executed even in this case?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Richard Stallman]

    Grep says it's in lisp/term/xterm.el

Does this mean that the ssh connection was made from within an xterm?
It seems that the xterm is generating the proper response,
which goes back thru ssh to Emacs.

So why doesn't Emacs process it correctly?

      emacs -nw -q --no-site-file --eval '(inhibit-splash-screen t)'

    Is lisp/term/xterm.el executed even in this case?

Yes.

Re: ";0c" in terminal with slow connections

[Dan Nicolaescu]

Richard Stallman <rms@gnu.org> writes:

  >     Grep says it's in lisp/term/xterm.el
  > 
  > Does this mean that the ssh connection was made from within an xterm?
  > It seems that the xterm is generating the proper response,
  > which goes back thru ssh to Emacs.
  > 
  > So why doesn't Emacs process it correctly?

Emacs uses a read-event with a timer for processing the response,
maybe the response does come back before the timer expires, so it is
processed as normal input.

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

On 2007-09-05 22:34:28 -0700, Dan Nicolaescu wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
>   >     Grep says it's in lisp/term/xterm.el
>   > 
>   > Does this mean that the ssh connection was made from within an xterm?

Yes (either xterm or another xterm-compatible terminal).

>   > It seems that the xterm is generating the proper response,
>   > which goes back thru ssh to Emacs.
>   > 
>   > So why doesn't Emacs process it correctly?
> 
> Emacs uses a read-event with a timer for processing the response,
> maybe the response does come back before the timer expires, so it is
                                    ^^^^^^ after
> processed as normal input.

Yes, I think this is the problem, as it occurs only with connections
having a high latency (such as ADSL). Is a timer really necessary?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Davis Herring]

>>   > It seems that the xterm is generating the proper response,
>>   > which goes back thru ssh to Emacs.
>>   >
>>   > So why doesn't Emacs process it correctly?
>>
>> Emacs uses a read-event with a timer for processing the response,
>> maybe the response does come back before the timer expires, so it is
>                                     ^^^^^^ after
>> processed as normal input.
>
> Yes, I think this is the problem, as it occurs only with connections
> having a high latency (such as ADSL). Is a timer really necessary?

This is of course what I was worried about (and is exactly Stefan's
example) in the thread about this a month ago:
http://lists.gnu.org/archive/html/emacs-devel/2007-08/msg00127.html

Would it be better to not have a timeout and just see whether the next
event read looks like an expected response from the terminal?  Of course,
if the user is typing while this code runs, the terminal's response may be
several characters later in the input stream...

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: ";0c" in terminal with slow connections

[Richard Stallman]

    > Emacs uses a read-event with a timer for processing the response,
    > maybe the response does come back before the timer expires, so it is
					^^^^^^ after
    > processed as normal input.

    Yes, I think this is the problem, as it occurs only with connections
    having a high latency (such as ADSL). Is a timer really necessary?

If there is a chance that the terminal won't respond at all,
the timer would enable Enacs to avoid hanging.  Is that the
reason for the timer?

What is the purpose of making this query in the first place?
Maybe the best thing to do is get rid of it.

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

On 2007-09-07 02:32:21 -0400, Richard Stallman wrote:
>     > Emacs uses a read-event with a timer for processing the response,
>     > maybe the response does come back before the timer expires, so it is
> 					^^^^^^ after
>     > processed as normal input.
> 
>     Yes, I think this is the problem, as it occurs only with connections
>     having a high latency (such as ADSL). Is a timer really necessary?
> 
> If there is a chance that the terminal won't respond at all,
> the timer would enable Enacs to avoid hanging.  Is that the
> reason for the timer?

Emacs should first make sure that the sequence is supported,
at least to avoid erratic behavior with other terminals.

> What is the purpose of making this query in the first place?
> Maybe the best thing to do is get rid of it.

I think so.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Dan Nicolaescu]

Richard Stallman <rms@gnu.org> writes:

  >     > Emacs uses a read-event with a timer for processing the response,
  >     > maybe the response does come back before the timer expires, so it is
  > 					^^^^^^ after
  >     > processed as normal input.
  > 
  >     Yes, I think this is the problem, as it occurs only with connections
  >     having a high latency (such as ADSL). Is a timer really necessary?
  > 
  > If there is a chance that the terminal won't respond at all,
  > the timer would enable Enacs to avoid hanging.  Is that the
  > reason for the timer?

The reason for the timer is to avoid hanging.

  > What is the purpose of making this query in the first place?

The query is used to find out the version of xterm being used.

The reason is to enable the modifyOtherKeys feature in new enough
xterms. With that feature enabled key combinations C-. C-,
and other keys that are normally not usable in a terminal generate
distinct keys and emacs can bind them. Only newer versions of xterm
can do this and AFAIK no other terminal emulators that set TERM to
xterm can do it.

If we just send the escape sequence to enable modifyOtherKeys on a
terminal that does not support it, the escape sequence will appear as
junk on the screen. 

  > Maybe the best thing to do is get rid of it.

Re: ";0c" in terminal with slow connections

[Richard Stallman]

      > If there is a chance that the terminal won't respond at all,
      > the timer would enable Enacs to avoid hanging.  Is that the
      > reason for the timer?

    The reason for the timer is to avoid hanging.

In what cases would it hang?  Are these old xterm versions?

      > What is the purpose of making this query in the first place?

    The query is used to find out the version of xterm being used.

    If we just send the escape sequence to enable modifyOtherKeys on a
    terminal that does not support it, the escape sequence will appear as
    junk on the screen. 

Emacs clears the screen at startup.  Maybe it is ok if that happens
on old versions of xterm, until they are replaced.

Re: ";0c" in terminal with slow connections

[Dan Nicolaescu]

Richard Stallman <rms@gnu.org> writes:

  >       > If there is a chance that the terminal won't respond at all,
  >       > the timer would enable Enacs to avoid hanging.  Is that the
  >       > reason for the timer?
  > 
  >     The reason for the timer is to avoid hanging.
  > 
  > In what cases would it hang?  Are these old xterm versions?

I don't remember finding any terminal where it would hang. But almost
all terminal emulators call themselves xterm nowadays, so there's a
chance that somebody does not implement this correctly.

  >       > What is the purpose of making this query in the first place?
  > 
  >     The query is used to find out the version of xterm being used.
  > 
  >     If we just send the escape sequence to enable modifyOtherKeys on a
  >     terminal that does not support it, the escape sequence will appear as
  >     junk on the screen. 
  > 
  > Emacs clears the screen at startup.  Maybe it is ok if that happens
  > on old versions of xterm, until they are replaced.

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

On 2007-09-07 07:13:46 -0700, Dan Nicolaescu wrote:
> The reason is to enable the modifyOtherKeys feature in new enough
> xterms. With that feature enabled key combinations C-. C-,
> and other keys that are normally not usable in a terminal generate
> distinct keys and emacs can bind them. Only newer versions of xterm
> can do this and AFAIK no other terminal emulators that set TERM to
> xterm can do it.
> 
> If we just send the escape sequence to enable modifyOtherKeys on a
> terminal that does not support it, the escape sequence will appear as
> junk on the screen. 

Why not let the user configure his xterm the way he likes?

If you enable modifyOtherKeys, won't the configuration change
for other applications running in xterm?

What if the user has set up VT100 translations corresponding
to C-. C-, and so on?

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Dan Nicolaescu]

Vincent Lefevre <vincent@vinc17.org> writes:

  > On 2007-09-07 07:13:46 -0700, Dan Nicolaescu wrote:
  > > The reason is to enable the modifyOtherKeys feature in new enough
  > > xterms. With that feature enabled key combinations C-. C-,
  > > and other keys that are normally not usable in a terminal generate
  > > distinct keys and emacs can bind them. Only newer versions of xterm
  > > can do this and AFAIK no other terminal emulators that set TERM to
  > > xterm can do it.
  > > 
  > > If we just send the escape sequence to enable modifyOtherKeys on a
  > > terminal that does not support it, the escape sequence will appear as
  > > junk on the screen. 
  > 
  > Why not let the user configure his xterm the way he likes?

You are missing the point, this is done because we would like the users
to transparently take advantage of this feature without having to
configure anything.

  > If you enable modifyOtherKeys, won't the configuration change
  > for other applications running in xterm?

Emacs restores the setting on suspend and quit.

  > What if the user has set up VT100 translations corresponding
  > to C-. C-, and so on?

I am not sure what is your point here.

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

On 2007-09-08 01:35:14 -0700, Dan Nicolaescu wrote:
> Vincent Lefevre <vincent@vinc17.org> writes:
>   > Why not let the user configure his xterm the way he likes?
> 
> You are missing the point, this is done because we would like the users
> to transparently take advantage of this feature without having to
> configure anything.

But what about users who are used to the current configuration
or have their own configuration? If this feature is so important
to have it without configuring anything, then why doesn't xterm
enable it by default?

>   > If you enable modifyOtherKeys, won't the configuration change
>   > for other applications running in xterm?
> 
> Emacs restores the setting on suspend and quit.

This is not sufficient:
  * Emacs won't be able to do anything for screen users, after
    detaching a session running Emacs or after switching the
    screen window.
  * If Emacs crashes (this hasn't happened yet with Emacs22,
    though), the configuration will not be restored.
  * In case of a SSH or dialup connection, if the connection closes
    because of some network problem (this occurs quite frequently),
    the configuration will not be restored either.

>   > What if the user has set up VT100 translations corresponding
>   > to C-. C-, and so on?
> 
> I am not sure what is your point here.

The user can bind some key sequences (in general, ones that are not
used in the terminal itself) to some xterm functions (e.g. to do
scrolling, paste strings and so on). This is done with the VT100
translations resource. What is the effect of modifyOtherKeys on
these bindings? (I don't have a xterm here to try...)

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: ";0c" in terminal with slow connections

[Dan Nicolaescu]

[Maybe you are not aware, but your tone sounds irritating. Please be
more careful on this mailing list.]

Vincent Lefevre <vincent@vinc17.org> writes:

  > On 2007-09-08 01:35:14 -0700, Dan Nicolaescu wrote:
  > > Vincent Lefevre <vincent@vinc17.org> writes:
  > >   > Why not let the user configure his xterm the way he likes?
  > > 
  > > You are missing the point, this is done because we would like the users
  > > to transparently take advantage of this feature without having to
  > > configure anything.
  > But what about users who are used to the current configuration
  > or have their own configuration? 

Can you please give a precise example of what you mean?

  > If this feature is so important to have it without configuring
  > anything, then why doesn't xterm enable it by default?

Don't know, please find out from the xterm guys and report back here.

  > >   > If you enable modifyOtherKeys, won't the configuration change
  > >   > for other applications running in xterm?
  > > 
  > > Emacs restores the setting on suspend and quit.
  > 
  > This is not sufficient:
  >   * Emacs won't be able to do anything for screen users, after
  >     detaching a session running Emacs or after switching the
  >     screen window.

Screen sets TERM to screen, so this won't affect such users.

  >   * If Emacs crashes (this hasn't happened yet with Emacs22,
  >     though), the configuration will not be restored.
  >   * In case of a SSH or dialup connection, if the connection closes
  >     because of some network problem (this occurs quite frequently),
  >     the configuration will not be restored either.

We can only do a best effort.

  > >   > What if the user has set up VT100 translations corresponding
  > >   > to C-. C-, and so on?
  > > 
  > > I am not sure what is your point here.
  > 
  > The user can bind some key sequences (in general, ones that are not
  > used in the terminal itself) to some xterm functions (e.g. to do
  > scrolling, paste strings and so on). This is done with the VT100
  > translations resource. What is the effect of modifyOtherKeys on
  > these bindings? (I don't have a xterm here to try...)

You seem to know about this stuff, why don't you try it and report any
problems that you find here.

Re: ";0c" in terminal with slow connections

[Vincent Lefevre]

Hi,

On 2007-09-08 09:17:03 -0700, Dan Nicolaescu wrote:
> [Maybe you are not aware, but your tone sounds irritating. Please be
> more careful on this mailing list.]

Sorry if you felt that. I also found your tone irritating ("You are
missing the point..."). I really hate when I spent time to configure
my software (based on their documentation, standards and so on) and
some other software, trying to be smarter than the user, silently
overrides my configuration (and breaks the configuration of other
software at the same time).

> Vincent Lefevre <vincent@vinc17.org> writes:
>   > On 2007-09-08 01:35:14 -0700, Dan Nicolaescu wrote:
>   > > Vincent Lefevre <vincent@vinc17.org> writes:
>   > >   > Why not let the user configure his xterm the way he likes?
>   > > 
>   > > You are missing the point, this is done because we would like
>   > > the users to transparently take advantage of this feature
>   > > without having to configure anything.
>   > But what about users who are used to the current configuration
>   > or have their own configuration? 
> 
> Can you please give a precise example of what you mean?

e.g. with VT100 translations, as I mentioned below. (Now, I think
that VT100 translations have the precedence.)

But the user could have also set the configuration related to
modifyOtherKeys via the X11 resources (e.g. .app-defaults/XTerm).

>   > If this feature is so important to have it without configuring
>   > anything, then why doesn't xterm enable it by default?
> 
> Don't know, please find out from the xterm guys and report back here.

I'll ask Thomas about that.

>   > This is not sufficient:
>   >   * Emacs won't be able to do anything for screen users, after
>   >     detaching a session running Emacs or after switching the
>   >     screen window.
> 
> Screen sets TERM to screen, so this won't affect such users.

OK. Note that the value of TERM is configurable, though; and nothing
prevents the user from setting TERM to 'xterm-something' (and this
is not wrong as screen supports conventional xterm escape sequences
in some way). There is even documentation that says that this is OK.
For instance, http://linux.die.net/man/1/nsc says: Within
putty/screen, "TERM=xterm-color" works well.

I suppose that for SGR, screen resets the status when switching the
window or detaching the session. I don't know if it does this with
modifyOtherKeys, but this is not as standard (e.g. no terminfo
interface, AFAIK), and could lead to problems.

>   >   * If Emacs crashes (this hasn't happened yet with Emacs22,
>   >     though), the configuration will not be restored.
>   >   * In case of a SSH or dialup connection, if the connection closes
>   >     because of some network problem (this occurs quite frequently),
>   >     the configuration will not be restored either.
> 
> We can only do a best effort.

Only the user could configure his shell and whatever program that
calls Emacs in order to restore the configuration. But again, this
could lead to problems with some terminals, if the sequence does
something else.

>   > The user can bind some key sequences (in general, ones that are not
>   > used in the terminal itself) to some xterm functions (e.g. to do
>   > scrolling, paste strings and so on). This is done with the VT100
>   > translations resource. What is the effect of modifyOtherKeys on
>   > these bindings? (I don't have a xterm here to try...)
> 
> You seem to know about this stuff, why don't you try it and report any
> problems that you find here.

I'll try to do that tonight or during the week.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)