[dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Richard Stallman]

Would someone please DTRT and ack?

------- Start of forwarded message -------
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY 
	autolearn=failed version=3.1.0
Date: Sun, 18 Mar 2007 19:34:52 -0400
To: bug-gnu-emacs@gnu.org
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
From: Rich Felker <dalias@aerifal.cx>
Subject: ansi-term \e[J causes spurious newline [revised report]

On versions of GNU emacs I have tested (21.1 and cvs unicode-2
branch), the "ansi-term" terminal emulator (M-x ansi-term) exhibits
incorrect terminal behavior when given the ESC [ J sequence. In
addition to clearing to the end of the screen, it moves the cursor to
the beginning of the next line if the cursor is not already at the
beginning of a line. To test this, use the following shell command
from a shell running in ansi-term:

echo -e 'hello\e[Jworld'

On a vt100/ansi/ecma compatible terminal, this should leave
"helloworld" visible on the screen, with everything afterward cleared.
On GNU emacs' ansi-term, it prints hello on one line and world on the
next, after clearing to the end of the screen.

This bug makes busybox shells' cmdedit functionality essentially
unusable, since it depends on ESC [ J leaving the cursor in the
correct position. Other programs may be affected too.

Removing the calls to term-unwrap-line from term-erase-in-display (in
term.el) fixes the problem, but I don't know if this has any bad
side-effects.

Also, in reading this code, I noticed that the sequence ESC [ 1 J
seems to only erase up to the point. To emulate VT100's Roman-inspired
off-by-one idiosyncracies, it should erase up to point+1, i.e. also
erase the character "under" the terminal cursor.

~Rich


_______________________________________________
bug-gnu-emacs mailing list
bug-gnu-emacs@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-gnu-emacs
------- End of forwarded message -------

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Chong Yidong]

> From: Rich Felker <dalias@aerifal.cx>
> Subject: ansi-term \e[J causes spurious newline [revised report]
> To: bug-gnu-emacs@gnu.org
>
> On versions of GNU emacs I have tested (21.1 and cvs unicode-2
> branch), the "ansi-term" terminal emulator (M-x ansi-term) exhibits
> incorrect terminal behavior when given the ESC [ J sequence. In
> addition to clearing to the end of the screen, it moves the cursor to
> the beginning of the next line if the cursor is not already at the
> beginning of a line. To test this, use the following shell command
> from a shell running in ansi-term:
>
> echo -e 'hello\e[Jworld'
>
> On a vt100/ansi/ecma compatible terminal, this should leave
> "helloworld" visible on the screen, with everything afterward cleared.
> On GNU emacs' ansi-term, it prints hello on one line and world on the
> next, after clearing to the end of the screen.
>
> Removing the calls to term-unwrap-line from term-erase-in-display (in
> term.el) fixes the problem, but I don't know if this has any bad
> side-effects.

Looking through the code, I think the calls to term-unwrap-line should
be removed.  The note in the docstring of term-erase-in-display that
it "should only be called when point is at the start of a screen line"
is also false; this condition generally doesn't hold in situations
where this function is called, and if we remove the term-unwrap-line
calls, it's not necessary at all.

Cursory testing seems to indicate that ansi-term behaves fine without
the term-unwrap-line calls.

What do you think?

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

Chong Yidong <cyd@stupidchicken.com> writes:

  > > From: Rich Felker <dalias@aerifal.cx>
  > > Subject: ansi-term \e[J causes spurious newline [revised report]
  > > To: bug-gnu-emacs@gnu.org
  > >
  > > On versions of GNU emacs I have tested (21.1 and cvs unicode-2
  > > branch), the "ansi-term" terminal emulator (M-x ansi-term) exhibits
  > > incorrect terminal behavior when given the ESC [ J sequence. In
  > > addition to clearing to the end of the screen, it moves the cursor to
  > > the beginning of the next line if the cursor is not already at the
  > > beginning of a line. To test this, use the following shell command
  > > from a shell running in ansi-term:
  > >
  > > echo -e 'hello\e[Jworld'
  > >
  > > On a vt100/ansi/ecma compatible terminal, this should leave
  > > "helloworld" visible on the screen, with everything afterward cleared.
  > > On GNU emacs' ansi-term, it prints hello on one line and world on the
  > > next, after clearing to the end of the screen.
  > >
  > > Removing the calls to term-unwrap-line from term-erase-in-display (in
  > > term.el) fixes the problem, but I don't know if this has any bad
  > > side-effects.

In general tests using "echo" are not good enough, the correct way to
do it is to use "tput", there are some subtleties involved...

I cannot reproduce this problem using tput.

  > Looking through the code, I think the calls to term-unwrap-line should
  > be removed.  The note in the docstring of term-erase-in-display that
  > it "should only be called when point is at the start of a screen line"
  > is also false; this condition generally doesn't hold in situations

\e[J is the terminfo "ed" string, the terminfo manual says:

       Ed is only defined from the first column of a line.  (Thus, it
       can be simulated by a request to delete a large number of
       lines, if a true ed is not available.) 

So I think that the code is correct. (and the behavior requested by
the echo command above is undefined).


  > where this function is called, and if we remove the term-unwrap-line
  > calls, it's not necessary at all.

  > Cursory testing seems to indicate that ansi-term behaves fine without
  > the term-unwrap-line calls.
  > 
  > What do you think?

I think that nothing should be changed. If the problem can be
reproduced using tput, then I can look into it. 

           --dan

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Chong Yidong]

Dan Nicolaescu <dann@ics.uci.edu> writes:

>   > > echo -e 'hello\e[Jworld'
>   > >
>   > > On a vt100/ansi/ecma compatible terminal, this should leave
>   > > "helloworld" visible on the screen, with everything afterward cleared.
>   > > On GNU emacs' ansi-term, it prints hello on one line and world on the
>   > > next, after clearing to the end of the screen.
>
> In general tests using "echo" are not good enough, the correct way to
> do it is to use "tput", there are some subtleties involved...

On the other hand, using "echo" in ansi-term does produce a different
result from doing it in xterm.  Isn't that divergent behavior?

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

Chong Yidong <cyd@stupidchicken.com> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > 
  > >   > > echo -e 'hello\e[Jworld'
  > >   > >
  > >   > > On a vt100/ansi/ecma compatible terminal, this should leave
  > >   > > "helloworld" visible on the screen, with everything afterward cleared.
  > >   > > On GNU emacs' ansi-term, it prints hello on one line and world on the
  > >   > > next, after clearing to the end of the screen.
  > >
  > > In general tests using "echo" are not good enough, the correct way to
  > > do it is to use "tput", there are some subtleties involved...
  > 
  > On the other hand, using "echo" in ansi-term does produce a different
  > result from doing it in xterm.  Isn't that divergent behavior?

It different behavior for undefined behavior, so it's fine. 

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Miles Bader]

Dan Nicolaescu <dann@ics.uci.edu> writes:
> \e[J is the terminfo "ed" string, the terminfo manual says:
>
>        Ed is only defined from the first column of a line.  (Thus, it
>        can be simulated by a request to delete a large number of
>        lines, if a true ed is not available.) 
...
>   > On the other hand, using "echo" in ansi-term does produce a different
>   > result from doing it in xterm.  Isn't that divergent behavior?
>
> It different behavior for undefined behavior, so it's fine. 

Is it the escape sequence "\e[J" actually undefined on a real terminal
in that case, or is merely the more abstract "ed" terminfo capability
which is undefined in that case?

[AFAIK, "term.el" tries to emulate a terminal, not merely the subset of
the terminal as viewed through terminfo.]

-Miles

-- 
Americans are broad-minded people.  They'll accept the fact that a person can
be an alcoholic, a dope fiend, a wife beater, and even a newspaperman, but if a
man doesn't drive, there is something wrong with him.  -- Art Buchwald

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

Miles Bader <miles.bader@necel.com> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > > \e[J is the terminfo "ed" string, the terminfo manual says:
  > >
  > >        Ed is only defined from the first column of a line.  (Thus, it
  > >        can be simulated by a request to delete a large number of
  > >        lines, if a true ed is not available.) 
  > ...
  > >   > On the other hand, using "echo" in ansi-term does produce a different
  > >   > result from doing it in xterm.  Isn't that divergent behavior?
  > >
  > > It different behavior for undefined behavior, so it's fine. 
  > 
  > Is it the escape sequence "\e[J" actually undefined on a real terminal
  > in that case, or is merely the more abstract "ed" terminfo capability
  > which is undefined in that case?

I am not sure what the question is... :-(
In any case I am not aware of any documentation for the different
escape sequences other than terminfo (or the terminal emulator
actual code, if that counts as documentation).

  > [AFAIK, "term.el" tries to emulate a terminal, not merely the subset of
  > the terminal as viewed through terminfo.]

I don't think so, term.el does not try to emulate a terminal, it
implements a terminal called "eterm-color" now that happens to have
terminfo strings very similar to xterm, but it never claimed to be an
emulator for any specific terminal. And IMO it shouldn't. 

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Miles Bader]

Dan Nicolaescu <dann@ics.uci.edu> writes:
>   > Is it the escape sequence "\e[J" actually undefined on a real terminal
>   > in that case, or is merely the more abstract "ed" terminfo capability
>   > which is undefined in that case?
>
> I am not sure what the question is... :-(
> In any case I am not aware of any documentation for the different
> escape sequences other than terminfo (or the terminal emulator
> actual code, if that counts as documentation).

I mean that you described the preconditions of the terminfo "ed"
capability, but that's not what term.el seems to be modelled after (to
degree that it's modelled after anything :-).  Terminfo is an
abstraction, and so tries to restrict the assumptions people make about
capabilities to some practical common denominator of the many types of
terminals it supports -- however many programs do not use terminfo; for
instance they may directly use ANSI escape sequences instead.

term.el says:

   ;;; It emulates (most of the features of) a VT100/ANSI-style terminal.

Elsewhere it mentions xterm in passing, and that's probably the most
relevant modern example of a "VT100/ANSI-style terminal".

I'm not saying it should exactly emulate any existing terminal -- that's
probably impractical -- but neither should it restrict its emulation to
only those sequences terminfo might produce.  For term.el to be
practical, we should care whether term.el follows common xterm and
VT100/ansi practice, even in cases where the behavior in question would
never be produced by terminfo.

-Miles

-- 
My books focus on timeless truths.  -- Donald Knuth

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

Miles Bader <miles.bader@necel.com> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > >   > Is it the escape sequence "\e[J" actually undefined on a real terminal
  > >   > in that case, or is merely the more abstract "ed" terminfo capability
  > >   > which is undefined in that case?
  > >
  > > I am not sure what the question is... :-(
  > > In any case I am not aware of any documentation for the different
  > > escape sequences other than terminfo (or the terminal emulator
  > > actual code, if that counts as documentation).
  > 
  > I mean that you described the preconditions of the terminfo "ed"
  > capability, but that's not what term.el seems to be modelled after (to
  > degree that it's modelled after anything :-).  Terminfo is an
  > abstraction, and so tries to restrict the assumptions people make about
  > capabilities to some practical common denominator of the many types of
  > terminals it supports -- however many programs do not use terminfo; for
  > instance they may directly use ANSI escape sequences instead.
  > 
  > term.el says:
  > 
  >    ;;; It emulates (most of the features of) a VT100/ANSI-style terminal.

So it emulates most of the features, NOT the terminal itself... AFAIK
there's no real effort in the code to emulate anything else other than
the documented terminfo behavior. (There is some commented out code
that I added that emulates non-terminfo stuff. I added that so that I
can run a hacked up version of vttest to check if things still work
when changing something).

  > I'm not saying it should exactly emulate any existing terminal -- that's
  > probably impractical -- but neither should it restrict its emulation to
  > only those sequences terminfo might produce.  For term.el to be
  > practical, we should care whether term.el follows common xterm and
  > VT100/ansi practice, even in cases where the behavior in question would
  > never be produced by terminfo.

Well, that's a rat hole that I don't think we want to get into.  Not at
this point in the release cycle. If users show that some features are
required by real applications to run correctly, then we can consider
it. If it's just: "'echo XYZ' works differently ", then it's not
really important.

The code that started this was documented to work the way
it does since 1994. The proposed change fixes the issue at hand, but: 
1. I am not convinced the change is correct in all cases.
2. Risking breaking something else this close to a release is not a
great idea. 
3. It's document undefined behavior created by an artificial testcase.

I can look into this after the release.

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Miles Bader]

Dan Nicolaescu <dann@ics.uci.edu> writes:
> I can look into this after the release.

That seems the safest to me too.

-Miles

-- 
It wasn't the Exxon Valdez captain's driving that caused the Alaskan oil spill.
It was yours.  [Greenpeace advertisement, New York Times, 25 February 1990]

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Kim F. Storm]

Dan Nicolaescu <dann@ics.uci.edu> writes:

> Well, that's a rat hole that I don't think we want to get into.  Not at
> this point in the release cycle. If users show that some features are
> required by real applications to run correctly, then we can consider
> it.

The reporter specifically mentioned busybox command line editing.

We cannot expect embedded systems to rely on termcap/terminfo.
Rather they will issue hardcoded VT100 escape sequences - like
ESC [ J, and expect the terminal emulator to DTRT...

I've tested the 'echo' example with both xterm and minicom (which I
use with embedded systems).  Both behave identically - for ESC [ J
and for ESC [ 1 J -- and in accordance with the reporter's expectations
for a "true" VT100 terminal.

So you may claim that the behaviour is "undefined" - but practical
tests shows "consistent" behaviour, different from what Emacs term
does.

IMO, we should fix term to behave like other VT100 emulators.
The current behaviour is clearly broken -- besides, fixing this
doesn't change the semantics of the only case where it was defined
before.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

storm@cua.dk (Kim F. Storm) writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > 
  > > Well, that's a rat hole that I don't think we want to get into.  Not at
  > > this point in the release cycle. If users show that some features are
  > > required by real applications to run correctly, then we can consider
  > > it.
  > 
  > The reporter specifically mentioned busybox command line editing.

Where? The message that I was CCed on only mentioned "echo".

  > We cannot expect embedded systems to rely on termcap/terminfo.
  > Rather they will issue hardcoded VT100 escape sequences - like
  > ESC [ J, and expect the terminal emulator to DTRT...
  > 
  > I've tested the 'echo' example with both xterm and minicom (which I
  > use with embedded systems).  Both behave identically - for ESC [ J
  > and for ESC [ 1 J -- and in accordance with the reporter's expectations
  > for a "true" VT100 terminal.
  > 
  > So you may claim that the behaviour is "undefined" - but practical
  > tests shows "consistent" behaviour, different from what Emacs term
  > does.
  > 
  > IMO, we should fix term to behave like other VT100 emulators.
  > The current behaviour is clearly broken -- besides, fixing this
  > doesn't change the semantics of the only case where it was defined
  > before.

I will look at that after the release. I don't think the proposed fix
is correct and I don't have time to deal with it now. 

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Kim F. Storm]

Dan Nicolaescu <dann@ics.uci.edu> writes:

>   > The reporter specifically mentioned busybox command line editing.
>
> Where? The message that I was CCed on only mentioned "echo".

Here's the original report.

From: Rich Felker <dalias@aerifal.cx>
Subject: ansi-term \e[J causes spurious newline [revised report]
Newsgroups: gmane.emacs.bugs
To: bug-gnu-emacs@gnu.org
Date: Sun, 18 Mar 2007 19:34:52 -0400

On versions of GNU emacs I have tested (21.1 and cvs unicode-2
branch), the "ansi-term" terminal emulator (M-x ansi-term) exhibits
incorrect terminal behavior when given the ESC [ J sequence. In
addition to clearing to the end of the screen, it moves the cursor to
the beginning of the next line if the cursor is not already at the
beginning of a line. To test this, use the following shell command
from a shell running in ansi-term:

echo -e 'hello\e[Jworld'

On a vt100/ansi/ecma compatible terminal, this should leave
"helloworld" visible on the screen, with everything afterward cleared.
On GNU emacs' ansi-term, it prints hello on one line and world on the
next, after clearing to the end of the screen.

This bug makes busybox shells' cmdedit functionality essentially
unusable, since it depends on ESC [ J leaving the cursor in the
correct position. Other programs may be affected too.

Removing the calls to term-unwrap-line from term-erase-in-display (in
term.el) fixes the problem, but I don't know if this has any bad
side-effects.

Also, in reading this code, I noticed that the sequence ESC [ 1 J
seems to only erase up to the point. To emulate VT100's Roman-inspired
off-by-one idiosyncracies, it should erase up to point+1, i.e. also
erase the character "under" the terminal cursor.

~Rich

 
-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Richard Stallman]

    Looking through the code, I think the calls to term-unwrap-line should
    be removed.  The note in the docstring of term-erase-in-display that
    it "should only be called when point is at the start of a screen line"
    is also false; this condition generally doesn't hold in situations
    where this function is called, and if we remove the term-unwrap-line
    calls, it's not necessary at all.

    Cursory testing seems to indicate that ansi-term behaves fine without
    the term-unwrap-line calls.

    What do you think?

I do not know that code, but if we do eventually make the change you
recommend, please put in a comment explaining that term-unwrap-line
was formerly called there, and a pointer to the test case for which
that call was removed.  That way, if this change breaks another case,
we will see what to do.

Re: [dalias@aerifal.cx: ansi-term \e[J causes spurious newline [revised report]]

[Dan Nicolaescu]

Chong Yidong <cyd@stupidchicken.com> writes:

  > > From: Rich Felker <dalias@aerifal.cx>
  > > Subject: ansi-term \e[J causes spurious newline [revised report]
  > > To: bug-gnu-emacs@gnu.org
  > >
  > > On versions of GNU emacs I have tested (21.1 and cvs unicode-2
  > > branch), the "ansi-term" terminal emulator (M-x ansi-term) exhibits
  > > incorrect terminal behavior when given the ESC [ J sequence. In
  > > addition to clearing to the end of the screen, it moves the cursor to
  > > the beginning of the next line if the cursor is not already at the
  > > beginning of a line. To test this, use the following shell command
  > > from a shell running in ansi-term:
  > >
  > > echo -e 'hello\e[Jworld'
  > >
  > > On a vt100/ansi/ecma compatible terminal, this should leave
  > > "helloworld" visible on the screen, with everything afterward cleared.
  > > On GNU emacs' ansi-term, it prints hello on one line and world on the
  > > next, after clearing to the end of the screen.
  > >
  > > Removing the calls to term-unwrap-line from term-erase-in-display (in
  > > term.el) fixes the problem, but I don't know if this has any bad
  > > side-effects.
  > 
  > Looking through the code, I think the calls to term-unwrap-line should
  > be removed.  The note in the docstring of term-erase-in-display that
  > it "should only be called when point is at the start of a screen line"
  > is also false; this condition generally doesn't hold in situations
  > where this function is called, and if we remove the term-unwrap-line
  > calls, it's not necessary at all.
  > 
  > Cursory testing seems to indicate that ansi-term behaves fine without
  > the term-unwrap-line calls.
  > 
  > What do you think?

I checked in a patch that I hope fixes the problem.