bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Thomas Fitzsimmons]

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':

$ emacs -Q
Create term-trailing-spaces.c:
#include <stdio.h>

int main (int argc, char* argv[])
{
  int i = 0;
  while (i < 10)
    {
      printf ("aaaaaaaaaaaaaaaaaaaaaaa\n\rbbbbbbbbbbbb\n\r");
      i++;
    }
  return 0;
}

M-x term
Run program: /bin/bash
RET
C-c M-x eval-expression
(setq show-trailing-whitespace t)
$ gcc -o term-trailing-spaces term-trailing-spaces.c
$ stty -opost
$ ./term-trailing-spaces
Trailing spaces are shown in red.
$ stty opost
$ ./term-trailing-spaces
No trailing spaces are shown.

I'm using serial-term to connect to an embedded Linux board's serial
console.  During Linux boot the terminal doesn't post-process output
(equivalent to stty -opost) so long-line/shorter-line combinations like
in the test case result in spaces being appended to the end of the
buffer.  This is OK in char mode but in line mode the spaces stack up at
the end of the buffer and interfere with terminal I/O.  If I
accidentally leave term-mode in line mode during boot I have to kill the
buffer and restart serial-term.

I traced through term-emulate-terminal to:

   ((eq char ?\r)  ;; (terminfo: cr)
    (term-vertical-motion 0)
    (setq term-current-column term-start-line-column))

term-vertical-motion expands to vertical-motion and it's that call that
results in the spaces being inserted in the buffer.

I worked around the issue by removing, after-the-fact, spaces inserted
by the (vertical-motion 0) call.  A more efficient solution would be to
prevent vertical-motion from inserting the spaces in the first place,
but that happens deep within Emacs somewhere.  I'm hoping a terminal
expert can suggest where to look.

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/23.2/etc/DEBUG.

In GNU Emacs 23.2.1 (i386-redhat-linux-gnu, GTK+ Version 2.20.1)
 of 2010-06-03 on x86-07.phx2.fedoraproject.org
Windowing system distributor `Fedora Project', version 11.0.10800000
configured using `configure  '--build=i386-redhat-linux-gnu' '--host=i386-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-dbus' '--with-gif' '--with-jpeg' '--with-png' '--with-rsvg' '--with-tiff' '--with-xft' '--with-xpm' '--with-x-toolkit=gtk' 'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu' 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=at
 om -fasynchronous-unwind-tables''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Term

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
o C-g M-x t e r m <return> <return> C-c M-x C-g C-x 
o C-x C-c C-g C-g C-x k C-g C-c M-x M-p C-k s w i t 
<tab> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> k i l l - b u <tab> <return> <return> y 
e s <return> C-g C-g M-x t e r m <return> <return> 
C-c M-x e v a l - e x p <tab> <return> ( s e t q SPC 
s h o w - t r a i l i n g - s h i w <backspace> <backspace> 
<backspace> <backspace> w h i t e s p a c e SPC t ) 
<return> g c c SPC - o SPC t e r m - t r a i l i n 
g - s p a c e s SPC t e r m - t r a i l i n g - s p 
a <tab> C-a C-k c d SPC s o u <tab> t e r m <tab> <return> 
g c c SPC - o SPC t e r m - t r a i l i <tab> SPC t 
e r m - t r a i l i <tab> . c <return> s t t y SPC 
- o p o s t <return> . / t e r m - t r a i l i <tab> 
<return> s t t y SPC o p o s t <return> . / t e r m 
- t r a i l i n g - s p a <tab> <return> . / t e r 
m - t r a i l i <tab> <return> C-c M-x r e p o r t 
- e m <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Quit [5 times]
t

Load-path shadows:
/usr/share/emacs/site-lisp/flim/md4 hides /usr/share/emacs/23.2/lisp/md4
/usr/share/emacs/site-lisp/flim/sha1 hides /usr/share/emacs/23.2/lisp/sha1
/usr/share/emacs/site-lisp/flim/hex-util hides /usr/share/emacs/23.2/lisp/hex-util
/usr/share/emacs/site-lisp/flim/ntlm hides /usr/share/emacs/23.2/lisp/net/ntlm
/usr/share/emacs/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/23.2/lisp/net/sasl-ntlm
/usr/share/emacs/site-lisp/flim/hmac-def hides /usr/share/emacs/23.2/lisp/net/hmac-def
/usr/share/emacs/site-lisp/flim/hmac-md5 hides /usr/share/emacs/23.2/lisp/net/hmac-md5
/usr/share/emacs/site-lisp/flim/sasl-digest hides /usr/share/emacs/23.2/lisp/net/sasl-digest
/usr/share/emacs/site-lisp/flim/sasl-cram hides /usr/share/emacs/23.2/lisp/net/sasl-cram
/usr/share/emacs/site-lisp/flim/sasl hides /usr/share/emacs/23.2/lisp/net/sasl

Features:
(shadow sort mail-extr message idna sendmail regexp-opt ecomplete rfc822
mml easymenu mml-sec password-cache mm-decode mm-bodies mm-encode
mailcap mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev
nnheader gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 sha1-el hex-util hashcash mail-utils emacsbug
term disp-table ehelp electric ring tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd font-setting tool-bar dnd fontset
image fringe lisp-mode register page menu-bar rfn-eshadow timer select
scroll-bar mldrag mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev loaddefs button minibuffer faces cus-face files
text-properties overlay md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote make-network-process dbusbind
system-font-setting font-render-setting gtk x-toolkit x multi-tty emacs)

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Chong Yidong]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> term-vertical-motion expands to vertical-motion and it's that call that
> results in the spaces being inserted in the buffer.
>
> I worked around the issue by removing, after-the-fact, spaces inserted
> by the (vertical-motion 0) call.  A more efficient solution would be to
> prevent vertical-motion from inserting the spaces in the first place,
> but that happens deep within Emacs somewhere.  I'm hoping a terminal
> expert can suggest where to look.

That's strange.  vertical-motion should not affect the buffer contents;
it only moves point.

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Thomas Fitzsimmons]

Chong Yidong <cyd@stupidchicken.com> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> term-vertical-motion expands to vertical-motion and it's that call that
>> results in the spaces being inserted in the buffer.
>>
>> I worked around the issue by removing, after-the-fact, spaces inserted
>> by the (vertical-motion 0) call.  A more efficient solution would be to
>> prevent vertical-motion from inserting the spaces in the first place,
>> but that happens deep within Emacs somewhere.  I'm hoping a terminal
>> expert can suggest where to look.
>
> That's strange.  vertical-motion should not affect the buffer contents;
> it only moves point.

Yeah, it is strange.  Were you able to replicate on your setup the
behavior I described?

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Thomas Fitzsimmons]

Chong Yidong <cyd@stupidchicken.com> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> term-vertical-motion expands to vertical-motion and it's that call that
>> results in the spaces being inserted in the buffer.
>>
>> I worked around the issue by removing, after-the-fact, spaces inserted
>> by the (vertical-motion 0) call.  A more efficient solution would be to
>> prevent vertical-motion from inserting the spaces in the first place,
>> but that happens deep within Emacs somewhere.  I'm hoping a terminal
>> expert can suggest where to look.
>
> That's strange.  vertical-motion should not affect the buffer contents;
> it only moves point.

I did some more tracing and it's not vertical-motion.  Here's a
simplified test case:

/*
  M-x term
  (Run program: /bin/bash)
  RET
  C-c M-x eval-expression
  (setq show-trailing-whitespace t)
  $ gcc -o term-trailing-spaces term-trailing-spaces.c
  $ stty -opost
  $ ./term-trailing-spaces
  (trailing spaces shown in red)
  $ stty opost
  $ ./term-trailing-spaces
  (no trailing spaces)
*/

#include <stdio.h>

int main (int argc, char* argv[])
{
  printf ("\n");
  return 0;
}

With stty -opost, the new test case produces (dots represent spaces):

$ ./term-trailing-spaces
........................
........................$.

That matches gnome-terminal's visual output for the same test.  I can't
tell if it uses spaces to produce that output.

I can eliminate the trailing spaces with this change:

Index: term.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/term.el,v
retrieving revision 1.115
diff -u -r1.115 term.el
--- term.el	13 Mar 2009 01:43:03 -0000	1.115
+++ term.el	20 Nov 2010 23:28:57 -0000
@@ -2619,7 +2619,7 @@
     (save-excursion
       (end-of-line)
       (setq point-at-eol (point)))
-    (move-to-column term-current-column t)
+    (move-to-column term-current-column)
     ;; If move-to-column extends the current line it will use the face
     ;; from the last character on the line, set the face for the chars
     ;; to default.

But then the output is:

$ ./term-trailing-spaces

$.

which doesn't match gnome-terminal visually, so I don't think that's a
correct solution.

When the terminal is in char mode the trailing spaces don't affect
input.  When the terminal is in line mode, input problems happen as a
result of the trailing spaces.

I think the reason it's a problem is explained in the
term-pending-delete-marker comment:

(defvar term-pending-delete-marker) ;; New user input in line mode needs to
;;		be deleted, because it gets echoed by the inferior.
;;		To reduce flicker, we defer the delete until the next output.

Maybe the correct solution is whenever (move-to-column ... t) is called
in line mode, term-pending-delete-marker needs to be used to delete the
extra spaces.

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Thomas Fitzsimmons]

This issue affected me when I was working with the RedBoot bootloader.
The boards I'm currently working on instead use the U-Boot bootloader.
I re-tested booting a U-Boot board under serial-term in line mode, and I
didn't see the terminal-trashing trailing spaces.  So I think this was a
bad interaction between RedBoot's terminal handling and term.el.  It's
pretty obscure, and it's not affecting my day-to-day work anymore, so I
would be fine with this bug being closed as wontfix.

bug#7426: 23.2; term.el: After running stty -opost, printing \n\r inserts trailing spaces

[Thomas Fitzsimmons]

tags 7426 wontfix
thanks

Tagging wontfix and closing.

Thomas