bug#23594: 25.0.94; Display errors on Linux tty

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

In GNU Emacs 25.0.94.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.18.7)
 of 2016-05-17 built on acm
Configured using:
 'configure --with-tiff=no --with-gif=no --with-gpm'

Configured features:
XPM JPEG PNG RSVG SOUND GPM DBUS GSETTINGS NOTIFY ACL GNUTLS LIBXML2
FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_GB.utf8
  locale-coding-system: utf-8-unix

Major mode: C/l

Minor modes in effect:
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

(This is the latest pretest version.)

emacs -Q
C-x b foo.c
M-x c-mode

Type the following five lines into the buffer:
#########################################################################
char foo [] = "foo";

#define FOO BAR

char bar [] = "bar";
#########################################################################

Move point to just after "BAR" on line 3.
C-u C-x =

There are now two display bugs evident:
1. The vertical divider between the two windows is missing a piece just
after the line "  decomposition: (10) ('" in *Help*.

2. In foo.c, the actual text display is out of sync with the text cursor.
The text has been scrolled one line up, but the cursor has remained at
the same position in the window, apparently at the end of a non-existent
line of spaces.  The cursor can be freely moved, but is one line out of
sync with the displayed text.

Type C-l one or more times.  The foo.c window becomes increasingly
confused.  Type <backspace>.  This causes even more confusion.

The bugs are not evident in X-Windows.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Eli Zaretskii]

> Date: Sat, 21 May 2016 10:47:20 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> emacs -Q
> C-x b foo.c
> M-x c-mode
> 
> Type the following five lines into the buffer:
> #########################################################################
> char foo [] = "foo";
> 
> #define FOO BAR
> 
> char bar [] = "bar";
> #########################################################################
> 
> Move point to just after "BAR" on line 3.
> C-u C-x =
> 
> There are now two display bugs evident:
> 1. The vertical divider between the two windows is missing a piece just
> after the line "  decomposition: (10) ('" in *Help*.
> 
> 2. In foo.c, the actual text display is out of sync with the text cursor.
> The text has been scrolled one line up, but the cursor has remained at
> the same position in the window, apparently at the end of a non-existent
> line of spaces.  The cursor can be freely moved, but is one line out of
> sync with the displayed text.

I see this, but I don't understand why you needed the c-mode stuff and
the text.  the same problem happens in a buffer in Fundamental mode
when you invoke "C-u C-x =" at a newline.

This bug exists since we started showing the 'decomposition' of
characters in Emacs 24.1.  With LF, we send a literal LF character to
the screen.

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello, Eli.

On Sat, May 21, 2016 at 08:22:05PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 21 May 2016 10:47:20 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > emacs -Q
> > C-x b foo.c
> > M-x c-mode

> > Type the following five lines into the buffer:
> > #########################################################################
> > char foo [] = "foo";

> > #define FOO BAR

> > char bar [] = "bar";
> > #########################################################################

> > Move point to just after "BAR" on line 3.
> > C-u C-x =

> > There are now two display bugs evident:
> > 1. The vertical divider between the two windows is missing a piece just
> > after the line "  decomposition: (10) ('" in *Help*.

> > 2. In foo.c, the actual text display is out of sync with the text cursor.
> > The text has been scrolled one line up, but the cursor has remained at
> > the same position in the window, apparently at the end of a non-existent
> > line of spaces.  The cursor can be freely moved, but is one line out of
> > sync with the displayed text.

> I see this, but I don't understand why you needed the c-mode stuff and
> the text.  the same problem happens in a buffer in Fundamental mode
> when you invoke "C-u C-x =" at a newline.

Hmm.  Maybe I wanted to take some of the blame.  ;-)

> This bug exists since we started showing the 'decomposition' of
> characters in Emacs 24.1.  With LF, we send a literal LF character to
> the screen.

OK.  The next question is is it easy to fix?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Eli Zaretskii]

> Date: Sat, 21 May 2016 19:01:10 +0000
> Cc: 23594@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > This bug exists since we started showing the 'decomposition' of
> > characters in Emacs 24.1.  With LF, we send a literal LF character to
> > the screen.
> 
> OK.  The next question is is it easy to fix?

Yes.  We should not send control characters to that buffer.

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello, Eli.

On Sat, May 21, 2016 at 10:09:38PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 21 May 2016 19:01:10 +0000
> > Cc: 23594@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > This bug exists since we started showing the 'decomposition' of
> > > characters in Emacs 24.1.  With LF, we send a literal LF character to
> > > the screen.

> > OK.  The next question is is it easy to fix?

> Yes.  We should not send control characters to that buffer.

OK, I'm looking at it.

Is there any chance of getting a patch for this into Emacs 25.1?  I know
this is nothing new, but it's very ugly.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello, Eli.

On Sat, May 21, 2016 at 10:09:38PM +0300, Eli Zaretskii wrote:
> > > This bug exists since we started showing the 'decomposition' of
> > > characters in Emacs 24.1.  With LF, we send a literal LF character
> > > to the screen.

That's only half the story.  The literal LF doesn't seem to be the
problem.  Rather, it's got a 'composition text-property attached to it.
The string we're trying to display is

#("  decomposition: (10) ('\n')\n" 24 25 (composition (0 1 [9 10 9])))
                           ^
			   |
			   24

What is this composition trying to do?  The [9 10 9] is [\t \n \t].

The whole thing is a result of calling
    (char-code-property-description 'decomposition '(10))

Under X-Windows, the same string is displayed, this time successfully.
The call
    (inseert #("  decomposition: .... [9 10 9])))
works on X-Windows, the "\n" with the composition property being
displayed as a square box.

> > OK.  The next question is is it easy to fix?

> Yes.  We should not send control characters to that buffer.

It seems to me there is a bug in the display engine here: the same
string which is displayed successfully in X-Windows goes badly wrong on
a Linux tty.

Comments?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Eli Zaretskii]

> Date: Sun, 22 May 2016 09:38:13 +0000
> Cc: 23594@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> On Sat, May 21, 2016 at 10:09:38PM +0300, Eli Zaretskii wrote:
> > > > This bug exists since we started showing the 'decomposition' of
> > > > characters in Emacs 24.1.  With LF, we send a literal LF character
> > > > to the screen.
> 
> That's only half the story.  The literal LF doesn't seem to be the
> problem.

What do you mean by "literal LF"?  A newline ends a line, and is never
displayed at all, the display engine "swallows" it so it disappears
without a trace, and instead instructs the terminal to move to the
next line.

> Rather, it's got a 'composition text-property attached to it.
> The string we're trying to display is
> 
> #("  decomposition: (10) ('\n')\n" 24 25 (composition (0 1 [9 10 9])))
>                            ^
> 			   |
> 			   24
> 
> What is this composition trying to do?  The [9 10 9] is [\t \n \t].

It tries to prevent the character from being composed with surrounding
ones, so that we could display there combining marks and other similar
stuff.

> Under X-Windows, the same string is displayed, this time successfully.
> The call
>     (inseert #("  decomposition: .... [9 10 9])))
> works on X-Windows, the "\n" with the composition property being
> displayed as a square box.

Why is that a "success", exactly?  Is the user supposed to guess that
the box represents a LF?

> > > OK.  The next question is is it easy to fix?
> 
> > Yes.  We should not send control characters to that buffer.
> 
> It seems to me there is a bug in the display engine here: the same
> string which is displayed successfully in X-Windows goes badly wrong on
> a Linux tty.
> 
> Comments?

IMO, there's no such thing as a successful display of an unadorned LF
(or any other control character), on any kind of terminal.  What would
be the graphics for that?

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello, Eli.

On Sun, May 22, 2016 at 06:27:42PM +0300, Eli Zaretskii wrote:
> > On Sat, May 21, 2016 at 10:09:38PM +0300, Eli Zaretskii wrote:
> > > > > This bug exists since we started showing the 'decomposition' of
> > > > > characters in Emacs 24.1.  With LF, we send a literal LF character
> > > > > to the screen.

> > That's only half the story.  The literal LF doesn't seem to be the
> > problem.

> What do you mean by "literal LF"?  A newline ends a line, and is never
> displayed at all, the display engine "swallows" it so it disappears
> without a trace, and instead instructs the terminal to move to the
> next line.

I mean (insert "\n") works without problems, even though it's more like a
command, and isn't a glyph.

> > Rather, it's got a 'composition text-property attached to it.
> > The string we're trying to display is

> > #("  decomposition: (10) ('\n')\n" 24 25 (composition (0 1 [9 10 9])))
> >                            ^
> > 			       |
> > 			       24

> > What is this composition trying to do?  The [9 10 9] is [\t \n \t].

> It tries to prevent the character from being composed with surrounding
> ones, so that we could display there combining marks and other similar
> stuff.

OK.

> > Under X-Windows, the same string is displayed, this time successfully.
> > The call
> >     (inseert #("  decomposition: .... [9 10 9])))
> > works on X-Windows, the "\n" with the composition property being
> > displayed as a square box.

> Why is that a "success", exactly?  Is the user supposed to guess that
> the box represents a LF?

It's a "success" in contrast to the failure on a Linux virtual terminal,
where text in the other window ends up displayed where it isn't situated.

> > > > OK.  The next question is is it easy to fix?

> > > Yes.  We should not send control characters to that buffer.

> > It seems to me there is a bug in the display engine here: the same
> > string which is displayed successfully in X-Windows goes badly wrong on
> > a Linux tty.

> > Comments?

> IMO, there's no such thing as a successful display of an unadorned LF
> (or any other control character), on any kind of terminal.  What would
> be the graphics for that?

I suppose one answer would be that glyph with a tiny "L" in the top left
corner and a tiny "F" in the bottom right corner.  But we couldn't use
that, since we don't know that the current terminal can display it.
Another answer would be some standard substitute character (like an
inverse question mark, or whatever).

But the real point seems to me to be that `insert'ing the above string
(with the composition property) completely fouls up the display in the
other window.  I'm guessing here, but could it be that the display engine
is actually outputting a raw 0x0a byte rather than handling it sensibly?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Eli Zaretskii]

> Date: Sun, 22 May 2016 16:16:51 +0000
> Cc: 23594@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> could it be that the display engine is actually outputting a raw
> 0x0a byte

Of course!  Isn't that what I said at the beginning of this
discussion?

> rather than handling it sensibly?

It was forbidden from doing "sensible" things by the composition
trick.

I think the best fix that is safe for Emacs 25.1 is simply to avoid
displaying the "decomposition" part for low control characters -- they
cannot be composed anyway, AFAIU.

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello, Eli.

On Sun, May 22, 2016 at 07:50:24PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 22 May 2016 16:16:51 +0000
> > Cc: 23594@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > could it be that the display engine is actually outputting a raw
> > 0x0a byte

> Of course!  Isn't that what I said at the beginning of this
> discussion?

Yes it was, but that was rather a long time ago.  ;-)  I've been poring
over code since then, which left me somewhat confused.  It's difficult
stuff.

> > rather than handling it sensibly?

> It was forbidden from doing "sensible" things by the composition
> trick.

OK.

> I think the best fix that is safe for Emacs 25.1 is simply to avoid
> displaying the "decomposition" part for low control characters -- they
> cannot be composed anyway, AFAIU.

OK, I'll come with a patch to do that.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23594: 25.0.94; Display errors on Linux tty

[Alan Mackenzie]

Hello again, Eli.

On Sun, May 22, 2016 at 07:50:24PM +0300, Eli Zaretskii wrote:

[ .... ]

> I think the best fix that is safe for Emacs 25.1 is simply to avoid
> displaying the "decomposition" part for low control characters -- they
> cannot be composed anyway, AFAIU.

I've committed a patch to emacs-25 which does exactly that.

Since it's a workaround rather than a fix, I think it would be better to
leave the bug open for now.

-- 
Alan Mackenzie (Nuremberg, Germany).