bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Bruno CHARRON]

Running 'emacs -Q', I load the latest 'term.el' in master (commit
bd36ab5), then set term-suppress-hard-newline to 't' and launch 'term'
with zsh as shell.
When typing a command which overflows the screen width, the shell sends
a "soft-wrap" after typing a character on the last column, say 'x
^M^[[K' after typing 'x' but 'term' does not wrap, it continues the same
row.
Then when hitting backspace, the shell asks to go up a row '^[[A' and
the 'term' state becomes inconsistent as it did not go down before.
The reason seems to be that at line 2912 (term.el, commit bd36ab5), 'x '
is inserted which makes the cursor go one column past the edge of the
screen and the wrapping at line 2936 only checks that the cursor is
exactly at the edge of the screen.
When term-suppress-hard-newline is nil, the wrapping is properly done
thanks to the '>' condition at line 2903.

In GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.18.9)
 of 2018-05-29 built on lgw01-amd64-038
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: Ubuntu 16.04.5 LTS

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Noam Postavsky]

Bruno CHARRON <bruno@charron.email> writes:

> Running 'emacs -Q', I load the latest 'term.el' in master (commit
> bd36ab5), then set term-suppress-hard-newline to 't' and launch 'term'
> with zsh as shell.
> When typing a command which overflows the screen width, the shell sends
> a "soft-wrap" after typing a character on the last column, say 'x
> ^M^[[K' after typing 'x' but 'term' does not wrap, it continues the same
> row.

Can you explain exactly what you're typing in?  I don't understand what
'x^M^[[K' means here.  Is specific to zsh?  Is it specific to your
prompt settings?

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Bruno CHARRON]

On Fri, Nov 2, 2018 at 7:53 AM Noam Postavsky <npostavs@gmail.com> wrote:
> Can you explain exactly what you're typing in?  I don't understand what
> 'x^M^[[K' means here.  Is specific to zsh?  Is it specific to your
> prompt settings?

Sorry the line break ate the space, the shell response is 'x ^M^[[K' and the
issue is with the space.

Basically I am typing 'x' repeatedly in the command line until it reaches the
right screen edge.
Setting term-log-buffer as t, I can see what the shell is sending to emacs in
the Messages buffer.
When I type 'x' in the middle of the screen, the shell responds 'x' to print an
'x' at the current cursor position.
When I type 'x' on the last column, the shell responds 'x ^M^[[K', which I could
understand with this explanation [1].
It seems to be the standard way to ask the terminal to wrap the line under
uncertainty on its behavior.
First it asks to insert 'x' on the last column, then some terminals will wrap
then but just in case it asks to insert an additional ' ' to force wrapping then
erases the new line (carriage return '^M' then erase to end of line '^[[K', see
[2]).
If term.el processes 'x' first then ' ', it will wrap when processing the ' '
but when term-suppress-hard-newline is t, it processes both at the same time and
doesn't wrap due to the reason explained in the original post.

When the command line has some attributes, e.g. bold red, the shell sends
'^[[1m^[[31mx^[[m^[[39m ^M^[[K', which means turn on bold ('^[[1m') red
('^[[31m'), insert 'x', reset attributes ('^[[m') and set default color
('^[[39m'), insert space, then '^M^[[K' as before.
Interestingly, in that case, term.el will first process the 'x' then the ' '
because there are control commands in between, and there is no issue (it wraps).

See a gif of that behavior in [3].

The behavior is the same with 'bash --norc', although it only returns 'x ^M', no
'^[[K'.
It does not depend on my prompt and also happens with 'zsh -f'.

[1] https://stackoverflow.com/a/31360700
[2] http://man7.org/linux/man-pages/man4/console_codes.4.html
[3] https://i.imgur.com/1dIQ8c6.gif

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Noam Postavsky]

found 33230 24.4
tags 33230 + confirmed
quit

Bruno CHARRON <bruno@charron.email> writes:

> Basically I am typing 'x' repeatedly in the command line until it reaches the
> right screen edge.
> Setting term-log-buffer as t, I can see what the shell is sending to emacs in
> the Messages buffer.
> When I type 'x' in the middle of the screen, the shell responds 'x' to print an
> 'x' at the current cursor position.
> When I type 'x' on the last column, the shell responds 'x ^M^[[K', which I could
> understand with this explanation [1].
> It seems to be the standard way to ask the terminal to wrap the line under
> uncertainty on its behavior.
> First it asks to insert 'x' on the last column, then some terminals will wrap
> then but just in case it asks to insert an additional ' ' to force wrapping then
> erases the new line (carriage return '^M' then erase to end of line '^[[K', see
> [2]).
> If term.el processes 'x' first then ' ', it will wrap when processing the ' '
> but when term-suppress-hard-newline is t, it processes both at the same time and
> doesn't wrap due to the reason explained in the original post.

Ah, got it.  I got mixed up a few times, because I didn't realize it
only happens after the first line (otherwise there's nowhere to go back
up to).  I can reproduce this also with 24.4 (when
term-suppress-hard-newline was introduced).

It seems like this option is somewhat incompatible with shells, it's not
clear what the right behaviour would be.  You say the problem is that
there is no wrapping, but isn't term-suppress-hard-newline exactly
intended to suppress this kind of wrapping?

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Bruno CHARRON]

> It seems like this option is somewhat incompatible with shells, it's not
> clear what the right behaviour would be.  You say the problem is that
> there is no wrapping, but isn't term-suppress-hard-newline exactly
> intended to suppress this kind of wrapping?

The option is said to "mean[s] text can automatically reflow if the
window is resized" and I think the behavior of libvte with the rewrap
option implements that well. You can see in [1] the behavior with and
without the option in gnome-terminal 3.18.3. The option is extensively
documented here [3].

For emacs, the rewrapping is currently done well with
term-suppress-hard-newline for all lines except the command currently
edited. One solution, though non-trivial, may be to interpret the
cursor movements (from the shell) which fall within the "region" of
the last (emacs) line, corresponding to several (shell) lines if there
was wrapping, as movements within the last (emacs) line. Emacs moves
to a new line only if the shell movements take it away from the
current "region".

For example, with width 20, after typing x repeatedly, _ is the cursor
on the last column (emacs current col: 19)
$ xxxxxxxxxxxxxxxxx_
Here, the last emacs line corresponds to a "region" of one (shell) line.
Then typing x, the shell (bash) sends "x ^M" and thinks the display is
$ xxxxxxxxxxxxxxxxxx
_
but emacs still displays a single line, visually wrapped or truncated
according to emacs' config
$ xxxxxxxxxxxxxxxxxxx_
to get there, it first processes "x ", the cursor is now at column 21,
then it processes ^M and since the width is 20, it moves the cursor to
the beginning of the current (shell) line which is at column 21 - (21
% 20) = 20.
Here, the last emacs line corresponds to a "region" of two (shell) lines.
Then after hitting backspace, the shell sends
"^[[A^[[19C^[[K^M^J^M^[[K^[[A^[[19C" and thinks the display is
$ xxxxxxxxxxxxxxxxx_

(with an empty line)
and emacs shows
$ xxxxxxxxxxxxxxxxx_
to get there, it virtually moves up one line (but actually goes 20
columns left due to width = 20 so current col: 20 - 20 = 0) due to
^[[A, then 19 columns right (current col: 0 + 19 = 19) due to ^[[19C
then erases up to the end of the virtual line (before col (19 + 20) -
((19 + 20) % 20) = 20) due to ^[[K, then moves back to the beginning
of the virtual line (current col: 19 - (19 % 20) = 0) due to ^M, moves
down one virtual line (current col: 0 + 20 = 20) due to ^K, erases up
to the end of the virtual line (before col (20 + 20) - ((20 + 20) %
20) = 40) due to ^[[K, moves up one virtual line (current col 20 - 20
= 0) due to ^[[A, then 19 columns right (current col: 0 + 19 = 19).
Here, the last emacs line corresponds to a "region" of only one
(shell) lines, as the second (shell) line is empty (due to ^M^[[K) and
does not contain the cursor, so that if we execute and the shell sends
a newline ^J to print the output, the output will fall within a new
(emacs) line.

Definitely non-trivial but I don't have another solution in mind that
would give the expected behavior. I don't know how Visual Line works
but it seems the idea is similar so maybe it can be reused? Instead of
remapping the user's cursor movement to movements on the visual lines,
we would like to remap the shell's cursor movement to movements on the
visual lines instead of the logical lines.

Another issue I found that may need to be addressed to get a behavior
similar to gnome-terminal above is that the shell is not aware
(checking $COLUMS) of when the frame is resized, only when its window
is resized and there is another window on the side.

[1] https://imgur.com/a/7IwZfmy
[2] https://gitlab.gnome.org/GNOME/vte/blob/master/doc/rewrap.txt

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

 > Another issue I found that may need to be addressed to get a behavior
 > similar to gnome-terminal above is that the shell is not aware
 > (checking $COLUMS) of when the frame is resized, only when its window
 > is resized and there is another window on the side.

FWIW this is Bug#32720 for which I am responsible.  It will be fixed
in Emacs 26.2 by reestablishing the behavior of Emacs 25.

martin

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 225 bytes --]

 > FWIW this is Bug#32720 for which I am responsible.  It will be fixed
 > in Emacs 26.2 by reestablishing the behavior of Emacs 25.

Attached find a patch for this against the release branch.  Please try
it.

Thanks, martin

[-- Attachment #2: run-window-configuration-change.diff --]
[-- Type: text/plain, Size: 1849 bytes --]

diff --git a/src/frame.c b/src/frame.c
index 0a6ca26..8076eba 100644
--- a/src/frame.c
+++ b/src/frame.c
@@ -779,6 +779,8 @@ struct frame *
 
   unblock_input ();
 
+  run_window_configuration_change_hook (f);
+
 #ifdef HAVE_WINDOW_SYSTEM
   {
     /* Adjust size of F's child frames.  */
diff --git a/src/window.c b/src/window.c
index 9026a7b..ef88fde 100644
--- a/src/window.c
+++ b/src/window.c
@@ -57,7 +57,6 @@ static bool foreach_window_1 (struct window *,
 static bool window_resize_check (struct window *, bool);
 static void window_resize_apply (struct window *, bool);
 static void select_window_1 (Lisp_Object, bool);
-static void run_window_configuration_change_hook (struct frame *);
 
 static struct window *set_window_fringes (struct window *, Lisp_Object,
 					  Lisp_Object, Lisp_Object);
@@ -3319,7 +3318,7 @@ depends on the value of (window-start WINDOW), so if calling this
     Fselect_frame (frame, Qt);
 }
 
-static void
+void
 run_window_configuration_change_hook (struct frame *f)
 {
   ptrdiff_t count = SPECPDL_INDEX ();
diff --git a/src/window.h b/src/window.h
index 629283a..bc6d8ab 100644
--- a/src/window.h
+++ b/src/window.h
@@ -1016,6 +1016,7 @@ extern Lisp_Object window_from_coordinates (struct frame *, int, int,
 extern void shrink_mini_window (struct window *, bool);
 extern int window_relative_x_coord (struct window *, enum window_part, int);
 
+void run_window_configuration_change_hook (struct frame *f);
 void run_window_size_change_functions (Lisp_Object);
 
 /* Make WINDOW display BUFFER.  RUN_HOOKS_P means it's allowed
diff --git a/src/xfns.c b/src/xfns.c
index 1381fee..574f0bb 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -1599,4 +1599,5 @@ struct mouse_cursor_data {
     }
 #endif /* not USE_X_TOOLKIT && not USE_GTK */
   adjust_frame_glyphs (f);
+  run_window_configuration_change_hook (f);
 }

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Bruno CHARRON]

>  > FWIW this is Bug#32720 for which I am responsible.  It will be fixed
>  > in Emacs 26.2 by reestablishing the behavior of Emacs 25.
>
> Attached find a patch for this against the release branch.  Please try
> it.

Thanks martin, I confirm that after applying your patch on the 26.1
release, COLUMNS is correctly updated in the shell at each frame
resize while it is not without the patch.

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

 >> Attached find a patch for this against the release branch.  Please try
 >> it.
 >
 > Thanks martin, I confirm that after applying your patch on the 26.1
 > release, COLUMNS is correctly updated in the shell at each frame
 > resize while it is not without the patch.

Eli - is this OK to install on the release branch?  It restores the
behavior of Emacs 25 to fix a bug introduced in 26.1.

martin

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Eli Zaretskii]

> Date: Tue, 13 Nov 2018 10:07:53 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: Noam Postavsky <npostavs@gmail.com>, 33230@debbugs.gnu.org, 
>  "eliz@gnu.org" <eliz@gnu.org>
> 
>  >> Attached find a patch for this against the release branch.  Please try
>  >> it.
>  >
>  > Thanks martin, I confirm that after applying your patch on the 26.1
>  > release, COLUMNS is correctly updated in the shell at each frame
>  > resize while it is not without the patch.
> 
> Eli - is this OK to install on the release branch?  It restores the
> behavior of Emacs 25 to fix a bug introduced in 26.1.

I'm too confused about this to be able to make up my mind.  Please
help me figure out what am I missing.

You said up-thread:

> > Another issue I found that may need to be addressed to get a behavior
> > similar to gnome-terminal above is that the shell is not aware
> > (checking $COLUMS) of when the frame is resized, only when its window
> > is resized and there is another window on the side.
> 
> FWIW this is Bug#32720 for which I am responsible.  It will be fixed
> in Emacs 26.2 by reestablishing the behavior of Emacs 25.

And your proposed change runs window-configuration-change-hook in two
additional places.

My confusion about this is twofold: (1) the original report for this
bug doesn't seem to involve any resizing of a frame, and (2) the ELisp
manual explicitly says that "resizing the frame or individual windows
do not count as configuration changes", and thus this hook shouldn't
be run when the frame is resized.

So how does the proposed change fix the problem at hand, and why do
you want to do exactly what the ELisp manual says we don't?

Thanks.

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

 >> FWIW this is Bug#32720 for which I am responsible.  It will be fixed
 >> in Emacs 26.2 by reestablishing the behavior of Emacs 25.
 >
 > And your proposed change runs window-configuration-change-hook in two
 > additional places.

Right.

 > My confusion about this is twofold: (1) the original report for this
 > bug doesn't seem to involve any resizing of a frame,

At the end it says:

   Another issue I found that may need to be addressed to get a behavior
   similar to gnome-terminal above is that the shell is not aware
   (checking $COLUMS) of when the frame is resized, only when its window
   is resized and there is another window on the side.

 > and (2) the ELisp
 > manual explicitly says that "resizing the frame or individual windows
 > do not count as configuration changes", and thus this hook shouldn't
 > be run when the frame is resized.
 >
 > So how does the proposed change fix the problem at hand,

By running 'window-configuration-change-hook' for frame resizes as
with Emacs 25.

 > and why do
 > you want to do exactly what the ELisp manual says we don't?

The Elisp manual doesn't represent the facts because we still run the
hook when resizing single windows.  The idea behind that text was to
avoid that new code runs 'window-configuration-change-hook' to trace
window size changes because that is unreliable (not all size changes
are caught) and costly (it's often run when no sizes changed at all).
Also NEWS warned that

*** Resizing a frame no longer runs 'window-configuration-change-hook'.
'window-size-change-functions' should be used instead.

and I checked known clients of 'window-configuration-change-hook'
whether they should call 'window-size-change-functions' instead.
Little did I expect to find such a client in window.el though, so this
went unnoticed.

We could add a call to 'window-size-change-functions' as Gary proposed
in the report of Bug#32720.  But then 'window--adjust-process-windows'
would be run by both 'window-configuration-change-hook' and
'window-size-change-functions' effectively increasing the number of
calls of that function instead of decreasing it.  If you prefer that
solution we can certainly do it.

martin

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Eli Zaretskii]

> Date: Wed, 14 Nov 2018 09:32:57 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: bruno@charron.email, npostavs@gmail.com, 33230@debbugs.gnu.org
> 
> At the end it says:
> 
>    Another issue I found that may need to be addressed to get a behavior
>    similar to gnome-terminal above is that the shell is not aware
>    (checking $COLUMS) of when the frame is resized, only when its window
>    is resized and there is another window on the side.
> 
>  > and (2) the ELisp
>  > manual explicitly says that "resizing the frame or individual windows
>  > do not count as configuration changes", and thus this hook shouldn't
>  > be run when the frame is resized.
>  >
>  > So how does the proposed change fix the problem at hand,
> 
> By running 'window-configuration-change-hook' for frame resizes as
> with Emacs 25.
> 
>  > and why do
>  > you want to do exactly what the ELisp manual says we don't?
> 
> The Elisp manual doesn't represent the facts because we still run the
> hook when resizing single windows.  The idea behind that text was to
> avoid that new code runs 'window-configuration-change-hook' to trace
> window size changes because that is unreliable (not all size changes
> are caught) and costly (it's often run when no sizes changed at all).
> Also NEWS warned that
> 
> *** Resizing a frame no longer runs 'window-configuration-change-hook'.
> 'window-size-change-functions' should be used instead.
> 
> and I checked known clients of 'window-configuration-change-hook'
> whether they should call 'window-size-change-functions' instead.
> Little did I expect to find such a client in window.el though, so this
> went unnoticed.

So you propose to fix something not directly related to the current
bug by reverting to pre-Emacs 26 behavior, and also make NEWS and the
manual consistent with the change?  If so, I'm okay with doing that on
the emacs-26 branch.

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

 > Thanks martin, I confirm that after applying your patch on the 26.1
 > release, COLUMNS is correctly updated in the shell at each frame
 > resize while it is not without the patch.

I applied a different fix provided by Gary Fredericks to the release
branch now.  Please test that.

Thanks, martin

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Bruno Charron]

> I applied a different fix provided by Gary Fredericks to the release
> branch now.  Please test that.

Thanks martin, I confirm that COLUMNS is correctly updated in the
shell at each frame resize on the release branch while it is not
updated when going back to commit d6542ea for example.

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[martin rudalics]

 > Thanks martin, I confirm that COLUMNS is correctly updated in the
 > shell at each frame resize on the release branch while it is not
 > updated when going back to commit d6542ea for example.

Many thanks for the prompt confirmation.

martin

bug#33230: 26.1; Soft-wrap issue in term.el with term-suppress-hard-newline

[Lars Ingebrigtsen]

Bruno Charron <bruno@charron.email> writes:

>> I applied a different fix provided by Gary Fredericks to the release
>> branch now.  Please test that.
>
> Thanks martin, I confirm that COLUMNS is correctly updated in the
> shell at each frame resize on the release branch while it is not
> updated when going back to commit d6542ea for example.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

This bug was then left open, but skimming this bug report, it seems like
the problem was fixed, so I'm closing it now.  If I misunderstood,
please respond to the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no