bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

This is a spin-off from bug#20611: 24.4; mutt slow in ansi-term.

I'm logging a new bug report by request, although it seems to me that
this is still the same bug (and 20611 is still open), so I suggest that
they can both be closed together once a fix is committed?

I've experimented further with this issue since my comments in 20611,
and my current observations follow...

In certain circumstances, a full-screen redraw in a term.el terminal is
incredibly slow, apparently increasing with the (width x height) number
of characters in the terminal -- certainly things are dramatically worse
when the term window is full-screen, than when it is only small; and as
observed below, the slow-down is worse than linear with respect to the
number of characters being drawn -- each block of characters drawn takes
noticeably more time to draw than the previous block did).

The issue affects both GUI and terminal Emacs, and affects all terminal
buffers generated by term.el (i.e. this is not specific to `ansi-term'
as it was treated in the original bug, but will also affect `term' and
any other similar wrapper command which creates a terminal and runs a
given program in it.)

In my case a full-screen terminal is (210x48) characters, for which the
initial screen redraw when running Debian package manager command
"dpkg-reconfigure postfix" takes ~7 seconds (with the redraw occurring
in three 'blocks' of visible activity, each of which 'instantly' draws
the next X characters of the display, followed by some seconds of
waiting before the next burst of visible activity.

Conversely with a terminal size of (80x21) characters, performance is
much better, with a redraw taking ~1 second and not obviously divided
into blocks, but all happening at once. I would say that the block
size seen in the large terminal would exceed the number of characters
visible in the small terminal, so it is not surprising to see it all
drawn in a single block of activity.)

Curiously, I've just noticed that the redraw in the small case takes
~1 second usually; but if I expand the window and cause a slow redraw,
and then reduce it to the original size and try again, the redraw is
now much slower (~2 seconds instead of ~1).

My speculation is that the amount of preceding text in the buffer is
having an effect, because switching from a large window to a small
window meant that a lot of the text which had been visible in the large
window was no longer visible, yet still present if you scrolled back in
the buffer.

I've seemingly confirmed this by repeating the process (1. small and
quick-ish; 2. large and extremely slow; 3. small again and not as quick
as (1)); then deleting the contents of the buffer prior to the final
shell prompt and repeating the command a fourth time, for which I once
again observed a quick-ish response as per (1).

Likewise, adding a lot of preceding text into a large-sized term buffer
caused the command to slow down even more. Testing in GUI Emacs with an
even larger area (234x53) I was seeing ~10 second redraws initially, and
that increased to ~17 seconds after dumping a bunch of other text into
the buffer.

Furthermore, I see now that the processing time increases for each
block displayed during the redraw. I assume all but the final block has
the same number of characters, yet the approximate elapsed time when
each appears on screen is as follows:

~1 second
~4 seconds
~9 seconds
~10 seconds (n.b. the final block is much smaller than the others)

I believe those first three blocks are the same number of characters,
yet the processing times needed to draw them are roughly 1, 3, and 5
seconds respectively.

(At this point I would hazard a guess that each character drawn is also
processing, to some extent, all of the preceding characters?)

I haven't tried to delve into the term.el code at all (and I am not
familiar with terminal emulation in any case), so I can only hope that
these observations might help to explain the root cause of this issue.

As established in bug#20611, disabling the bidi support (by either
setting bidi-paragraph-direction to 'left-to-right, or by setting
bidi-display-reordering to nil) improves the speed of the slow redraws
by an order of magnitude.

The proposed fix for that bug sets bidi-paragraph-direction in the
`ansi-term' command, but this is not general enough. My initial thought
is to set the value in `term-mode' instead.

I am not familiar with bidi concerns, and do not know whether there is
ever a use-case for bidi being active in a term-mode buffer, but I note
the following comment in the change committed for the original bug:

   ;; There's a larger problem here with supporting bidirectional text:
   ;; the application that writes to the terminal could have its own
   ;; ideas about displaying bidirectional text, and might not want us
   ;; reordering the text or deciding on base paragraph direction.  One
   ;; such application is Emacs in TTY mode...  FIXME.



-Phil









In GNU Emacs 25.0.95.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d 
scroll bars)
  of 2016-06-18 built on shodan
Repository revision: 2ad3d0182df68f00cca57584807f721c3c5c96f1
Windowing system distributor 'The X.Org Foundation', version 11.0.11701000
System Description:	Ubuntu 15.04

Configured using:
  'configure --prefix=/home/phil/emacs/trunk/usr/local
  --with-x-toolkit=lucid --without-sound'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
   value of $LANG: en_NZ.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
GNU Emacs 25.0.95.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll 
bars) of 2016-06-18
Mark activated

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 88249 8416)
  (symbols 48 20114 0)
  (miscs 40 46 146)
  (strings 32 15667 3956)
  (string-bytes 1 429343)
  (vectors 16 11737)
  (vector-slots 8 429691 5060)
  (floats 8 166 283)
  (intervals 56 223 0)
  (buffers 976 11)
  (heap 1024 29329 1004))

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Sun, 19 Jun 2016 22:04:38 +1200
> 
> This is a spin-off from bug#20611: 24.4; mutt slow in ansi-term.
> 
> I'm logging a new bug report by request, although it seems to me that
> this is still the same bug (and 20611 is still open), so I suggest that
> they can both be closed together once a fix is committed?
> 
> I've experimented further with this issue since my comments in 20611,
> and my current observations follow...

Thanks.

My problem is that I don't see these slow redraws.  I tried in a TTY
session on a GNU/Linux system, using "ls ~" as the command inside
term-mode (my home directory on that system produces a 1500-line list
of files).  I don't see slow redraws, and don't see any perceptible
speed-up when I set bidi-paragraph-direction to left-to-right.

Can you reproduce the problem using 'ls'?

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

On 2016-06-20 04:26, Eli Zaretskii wrote:
> My problem is that I don't see these slow redraws.  I tried in a TTY
> session on a GNU/Linux system, using "ls ~" as the command inside
> term-mode (my home directory on that system produces a 1500-line list
> of files).  I don't see slow redraws, and don't see any perceptible
> speed-up when I set bidi-paragraph-direction to left-to-right.
> 
> Can you reproduce the problem using 'ls'?

No, the majority of commands do respond quickly.

I suspect you'll need to do something which repaints the entire 
terminal.

The examples I know of are starting the "mutt" email client (as per the
original report) or, for Debian, running "dpkg-reconfigure" for some
package with a configuration menu. I'm sure there will be plenty of
others, but I'm afraid I can't suggest a more commonly-available example
at the moment. If you have anything which provides a full-screen 
terminal
UI, however, give that a try.

Both previous examples are drawing a background colour (before 
eventually
drawing some text over the top, which happens quickly). Perhaps there 
are
a ton of escape sequences being processed for the colours? (I would have
thought the colouring was just a single ON and OFF wrapping the entire
redraw, but I don't know how these things actually work).

I've just tried installing and running "mc", which was another example I
thought I'd seen mentioned somewhere. That's GNU Midnight Commander.  
This
one is curious in that its initial redraw (with a background colour) is
pretty fast regardless; however when you quit (type "exit" at the 
prompt),
there's a slow redraw to get back to the shell (but also a bit faster 
than
my other examples). I guess it simply depends on exactly what each
application is doing. GNU Midnight Commander is a file manager. I think
"mc" followed by "exit" should be a pretty easy/safe test for you to 
try?

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

The "lynx" web browser is another example which may be
convenient to try.

Starting lynx causes a slow redraw.

Exit with 'q' (and 'y' to confirm).

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> Date: Mon, 20 Jun 2016 11:41:17 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: 23801@debbugs.gnu.org
> 
> > Can you reproduce the problem using 'ls'?
> 
> No, the majority of commands do respond quickly.
> 
> I suspect you'll need to do something which repaints the entire 
> terminal.

I don't understand what that means.  The output from "ls" repaints the
entire terminal as well, as it produces 1500 lines, much more than is
shown in the window.

Perhaps you mean cursor motion commands, i.e. escape sequences that
move the cursor up and down the screen, not just down?  But if that is
the reason, I'd expect to see its signs in the profile, which was not
the case, I think.  If you load term.el (not the .elc file!), and then
run your experiments under the profiler (profiler-start), what does
profiler-report produce?

> Both previous examples are drawing a background colour (before 
> eventually
> drawing some text over the top, which happens quickly). Perhaps there 
> are
> a ton of escape sequences being processed for the colours?

Probably, but I don't see how bidirectional display could slow down
this processing, because AFAIU these escape sequences are converted to
faces that are put on the displayed text, something that doesn't
involve the bidirectional reordering for display at all.

> I guess it simply depends on exactly what each application is
> doing. GNU Midnight Commander is a file manager. I think "mc"
> followed by "exit" should be a pretty easy/safe test for you to try?

My problem is precisely that I cannot figure out what could the
application be doing that would be so profoundly affected by
bidirectional display.

Does making the screen buffer (term-buffer-maximum-size) smaller help
in any way?

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

Progress...

I believe the terminal screen refreshes which exhibit the issue
are performed by inserting spaces (with a background colour).

It turns out that we can demonstrate this issue without any other
programs, simply by inserting spaces (or tabs).

Most critically, inserting non-whitespace characters does NOT
cause performance issues! (At least not the ones I've tried.)

(This presumably explains why midnight commander's initial draw
was so comparatively speedy, as it was drawing the final content
from the outset, rather than clearing the screen first and then
writing the content.)


Recipe:

emacs -Q (and maximise the terminal or GUI frame)
M-x term (and run a shell)
printf "%10000s" x

(to insert an 'x' preceded by 9,999 spaces.)

That's not as obvious without a background colour, but we can
provide that easily enough:

printf "\033[41m%10000s\033[47m" x

(Drawing in red, and reverting to a white background. Use 40m
in the final sequence if you need to revert to black instead.)


Bash supports numeric prefix arguments for repetition just like
Emacs, so you can also insert lots of spaces like so:

M-10000 SPC

You can insert 1,000 TABs with:
M-1000 C-v C-i


With this approach we are entering (but not yet submitting) a
command, and I further note that using C-u at this point to erase
our input is also really slow to complete -- but again, only when
it is whitespace being 'erased'.

e.g. M-10000 x C-u is perfectly speedy.

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 21 Jun 2016 02:35:46 +1200
> 
> I believe the terminal screen refreshes which exhibit the issue
> are performed by inserting spaces (with a background colour).

Is background color really necessary?

> It turns out that we can demonstrate this issue without any other
> programs, simply by inserting spaces (or tabs).
> 
> Most critically, inserting non-whitespace characters does NOT
> cause performance issues! (At least not the ones I've tried.)

Thanks, this finally begins to make sense.  I will look into this when
I have time.

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

On 21/06/16 02:28, Eli Zaretskii wrote:
>> I suspect you'll need to do something which repaints the entire
>> terminal.
>
> I don't understand what that means.  The output from "ls" repaints the
> entire terminal as well, as it produces 1500 lines, much more than is
> shown in the window.

Yes, in hindsight the slow redraws with a background colour were so
blatantly drawing every single character in the terminal that I was
imagining that other kinds of command output (with less colourful
results) did not need to do that; and that the problem was therefore
connected to the slow redraws processing many more characters than
was necessary for other kinds of screen update. Now that I've (AFAICS)
isolated the issue to whitespace, I can see that my earlier presumption
doesn't actually make much sense. Apologies for the confusion.

Let me know if you'd still like me to provide profiler or elp results?
I'm well and truly done for the night, but I can follow up tomorrow.
(I'm hoping that with the prior email you're now able to replicate
the issue, though?)

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 21 Jun 2016 02:35:46 +1200
> 
> emacs -Q (and maximise the terminal or GUI frame)
> M-x term (and run a shell)
> printf "%10000s" x
> 
> (to insert an 'x' preceded by 9,999 spaces.)
> 
> That's not as obvious without a background colour, but we can
> provide that easily enough:
> 
> printf "\033[41m%10000s\033[47m" x
> 
> (Drawing in red, and reverting to a white background. Use 40m
> in the final sequence if you need to revert to black instead.)
> 
> 
> Bash supports numeric prefix arguments for repetition just like
> Emacs, so you can also insert lots of spaces like so:
> 
> M-10000 SPC
> 
> You can insert 1,000 TABs with:
> M-1000 C-v C-i
> 
> 
> With this approach we are entering (but not yet submitting) a
> command, and I further note that using C-u at this point to erase
> our input is also really slow to complete -- but again, only when
> it is whitespace being 'erased'.
> 
> e.g. M-10000 x C-u is perfectly speedy.

Can you try this in an Emacs built from the latest master branch of
the Emacs Git repository?  It could be that the current development
sources already provide improvement in these cases.

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

On 22/06/16 06:47, Eli Zaretskii wrote:
> Can you try this in an Emacs built from the latest master branch of
> the Emacs Git repository?  It could be that the current development
> sources already provide improvement in these cases.

Indeed, the master branch emacs -Q seems as quick as 24 and 25 do
with their bidi support disabled.

Excellent!

Presumably these fixes aren't going to make it into 25.1. Should
I anticipate that 25.2 will include them?

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> Cc: 23801@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Thu, 23 Jun 2016 01:13:08 +1200
> 
> On 22/06/16 06:47, Eli Zaretskii wrote:
> > Can you try this in an Emacs built from the latest master branch of
> > the Emacs Git repository?  It could be that the current development
> > sources already provide improvement in these cases.
> 
> Indeed, the master branch emacs -Q seems as quick as 24 and 25 do
> with their bidi support disabled.

Thanks for testing.  So I think this bug can be closed.

> Presumably these fixes aren't going to make it into 25.1. Should
> I anticipate that 25.2 will include them?

We are still debating whether 25.2 will be produced from the master
branch or from emacs-25.

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Phil Sainty]

On 23/06/16 03:27, Eli Zaretskii wrote:
> Thanks for testing.  So I think this bug can be closed.

Yes, agreed.


Bug #20611 can potentially be closed as well, with the caveat
that there was a workaround commit made for that one, and it
is presumably important to ensure that the workaround will not
get merged to master later on, being unnecessary in that branch.

(Although I'd still be inclined to shift that workaround to act
in term-mode-hook rather than in ansi-term. I'll follow up in
that bug, though.)


 > We are still debating whether 25.2 will be produced from the
 > master branch or from emacs-25.

Ok. Either way, if we have the workaround in effect in emacs-25,
and the real fix in master, this problem is dealt with for the
upcoming releases.


thanks,
-Phil

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> Cc: 23801@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Sun, 26 Jun 2016 12:35:40 +1200
> 
> On 23/06/16 03:27, Eli Zaretskii wrote:
> > Thanks for testing.  So I think this bug can be closed.
> 
> Yes, agreed.
> 
> 
> Bug #20611 can potentially be closed as well, with the caveat
> that there was a workaround commit made for that one, and it
> is presumably important to ensure that the workaround will not
> get merged to master later on, being unnecessary in that branch.

Done.

> (Although I'd still be inclined to shift that workaround to act
> in term-mode-hook rather than in ansi-term. I'll follow up in
> that bug, though.)

Also done.

bug#23801: 25.0.95; term.el redraws extremely slow with bidi support enabled, and large buffers

[Eli Zaretskii]

> Cc: 23801@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Sun, 26 Jun 2016 12:35:40 +1200
> 
> On 23/06/16 03:27, Eli Zaretskii wrote:
> > Thanks for testing.  So I think this bug can be closed.
> 
> Yes, agreed.

Done.