Re: Excessive redisplay from lots of process output

[Spencer Baugh <sbaugh@janestreet.com>]

On Fri, Feb 17, 2023 at 11:06 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Cc: azeng@janestreet.com
> > Date: Fri, 17 Feb 2023 10:16:11 -0500
> >
> > Emacs calls redisplay every time it reads process output.  This causes
> > Emacs to become painfully slow when these two conditions are met:
> >
> > - When redisplay is expensive (e.g. when using ssh X forwarding)
> >
> > - And when there is frequent process output, say hundreds or thousands
> >   of times a second (e.g. ERC receiving network traffic, shells
> >   running commands which log heavily, other things which run in the
> >   background)
>
> This cannot be the whole story, because usually redisplay exits almost
> immediately after being called, having determined that nothing needs
> to be changed on display.
>
> So the important part of your situation that you didn't disclose is:
> why does Emacs in your case perform the expensive redisplay each time
> redisplay is entered?  Can you figure that out by stepping through
> redisplay_internal?  (Let me know if you need guidance.)

Oh, this was the key information for me to realize what was *really*
going on.  In Emacs 27, in-progress key sequences get replayed each
time there is process output to read, see bug#32922. This causes
redisplay to have actual work to do each time it reads process output.
But this only happens mid key-sequence, so it's not really a general
issue with Emacs performance, but it *does* cause the X server to get
swamped and therefore slows down X in general, including the window
manager and other X clients.

But fortunately that key-sequence replaying behavior was deleted in
Emacs 28, just as a general cleanup, see bug#5803.  Which helpfully
means that redisplay is once again always cheap, so the bad X
performance of having lots of process output disappears.

Sorry, a mix of upgrading to Emacs 28 and also applying this patch
made it not clear what was helping performance.  But now it is clear.

> Not sure here is the right place for this kind of discussion, btw.  It
> might be better to submit a bug report with all the relevant details,
> and discuss this on the bug tracker.
>
> > I realize that reading process output can trigger filters which can
> > change window and frame configurations, which in turn means redisplay
> > needs to happen.  But isn't there some way we could improve this
> > situation?  Right now these redisplays are causing serious
> > user-visible performance degradation.
>
> There are many variables and flags Emacs uses to decide when something
> changed that needs the display to be updated.  Those should generally
> prevent Emacs from running the expensive parts of redisplay if they
> are unnecessary.  The question is why this doesn't work in your case.
>
> IOW, it is better to consider this as some specific failure in some
> particular situation, and find why it happens, instead of considering
> this as some general failure of the Emacs design.  Design-wise, things
> are fine; it's something in your specific case that could fail the
> existing mechanisms, and the question is: what is that and why it
> fails?

Oh yes, that makes sense, now I see, thanks for the explanation.