bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Mark H Weaver]

On a GNU Guix system, after upgrading my kernel from Linux-libre 5.10.19
to 5.10.20, Emacs Shell mode now performs extremely poorly, specifically
when displaying the output from commands.  For example, using "cat" to
print a 14 kilobyte text file on a Thinkpad X200, Emacs Shell mode now
requires approximately 11 seconds to display the output.  That's ~1200
characters per second.

It appears to me that Emacs might now be performing a display update for
each character received from the pseudo-tty, instead of processing
larger blocks at a time, as it has previously done.

Other programs I've tried that use pseudo-ttys don't seem to have this
problem with the new kernel.  For example, the performance of GNU Screen
4.8.0 is not noticeably affected.  Ditto for GNOME Terminal 3.34.2,
based on VTE 0.60.3.

The problem is reproducible using "emacs -Q", and does not seem to
depend on the windowing system.  It happens both within X using an Emacs
built with --with-x-toolkit=no (the "emacs-no-x-toolkit" package in
Guix), and also within a Linux text console using an Emacs built without
X support at all (the "emacs-no-x" package in Guix).

I've skimmed the list of changes between Linux 5.10.19 and 5.10.20, and
found these two commits that might be relevant:

tty: implement read_iter
<https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=41c6f6b926d0e712d0321f8a8f6511fea748e814>

tty: convert tty_ldisc_ops 'read()' function to take a kernel pointer
<https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=279e54536ddbb4dbd337fca74926b68651160043>

These changes are also included in Linux-libre 5.11.3 (the newest
release), but I haven't yet tried that version.

I'm currently building a variant of Linux-libre 5.10.20 with these two
commits reverted, to see if that makes the problem go away, and I will
report back when I have those results.

See below for some details generated by M-x report-emacs-bug.  I could
provide further details, including the full build log of the Emacs
binary I'm using, or instructions for how to build precisely the same
Emacs binary using GNU Guix, but I suspect that this bug is insensitive
to such details.

      Mark


In GNU Emacs 27.1 (build 1, x86_64-unknown-linux-gnu)
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure
 CONFIG_SHELL=/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash
 SHELL=/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash
 --prefix=/gnu/store/ack6blyzqgwx3cfsxk0drgj61xghri0x-emacs-no-x-27.1
 --enable-fast-install --build=x86_64-unknown-linux-gnu --with-modules
 --disable-build-details'

Configured features:
SOUND NOTIFY INOTIFY ACL GNUTLS LIBXML2 ZLIB MODULES THREADS JSON
PDUMPER GMP

Important settings:
  value of $EMACSLOADPATH: /root/.guix-profile/share/emacs/site-lisp:/root/.guix-profile/share/emacs/27.1/lisp
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow regexp-opt sort mail-extr emacsbug message rmc puny dired
dired-loaddefs format-spec rfc822 mml easymenu mml-sec password-cache
epa derived epg epg-config gnus-util rmail tool-bar rmail-loaddefs
text-property-search time-date subr-x seq byte-opt gv bytecomp
byte-compile cconv mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils t-mouse term/linux
disp-table tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads inotify multi-tty make-network-process emacs)

Memory information:
((conses 16 42993 4300)
 (symbols 48 5872 1)
 (strings 32 16278 1702)
 (string-bytes 1 528232)
 (vectors 16 6851)
 (vector-slots 8 82979 4968)
 (floats 8 17 248)
 (intervals 56 175 0)
 (buffers 1000 11))

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> I've skimmed the list of changes between Linux 5.10.19 and 5.10.20, and
> found these two commits that might be relevant:
>
> tty: implement read_iter
> <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=41c6f6b926d0e712d0321f8a8f6511fea748e814>
>
> tty: convert tty_ldisc_ops 'read()' function to take a kernel pointer
> <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=279e54536ddbb4dbd337fca74926b68651160043>
>
> These changes are also included in Linux-libre 5.11.3 (the newest
> release), but I haven't yet tried that version.
>
> I'm currently building a variant of Linux-libre 5.10.20 with these two
> commits reverted, to see if that makes the problem go away, and I will
> report back when I have those results.

Reverting the two commits cited above fixes the problem.

       Mark

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Date: Sat, 06 Mar 2021 21:39:54 -0500
> 
> Mark H Weaver <mhw@netris.org> writes:
> 
> > I've skimmed the list of changes between Linux 5.10.19 and 5.10.20, and
> > found these two commits that might be relevant:
> >
> > tty: implement read_iter
> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=41c6f6b926d0e712d0321f8a8f6511fea748e814>
> >
> > tty: convert tty_ldisc_ops 'read()' function to take a kernel pointer
> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=279e54536ddbb4dbd337fca74926b68651160043>
> >
> > These changes are also included in Linux-libre 5.11.3 (the newest
> > release), but I haven't yet tried that version.
> >
> > I'm currently building a variant of Linux-libre 5.10.20 with these two
> > commits reverted, to see if that makes the problem go away, and I will
> > report back when I have those results.
> 
> Reverting the two commits cited above fixes the problem.

So does this mean this isn't an Emacs issue, but the issue with that
kernel?  Or is there still something you'd like us to look into on the
Emacs side?

Thanks.

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Mark H Weaver]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mark H Weaver <mhw@netris.org>
>> Date: Sat, 06 Mar 2021 21:39:54 -0500
>> 
>> Mark H Weaver <mhw@netris.org> writes:
>> 
>> > I've skimmed the list of changes between Linux 5.10.19 and 5.10.20, and
>> > found these two commits that might be relevant:
>> >
>> > tty: implement read_iter
>> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=41c6f6b926d0e712d0321f8a8f6511fea748e814>
>> >
>> > tty: convert tty_ldisc_ops 'read()' function to take a kernel pointer
>> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=279e54536ddbb4dbd337fca74926b68651160043>
>> >
>> > These changes are also included in Linux-libre 5.11.3 (the newest
>> > release), but I haven't yet tried that version.
>> >
>> > I'm currently building a variant of Linux-libre 5.10.20 with these two
>> > commits reverted, to see if that makes the problem go away, and I will
>> > report back when I have those results.
>> 
>> Reverting the two commits cited above fixes the problem.
>
> So does this mean this isn't an Emacs issue, but the issue with that
> kernel?

That's not clear to me.  What I know is that these changes to Linux's
TTY subsystem, authored by Linus Torvalds himself and recently included
in upstream Linux 5.11.3 and 5.10.20, have lead to this regression in
Emacs.  This might simply be a kernel bug, or it could be that Emacs is
making an improper assumption about how the kernel behaves.

> Or is there still something you'd like us to look into on the
> Emacs side?

I would suggest investigating to find out what's going wrong here, and
then:

(1) if it can reasonably be considered a kernel regression, submit a bug
    report to the Linux developers, and

(2) if Emacs is found to behave suboptimally in its interactions with
    the kernel, or if the Linux developers do not soon deploy their own
    fix, modify Emacs as needed to avoid the problem.

What do you think?

    Thanks,
      Mark

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: 46978@debbugs.gnu.org
> Date: Sun, 07 Mar 2021 03:50:15 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Mark H Weaver <mhw@netris.org>
> >> Date: Sat, 06 Mar 2021 21:39:54 -0500
> >> 
> >> Mark H Weaver <mhw@netris.org> writes:
> >> 
> >> > I've skimmed the list of changes between Linux 5.10.19 and 5.10.20, and
> >> > found these two commits that might be relevant:
> >> >
> >> > tty: implement read_iter
> >> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=41c6f6b926d0e712d0321f8a8f6511fea748e814>
> >> >
> >> > tty: convert tty_ldisc_ops 'read()' function to take a kernel pointer
> >> > <https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.20&id=279e54536ddbb4dbd337fca74926b68651160043>
> >> >
> >> > These changes are also included in Linux-libre 5.11.3 (the newest
> >> > release), but I haven't yet tried that version.
> >> >
> >> > I'm currently building a variant of Linux-libre 5.10.20 with these two
> >> > commits reverted, to see if that makes the problem go away, and I will
> >> > report back when I have those results.
> >> 
> >> Reverting the two commits cited above fixes the problem.
> >
> > So does this mean this isn't an Emacs issue, but the issue with that
> > kernel?
> 
> That's not clear to me.  What I know is that these changes to Linux's
> TTY subsystem, authored by Linus Torvalds himself and recently included
> in upstream Linux 5.11.3 and 5.10.20, have lead to this regression in
> Emacs.  This might simply be a kernel bug, or it could be that Emacs is
> making an improper assumption about how the kernel behaves.

Emacs doesn't assume anything about the kernel, it simply reads from
the PTY which serves as stdout for the subprocess.  Your description
seems to indicate that the reads stall, because that's the only
explanation I can come up with to the fact that we display 'cat's
output one character at a time.

Do these changes affect the way characters are written to the PTY?
For example, do they affect the frequency with which stuff is written,
or the batching of the bytes (i.e. how many bytes are written in one
go)?  Or maybe they affect the read side of the PTY?

Does it help to set process-connection-type to nil?

Does it help to play with process-adaptive-read-buffering?

> I would suggest investigating to find out what's going wrong here, and
> then:

If someone here knows enough about this part of the Linux kernel to
understand its effect on Emacs communications with subprocess, I hope
they speak up soon.  I looked at the two commits you mentioned and
didn't see anything that immediately caught my eye, but then I have no
idea how TTYs work on Linux at that level.

Failing that, my suggestion is to inform the kernel developers of the
problem and see if they have any suggestions for where in Emacs to
look for the cause, if at all.

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Mark H Weaver]

Linux-libre 5.10.21 and 5.11.4 have since been released, each including
another batch of 5 commits by Linus to the TTY subsystem (with commit
logs acknowledging problems in the earlier commits), and I'm glad to
report that the performance regression no longer occurs in 5.10.21.
I'll assume that it's fixed in 5.11.4 as well, and close this bug now.

      Thanks!
        Mark

bug#46978: 27.1; Shell mode performs extremely poorly in Linux-libre 5.10.20

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: 46978-done@debbugs.gnu.org
> Date: Sun, 07 Mar 2021 23:58:24 -0500
> 
> Linux-libre 5.10.21 and 5.11.4 have since been released, each including
> another batch of 5 commits by Linus to the TTY subsystem (with commit
> logs acknowledging problems in the earlier commits), and I'm glad to
> report that the performance regression no longer occurs in 5.10.21.
> I'll assume that it's fixed in 5.11.4 as well, and close this bug now.

Great, thanks for telling us.