bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

[Frédéric Giquel via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 5618 bytes --]

Hi,

While trying to understand the reason for slow completion with lsp-mode
in some situations, I noticed that the language server response is
split into 64k chunks despite setting `read-process-output-max' to 1M
(as recommended by lsp-mode maintainers).

The problem is not lsp-mode specific and can be reproduced without
additionnal packages by using the two scripts attached to this email:
- "write-server.py" is the server: it waits for a number
(requested_size) on its input and write a string of "requested_size"
bytes on its output. The file must be executable and located in the
PATH.
- "real-pipe-size.el" launch "write-server.py" and communicate with the
process using different values for `read-process-output-max'. It prints
the maximum chunk size used for each value.

On my system, I got the following output executing 'emacs -Q --script
~/Projets/emacs-pipe/real-pipe-size.el':
Real pipe size (with read-process-output-max = 4096): 4096
Real pipe size (with read-process-output-max = 32768): 32768
Real pipe size (with read-process-output-max = 65536): 65536
Real pipe size (with read-process-output-max = 65537): 65536
Real pipe size (with read-process-output-max = 204800): 65536
Real pipe size (with read-process-output-max = 1048576): 65536
Real pipe size (with read-process-output-max = 2097152): 65536

This 64k limit is GNU/Linux specific and is explained in "Pipe
capacity" section of pipe(7) manpage and can be changed calling
`fcntl'. So I added a call to `fcntl' for `F_SETPIPE_SZ' in
`create_process' function just after other uses of `fcntl' (I'm not
really sure if it's the right place). The patch is attached to this
email.

With the patch, the limit is now the value of `/proc/sys/fs/pipe-max-
size' (which default to 1M and can be change by users with the
CAP_SYS_RESOURCE capability). Output of 'emacs -Q --script
~/Projets/emacs-pipe/real-pipe-size.el' is now:
Real pipe size (with read-process-output-max = 4096): 4096
Real pipe size (with read-process-output-max = 32768): 32768
Real pipe size (with read-process-output-max = 65536): 65536
Real pipe size (with read-process-output-max = 65537): 65537
Real pipe size (with read-process-output-max = 204800): 204800
Real pipe size (with read-process-output-max = 1048576): 1048576
Real pipe size (with read-process-output-max = 2097152): 65536

And my initial problem of slow completion with lsp-mode is fixed.

P.S. I signed the FSF copyright assignment agreement for Emacs last
year.

In GNU Emacs 28.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33,
cairo version 1.16.0)
 of 2022-05-30 built on guiriden
Repository revision: 5a223c7f2ef4c31abbd46367b6ea83cd19d30aa7
Repository branch: heads/emacs-28.1
Windowing system distributor 'The X.Org Foundation', version
11.0.12201000
System Description: Debian GNU/Linux bookworm/sid

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: fr_FR.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-
loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-
utils
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-
dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj
charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process
emacs)

Memory information:
((conses 16 50603 6602)
 (symbols 48 6632 1)
 (strings 32 18327 1513)
 (string-bytes 1 607344)
 (vectors 16 13715)
 (vector-slots 8 184638 11282)
 (floats 8 21 47)
 (intervals 56 195 0)
 (buffers 992 10))


[-- Attachment #2: write-server.py --]
[-- Type: text/x-python3, Size: 175 bytes --]

#! /usr/bin/env python3

import sys

while True:
    requested_size = int(sys.stdin.readline())
    write_size = sys.stdout.write("A" * requested_size)
    sys.stdout.flush()

[-- Attachment #3: real-pipe-size.el --]
[-- Type: text/x-emacs-lisp, Size: 1388 bytes --]

(defvar show-details nil)

;; Values to use for `read-process-output-max'
(defvar read-process-output-max-list
  `(
    4096               ;; default value
    ,(* 32 1024)       ;; 32k
    ,(* 64 1024)       ;; 64k
    ,(+ 1 (* 64 1024)) ;; 64k + 1
    ,(* 200 1024)      ;; 200k
    ,(* 1024 1024)     ;; 1M
    ,(* 2 1024 1024)   ;; 2M
    ))

(defun filter-set-max-chunk-size (proc string)
  (when show-details
    (message "... chunk size: %s" (length string)))
  (setq max-chunk-size (max (length string) max-chunk-size)))

(defun calculate-real-pipe-size (requested-pipe-size)
  (let* ((read-process-output-max requested-pipe-size)
         (max-chunk-size 0)
         (proc (make-process
                :name "calculate-pipe-size"
                :buffer nil
                :command '("write-server.py")
                :connection-type 'pipe
                :filter 'filter-set-max-chunk-size)))
    (when show-details
      (message "== Calculating real pipe size with read-process-output-max = %s ==" read-process-output-max))
    (process-send-string "calculate-pipe-size" (concat (number-to-string requested-pipe-size) "\n"))
    (accept-process-output proc)
    (delete-process proc)
    max-chunk-size))

(dolist (pipe-size read-process-output-max-list)
  (message "Real pipe size (with read-process-output-max = %s): %s" pipe-size (calculate-real-pipe-size pipe-size)))

[-- Attachment #4: pipe-size.patch --]
[-- Type: text/x-patch, Size: 1125 bytes --]

diff --git a/src/process.c b/src/process.c
index 8b587aaa4e..703352e59e 100644
--- a/src/process.c
+++ b/src/process.c
@@ -2155,6 +2155,10 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   fcntl (inchannel, F_SETFL, O_NONBLOCK);
   fcntl (outchannel, F_SETFL, O_NONBLOCK);
 
+#if defined(GNU_LINUX) && defined(F_SETPIPE_SZ)
+  fcntl (inchannel, F_SETPIPE_SZ, read_process_output_max);
+#endif
+
   /* Record this as an active process, with its channels.  */
   eassert (0 <= inchannel && inchannel < FD_SETSIZE);
   chan_process[inchannel] = process;
@@ -8571,7 +8575,10 @@ syms_of_process (void)
   DEFVAR_INT ("read-process-output-max", read_process_output_max,
 	      doc: /* Maximum number of bytes to read from subprocess in a single chunk.
 Enlarge the value only if the subprocess generates very large (megabytes)
-amounts of data in one go.  */);
+amounts of data in one go.
+
+On GNU/Linux system, the value should not exceed
+`/proc/sys/fs/pipe-max-size'. See pipe(7) manpage for details. */);
   read_process_output_max = 4096;
 
   DEFSYM (Qinternal_default_interrupt_process,

bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

[Eli Zaretskii]

> Date: Tue, 31 May 2022 13:09:38 +0200
> From:  Frédéric Giquel via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> +#if defined(GNU_LINUX) && defined(F_SETPIPE_SZ)
> +  fcntl (inchannel, F_SETPIPE_SZ, read_process_output_max);
> +#endif

Shouldn't this be done only for pipe connections to the subprocess?

Thanks.

bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

[Frédéric Giquel via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

Le mardi 31 mai 2022 à 15:00 +0300, Eli Zaretskii a écrit :
> > Date: Tue, 31 May 2022 13:09:38 +0200
> > From:  Frédéric Giquel via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> > 
> > +#if defined(GNU_LINUX) && defined(F_SETPIPE_SZ)
> > +  fcntl (inchannel, F_SETPIPE_SZ, read_process_output_max);
> > +#endif
> 
> Shouldn't this be done only for pipe connections to the subprocess?

You're right. Setting pipe size is useless for pty connection.
Here is the new patch.

[-- Attachment #2: pipe-size-v2.patch --]
[-- Type: text/x-patch, Size: 1081 bytes --]

diff --git a/src/process.c b/src/process.c
index 8b587aaa4e..eae7b26eb3 100644
--- a/src/process.c
+++ b/src/process.c
@@ -2132,6 +2132,10 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
       inchannel = p->open_fd[READ_FROM_SUBPROCESS];
       forkout = p->open_fd[SUBPROCESS_STDOUT];
 
+#if defined(GNU_LINUX) && defined(F_SETPIPE_SZ)
+      fcntl (inchannel, F_SETPIPE_SZ, read_process_output_max);
+#endif
+
       if (!NILP (p->stderrproc))
 	{
 	  struct Lisp_Process *pp = XPROCESS (p->stderrproc);
@@ -8571,7 +8575,10 @@ syms_of_process (void)
   DEFVAR_INT ("read-process-output-max", read_process_output_max,
 	      doc: /* Maximum number of bytes to read from subprocess in a single chunk.
 Enlarge the value only if the subprocess generates very large (megabytes)
-amounts of data in one go.  */);
+amounts of data in one go.
+
+On GNU/Linux system, the value should not exceed
+`/proc/sys/fs/pipe-max-size'. See pipe(7) manpage for details. */);
   read_process_output_max = 4096;
 
   DEFSYM (Qinternal_default_interrupt_process,

bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

[Lars Ingebrigtsen]

Frédéric Giquel <frederic.giquel@laposte.net> writes:

> You're right. Setting pipe size is useless for pty connection.
> Here is the new patch.

Thanks; pushed to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no