bug#12440: 24.2; process-send-string only sends 4K for long strings

bug#12440: 24.2; process-send-string only sends 4K for long strings

[Hendrik Tews]

Hi,

in Emacs 24.2, the following code

  (let* ((process-connection-type t)
         (process (start-process "cat" (current-buffer) "cat")))
    (process-send-string process (format "%s\n" (make-string 6000 ?a))))

inserts only 4095 characters in the current buffer. Setting
process-connection-type to nil will insert all characters. In Emacs 23
the code works always as expected.

This is probably related to Emacs bug #7078. I cannot reproduce the
problem described there (with 257 characters), probably because the
buffer size was increased to 4K.

Bye,

Hendrik Tews



In GNU Emacs 24.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-09-13 on blau
Windowing system distributor `The XFree86 Project, Inc', version 11.0.40300000
Configured using:
 `configure '--prefix' '/usr/local/stow/emacs-24.2' '--with-xpm=no'
 '--with-jpeg=no' '--with-gif=no' '--with-tiff=no''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<return> C-x s ! ! <backspace> C-x s ! <f9> C-x b T 
O <tab> <return> C-x C-v <return> <down> <down> <down> 
<down> <down> <down> <down> C-SPC <down> <down> M-w 
C-x b <return> <backspace> C-y <left> C-j M-x e m a 
c s - v e r <tab> <return> <up> <up> <down> <down> 
<down> C-k C-k <up> <up> <up> <M-right> <M-right> <M-right> 
<M-right> <M-right> <M-right> <M-right> <M-right> <M-right> 
<M-right> <backspace> <backspace> <backspace> 6 0 0 
0 C-e C-j <down> <down> <down> C-SPC C-e M-= M-< A-f 
<down> <down> <down> <C-down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <up> <M-right> * <right> <right> C-o ( p r o 
c e s s - c o n n M-/ <left> <left> C-h v <return> 
C-e SPC n i l ) <down> <tab> C-e <backspace> ) <down> 
<down> <down> <down> <down> <down> C-a C-k <up> <up> 
<up> <up> <up> C-e <backspace> ) C-j C-x 1 <down> <down> 
<down> <down> <down> <down> C-SPC C-e M-= C-a C-k <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> C-e <left> 
<M-backspace> t <down> <down> C-e C-j <down> <down> 
<down> <down> <down> <down> <down> <down> <down> C-SPC 
C-e M-= C-a C-SPC <right> M-= C-a C-SPC C-e M-= C-a 
M-f C-a M-1 C-f C-a M-4 M-0 M-9 M-6 C-f C-a <up> C-a 
M-4 M-0 M-9 M-5 C-f C-a <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <down> C-k 
C-k C-k C-k C-k C-x b T O <tab> <return> <up> <up> 
C-y <return> C-k C-k C-k C-k C-o C-x C-s <up> <up> 
<up> <up> <down> M-x e m a c s - v e r <tab> <return> 
M-x r e p o r t - e m <tab> <return>

Recent messages:
Mark set
Region has 1 line, 1 word, and 4095 characters.
Mark activated
Region has 1 line, 1 word, and 1 character.
Mark activated
Region has 1 line, 1 word, and 4095 characters.
Mark set
Saving file /home/tews/src/pg/TODO...
Wrote /home/tews/src/pg/TODO
GNU Emacs 24.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.10) of 2012-09-13 on blau

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils jka-compr find-func etags debug dabbrev parse-time
vc-cvs tabulated-list pp help-mode view which-func imenu coq edmacro
kmacro coq-smie-lexer smie coq-indent coq-abbrev coq-local-vars
local-vars-list coq-syntax regexp-opt coq-db holes proof proof-shell
pg-user completion pg-goals pg-response proof-toolbar pg-assoc
proof-tree proof-script proof-menu cus-edit cus-start cus-load wid-edit
span proof-auxmodes proof-utils scomint proof-syntax advice help-fns
advice-preload bufhist easy-mmode ring proof-splash derived pg-custom
proof-config proof-faces proof-useropts pg-pamacs proof-compat easymenu
proof-site proof-autoloads cl pg-vars misearch multi-isearch time-date
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
make-network-process dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

bug#12440: 24.2; process-send-string only sends 4K for long strings

[Lars Ingebrigtsen]

Hendrik Tews <tews@os.inf.tu-dresden.de> writes:

> in Emacs 24.2, the following code
>
>   (let* ((process-connection-type t)
>          (process (start-process "cat" (current-buffer) "cat")))
>     (process-send-string process (format "%s\n" (make-string 6000 ?a))))
>
> inserts only 4095 characters in the current buffer. Setting
> process-connection-type to nil will insert all characters.

I can confirm that this bug is still present in Emacs 27.

It's odd that this doesn't cause more problems than it does in practice,
but I guess it's rare that we talk to processes this way.

The bug was apparently introduced by the patch below, but I have to
admit when reading it, I don't quite understand what's going on...

commit 2b0a91e78f83fb446cc38efb99399e83ad2cded3
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Apr 12 22:07:48 2010 -0400

    Try to solve the problem of spurious EOF chars in long lines of text
    sent to interactive subprocesses.
    * sysdep.c (child_setup_tty): Do not enable ICANON any more.
    (system_process_attributes): Remove unused var `ttotal'.
    * process.c (send_process): Don't bother breaking long line with EOF
    chars when talking to ttys any more.
    (wait_reading_process_output): Output a warning when called in such
    a way that it could block without being interruptible.

diff --git a/src/ChangeLog b/src/ChangeLog
index ad88dc8311..f9567b1308 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,5 +1,14 @@
 2010-04-13  Stefan Monnier  <monnier@iro.umontreal.ca>
 
+	Try to solve the problem of spurious EOF chars in long lines of text
+	sent to interactive subprocesses.
+	* sysdep.c (child_setup_tty): Do not enable ICANON any more.
+	(system_process_attributes): Remove unused var `ttotal'.
+	* process.c (send_process): Don't bother breaking long line with EOF
+	chars when talking to ttys any more.
+	(wait_reading_process_output): Output a warning when called in such
+	a way that it could block without being interruptible.
+
 	Try to detect file modification within the same second.
 	* buffer.h (struct buffer): New field modtime_size.
 	* buffer.c (reset_buffer): Initialize it.
diff --git a/src/process.c b/src/process.c
index 34aa2c4fcf..7e8f4cc57b 100644
--- a/src/process.c
+++ b/src/process.c
@@ -4643,6 +4643,10 @@ wait_reading_process_output (time_limit, microsecs, read_kbd, do_display,
   FD_ZERO (&Connecting);
 #endif
 
+  if (time_limit == 0 && wait_proc && !NILP (Vinhibit_quit)
+      && !(CONSP (wait_proc->status) && EQ (XCAR (wait_proc->status), Qexit)))
+    message ("Blocking call to accept-process-output with quit inhibited!!");
+
   /* If wait_proc is a process to watch, set wait_channel accordingly.  */
   if (wait_proc != NULL)
     wait_channel = wait_proc->infd;
@@ -5768,34 +5772,6 @@ send_process (proc, buf, len, object)
 	{
 	  int this = len;
 
-	  /* Decide how much data we can send in one batch.
-	     Long lines need to be split into multiple batches.  */
-	  if (p->pty_flag)
-	    {
-	      /* Starting this at zero is always correct when not the first
-		 iteration because the previous iteration ended by sending C-d.
-		 It may not be correct for the first iteration
-		 if a partial line was sent in a separate send_process call.
-		 If that proves worth handling, we need to save linepos
-		 in the process object.  */
-	      int linepos = 0;
-	      unsigned char *ptr = (unsigned char *) buf;
-	      unsigned char *end = (unsigned char *) buf + len;
-
-	      /* Scan through this text for a line that is too long.  */
-	      while (ptr != end && linepos < pty_max_bytes)
-		{
-		  if (*ptr == '\n')
-		    linepos = 0;
-		  else
-		    linepos++;
-		  ptr++;
-		}
-	      /* If we found one, break the line there
-		 and put in a C-d to force the buffer through.  */
-	      this = ptr - buf;
-	    }
-
 	  /* Send this batch, using one or more write calls.  */
 	  while (this > 0)
 	    {
@@ -5899,11 +5875,6 @@ send_process (proc, buf, len, object)
 	      len -= rv;
 	      this -= rv;
 	    }
-
-	  /* If we sent just part of the string, put in an EOF (C-d)
-	     to force it through, before we send the rest.  */
-	  if (len > 0)
-	    Fprocess_send_eof (proc);
 	}
     }
   else
diff --git a/src/sysdep.c b/src/sysdep.c
index 37e7dfbaf9..506af23ef3 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -529,8 +529,6 @@ child_setup_tty (out)
 #endif
   s.main.c_oflag &= ~TAB3;	/* Disable tab expansion */
   s.main.c_cflag = (s.main.c_cflag & ~CSIZE) | CS8; /* Don't strip 8th bit */
-  s.main.c_lflag |= ICANON;	/* Enable erase/kill and eof processing */
-  s.main.c_cc[VEOF] = 04;	/* insure that EOF is Control-D */
   s.main.c_cc[VERASE] = CDISABLE;	/* disable erase processing */
   s.main.c_cc[VKILL] = CDISABLE;	/* disable kill processing */
 
@@ -560,7 +558,6 @@ child_setup_tty (out)
   /* rms: Formerly it set s.main.c_cc[VINTR] to 0377 here
      unconditionally.  Then a SIGNALS_VIA_CHARACTERS conditional
      would force it to 0377.  That looks like duplicated code.  */
-  s.main.c_cc[VEOL] = CDISABLE;
   s.main.c_cflag = (s.main.c_cflag & ~CBAUD) | B9600; /* baud rate sanity */
 #endif /* AIX */
 
@@ -573,6 +570,18 @@ child_setup_tty (out)
   s.main.sg_kill = 0377;
   s.lmode = LLITOUT | s.lmode;        /* Don't strip 8th bit */
 
+  /* We used to enable ICANON (and set VEOF to 04), but this leads to
+     problems where process.c wants to send EOFs every once in a while
+     to force the output, which leads to weird effects when the
+     subprocess has disabled ICANON and ends up seeing those spurious
+     extra EOFs.  So we don't send EOFs any more in
+     process.c:send_process, and instead we disable ICANON by default,
+     so if a subsprocess sets up ICANON, it's his problem (or the Elisp
+     package that talks to it) to deal with lines that are too long.  */
+  s.main.c_lflag &= ~ICANON;	/* Disable line editing and eof processing */
+  s.main.c_cc[VMIN] = 1;
+  s.main.c_cc[VTIME] = 0;
+
 #endif /* not HAVE_TERMIO */
 
   EMACS_SET_TTY (out, &s, 0);
@@ -3344,7 +3353,7 @@ system_process_attributes (Lisp_Object pid)
   unsigned long minflt, majflt, cminflt, cmajflt, vsize;
   time_t sec;
   unsigned usec;
-  EMACS_TIME tnow, tstart, tboot, telapsed,ttotal;
+  EMACS_TIME tnow, tstart, tboot, telapsed;
   double pcpu, pmem;
   Lisp_Object attrs = Qnil;
   Lisp_Object cmd_str, decoded_cmd, tem;


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#12440: 24.2; process-send-string only sends 4K for long strings

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 07 Oct 2019 16:53:24 +0200
> Cc: 12440@debbugs.gnu.org
> 
> Hendrik Tews <tews@os.inf.tu-dresden.de> writes:
> 
> > in Emacs 24.2, the following code
> >
> >   (let* ((process-connection-type t)
> >          (process (start-process "cat" (current-buffer) "cat")))
> >     (process-send-string process (format "%s\n" (make-string 6000 ?a))))
> >
> > inserts only 4095 characters in the current buffer. Setting
> > process-connection-type to nil will insert all characters.
> 
> I can confirm that this bug is still present in Emacs 27.
> 
> It's odd that this doesn't cause more problems than it does in practice,
> but I guess it's rare that we talk to processes this way.

IMO, when a Lisp program intends to communicate with a subprocess this
way, it should indeed set process-connect-type to nil.  It makes no
sense to me to use PTYs to communicate massive amounts of data in
batch-like manner.