bug#24531: process-send-string seems to truncate lines over 4096 characters

bug#24531: process-send-string seems to truncate lines over 4096 characters

[talwrii talwrii]

[-- Attachment #1.1: Type: text/plain, Size: 1689 bytes --]

=== Repro ===

i. Download attached el file
ii. Read it to ensure I am not hacking your computer
iii. In the same directory run

emacs -Q --eval '(progn (load-file "line-repro.el") (message (format "%S"
(repro))))' -nw --batch

As the last line of output I get

(3005 4100 6006)

I wouild expect this 4100 to be 5005

The third value indicates it is the line that gets truncated rather than
the string

=== Version details ===

In GNU Emacs 24.5.1 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of
2016-04-08 on binet, modified by Debian Windowing system distributor `The
X.Org Foundation', version 11.0.11804000 System Description: Debian
GNU/Linux testing (stretch) Configured using: `configure --build
x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
--libexecdir=/usr/lib --localstatedir=/var/lib --infodir=/usr/share/info
--mandir=/usr/share/man --with-pop=yes
--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.5/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.5/site-lisp:/usr/share/emacs/site-lisp
--build x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
--libexecdir=/usr/lib --localstatedir=/var/lib --infodir=/usr/share/info
--mandir=/usr/share/man --with-pop=yes
--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.5/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.5/site-lisp:/usr/share/emacs/site-lisp
--with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g
-O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro' Important
settings: value of $LANG: en_GB.UTF-8 locale-coding-system: utf-8-unix

[-- Attachment #1.2: Type: text/html, Size: 1994 bytes --]

[-- Attachment #2: line-repro.el --]
[-- Type: text/x-emacs-lisp, Size: 1741 bytes --]


(defun s-repeat (num s)
  ;; extracted from s.el
  "Make a string of S repeated NUM times."
  (let (ss)
    (while (> num 0)
      (setq ss (cons s ss))
      (setq num (1- num)))
    (apply 'concat ss)))

(defun s-concat (&rest strings)
  ;; extracted from s.el
  "Join all the string arguments into one string."
  (apply 'concat strings))

(setq tee-program "
import fcntl
import os
import select
import sys

fd = sys.stdin.fileno()
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

filename, = sys.argv[1:]

with open(filename, 'w') as stream:
    while True:
        select.select([sys.stdin], [], [sys.stdin], 0.2)
        data = sys.stdin.read()
        if data == '':
            break
        else:
            stream.write(data)
            stream.flush()
")

(defun repro ()
  (list
     (length (write-to-file (s-repeat 3000 "a")))
     (length (write-to-file (s-repeat 5000 "a")))
     (length (write-to-file (s-concat (s-repeat 3000 "a") "\n" (s-repeat 3000 "a"))))
     ))

(defun write-to-file (input)
  (let (p)

    (if (not (file-exists-p "/tmp/bettertee.py"))
        (write-region tee-program nil "/tmp/bettertee.py"))

    (setq p (start-process "bocp" "*bocp*"
                           "python" "/tmp/bettertee.py" "/tmp/line-repro-bug"))
    (process-send-string p input)
    (process-send-string p "\nnext")
    (process-send-eof p)

    (message (format "Waiting for process to die"))
    (while (process-live-p p)
      (sit-for 0.1))
    (message "Process is dead")
    (get-string-from-file "/tmp/line-repro-bug")
    ))

(defun get-string-from-file (filePath)
  "Return filePath's file content."
  (with-temp-buffer
    (insert-file-contents filePath) (buffer-string)))

bug#24531: process-send-string seems to truncate lines over 4096 characters

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]


() talwrii talwrii <talwrii@gmail.com>
() Sat, 24 Sep 2016 18:38:18 -0500

   (3005 4100 6006)

   I wouild expect this 4100 to be 5005

See variable ‘process-connection-type’.  When i add:

 (setq process-connection-type nil)

to the beginning of line-repo.el, i see the expected output.

--
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (type via)
   (case type
     (technical (eq 'mailing-list via))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

bug#24531: process-send-string seems to truncate lines over 4096 characters

[Andreas Schwab]

On Sep 26 2016, Thien-Thi Nguyen <ttn@gnu.org> wrote:

> () talwrii talwrii <talwrii@gmail.com>
> () Sat, 24 Sep 2016 18:38:18 -0500
>
>    (3005 4100 6006)
>
>    I wouild expect this 4100 to be 5005
>
> See variable ‘process-connection-type’.  When i add:
>
>  (setq process-connection-type nil)
>
> to the beginning of line-repo.el, i see the expected output.

That could be the effect of the TTY ICANON handling.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#24531: process-send-string seems to truncate lines over 4096 characters

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 24531@debbugs.gnu.org, 6149@debbugs.gnu.org, jidanni@jidanni.org
>> Date: Thu, 20 Jul 2023 17:21:53 -0400
>> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> > I see that this bug is about 13 years old.  I think there's a pretty
>> > obvious solution: process-connection-type should default to nil.
>> 
>> In practice, that can't fly because it'll break existing code.
>
> Indeed.

I agree that we probably can't change the default.  However...

> A large portion (I think the majority, but I'm not sure) of
> Lisp programs that use bidirectional communications with async
> subprocesses actually _want_ the PTY interface, because they act as
> GUI front-ends to other programs.  Think "M-x term", "M-x gdb",
> inferior-python-mode, etc.  Even "M-x grep" and the likes need that
> because they rely on default color output, which only happens if Grep
> is connected to a terminal device.  Some of the Emacs features based
> on this don't work or don't work well on MS-Windows because Windows
> only supports pipes.  Do we really want such semi-broken behavior on
> GNU and Unix systems?
>
> The number of applications that (a) don't need console-like behavior
> and (b) need to send larger-than-4KB buffers to sub-processes is quite
> small.  Which is why this issue comes up only very rarely.  So making
> pipes the default will fix a very small fraction of applications, and
> break the vast majority -- clearly a wrong balance.

I see your point, but at the same time, the PTY interface on its own is
not sufficient to make these applications work, not at all.  Specialized
modes are necessary to make M-x term (to implement a terminal) and M-x
grep (to parse ANSI color codes) and other such programs work.  Running
things in a PTY without such specialized code doesn't give you anything,
AFAIK, because a PTY alone is far from enough to make the Emacs end
behave like a terminal.  So such programs need to be aware and careful
about such things anyway, and need additional infrastructure on top of
make-process.  So the default being "pty" gives such programs very
little: it doesn't save them any complexity.

Programs that just want to do some data processing with a subprocess, on
the other hand, work fine with just make-process alone, they need no
additional infrastructure, just process-send-string and reading directly
from the process buffer.  The default being "pipe" would take away a big
footgun for such programs, since it's easy to forget that and then have
a silently wrong program which will fail once you get large input.

>> Also I don't think either value of `process-connection-type` is a good
>> option.  IOW, I think that the connection type should be a mandatory
>> argument when creating an async process (except maybe for those
>> processes with no input/output).
>
> If we go that way, we should start by specifying :connection-type for
> all the uses of make-process and start-process we have in the core.
> It's a large job, but before it is done we cannot in good faith make
> such an incompatible transition.

I can do that.

However, what about my patch adding a warning about this to
process-send-string?  I think that is independently valuable.  Right now
we have no documentation of this problem...

>> So maybe, the default value of `process-connection-type` should be
>> `unspecified` and the process creation code should emit a warning when
>> creating a process whose connection type is `unspecified` (just
>> a warning, tho: it should then pursue execution as if that value was t,
>> as usual, so as to preserve compatibility).
>
> Something like that, yes.
>
> But I'm actually wondering how come modern Linux kernels don't have a
> way of lifting this restriction, or at least enlarging the limit so it
> makes the problem even less frequent.  Is there some inherent
> limitation that this must be 4KB and nothing larger?

Unfortunately from looking at Linux the limit of 4096 seems to be
hardcoded.

bug#24531: process-send-string seems to truncate lines over 4096 characters

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  24531@debbugs.gnu.org,
>    6149@debbugs.gnu.org,  jidanni@jidanni.org
> Date: Fri, 21 Jul 2023 09:58:38 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The number of applications that (a) don't need console-like behavior
> > and (b) need to send larger-than-4KB buffers to sub-processes is quite
> > small.  Which is why this issue comes up only very rarely.  So making
> > pipes the default will fix a very small fraction of applications, and
> > break the vast majority -- clearly a wrong balance.
> 
> I see your point, but at the same time, the PTY interface on its own is
> not sufficient to make these applications work, not at all.  Specialized
> modes are necessary to make M-x term (to implement a terminal) and M-x
> grep (to parse ANSI color codes) and other such programs work.  Running
> things in a PTY without such specialized code doesn't give you anything,
> AFAIK, because a PTY alone is far from enough to make the Emacs end
> behave like a terminal.  So such programs need to be aware and careful
> about such things anyway, and need additional infrastructure on top of
> make-process.  So the default being "pty" gives such programs very
> little: it doesn't save them any complexity.

That Emacs needs to do something doesn't invalidate my point.  My
point is that communications via a PTY is a necessary (though a
sufficient) condition for these features.  Basically, you cannot use
pipes for any interactive feature, because pipes are buffered.

> However, what about my patch adding a warning about this to
> process-send-string?  I think that is independently valuable.  Right now
> we have no documentation of this problem...

This should be documented in the ELisp manual, and in more detail, not
just as a vague warning.