bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

I was investigating why eudc-ldap hangs, and stripped the problem to the
following (did this in emacs -Q):

(let ((buf (get-buffer-create "my-process-buf")))
  (with-current-buffer buf
    (erase-buffer)
    (let* ((process-connection-type nil)
	   (proc (start-process "my-process" buf "/usr/bin/read" "-p" "enter something:")))
      (while (null (progn
		     (goto-char (point-min))
		     (re-search-forward "enter something:" (point-max) t)))
	(accept-process-output proc 1)
	(message "proc status: %s" (process-status proc)))
      (message "found prompt")
      (message "proc status: %s" (process-status proc))
      (process-send-string proc "asdf")
      (process-send-string proc "\n")
      (while (not (memq (process-status proc) '(exit signal)))
	(sit-for 0.1))
      (message "exited"))))

If I let-bind process-connection-type to t (use pty) in above code, then
it works normally:

*Messages* :
proc status: run
found prompt
proc status: run
exited
"exited"

Buffer "my-process-buf":
enter something:
Process my-process<2> finished

But if I let-bind process-connection-type to nil (so does ldap.el), then
it hangs, and after a few seconds wait and C-g, *Messages* has only
this: "proc status: run [6 times]", and buffer my-process-buf is empty.

This is emacs master built with ./configure CC=/usr/bin/gcc --without-x
--without-sound --with-ns --with-file-notification=no --with-gnutls
--with-json --with-modules --with-threads --with-xml2 && make

Also tried --without-threads - same behavior.

Thanks.

In GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin17.7.0, NS appkit-1561.60 Version 10.13.6 (Build 17G65))
 of 2018-10-15 built on fgunbin.playteam.ru
Repository revision: 3d91dc1bb5aeecda786ebe1805c33d14c8bd89fa
System Description:  Mac OS X 10.13.6

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Date: Mon, 15 Oct 2018 22:03:19 +0300
> 
> If I let-bind process-connection-type to t (use pty) in above code, then
> it works normally:
> 
> *Messages* :
> proc status: run
> found prompt
> proc status: run
> exited
> "exited"
> 
> Buffer "my-process-buf":
> enter something:
> Process my-process<2> finished
> 
> But if I let-bind process-connection-type to nil (so does ldap.el), then
> it hangs, and after a few seconds wait and C-g, *Messages* has only
> this: "proc status: run [6 times]", and buffer my-process-buf is empty.

The reason is almost certainly buffering: when the connection is via a
pipe, the subprocess writes in buffered mode, so it might take quite a
few characters of input before /usr/bin/read outputs something.  Try
using the -n or -N options, and see if that helps.

Why does ldap.el set process-connection-type to nil?

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 20/10/2018 13:09 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Date: Mon, 15 Oct 2018 22:03:19 +0300
>>
>> If I let-bind process-connection-type to t (use pty) in above code, then
>> it works normally:
>>
>> *Messages* :
>> proc status: run
>> found prompt
>> proc status: run
>> exited
>> "exited"
>>
>> Buffer "my-process-buf":
>> enter something:
>> Process my-process<2> finished
>>
>> But if I let-bind process-connection-type to nil (so does ldap.el), then
>> it hangs, and after a few seconds wait and C-g, *Messages* has only
>> this: "proc status: run [6 times]", and buffer my-process-buf is empty.
>
> The reason is almost certainly buffering: when the connection is via a
> pipe, the subprocess writes in buffered mode, so it might take quite a
> few characters of input before /usr/bin/read outputs something.  Try
> using the -n or -N options, and see if that helps.

Thanks for looking at this.
I tried wrapping read in stdbuf, but it didn't change anything:
(start-process "my-process" buf "stdbuf" "-o0" "/usr/bin/read" "-p" "enter something:")

read -n won't help either, because it can affect the number of chars
"read" reads - while in my example we are just waiting for prompt.

> Why does ldap.el set process-connection-type to nil?

I don't know, and it seems like it should not mingle with this setting
at all (why should it?)

I've CC'ed Thomas, who is the author of these lines (ldap.el line 649),
according to git blame.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 20/10/2018 13:09 +0300, Eli Zaretskii wrote:
>
>>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>>> Date: Mon, 15 Oct 2018 22:03:19 +0300
>>>
>>> If I let-bind process-connection-type to t (use pty) in above code, then
>>> it works normally:
>>>
>>> *Messages* :
>>> proc status: run
>>> found prompt
>>> proc status: run
>>> exited
>>> "exited"
>>>
>>> Buffer "my-process-buf":
>>> enter something:
>>> Process my-process<2> finished
>>>
>>> But if I let-bind process-connection-type to nil (so does ldap.el), then
>>> it hangs, and after a few seconds wait and C-g, *Messages* has only
>>> this: "proc status: run [6 times]", and buffer my-process-buf is empty.
>>
>> The reason is almost certainly buffering: when the connection is via a
>> pipe, the subprocess writes in buffered mode, so it might take quite a
>> few characters of input before /usr/bin/read outputs something.  Try
>> using the -n or -N options, and see if that helps.
>
> Thanks for looking at this.
> I tried wrapping read in stdbuf, but it didn't change anything:
> (start-process "my-process" buf "stdbuf" "-o0" "/usr/bin/read" "-p" "enter something:")
>
> read -n won't help either, because it can affect the number of chars
> "read" reads - while in my example we are just waiting for prompt.
>
>> Why does ldap.el set process-connection-type to nil?
>
> I don't know, and it seems like it should not mingle with this setting
> at all (why should it?)
>
> I've CC'ed Thomas, who is the author of these lines (ldap.el line 649),
> according to git blame.

The overall change was to send the password to ldapsearch -W via a pipe
instead of in the clear on the command line (which made it visible to
other users e.g., in ps output).

As for that specific line, I may have copied it from another part of
Emacs that reads a password via a pipe; when I look now at other places
that bind this variable to nil, I see comments like this one in
lisp/gnus/nntp.el:

    ;; A non-nil connection type results in mightily odd behavior where
    ;; (process-send-string proc "\^M") ends up sending a "\n" to the
    ;; ssh process.  --Stef
    ;; Also a nil connection allow ssh-askpass to work under X11.
    (let ((process-connection-type nil))
      (apply 'start-process "nntpd" buffer command))

Today I tested my setup (x86_64 GNU/Linux, OpenLDAP ldapsearch 2.4.40)
without setting process-connection-type to nil, and it still works.  The
documentation for that variable says that the fallback is to use a pipe
if all ptys are busy in which case I guess this would still fail for
you.

Your test case behaves the same way for me on x86_64 GNU/Linux.

Maybe our ldapsearch commands are behaving differently.  What version of
ldapsearch are you using?

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Thomas,

On 22/10/2018 21:53 -0400, Thomas Fitzsimmons wrote:
[..]
> As for that specific line, I may have copied it from another part of
> Emacs that reads a password via a pipe; when I look now at other places
> that bind this variable to nil, I see comments like this one in
> lisp/gnus/nntp.el:
>
>     ;; A non-nil connection type results in mightily odd behavior where
>     ;; (process-send-string proc "\^M") ends up sending a "\n" to the
>     ;; ssh process.  --Stef
>     ;; Also a nil connection allow ssh-askpass to work under X11.
>     (let ((process-connection-type nil))
>       (apply 'start-process "nntpd" buffer command))
>
> Today I tested my setup (x86_64 GNU/Linux, OpenLDAP ldapsearch 2.4.40)
> without setting process-connection-type to nil, and it still works.  The
> documentation for that variable says that the fallback is to use a pipe
> if all ptys are busy in which case I guess this would still fail for
> you.
>
> Your test case behaves the same way for me on x86_64 GNU/Linux.

Do you mean it fails, or it works ok (regardless of
process-connection-type value)?

> Maybe our ldapsearch commands are behaving differently.  What version of
> ldapsearch are you using?

I have OpenLDAP 2.4.28.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Thomas,
>
> On 22/10/2018 21:53 -0400, Thomas Fitzsimmons wrote:
> [..]
>> As for that specific line, I may have copied it from another part of
>> Emacs that reads a password via a pipe; when I look now at other places
>> that bind this variable to nil, I see comments like this one in
>> lisp/gnus/nntp.el:
>>
>>     ;; A non-nil connection type results in mightily odd behavior where
>>     ;; (process-send-string proc "\^M") ends up sending a "\n" to the
>>     ;; ssh process.  --Stef
>>     ;; Also a nil connection allow ssh-askpass to work under X11.
>>     (let ((process-connection-type nil))
>>       (apply 'start-process "nntpd" buffer command))
>>
>> Today I tested my setup (x86_64 GNU/Linux, OpenLDAP ldapsearch 2.4.40)
>> without setting process-connection-type to nil, and it still works.  The
>> documentation for that variable says that the fallback is to use a pipe
>> if all ptys are busy in which case I guess this would still fail for
>> you.
>>
>> Your test case behaves the same way for me on x86_64 GNU/Linux.
>
> Do you mean it fails, or it works ok (regardless of
> process-connection-type value)?

I meant its behaviour is the same for me on x86_64 GNU/Linux as what you
described its behaviour to be on Mac OS.  Without:

(process-connection-type nil)

the test case succeeds.  With it, the test case hangs.

However if I change the test case program from:

"/usr/bin/read" "-p" "enter something:"

to:

"ldapsearch" "-W"

it does not hang regardless of the process-connection-type setting.
Because it is missing the other options, after reading the password
ldapsearch just fails to contact the server, after which my-process-buf
contains...

...without (process-connection-type nil)):

   Enter LDAP Password: 
   ldap_sasl_interactive_bind_s: can't contact LDAP server (-1)
   
   Process my-process<4> exited abnormally with code 255

...with (process-connection-type nil)):

   Enter LDAP Password: ldap_sasl_interactive_bind_s: can't contact LDAP server (-1)
   
   Process my-process<4> exited abnormally with code 255

The only difference seems to be the missing newline.

I got the above results with both my system ldapsearch (2.4.40) and with
ldapseach 2.4.28 which I built from source.

Anyway, at least on my system, it seems that /usr/bin/read is not doing
the same thing as ldapsearch -W to read the string.

What happens when you try the test case with "ldapsearch" "-W"?

>> Maybe our ldapsearch commands are behaving differently.  What version of
>> ldapsearch are you using?
>
> I have OpenLDAP 2.4.28.

OK, I doubt the problem is related to OpenLDAP version differences, but
if it's easy it might be worth trying a newer one.

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Thomas,

On 23/10/2018 21:55 -0400, Thomas Fitzsimmons wrote:

>>> Your test case behaves the same way for me on x86_64 GNU/Linux.
>>
>> Do you mean it fails, or it works ok (regardless of
>> process-connection-type value)?
>
> I meant its behaviour is the same for me on x86_64 GNU/Linux as what you
> described its behaviour to be on Mac OS.  Without:
>
> (process-connection-type nil)
>
> the test case succeeds.  With it, the test case hangs.

Good that it can be "reproduced" at least in this form.

If I change args to "ldapsearch" "-W" (and prompt, accordingly), then

- with process-connection-type t, the test exits normally, and buffer
  contains the same text as for you:

  Enter LDAP Password: 
  ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)

  Process my-process exited abnormally with code 255

- with process-connection-type nil, the test hangs, and when I quit, the
  buffer contains:

  Process my-process stopped (tty output): 22

I see that /usr/bin/read is not the best thing to test with, because I
don't know exactly what it's doing.

My ldapsearch (2.4.28) is from macOS - they don't update it frequently.

I'll try a later version.  Other than that, I don't know what to do for
now :-(

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

This may be the clue to the problem:

man termios(4):

   If a process is in the foreground process group of its controlling termi-
   nal, read operations are allowed.  Any attempts by a process in a back-
   ground process group to read from its controlling terminal causes a
   SIGTTIN signal to be sent to the process's group unless one of the fol-
   lowing special cases apply:  If the reading process is ignoring or block-
   ing the SIGTTIN signal, or if the process group of the reading process is
   orphaned, the read(2) returns -1 with errno set to EIO and no signal is
   sent.  The default action of the SIGTTIN signal is to stop the process to
   which it is sent.

I output the process-attributes of the started process, and note tpgid
(ID of foreground process group on the process’s tty) and tty:

- p-c-t t

  attrs=((etime 0 0 7080 0) (start 23504 29912 167367 0) (nice . 0)
  (tpgid . 16419) (tty . ttys001) (pgrp . 16419) (ppid . 98669) (state
  . R) (comm . ldapsearch) (group . staff) (egid . 20) (user . fgunbin)
  (euid . 501))

 ldapsearch's pgrp=16419, it is using ttys001 for r/w, and it's in
 foreground process group on that tty (tpgid . 16419)

- p-c-t nil

  attrs=((etime 0 0 6177 0) (start 23504 29897 662718 0) (nice . 0)
  (tpgid . 98669) (tty . ttys000) (pgrp . 16416) (ppid . 98669) (state
  . R) (comm . ldapsearch) (group . staff) (egid . 20) (user . fgunbin)
  (euid . 501))

  ldapsearch's pgrp=16416, it is using ttys000 for r/w (??), and it is
  NOT in foreground process group on that tty - (tpgid . 98669).  98669
  is emacs, and emacs is in foreground - this is ok.

So the question is - why the process in the second case thinks that it
has a tty?  When it tries to use it, it receives SIGTTIN and is stopped,
as termios man page says.  My test case with p-c-t nil results in the
processed being stopped, indeed.

My knowledge of these things is very limited, forgive me if I'm clearly
wrong somewhere.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> This may be the clue to the problem:
>
> man termios(4):
>
>    If a process is in the foreground process group of its controlling termi-
>    nal, read operations are allowed.  Any attempts by a process in a back-
>    ground process group to read from its controlling terminal causes a
>    SIGTTIN signal to be sent to the process's group unless one of the fol-
>    lowing special cases apply:  If the reading process is ignoring or block-
>    ing the SIGTTIN signal, or if the process group of the reading process is
>    orphaned, the read(2) returns -1 with errno set to EIO and no signal is
>    sent.  The default action of the SIGTTIN signal is to stop the process to
>    which it is sent.
>
> I output the process-attributes of the started process, and note tpgid
> (ID of foreground process group on the process’s tty) and tty:
>
> - p-c-t t
>
>   attrs=((etime 0 0 7080 0) (start 23504 29912 167367 0) (nice . 0)
>   (tpgid . 16419) (tty . ttys001) (pgrp . 16419) (ppid . 98669) (state
>   . R) (comm . ldapsearch) (group . staff) (egid . 20) (user . fgunbin)
>   (euid . 501))
>
>  ldapsearch's pgrp=16419, it is using ttys001 for r/w, and it's in
>  foreground process group on that tty (tpgid . 16419)
>
> - p-c-t nil
>
>   attrs=((etime 0 0 6177 0) (start 23504 29897 662718 0) (nice . 0)
>   (tpgid . 98669) (tty . ttys000) (pgrp . 16416) (ppid . 98669) (state
>   . R) (comm . ldapsearch) (group . staff) (egid . 20) (user . fgunbin)
>   (euid . 501))
>
>   ldapsearch's pgrp=16416, it is using ttys000 for r/w (??), and it is
>   NOT in foreground process group on that tty - (tpgid . 98669).  98669
>   is emacs, and emacs is in foreground - this is ok.
>
> So the question is - why the process in the second case thinks that it
> has a tty?  When it tries to use it, it receives SIGTTIN and is stopped,
> as termios man page says.  My test case with p-c-t nil results in the
> processed being stopped, indeed.

Try the test case again but with the command "/usr/bin/tty" (with no
arguments).

For me (GNU/Linux, x86_64):

- p-c-t t

  /dev/pts/51

  Process my-process<4> finished

- p-c-t nil

  not a tty

  Process my-process<4> exited abnormally with code 1

(Meanwhile the test case hangs looking for the expected string until I
Ctrl-g it.)

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Tried on macOS, here's the output.

On 24/10/2018 12:20 -0400, Thomas Fitzsimmons wrote:

> Try the test case again but with the command "/usr/bin/tty" (with no
> arguments).
>
> For me (GNU/Linux, x86_64):
>
> - p-c-t t
>
>   /dev/pts/51
>
>   Process my-process<4> finished

  /dev/ttys003

  Process my-process finished

>
> - p-c-t nil
>
>   not a tty
>
>   Process my-process<4> exited abnormally with code 1

  not a tty

  Process my-process exited abnormally with code 1

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Thomas, what does (process-attributes (process-id proc)) output for you?
(say, right after the process creation).

The fact that the process is stopped by OS (in my case) makes me think
that the process is indeed trying to use tty in p-c-t nil case.  So it
could be bug in process creation.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Hi Filipp,

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Thomas, what does (process-attributes (process-id proc)) output for you?
> (say, right after the process creation).
>
> The fact that the process is stopped by OS (in my case) makes me think
> that the process is indeed trying to use tty in p-c-t nil case.  So it
> could be bug in process creation.

p-c-t t

((args . /usr/bin/ldapsearch -W) (pmem . 0.006538857941754344) (pcpu
. -0.0e+NaN) (etime 0 0 0 0) (rss . 1596) (vsize . 42680) (start 23504
58749 447508 202000) (thcount . 1) (nice . 0) (pri . 20) (ctime 0 0 0 0)
(cstime 0 0 0 0) (cutime 0 0 0 0) (time 0 0 0 0) (stime 0 0 0 0) (utime
0 0 0 0) (cmajflt . 0) (cminflt . 0) (majflt . 0) (minflt . 485) (tpgid
. [...]) (ttname . /dev/pts54) (sess . 29274) (pgrp . 29274) (ppid
. 29075) (state . S) (comm . ldapsearch) (group . [...]) (egid . [...])
(user . [...]) (euid . [...]))

p-c-t nil

((args . /usr/bin/ldapsearch -W) (pmem . 0.006538857941754344) (pcpu
. 0.0) (etime 0 1 10000 0) (rss . 1596) (vsize . 42684) (start 23504
58688 70097 688000) (thcount . 1) (nice . 0) (pri . 20) (ctime 0 0 0 0)
(cstime 0 0 0 0) (cutime 0 0 0 0) (time 0 0 0 0) (stime 0 0 0 0) (utime
0 0 0 0) (cmajflt . 0) (cminflt . 0) (majflt . 0) (minflt . 483) (tpgid
. [...]) (ttname . ) (sess . 29201) (pgrp . 29201) (ppid . 29075) (state
. S) (comm . ldapsearch) (group . [...]) (egid . [...]) (user . [...])
(euid . [...]))

I had been testing on Emacs 26.1, but noticed the difference:

(tty . ...) vs (ttname . ...)

and so retested on master tip to see if the Emacs version accounted for
that, but I get the same result.  "tty" isn't listed as a key in
src/process.c, so I'm don't know where it's coming from on Darwin.
However, ttname's value on my system does correspond to the
process-connection-type setting.

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 24/10/2018 18:07 -0400, Thomas Fitzsimmons wrote:

> Hi Filipp,
>
[...]
> However, ttname's value on my system does correspond to the
> process-connection-type setting.

Yes, so in your case child process doesn't have controlling terminal,
while in my case it does.

process.c Line 2069:

  #ifdef DARWIN_OS
    /* Darwin doesn't let us run setsid after a vfork, so use fork when
       necessary.  Also, reset SIGCHLD handling after a vfork, as
       apparently macOS can mistakenly deliver SIGCHLD to the child.  */
    if (pty_flag)
      pid = fork ();
    else
      {
        pid = vfork ();
        if (pid == 0)
          signal (SIGCHLD, SIG_DFL);
      }
  #else
    pid = vfork ();
  #endif

With process-connection-type nil, we call vfork() and then setsid()
apparently fails (at least to reset child's controlling terminal).

I tried to change the above to just use fork(), and got the correct
behavior.

If only I knew how to fix this correctly, without reverting to fork()
:-) Does anyone have ideas?  I'll try to look into this further myself,
and will write back.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org
> Date: Thu, 25 Oct 2018 18:51:17 +0300
> 
> With process-connection-type nil, we call vfork() and then setsid()
> apparently fails (at least to reset child's controlling terminal).
> 
> I tried to change the above to just use fork(), and got the correct
> behavior.
> 
> If only I knew how to fix this correctly, without reverting to fork()
> :-) Does anyone have ideas?  I'll try to look into this further myself,
> and will write back.

I'm probably missing something: why are you still trying to find a
solution, when one was already found?  What's wrong with setting
process-connection-type non-nil in this case, at least for Darwin?

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 25/10/2018 19:24 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org
>> Date: Thu, 25 Oct 2018 18:51:17 +0300
>>
>> With process-connection-type nil, we call vfork() and then setsid()
>> apparently fails (at least to reset child's controlling terminal).
>>
>> I tried to change the above to just use fork(), and got the correct
>> behavior.
>>
>> If only I knew how to fix this correctly, without reverting to fork()
>> :-) Does anyone have ideas?  I'll try to look into this further myself,
>> and will write back.
>
> I'm probably missing something: why are you still trying to find a
> solution, when one was already found?  What's wrong with setting
> process-connection-type non-nil in this case, at least for Darwin?

Yes, setting p-c-t to t worked from the start, but it's just a
workaround - there's still problem with p-c-t nil.

I'm currently running with this patch, which is based on Alan's fix in
callproc.c (call_process() had the same problem on Darwin).

Filipp

diff --git a/src/process.c b/src/process.c
index 6cda4f27ac..1f8810927d 100644
--- a/src/process.c
+++ b/src/process.c
@@ -2066,21 +2066,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   int volatile forkerr_volatile = forkerr;
   struct Lisp_Process *p_volatile = p;
 
-#ifdef DARWIN_OS
-  /* Darwin doesn't let us run setsid after a vfork, so use fork when
-     necessary.  Also, reset SIGCHLD handling after a vfork, as
-     apparently macOS can mistakenly deliver SIGCHLD to the child.  */
-  if (pty_flag)
-    pid = fork ();
-  else
-    {
-      pid = vfork ();
-      if (pid == 0)
-	signal (SIGCHLD, SIG_DFL);
-    }
-#else
   pid = vfork ();
-#endif
 
   current_dir = current_dir_volatile;
   lisp_pty_name = lisp_pty_name_volatile;
@@ -2091,15 +2077,35 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   p = p_volatile;
 
   pty_flag = p->pty_flag;
-
   if (pid == 0)
 #endif /* not WINDOWSNT */
     {
+#ifdef DARWIN_OS
+      /* Work around a macOS bug, where SIGCHLD is apparently
+	 delivered to a vforked child instead of to its parent.  See:
+	 https://lists.gnu.org/r/emacs-devel/2017-05/msg00342.html
+      */
+      signal (SIGCHLD, SIG_DFL);
+#endif
+
       /* Make the pty be the controlling terminal of the process.  */
 #ifdef HAVE_PTYS
       /* First, disconnect its current controlling terminal.
 	 Do this even if !PTY_FLAG; see Bug#30762.  */
+#ifdef DARWIN_OS
+      /* Darwin doesn't let us run setsid after a vfork, so use
+         TIOCNOTTY when necessary. */
+      {
+	int j = emacs_open (DEV_TTY, O_RDWR, 0);
+	if (j >= 0)
+	  {
+	    ioctl (j, TIOCNOTTY, 0);
+	    emacs_close (j);
+	  }
+      }
+#else
       setsid ();
+#endif
       /* Make the pty's terminal the controlling terminal.  */
       if (pty_flag && forkin >= 0)
 	{

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org, alan@idiocy.org
> Date: Thu, 25 Oct 2018 20:10:06 +0300
> 
> > I'm probably missing something: why are you still trying to find a
> > solution, when one was already found?  What's wrong with setting
> > process-connection-type non-nil in this case, at least for Darwin?
> 
> Yes, setting p-c-t to t worked from the start, but it's just a
> workaround - there's still problem with p-c-t nil.
> 
> I'm currently running with this patch, which is based on Alan's fix in
> callproc.c (call_process() had the same problem on Darwin).

I don't understand why: using nil process-connection-type for programs
that prompt the user is a bug anyway.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 25/10/2018 20:29 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org, alan@idiocy.org
>> Date: Thu, 25 Oct 2018 20:10:06 +0300
>>
>> > I'm probably missing something: why are you still trying to find a
>> > solution, when one was already found?  What's wrong with setting
>> > process-connection-type non-nil in this case, at least for Darwin?
>>
>> Yes, setting p-c-t to t worked from the start, but it's just a
>> workaround - there's still problem with p-c-t nil.
>>
>> I'm currently running with this patch, which is based on Alan's fix in
>> callproc.c (call_process() had the same problem on Darwin).
>
> I don't understand why: using nil process-connection-type for programs
> that prompt the user is a bug anyway.

I think these are separate problems.

Here's my explanation of this problem on Darwin:

Some time ago, setsid() was called only if (pty_flag).  Alan's commit
a13eaddce2ddbe3ba0b7f4c81715bc0fcdba99f6 (fixes bug 26397) changes
vfork() to fork() for this case - so we never call setsid() after
vfork().  Then Paul's commit 5c91ca8f30098cb2593ca375daa82d25aef03ad7
(fixes bug 30762) changes setsid() to be invoked always.  So it again
became possible that we call setsid() after vfork(), and this I believe
is the root case of the current problem.

My patch tries to fix that.  I don't think it matters here whether we
call ldapsearch with correct value of p-c-t or not, it just the case
which was missed.  And vfork() should perform better than fork(), so it
looks like an improvement.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Thu, 25 Oct 2018 21:10:25 +0300
> 
> On 25/10/2018 20:29 +0300, Eli Zaretskii wrote:
> 
> >> From: Filipp Gunbin <fgunbin@fastmail.fm>
> >> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org, alan@idiocy.org
> >> Date: Thu, 25 Oct 2018 20:10:06 +0300
> >>
> >> > I'm probably missing something: why are you still trying to find a
> >> > solution, when one was already found?  What's wrong with setting
> >> > process-connection-type non-nil in this case, at least for Darwin?
> >>
> >> Yes, setting p-c-t to t worked from the start, but it's just a
> >> workaround - there's still problem with p-c-t nil.
> >>
> >> I'm currently running with this patch, which is based on Alan's fix in
> >> callproc.c (call_process() had the same problem on Darwin).
> >
> > I don't understand why: using nil process-connection-type for programs
> > that prompt the user is a bug anyway.
> 
> I think these are separate problems.

They are, indeed; and I thought this bug report is about the problem
with ldapsearch.  If you think the way we start subprocesses on macOS
needs fixing, it should be a separate bug, most probably with a
different urgency and severity.

> So it again became possible that we call setsid() after vfork(), and
> this I believe is the root case of the current problem.

Once again, I believe the root cause is that process-connection-type
has the wrong value.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Eli,

On 25/10/2018 21:20 +0300, Eli Zaretskii wrote:

>> I think these are separate problems.
>
> They are, indeed; and I thought this bug report is about the problem
> with ldapsearch.  If you think the way we start subprocesses on macOS
> needs fixing, it should be a separate bug, most probably with a
> different urgency and severity.

Sure, filed bug #33154.

>> So it again became possible that we call setsid() after vfork(), and
>> this I believe is the root case of the current problem.
>
> Once again, I believe the root cause is that process-connection-type
> has the wrong value.

I have no opinion on the correctness of p-c-t in ldap-search-internal in
ldap.el.  If 33154 gets fixed, then code in ldap.el works regardless of
its value.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Thu, 25 Oct 2018 22:36:50 +0300
> 
> > Once again, I believe the root cause is that process-connection-type
> > has the wrong value.
> 
> I have no opinion on the correctness of p-c-t in ldap-search-internal in
> ldap.el.  If 33154 gets fixed, then code in ldap.el works regardless of
> its value.

Is this bug important to fix, or can it wait for Emacs 27?  If it
cannot wait, then setting process-connection-type non-nil is the only
way we can have it fixed on the release branch, because changes like
those which you propose for process.c can only go to master.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 25/10/2018 22:40 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org,  alan@idiocy.org
>> Date: Thu, 25 Oct 2018 22:36:50 +0300
>>
>> > Once again, I believe the root cause is that process-connection-type
>> > has the wrong value.
>>
>> I have no opinion on the correctness of p-c-t in ldap-search-internal in
>> ldap.el.  If 33154 gets fixed, then code in ldap.el works regardless of
>> its value.
>
> Is this bug important to fix, or can it wait for Emacs 27?  If it
> cannot wait, then setting process-connection-type non-nil is the only
> way we can have it fixed on the release branch, because changes like
> those which you propose for process.c can only go to master.

If someone could confirm that setting process-connection-type to t
(ldap.el, line 649) doesn't break it on GNU/Linux, then I think it's
good to have it on release branch.  This bug prevents at least
eudc-expand-inline with ldap on macOS from working.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Thu, 25 Oct 2018 23:47:00 +0300
> 
> > Is this bug important to fix, or can it wait for Emacs 27?  If it
> > cannot wait, then setting process-connection-type non-nil is the only
> > way we can have it fixed on the release branch, because changes like
> > those which you propose for process.c can only go to master.
> 
> If someone could confirm that setting process-connection-type to t
> (ldap.el, line 649) doesn't break it on GNU/Linux, then I think it's
> good to have it on release branch.  This bug prevents at least
> eudc-expand-inline with ldap on macOS from working.

We could make the change conditional on macOS, if we want to be
extra-careful.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: fitzsim@fitzsim.org,  33050@debbugs.gnu.org, alan@idiocy.org
>> Date: Thu, 25 Oct 2018 20:10:06 +0300
>> 
>> > I'm probably missing something: why are you still trying to find a
>> > solution, when one was already found?  What's wrong with setting
>> > process-connection-type non-nil in this case, at least for Darwin?
>> 
>> Yes, setting p-c-t to t worked from the start, but it's just a
>> workaround - there's still problem with p-c-t nil.
>> 
>> I'm currently running with this patch, which is based on Alan's fix in
>> callproc.c (call_process() had the same problem on Darwin).
>
> I don't understand why: using nil process-connection-type for programs
> that prompt the user is a bug anyway.

I'm interested to know why you think this is a bug (or even not
preferable) in the specific case of ldap.el.  When ldap.el calls
ldapsearch, ldapsearch doesn't prompt the user for the password
directly.  ldap.el (having pre-read the password into Elisp by various
means) runs the command, and, in response to its prompt, sends it the
password.

I re-read the Elisp "Asynchronous Processes" node; when I was deciding
how to set process-connection-type, I think I was following the
_underlined_ part in:

   [...] If available, ptys are usually preferable for processes visible
   to the user, as in Shell mode, because they allow for job control
   (‘C-c’, ‘C-z’, etc.) between the process and its children, and
   because interactive programs treat ptys as terminal devices, whereas
   pipes don’t support these features.  However, _for subprocesses used
   by Lisp programs for internal purposes, it is often better to use a
   pipe_, because pipes are more efficient, and because they are immune
   to stray character injections that ptys introduce for large (around
   500 byte) messages.  Also, the total number of ptys is limited on
   many systems and it is good not to waste them.

Given that ldapsearch can operate over a pipe (i.e., doesn't need
terminal features, seems to be unaffected by buffering), and is used for
internal purposes (i.e., is not visible to the user) isn't pipe mode
preferable for the reasons given there?

ldapsearch works for me on GNU/Linux either way, and I don't mind
changing process-connection-type to t on the release branch to work
around the Darwin behaviour.  But if ldap.el's current setting of
process-connection-type really is a flat out bug then the Elisp manual
might need clarification or stronger language about when not to set it
to "pipe mode".

Thanks,
Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: Filipp Gunbin <fgunbin@fastmail.fm>,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Thu, 25 Oct 2018 21:41:34 -0400
> 
> > I don't understand why: using nil process-connection-type for programs
> > that prompt the user is a bug anyway.
> 
> I'm interested to know why you think this is a bug (or even not
> preferable) in the specific case of ldap.el.  When ldap.el calls
> ldapsearch, ldapsearch doesn't prompt the user for the password
> directly.  ldap.el (having pre-read the password into Elisp by various
> means) runs the command, and, in response to its prompt, sends it the
> password.

First, I must apologize in advance: I might be mis-interpreting the
situation, because I don't use ldap.el or ldapsearch.  In that case,
what I said before and what I say below might not make sense.

That said, my rationale was simple: _any_ program that interacts with
the user will generally behave better when connected through pty than
when connected through a pipe, because the former looks and feels as a
terminal, and thus will cause the program to use the appropriate
buffering, will support terminal-only features like various escape
sequences, cursor control, job control, etc.  Using pipes will
probably work in most cases, but might fail in some of them, so good
engineering tells us to prefer pty's, like the ELisp manual says.

> I re-read the Elisp "Asynchronous Processes" node; when I was deciding
> how to set process-connection-type, I think I was following the
> _underlined_ part in:
> 
>    [...] If available, ptys are usually preferable for processes visible
>    to the user, as in Shell mode, because they allow for job control
>    (‘C-c’, ‘C-z’, etc.) between the process and its children, and
>    because interactive programs treat ptys as terminal devices, whereas
>    pipes don’t support these features.  However, _for subprocesses used
>    by Lisp programs for internal purposes, it is often better to use a
>    pipe_, because pipes are more efficient, and because they are immune
>    to stray character injections that ptys introduce for large (around
>    500 byte) messages.  Also, the total number of ptys is limited on
>    many systems and it is good not to waste them.
> 
> Given that ldapsearch can operate over a pipe (i.e., doesn't need
> terminal features, seems to be unaffected by buffering), and is used for
> internal purposes (i.e., is not visible to the user) isn't pipe mode
> preferable for the reasons given there?

Not in my opinion, no.  AFAIU, the efficiency considerations don't
really apply here, since the amount of data exchanged is small
(right?).

All in all, I consider the "however" part above too vague to be useful
(what exactly does "internal purposes" mean in this context?).  I
think it's even slightly misleading.  I'll think about rewording it to
make the point more clear and the decision easier.

> But if ldap.el's current setting of process-connection-type really
> is a flat out bug then the Elisp manual might need clarification or
> stronger language about when not to set it to "pipe mode".

Agreed.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Cc: Filipp Gunbin <fgunbin@fastmail.fm>,  33050@debbugs.gnu.org,  alan@idiocy.org
>> Date: Thu, 25 Oct 2018 21:41:34 -0400
>> 
>> > I don't understand why: using nil process-connection-type for programs
>> > that prompt the user is a bug anyway.
>> 
>> I'm interested to know why you think this is a bug (or even not
>> preferable) in the specific case of ldap.el.  When ldap.el calls
>> ldapsearch, ldapsearch doesn't prompt the user for the password
>> directly.  ldap.el (having pre-read the password into Elisp by various
>> means) runs the command, and, in response to its prompt, sends it the
>> password.
>
> First, I must apologize in advance: I might be mis-interpreting the
> situation, because I don't use ldap.el or ldapsearch.  In that case,
> what I said before and what I say below might not make sense.
>
> That said, my rationale was simple: _any_ program that interacts with
> the user will generally behave better when connected through pty than
> when connected through a pipe, because the former looks and feels as a
> terminal, and thus will cause the program to use the appropriate
> buffering, will support terminal-only features like various escape
> sequences, cursor control, job control, etc.  Using pipes will
> probably work in most cases, but might fail in some of them, so good
> engineering tells us to prefer pty's, like the ELisp manual says.

OK, this makes sense, thanks for providing the rationale.  Given that
ldapsearch does issue a human-readable prompt for the password, it was
likely written and tested assuming a pty.

>> I re-read the Elisp "Asynchronous Processes" node; when I was deciding
>> how to set process-connection-type, I think I was following the
>> _underlined_ part in:
>> 
>>    [...] If available, ptys are usually preferable for processes visible
>>    to the user, as in Shell mode, because they allow for job control
>>    (‘C-c’, ‘C-z’, etc.) between the process and its children, and
>>    because interactive programs treat ptys as terminal devices, whereas
>>    pipes don’t support these features.  However, _for subprocesses used
>>    by Lisp programs for internal purposes, it is often better to use a
>>    pipe_, because pipes are more efficient, and because they are immune
>>    to stray character injections that ptys introduce for large (around
>>    500 byte) messages.  Also, the total number of ptys is limited on
>>    many systems and it is good not to waste them.
>> 
>> Given that ldapsearch can operate over a pipe (i.e., doesn't need
>> terminal features, seems to be unaffected by buffering), and is used for
>> internal purposes (i.e., is not visible to the user) isn't pipe mode
>> preferable for the reasons given there?
>
> Not in my opinion, no.  AFAIU, the efficiency considerations don't
> really apply here, since the amount of data exchanged is small
> (right?).

Right.

> All in all, I consider the "however" part above too vague to be useful
> (what exactly does "internal purposes" mean in this context?).  I
> think it's even slightly misleading.  I'll think about rewording it to
> make the point more clear and the decision easier.

OK, sounds good.  One worry I have about always leaving
process-connection-type t is that, depending on the external system
state -- specifically whether or not all ptys are busy --
process-connection-type might not have any effect, and the underlying
process will rarely (and silently AFAICT) operate in pipe mode.  By
forcing process-connection-type nil, one is always testing in the same
known mode.

That said, it is probably a bad idea to warn or signal an error if all
ptys are busy in the p-c-t t case.  Anyway, I thought I'd mention it;
maybe the documentation update can address this point as well.

How about I make the change conditional on Mac OS on the release branch,
and unconditional on the master branch (once the fix for 33154 is
committed)?  That way users on the master branch will be able to test
pty mode before the next release, and we fix the release branch without
potentially destabilizing non-Darwin users.

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: fgunbin@fastmail.fm,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Fri, 26 Oct 2018 11:41:02 -0400
> 
> That said, it is probably a bad idea to warn or signal an error if all
> ptys are busy in the p-c-t t case.  Anyway, I thought I'd mention it;
> maybe the documentation update can address this point as well.

Right, thanks for pointing that out.

> How about I make the change conditional on Mac OS on the release branch,
> and unconditional on the master branch (once the fix for 33154 is
> committed)?

Sounds good to me, thanks.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> Date: Fri, 26 Oct 2018 20:20:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 33050@debbugs.gnu.org, fgunbin@fastmail.fm, alan@idiocy.org
> 
> > From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> > Cc: fgunbin@fastmail.fm,  33050@debbugs.gnu.org,  alan@idiocy.org
> > Date: Fri, 26 Oct 2018 11:41:02 -0400
> > 
> > That said, it is probably a bad idea to warn or signal an error if all
> > ptys are busy in the p-c-t t case.  Anyway, I thought I'd mention it;
> > maybe the documentation update can address this point as well.
> 
> Right, thanks for pointing that out.

I've updated the documentation, please see if the new text makes
sense.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 26 Oct 2018 20:20:14 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 33050@debbugs.gnu.org, fgunbin@fastmail.fm, alan@idiocy.org
>> 
>> > From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> > Cc: fgunbin@fastmail.fm,  33050@debbugs.gnu.org,  alan@idiocy.org
>> > Date: Fri, 26 Oct 2018 11:41:02 -0400
>> > 
>> > That said, it is probably a bad idea to warn or signal an error if all
>> > ptys are busy in the p-c-t t case.  Anyway, I thought I'd mention it;
>> > maybe the documentation update can address this point as well.
>> 
>> Right, thanks for pointing that out.
>
> I've updated the documentation, please see if the new text makes
> sense.

Yes, though I might still read that and choose a pipe for ldapsearch.
How about adding this sentence:

[...] these features.  A Lisp program should prefer a pty when
interacting with a subprocess that prompts the user for information
(e.g., a password) because that feature of the subprocess was probably
developed and tested assuming a pty.  However, for subprocesses used by
Lisp programs [...]

AIUI, this is your main rationale for using a pty for ldapsearch --
though feel free to reword it.

Thanks,
Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: 33050@debbugs.gnu.org,  fgunbin@fastmail.fm,  alan@idiocy.org
> Date: Sat, 27 Oct 2018 09:42:17 -0400
> 
> > I've updated the documentation, please see if the new text makes
> > sense.
> 
> Yes, though I might still read that and choose a pipe for ldapsearch.
> How about adding this sentence:
> 
> [...] these features.  A Lisp program should prefer a pty when
> interacting with a subprocess that prompts the user for information
> (e.g., a password) because that feature of the subprocess was probably
> developed and tested assuming a pty.  However, for subprocesses used by
> Lisp programs [...]

Wouldn't this just reiterate what the text already says?

						      [...]  However, for
  subprocesses used by Lisp programs for internal purposes (i.e., with no
  user interaction), where significant amounts of data need to be
  exchanged between the subprocess and the Lisp program, it is often
  better to use a pipe, because pipes are more efficient, and because they
  are immune to stray character injections that ptys introduce for large
  (around 500 byte) messages. [...]

This explicitly says that pipes are preferable when NO user
interaction is done, especially when large amounts of data are to be
exchanged with Emacs.  I don't think that asking for a password is a
special case in this context.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
>> Cc: 33050@debbugs.gnu.org,  fgunbin@fastmail.fm,  alan@idiocy.org
>> Date: Sat, 27 Oct 2018 09:42:17 -0400
>> 
>> > I've updated the documentation, please see if the new text makes
>> > sense.
>> 
>> Yes, though I might still read that and choose a pipe for ldapsearch.
>> How about adding this sentence:
>> 
>> [...] these features.  A Lisp program should prefer a pty when
>> interacting with a subprocess that prompts the user for information
>> (e.g., a password) because that feature of the subprocess was probably
>> developed and tested assuming a pty.  However, for subprocesses used by
>> Lisp programs [...]
>
> Wouldn't this just reiterate what the text already says?
>
> 						      [...]  However, for
>   subprocesses used by Lisp programs for internal purposes (i.e., with no
>   user interaction), where significant amounts of data need to be
>   exchanged between the subprocess and the Lisp program, it is often
>   better to use a pipe, because pipes are more efficient, and because they
>   are immune to stray character injections that ptys introduce for large
>   (around 500 byte) messages. [...]
>
> This explicitly says that pipes are preferable when NO user
> interaction is done, especially when large amounts of data are to be
> exchanged with Emacs.  I don't think that asking for a password is a
> special case in this context.

Then maybe change "(i.e., with no user interaction)", which could be
talking about the Lisp program itself or the subprocess, to "(i.e., when
the subprocess does not require user interaction)".  In the case of
ldapsearch, there is user interaction, but it's handled by the Lisp
program, then passed to the subprocess.  That parenthetical part is the
part that I consider in its current form to be a little unclear.

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Cc: 33050@debbugs.gnu.org,  fgunbin@fastmail.fm,  alan@idiocy.org
> Date: Sat, 27 Oct 2018 12:53:02 -0400
> 
> Then maybe change "(i.e., with no user interaction)", which could be
> talking about the Lisp program itself or the subprocess, to "(i.e., when
> the subprocess does not require user interaction)".

Done, thanks.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Thomas,

On 26/10/2018 11:41 -0400, Thomas Fitzsimmons wrote:

> One worry I have about always leaving process-connection-type t is
> that, depending on the external system state -- specifically whether
> or not all ptys are busy -- process-connection-type might not have any
> effect, and the underlying process will rarely (and silently AFAICT)
> operate in pipe mode.  By forcing process-connection-type nil, one is
> always testing in the same known mode.

I don't really understand why pty mode is better here than pipe mode.
Do we need job control, or escape sequences, or anything else specific
to pty?  If we use pty, won't these features, on the contrary, get in
the way somewhere?  We have to respond to only one prompt from
ldapsearch, and for that pipes should work well.  It's not like when the
user is interacting with the process (like in shell mode).  The user may
be unaware that external process is at all invoked.

Your suggestion to condionally fix this for Darwin on release looks
good.

As for 33154, I'll live with it for some more days, and wait for more
people to look at it.  And if all goes well, then I'll push it next
week.

Thanks.
Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Thomas,
>
> On 26/10/2018 11:41 -0400, Thomas Fitzsimmons wrote:
>
>> One worry I have about always leaving process-connection-type t is
>> that, depending on the external system state -- specifically whether
>> or not all ptys are busy -- process-connection-type might not have any
>> effect, and the underlying process will rarely (and silently AFAICT)
>> operate in pipe mode.  By forcing process-connection-type nil, one is
>> always testing in the same known mode.
>
> I don't really understand why pty mode is better here than pipe mode.
> Do we need job control, or escape sequences, or anything else specific
> to pty?  If we use pty, won't these features, on the contrary, get in
> the way somewhere?  We have to respond to only one prompt from
> ldapsearch, and for that pipes should work well.  It's not like when the
> user is interacting with the process (like in shell mode).  The user may
> be unaware that external process is at all invoked.

Yes, my inclination was (and is) to use pipe mode, but I don't know
enough about the implications of either mode, in practice, to
definitively say.

I searched the mailing list archives and current Emacs source code for
process-connection-type and there are many different places where it is
set to either t or nil to accommodate some specific operating system or
program.  Some of these might even have been workarounds for the Darwin
process.c bug you seem to have fixed properly.  Others, like the one I
saw for Solaris, would be harder to test.

Unless we can prove that pty mode can fail against ldapsearch -W in some
semi-realistic valid use case -- maybe a password with a weird
character, or a really really long password that gets buffered or
something -- or conversely that pipe mode can fail in some case where
pty mode would succeed, then I feel I have to defer to Eli's experience
and recommendation.  I need to at least give him a chance to update the
documentation; hopefully that will clarify when to use each mode
definitively.

(I also wonder what happens to a process (e.g., Bash) that needs a pty,
when all ptys are busy.  I haven't done any experiments to check.  At
least in one case I saw in the source code, ange-ftp.el, the program
would just hang in pipe mode.  In that case, it seems like there should
be a way to tell start-process to signal an error if it doesn't get the
desired pty.)

> Your suggestion to condionally fix this for Darwin on release looks
> good.

OK, can you try the attached patch before I push it to emacs-26?

> As for 33154, I'll live with it for some more days, and wait for more
> people to look at it.  And if all goes well, then I'll push it next
> week.

OK, sounds good.

Thanks,
Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-LDAP-Set-process-connection-type-to-t-on-Darwin.patch --]
[-- Type: text/x-diff, Size: 1263 bytes --]

From ce820679ecc4585dd6ab401f8f027e0e527b6092 Mon Sep 17 00:00:00 2001
From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
Date: Fri, 26 Oct 2018 16:53:19 -0400
Subject: [PATCH] LDAP: Set process-connection-type to t on Darwin

* lisp/net/ldap.el (ldap-search-internal): Set
process-connection-type to t on Darwin.  Do not merge to
master.  (Bug#33050)
---
 lisp/net/ldap.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/net/ldap.el b/lisp/net/ldap.el
index 7b47a54b9f..b106de02e9 100644
--- a/lisp/net/ldap.el
+++ b/lisp/net/ldap.el
@@ -646,7 +646,12 @@ ldap-search-internal
 	       (not (equal "" sizelimit)))
 	  (setq arglist (nconc arglist (list (format "-z%s" sizelimit)))))
       (if passwd
-	  (let* ((process-connection-type nil)
+	  ;; Work around Bug#33154, see also Bug#33050.  Leaving
+	  ;; process-connection-type at its default (typically t)
+	  ;; would probably be fine too, however this is the minimal
+	  ;; change on the release branch that fixes ldap.el on Darwin
+	  ;; and leaves other operating systems unchanged.
+	  (let* ((process-connection-type (eq system-type 'darwin))
 		 (proc-args (append arglist ldap-ldapsearch-args
 				    filter))
 		 (proc (apply #'start-process "ldapsearch" buf
-- 
2.11.0

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 26/10/2018 22:09 -0400, Thomas Fitzsimmons wrote:

> (I also wonder what happens to a process (e.g., Bash) that needs a pty,
> when all ptys are busy.  I haven't done any experiments to check.  At
> least in one case I saw in the source code, ange-ftp.el, the program
> would just hang in pipe mode.  In that case, it seems like there should
> be a way to tell start-process to signal an error if it doesn't get the
> desired pty.)

Yes, would be good to check this.

>> Your suggestion to condionally fix this for Darwin on release looks
>> good.
>
> OK, can you try the attached patch before I push it to emacs-26?

I tried it on emacs-26, and couldn't get eudc-expand-inline to work, but
I think that's a different unrelated problem (maybe auth-source reading
.authinfo or something).  Otherwise it looks safe.  I can look more into
it on Monday..

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 26/10/2018 22:09 -0400, Thomas Fitzsimmons wrote:
>
>> (I also wonder what happens to a process (e.g., Bash) that needs a pty,
>> when all ptys are busy.  I haven't done any experiments to check.  At
>> least in one case I saw in the source code, ange-ftp.el, the program
>> would just hang in pipe mode.  In that case, it seems like there should
>> be a way to tell start-process to signal an error if it doesn't get the
>> desired pty.)
>
> Yes, would be good to check this.

I tested ldapsearch after setting the maximum number of ptys to the
number of currently-busy ptys:

cat /proc/sys/kernel/pty/nr > /proc/sys/kernel/pty/max

It does get run in pipe mode silently (i.e., no warning messages are
generated), and as we already confirmed, ldapsearch happens to still
work when run in pipe mode.

>>> Your suggestion to condionally fix this for Darwin on release looks
>>> good.
>>
>> OK, can you try the attached patch before I push it to emacs-26?
>
> I tried it on emacs-26, and couldn't get eudc-expand-inline to work, but
> I think that's a different unrelated problem (maybe auth-source reading
> .authinfo or something).  Otherwise it looks safe.  I can look more into
> it on Monday..

OK, I guess file another bug if there's another issue preventing
eudc-expand-inline from working on emacs-26.

Thanks,
Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 29/10/2018 20:49 -0400, Thomas Fitzsimmons wrote:

> I tested ldapsearch after setting the maximum number of ptys to the
> number of currently-busy ptys:
>
> cat /proc/sys/kernel/pty/nr > /proc/sys/kernel/pty/max
>
> It does get run in pipe mode silently (i.e., no warning messages are
> generated), and as we already confirmed, ldapsearch happens to still
> work when run in pipe mode.

Thanks for testing this!

>>> OK, can you try the attached patch before I push it to emacs-26?
>>
>> I tried it on emacs-26, and couldn't get eudc-expand-inline to work, but
>> I think that's a different unrelated problem (maybe auth-source reading
>> .authinfo or something).  Otherwise it looks safe.  I can look more into
>> it on Monday..
>
> OK, I guess file another bug if there's another issue preventing
> eudc-expand-inline from working on emacs-26.

Yes, I'll get to it soon, thanks.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 26/10/2018 22:09 -0400, Thomas Fitzsimmons wrote:

> OK, can you try the attached patch before I push it to emacs-26?

Thomas, sorry for the late reply.

I finally checked your conditional patch on emacs-26 (after make
bootstrap) and it works.  So if no one objects, it can be committed (and
this bug resolved).

Bug 33154 was resolved, and we can also commit setting p-c-t to t
unconditionally on master.

We just need to make sure that change from emacs-26 doesn't propagate to
master.

Thanks.
Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org
> Date: Thu, 29 Nov 2018 02:09:27 +0300
> 
> On 26/10/2018 22:09 -0400, Thomas Fitzsimmons wrote:
> 
> > OK, can you try the attached patch before I push it to emacs-26?
> 
> Thomas, sorry for the late reply.
> 
> I finally checked your conditional patch on emacs-26 (after make
> bootstrap) and it works.  So if no one objects, it can be committed (and
> this bug resolved).
> 
> Bug 33154 was resolved, and we can also commit setting p-c-t to t
> unconditionally on master.
> 
> We just need to make sure that change from emacs-26 doesn't propagate to
> master.

The patch was installed on master already, by Paul, and the bug was
closed.

Why do we need this on emacs-26?

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 29/11/2018 09:21 +0200, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org
>> Date: Thu, 29 Nov 2018 02:09:27 +0300
>>
>> On 26/10/2018 22:09 -0400, Thomas Fitzsimmons wrote:
>>
>> > OK, can you try the attached patch before I push it to emacs-26?
>>
>> Thomas, sorry for the late reply.
>>
>> I finally checked your conditional patch on emacs-26 (after make
>> bootstrap) and it works.  So if no one objects, it can be committed (and
>> this bug resolved).
>>
>> Bug 33154 was resolved, and we can also commit setting p-c-t to t
>> unconditionally on master.
>>
>> We just need to make sure that change from emacs-26 doesn't propagate to
>> master.
>
> The patch was installed on master already, by Paul, and the bug was
> closed.
>
> Why do we need this on emacs-26?

Paul closed 33154, there was C-level change.  It goes only on master.

This bug is about process-connection-type in ldap-search-internal.  In
this thread, we seemingly agreed to set it to t conditionally for Darwin
on emacs-26, and unconditionally on master.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
> Date: Thu, 29 Nov 2018 11:46:27 +0300
> 
> > Why do we need this on emacs-26?
> 
> Paul closed 33154, there was C-level change.  It goes only on master.
> 
> This bug is about process-connection-type in ldap-search-internal.  In
> this thread, we seemingly agreed to set it to t conditionally for Darwin
> on emacs-26, and unconditionally on master.

I guess I'm confused regarding what you'd like to see on emacs-26.
Was there a patch to that effect posted in this discussion, and if so,
could you point me to it, please?

Thanks.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 29/11/2018 11:19 +0200, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
>> Date: Thu, 29 Nov 2018 11:46:27 +0300
>>
>> > Why do we need this on emacs-26?
>>
>> Paul closed 33154, there was C-level change.  It goes only on master.
>>
>> This bug is about process-connection-type in ldap-search-internal.  In
>> this thread, we seemingly agreed to set it to t conditionally for Darwin
>> on emacs-26, and unconditionally on master.
>
> I guess I'm confused regarding what you'd like to see on emacs-26.
> Was there a patch to that effect posted in this discussion, and if so,
> could you point me to it, please?

Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: eggert@cs.ucla.edu,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
> Date: Thu, 29 Nov 2018 12:51:09 +0300
> 
> > I guess I'm confused regarding what you'd like to see on emacs-26.
> > Was there a patch to that effect posted in this discussion, and if so,
> > could you point me to it, please?
> 
> Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89

Thanks, now pushed to the emacs-26 branch.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 29/11/2018 12:41 +0200, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: eggert@cs.ucla.edu,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
>> Date: Thu, 29 Nov 2018 12:51:09 +0300
>>
>> > I guess I'm confused regarding what you'd like to see on emacs-26.
>> > Was there a patch to that effect posted in this discussion, and if so,
>> > could you point me to it, please?
>>
>> Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89
>
> Thanks, now pushed to the emacs-26 branch.

Ok, I pushed the unconditional setting on master and am closing this
bug.

Thanks.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 29/11/2018 12:41 +0200, Eli Zaretskii wrote:
>
>>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>>> Cc: eggert@cs.ucla.edu,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
>>> Date: Thu, 29 Nov 2018 12:51:09 +0300
>>>
>>> > I guess I'm confused regarding what you'd like to see on emacs-26.
>>> > Was there a patch to that effect posted in this discussion, and if so,
>>> > could you point me to it, please?
>>>
>>> Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89
>>
>> Thanks, now pushed to the emacs-26 branch.
>
> Ok, I pushed the unconditional setting on master and am closing this
> bug.

Thanks Eli and Filipp for pushing these.  Filipp, on master is there any
reason not to just leave process-connection-type at its default value?
The patch I was going to push for this is attached.

Thomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ldap.el-Do-not-set-process-connection-type.patch --]
[-- Type: text/x-diff, Size: 1058 bytes --]

From 5e933b39d1ce8016a8d989cef99a19e12eb36039 Mon Sep 17 00:00:00 2001
From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
Date: Fri, 26 Oct 2018 16:59:09 -0400
Subject: [PATCH] ldap.el: Do not set process-connection-type

* lisp/net/ldap.el (ldap-search-internal): Leave
process-connection-type at its default value.  (Bug#33050)
---
 lisp/net/ldap.el | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lisp/net/ldap.el b/lisp/net/ldap.el
index 7b47a54..4b49f76 100644
--- a/lisp/net/ldap.el
+++ b/lisp/net/ldap.el
@@ -646,8 +646,9 @@ ldap-search-internal
 	       (not (equal "" sizelimit)))
 	  (setq arglist (nconc arglist (list (format "-z%s" sizelimit)))))
       (if passwd
-	  (let* ((process-connection-type nil)
-		 (proc-args (append arglist ldap-ldapsearch-args
+	  ;; Leave process-connection-type at its default value.  See
+	  ;; discussion in Bug#33050.
+	  (let* ((proc-args (append arglist ldap-ldapsearch-args
 				    filter))
 		 (proc (apply #'start-process "ldapsearch" buf
 			      ldap-ldapsearch-prog
-- 
1.8.3.1

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 03/12/2018 09:17 -0500, Thomas Fitzsimmons wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> On 29/11/2018 12:41 +0200, Eli Zaretskii wrote:
>>
>>>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>>>> Cc: eggert@cs.ucla.edu,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
>>>> Date: Thu, 29 Nov 2018 12:51:09 +0300
>>>>
>>>> > I guess I'm confused regarding what you'd like to see on emacs-26.
>>>> > Was there a patch to that effect posted in this discussion, and if so,
>>>> > could you point me to it, please?
>>>>
>>>> Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89
>>>
>>> Thanks, now pushed to the emacs-26 branch.
>>
>> Ok, I pushed the unconditional setting on master and am closing this
>> bug.
>
> Thanks Eli and Filipp for pushing these.  Filipp, on master is there any
> reason not to just leave process-connection-type at its default value?
> The patch I was going to push for this is attached.
>
> Thomas

I find explicit setting more descriptive, but if you think default is
better - please push :-)

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Thomas Fitzsimmons]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 03/12/2018 09:17 -0500, Thomas Fitzsimmons wrote:
>
>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>
>>> On 29/11/2018 12:41 +0200, Eli Zaretskii wrote:
>>>
>>>>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>>>>> Cc: eggert@cs.ucla.edu,  fitzsim@fitzsim.org,  33050@debbugs.gnu.org
>>>>> Date: Thu, 29 Nov 2018 12:51:09 +0300
>>>>>
>>>>> > I guess I'm confused regarding what you'd like to see on emacs-26.
>>>>> > Was there a patch to that effect posted in this discussion, and if so,
>>>>> > could you point me to it, please?
>>>>>
>>>>> Sure, it's in message https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33050#89
>>>>
>>>> Thanks, now pushed to the emacs-26 branch.
>>>
>>> Ok, I pushed the unconditional setting on master and am closing this
>>> bug.
>>
>> Thanks Eli and Filipp for pushing these.  Filipp, on master is there any
>> reason not to just leave process-connection-type at its default value?
>> The patch I was going to push for this is attached.
>>
>> Thomas
>
> I find explicit setting more descriptive, but if you think default is
> better - please push :-)

OK, pushed.  I put a comment in to reference this discussion.

Thomas

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org,  alan@idiocy.org
> Date: Sat, 27 Oct 2018 02:12:21 +0300
> 
> Thomas,

I'm not him, but since you asked...

> I don't really understand why pty mode is better here than pipe mode.
> Do we need job control, or escape sequences, or anything else specific
> to pty?

Yes, we want the buffering be as if ldapsearch was invoked from a
terminal.  Possibly also other features, but I don't know enough about
the program to tell.

Basically, if the program was written for interactive invocation
communicating through pty is always better.

> If we use pty, won't these features, on the contrary, get in
> the way somewhere?

If they get in the way, it's another bug, which needs to be analyzed
and solved.  It could be (although IMO improbable) that we decide to
go back to pipes, but then we will know why we did that, and we should
leave behind a comment with the explanation.

> Your suggestion to condionally fix this for Darwin on release looks
> good.

Right.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

On 27/10/2018 09:59 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  33050@debbugs.gnu.org,  alan@idiocy.org
>> Date: Sat, 27 Oct 2018 02:12:21 +0300
>>
>> Thomas,
>
> I'm not him, but since you asked...
>
>> I don't really understand why pty mode is better here than pipe mode.
>> Do we need job control, or escape sequences, or anything else specific
>> to pty?
>
> Yes, we want the buffering be as if ldapsearch was invoked from a
> terminal.  Possibly also other features, but I don't know enough about
> the program to tell.
>
> Basically, if the program was written for interactive invocation
> communicating through pty is always better.
>
>> If we use pty, won't these features, on the contrary, get in
>> the way somewhere?
>
> If they get in the way, it's another bug, which needs to be analyzed
> and solved.  It could be (although IMO improbable) that we decide to
> go back to pipes, but then we will know why we did that, and we should
> leave behind a comment with the explanation.

Thanks, got it.

Filipp

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Charles A. Roelli]

> From: Thomas Fitzsimmons <fitzsim@fitzsim.org>
> Date: Thu, 25 Oct 2018 21:41:34 -0400
> 
> I re-read the Elisp "Asynchronous Processes" node; when I was deciding
> how to set process-connection-type, I think I was following the
> _underlined_ part in:
>
> [...]
>
> 					  However, _for subprocesses used
>    by Lisp programs for internal purposes, it is often better to use a
>    pipe_, because pipes are more efficient, and because they are immune
>    to stray character injections that ptys introduce for large (around
>    500 byte) messages.

Does anyone recall where these "character injections" come from?  ISTR
that these no longer happen...  Maybe this is a reference to the code
deleted here:

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=2b0a91e78f83fb446cc38efb99399e83ad2cded3
(as seen, e.g., in Bug#32438)

Also, both "process-send-region" and "process-send-string" (in master)
claim to split their process input if it is longer than 500
/characters/ (not bytes) -- but reading through the code in process.c,
it's not clear where (or if) this still happens.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Eli Zaretskii]

> Date: Sat, 27 Oct 2018 22:28:15 +0200
> From: charles@aurox.ch (Charles A. Roelli)
> CC: eliz@gnu.org, 33050@debbugs.gnu.org, fgunbin@fastmail.fm,
> 	alan@idiocy.org
> 
> > 					  However, _for subprocesses used
> >    by Lisp programs for internal purposes, it is often better to use a
> >    pipe_, because pipes are more efficient, and because they are immune
> >    to stray character injections that ptys introduce for large (around
> >    500 byte) messages.
> 
> Does anyone recall where these "character injections" come from?  ISTR
> that these no longer happen...  Maybe this is a reference to the code
> deleted here:
> 
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=2b0a91e78f83fb446cc38efb99399e83ad2cded3
> (as seen, e.g., in Bug#32438)

Probably.

> Also, both "process-send-region" and "process-send-string" (in master)
> claim to split their process input if it is longer than 500
> /characters/ (not bytes) -- but reading through the code in process.c,
> it's not clear where (or if) this still happens.

pty_max_bytes was deleted as well, so I think you are right.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Charles A. Roelli]

> Date: Sun, 28 Oct 2018 18:00:49 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Does anyone recall where these "character injections" come from?  ISTR
> > that these no longer happen...  Maybe this is a reference to the code
> > deleted here:
> > 
> > https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=2b0a91e78f83fb446cc38efb99399e83ad2cded3
> > (as seen, e.g., in Bug#32438)
> 
> Probably.
> 
> > Also, both "process-send-region" and "process-send-string" (in master)
> > claim to split their process input if it is longer than 500
> > /characters/ (not bytes) -- but reading through the code in process.c,
> > it's not clear where (or if) this still happens.
> 
> pty_max_bytes was deleted as well, so I think you are right.

Aha, yes, pty_max_bytes was removed later after going unused for some time:

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=3d082a269ece18058ed82957f8a056822b39789e

IIUC, there might be a good case for bringing it back, with respect to
Bug#6149 and Bug#32438.  At least so that we can warn users (or
prevent them from) overflowing a pty input queue.

For the doc issue, I've opened Bug#33191.

bug#33050: 27.0.50; [macOS] Problem with process input with process-connection-type nil

[Filipp Gunbin]

Hi Alan,

Your commit [1] fixes usage of setsid() on macOS in call_process()
(callproc.c), seems that the same thing needs to be done in process.c in
create_process().  That setsid() in create_process() seems to be the
cause of this thread's bug.

WDYT?

[1]:
commit a13eaddce2ddbe3ba0b7f4c81715bc0fcdba99f6
Author: Alan Third <alan@idiocy.org>
Date:   Sun Apr 9 20:10:33 2017 +0100

Filipp