Sending EOF to process as part of comint-simple-send

Sending EOF to process as part of comint-simple-send

[Eli Zaretskii]

For the background, see

  https://lists.gnu.org/archive/html/help-gnu-emacs/2023-01/msg00097.html
  https://lists.gnu.org/archive/html/help-gnu-emacs/2023-01/msg00156.html

Note that this was on MS-Windows, where we communicate with
sub-processes via pipes, as PTYs are not available.

In comint-simple-send, we have:

    (comint-send-string proc send-string))
  (if (and comint-input-sender-no-newline
	   (not (string-equal string "")))
      (process-send-eof)))

Thus, if we send sub-process input without a newline, we then send EOF
to the sub-process, except when the string we send is empty.

I can only understand this logic if it assumes a Posix shell or other
Posix process with which we communicate via a PTY.  Because in that
case, we just send C-d to the process, and AFAIK that will not cause
the sub-process to finish except when C-d is the only character we
send.  But if the communications with the sub-process are via a pipe,
then process-send-eof says:

  If PROCESS is a network connection, or is a process communicating
  through a pipe (as opposed to a pty), then you cannot send any more
  text to PROCESS after you call this function.

The implementation closes the file descriptor through which we write
to the sub-process, which is supposed to produce EOF on input for the
sub-process, and interactive sub-processes will exit.  And even if the
sub-process doesn't exit, we will not be able to write to it anymore.

The change to send EOF was installed 21 years ago, as part of commit
004a541.  The only reference to some related issue that I can find is
this bug report:

  https://lists.gnu.org/archive/html/bug-gnu-emacs/2002-01/msg00195.html

which indeed describes a situation where we want to send to a process
input without ending it with a newline.  If you try that recipe on
MS-Windows, you see:

  d:\branch\src>od -t x1
  od -t x1
  10000000000 31 30 30 30
  0000004

  d:\branch\src>
  Process shell finished

whereas on GNU/Linux we get the expected:

  $ od -t x1
  10000000000 31 30 30 30
  0000004
  $

So I think comint-simple-send should only send EOF if the
communications with the process are via a PTY, at least if the
sub-process is a real process (as opposed to a network or serial or
pipe process).  Or maybe we should only refrain from sending EOF on
MS-Windows?

Comments?

Re: Sending EOF to process as part of comint-simple-send

[Jim Porter]

On 1/14/2023 4:18 AM, Eli Zaretskii wrote:
> In comint-simple-send, we have:
> 
>      (comint-send-string proc send-string))
>    (if (and comint-input-sender-no-newline
> 	   (not (string-equal string "")))
>        (process-send-eof)))
> 
> Thus, if we send sub-process input without a newline, we then send EOF
> to the sub-process, except when the string we send is empty.
> 
> I can only understand this logic if it assumes a Posix shell or other
> Posix process with which we communicate via a PTY.  Because in that
> case, we just send C-d to the process, and AFAIK that will not cause
> the sub-process to finish except when C-d is the only character we
> send.
[snip]
> So I think comint-simple-send should only send EOF if the
> communications with the process are via a PTY, at least if the
> sub-process is a real process (as opposed to a network or serial or
> pipe process).  Or maybe we should only refrain from sending EOF on
> MS-Windows?

I agree with this: I think we should only call 'process-send-eof' here 
when communicating via PTY. (To be precise, whenever the process's 
*stdin* is a PTY; as of Emacs 29, a process's stdin can be a PTY even if 
its stdout isn't, or vice versa. See commit d7b89ea407.)

I imagine you've looked into the relevant POSIX specs already, but just 
to explain my reasoning for why I agree... my reading of the POSIX 
spec[1] for EOF is that the code above is using it as a way to flush the 
I/O buffer for the PTY, which makes sense to me:

> EOF: Special character on input, which is recognized if the
> ICANON flag is set. When received, all the bytes waiting to be
> read are immediately passed to the process without waiting for a
> <newline>, and the EOF is discarded. Thus, if there are no bytes
> waiting (that is, the EOF occurred at the beginning of a line), a
> byte count of zero shall be returned from the read(),
> representing an end-of-file indication. If ICANON is set, the EOF
> character shall be discarded when processed.

However, that logic would only apply for PTYs, since a) this 
documentation is about the POSIX terminal interface, and b) as you said 
in your message, 'process-send-eof' doesn't actually send an EOF 
character when communicating via a pipe at all; it closes the file 
descriptor!

[1] 
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html#tag_11_01_09

Re: Sending EOF to process as part of comint-simple-send

[Eli Zaretskii]

> Date: Sat, 14 Jan 2023 21:27:56 -0800
> From: Jim Porter <jporterbugs@gmail.com>
> 
> > So I think comint-simple-send should only send EOF if the
> > communications with the process are via a PTY, at least if the
> > sub-process is a real process (as opposed to a network or serial or
> > pipe process).  Or maybe we should only refrain from sending EOF on
> > MS-Windows?
> 
> I agree with this: I think we should only call 'process-send-eof' here 
> when communicating via PTY. (To be precise, whenever the process's 
> *stdin* is a PTY; as of Emacs 29, a process's stdin can be a PTY even if 
> its stdout isn't, or vice versa. See commit d7b89ea407.)

Yes, I know.  The function process-tty-name can be asked to return
information about a specific stream.

> I imagine you've looked into the relevant POSIX specs already, but just 
> to explain my reasoning for why I agree... my reading of the POSIX 
> spec[1] for EOF is that the code above is using it as a way to flush the 
> I/O buffer for the PTY, which makes sense to me:

Right.  This doesn't work for pipes, so I intend to introduce a new
function that does fsync on the descriptor via which we write to the
sub-process, and call it there instead of process-send-eof.  Does this
make sense?

Re: Sending EOF to process as part of comint-simple-send

[Andreas Schwab]

On Jan 15 2023, Eli Zaretskii wrote:

> Right.  This doesn't work for pipes, so I intend to introduce a new
> function that does fsync on the descriptor via which we write to the
> sub-process, and call it there instead of process-send-eof.  Does this
> make sense?

You will probably get EINVAL.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sending EOF to process as part of comint-simple-send

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Jim Porter <jporterbugs@gmail.com>,  emacs-devel@gnu.org
> Date: Sun, 15 Jan 2023 09:40:09 +0100
> 
> On Jan 15 2023, Eli Zaretskii wrote:
> 
> > Right.  This doesn't work for pipes, so I intend to introduce a new
> > function that does fsync on the descriptor via which we write to the
> > sub-process, and call it there instead of process-send-eof.  Does this
> > make sense?
> 
> You will probably get EINVAL.

And the stuff will not be flushed?  Is there a better way of doing
that for pipe file descriptors?

Re: Sending EOF to process as part of comint-simple-send

[Andreas Schwab]

On Jan 15 2023, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Jim Porter <jporterbugs@gmail.com>,  emacs-devel@gnu.org
>> Date: Sun, 15 Jan 2023 09:40:09 +0100
>> 
>> On Jan 15 2023, Eli Zaretskii wrote:
>> 
>> > Right.  This doesn't work for pipes, so I intend to introduce a new
>> > function that does fsync on the descriptor via which we write to the
>> > sub-process, and call it there instead of process-send-eof.  Does this
>> > make sense?
>> 
>> You will probably get EINVAL.
>
> And the stuff will not be flushed?

There is nothing to flush.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sending EOF to process as part of comint-simple-send

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: jporterbugs@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 15 Jan 2023 11:08:29 +0100
> 
> On Jan 15 2023, Eli Zaretskii wrote:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: Jim Porter <jporterbugs@gmail.com>,  emacs-devel@gnu.org
> >> Date: Sun, 15 Jan 2023 09:40:09 +0100
> >> 
> >> On Jan 15 2023, Eli Zaretskii wrote:
> >> 
> >> > Right.  This doesn't work for pipes, so I intend to introduce a new
> >> > function that does fsync on the descriptor via which we write to the
> >> > sub-process, and call it there instead of process-send-eof.  Does this
> >> > make sense?
> >> 
> >> You will probably get EINVAL.
> >
> > And the stuff will not be flushed?
> 
> There is nothing to flush.

This is not useful.

I'm considering the following scenario:

  emacs -Q
  M-: (setq process-connection-type nil) RET
  M-x shell RET
  od -t x1 RET
  1000 C-c C-d

Instead of "C-c C-d" (which in the above scenario closes the
connection to the sub-process) I'm thinking of adding a command to
make sure the "1000" part is sent to 'od' without a newline.  Are you
saying that this will happen automatically on Posix systems?  Or are
you saying that there's no way to do that when communicating with
sub-processes via pipes?  Or are you saying something else?

Re: Sending EOF to process as part of comint-simple-send

[Andreas Schwab]

On Jan 15 2023, Eli Zaretskii wrote:

> Are you saying that this will happen automatically on Posix systems?

Sure.  What you write to the file descriptor will be immediately
available for read on the other side of the pipe.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sending EOF to process as part of comint-simple-send

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: jporterbugs@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 15 Jan 2023 11:32:22 +0100
> 
> On Jan 15 2023, Eli Zaretskii wrote:
> 
> > Are you saying that this will happen automatically on Posix systems?
> 
> Sure.  What you write to the file descriptor will be immediately
> available for read on the other side of the pipe.

Thanks.