How to flush output to process

How to flush output to process

[Frank Fischer]

Hi,

I have a question about communication with processes. I want to start
an external program from Emacs Lisp in a process (using
`start-file-process'). This program accepts input data from stdin and
writes its results to stdout. After sending data to the process using
`process-send-string' nothing happens if the data does not end in a
newline character. The problem is that in general I cannot end every
request I send to the process with a newline (a constraint of the
communication protocol) and using `process-send-eof' is also not an
option because this seems to close the connection completely (I want
to communicate with the process in several turns).

In something like C I would use fflush(...) to force the buffered data
to be sent to the process (and it works), but I have not been able to
find some comparable function for Emacs Lisp. 

Is there a way the flush the Emacs->process stream, i.e., force all
buffered data to be actually send to the process, from Emacs Lisp?

Thanks in advance,
Frank

Re: How to flush output to process

[Eli Zaretskii]

> From: Frank Fischer <frank-fischer@shadow-soft.de>
> Date: Sun, 6 Nov 2011 09:53:39 +0000 (UTC)
> 
> In something like C I would use fflush(...) to force the buffered data
> to be sent to the process (and it works), but I have not been able to
> find some comparable function for Emacs Lisp. 
> 
> Is there a way the flush the Emacs->process stream, i.e., force all
> buffered data to be actually send to the process, from Emacs Lisp?

I'm guessing that your analysis of the problem (and therefore the
solution you are seeking) is incorrect.  I think it's the sub-process
who is buffering input until it sees a newline.

To fix this, make sure your subprocess reads its stdin in raw mode,
not in cooked mode.

Re: How to flush output to process

[Frank Fischer]

Am Sun, 06 Nov 2011 06:07:17 -0500
schrieb Eli Zaretskii <eliz@gnu.org>:

> > From: Frank Fischer <frank-fischer@shadow-soft.de>
> > Date: Sun, 6 Nov 2011 09:53:39 +0000 (UTC)
> > 
> > In something like C I would use fflush(...) to force the buffered
> > data to be sent to the process (and it works), but I have not been
> > able to find some comparable function for Emacs Lisp. 
> > 
> > Is there a way the flush the Emacs->process stream, i.e., force all
> > buffered data to be actually send to the process, from Emacs Lisp?
> 
> I'm guessing that your analysis of the problem (and therefore the
> solution you are seeking) is incorrect.  I think it's the sub-process
> who is buffering input until it sees a newline.
> 
> To fix this, make sure your subprocess reads its stdin in raw mode,
> not in cooked mode.

Well, I cannot change the subprocess' code. But even if it is a problem
of the subprocess, why does it work if I communicate with the
subprocess from some other program, i.e., not starting the process from
Emacs but from some other programming language, AND call the flush
function of the corresponding stream?

Btw, I would be surprised if it is indeed a problem of the subprocess
as it is designed to be used in this way.

Frank

Re: How to flush output to process

[Thien-Thi Nguyen]

() Frank Fischer <frank-fischer@shadow-soft.de>
() Sun, 6 Nov 2011 12:38:20 +0100

   Btw, I would be surprised if it is indeed a problem of the
   subprocess as it is designed to be used in this way.

If the subprocess is comfortable ignoring signals from Emacs,
you might try using pipes instead of ptys to talk to it.
That would probably eliminate raw/cooked misunderstandings.

See ‘process-connection-type’ (info "(elisp) Asynchronous Processes").

Re: How to flush output to process

[Frank Fischer]

Am Sun, 06 Nov 2011 22:08:32 +0100
schrieb Thien-Thi Nguyen <ttn@gnuvola.org>:

> () Frank Fischer <frank-fischer@shadow-soft.de>
> () Sun, 6 Nov 2011 12:38:20 +0100
> 
>    Btw, I would be surprised if it is indeed a problem of the
>    subprocess as it is designed to be used in this way.
> 
> If the subprocess is comfortable ignoring signals from Emacs,
> you might try using pipes instead of ptys to talk to it.
> That would probably eliminate raw/cooked misunderstandings.
> 
> See ‘process-connection-type’ (info "(elisp) Asynchronous Processes").

I do use pipes, but the problem remains.

Frank

Re: How to flush output to process

[Thien-Thi Nguyen]

() Frank Fischer <frank-fischer@shadow-soft.de>
() Sun, 6 Nov 2011 22:56:07 +0100

   I do use pipes, but the problem remains.

Perhaps you need to specify the ‘no-conversion’ coding system by frobbing
‘coding-system-for-write’ (info "(elisp) Specifying Coding Systems"), or
something along those lines.

Re: How to flush output to process

[Frank Fischer]

Am Mon, 07 Nov 2011 09:49:52 +0100
schrieb Thien-Thi Nguyen <ttn@gnuvola.org>:

> () Frank Fischer <frank-fischer@shadow-soft.de>
> () Sun, 6 Nov 2011 22:56:07 +0100
> 
>    I do use pipes, but the problem remains.
> 
> Perhaps you need to specify the ‘no-conversion’ coding system by
> frobbing ‘coding-system-for-write’ (info "(elisp) Specifying Coding
> Systems"), or something along those lines.
> 

I tried it again today and now it work - no idea what I did differently
last time. Thank you for your advices and sorry for bothering.

Frank

Re: How to flush output to process

[Thien-Thi Nguyen]

() Frank Fischer <frank-fischer@shadow-soft.de>
() Mon, 7 Nov 2011 20:29:15 +0100

   I tried it again today and now it work - no idea what I did differently
   last time. Thank you for your advices and sorry for bothering.

I used to run my computer 24x7, with the consequence that Emacs uptime was
many-monthed.  Being conscientious, i made sure to sync changes in *scratch*
w/ (what i presumed to be) analogous changes in ~/.emacs.  Then one fateful
day, i typed ‘C-x C-c’ by mistake.  Hundreds of buffers gone.  Bummer.

Moment of truth: time to restart Emacs.  You can guess the outcome: errors
(multiple) in ~/.emacs, errors (multiple) in filenames and other linkages,
errors errors everywhere.  Major bummer.

So now i know being conscientious is not something to be so proud of.  :-D

Re: How to flush output to process

[Frank Fischer]

On 2011-11-06, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () Frank Fischer <frank-fischer@shadow-soft.de>
> () Sun, 6 Nov 2011 12:38:20 +0100
>
>    Btw, I would be surprised if it is indeed a problem of the
>    subprocess as it is designed to be used in this way.
>
> If the subprocess is comfortable ignoring signals from Emacs,
> you might try using pipes instead of ptys to talk to it.
> That would probably eliminate raw/cooked misunderstandings.
>
> See ‘process-connection-type’ (info "(elisp) Asynchronous Processes").

Using pipes seems to work locally. But when starting the process using
tramp a pty has to be used. Any ideas what to do in this case?

Frank

Re: How to flush output to process

[Helmut Eller]

* Frank Fischer [2011-11-06 11:38] writes:

> Am Sun, 06 Nov 2011 06:07:17 -0500
> schrieb Eli Zaretskii <eliz@gnu.org>:
>
>> > From: Frank Fischer <frank-fischer@shadow-soft.de>
>> > Date: Sun, 6 Nov 2011 09:53:39 +0000 (UTC)
>> > 
>> > In something like C I would use fflush(...) to force the buffered
>> > data to be sent to the process (and it works), but I have not been
>> > able to find some comparable function for Emacs Lisp. 
>> > 
>> > Is there a way the flush the Emacs->process stream, i.e., force all
>> > buffered data to be actually send to the process, from Emacs Lisp?
>> 
>> I'm guessing that your analysis of the problem (and therefore the
>> solution you are seeking) is incorrect.  I think it's the sub-process
>> who is buffering input until it sees a newline.
>> 
>> To fix this, make sure your subprocess reads its stdin in raw mode,
>> not in cooked mode.
>
> Well, I cannot change the subprocess' code. But even if it is a problem
> of the subprocess, why does it work if I communicate with the
> subprocess from some other program, i.e., not starting the process from
> Emacs but from some other programming language, AND call the flush
> function of the corresponding stream?

process-send-string does no buffering.

> Btw, I would be surprised if it is indeed a problem of the subprocess
> as it is designed to be used in this way.

Maybe it's an end-of-line conversion issue.  The subprocess may expect
\r\n on end-of-line.  What's the process-coding-system?

Helmut