asynchronous process management on WinWP; buffering problem?

asynchronous process management on WinWP; buffering problem?

[Peter Tury]

Hi,

I try to control program xxx on MS WinXP from EmacsW32 (cvs). xxx runs in
DOS window normally and after a command it reads from stdin (until
newline), handles the input and, after another command, writes result to
stdout.

However it seems that some buffering (in Emacs? in MS Windows'
stdin/stdout?) tries to kill me: I can't see the results (from stdout) in
the associated buffer, only if it is long enough. At least it seems so: if
I send many "\n"-s (to "generate" a lot of prompts (to fill up the
buffer?)) then I get everything together after a while (of course, if I run
the program normally (=in a DOS window) everything works fine)... I hope I
could handle this through filter functions, but...

...my bigger problem is that it seems that some (too long?) data cannot be
sent correctly.

My question: do you know some easy "tricks" how to log/save/loopback/...
every bytes sent by Emacs to (the stdin for) an async. process? xxx doesn't
writes them back to stdout, so I can't see now what is sent exactly, but I
see the error message -- what shouldn't happen, since the input I send from
Emacs is good?

My problematic input e.g.: (process-send-string h248-process "43 6F 6E 74
65 78 74 20 3D 20 24 20 7B 20 50 72 69 6F 72 69 74 79 20 3D 20 31 35 2C 20
41 64 64 20 3D 20 24 20 7B 20 4D 65 64 69 61 20 7B 20 4C 6F 63 61 6C 43 6F
6E 74 72 6F 6C 20 7B 20 4D 6F 64 65 20 3D 20 53 65 6E 64 52 65 63 65 69 76
65 2C 20 52 65 73 65 72 76 65 64 56 61 6C 75 65 20 3D 20 4F 46 46 2C 20 52
65 73 65 72 76 65 64 47 72 6F 75 70 20 3D 20 4F 46 46 2C 20 62 63 70 2F 42
4E 43 43 68 61 72 20 3D 20 41 61 6C 32 20 2C 20 74 68 72 65 65 67 75 70 2F
6D 6F 64 65 20 3D 20 53 75 70 70 20 2C 20 74 68 72 65 65 67 75 70 2F 75 70
76 65 72 73 69 6F 6E 73 20 3D 20 32 20 2C 20 74 68 72 65 65 67 75 70 2F 64
65 6C 65 72 72 73 64 75 20 3D 20 59 65 73 20 2C 20 74 68 72 65 65 67 75 70
2F 69 6E 74 65 72 66 61 63 65 20 3D 20 43 4E 20 2C 20 74 68 72 65 65 67 75
70 2F 69 6E 69 74 64 69 72 20 3D 20 49 6E 20 20 7D 20 2C 20 4C 6F 63 61 6C
20 7B 20 6D 3D 61 75 64 69 6F 20 2D 20 2D 20 2D 63 3D 41 54 4D 20 4E 53 41
50 20 24 61 3D 76 73 65 6C 3A 55 4D 54 53 2D 41 4D 52 20 2D 20 2D 20 61 3D
63 6F 64 65 63 63 6F 6E 66 69 67 3A 30 32 30 35 41 35 41 35 30 34 61 3D 65
65 63 69 64 3A 24 0A 20 7D 20 20 7D 20 20 7D 20 7D 20 \n\n")

I tried both pipe and pty process-connection-type.

It seems as if somewhere (before the real end) a "\n" would be sent (what
stops reading of the stdin in xxx, thus the got data (=the "first half") is
obviuosly incorrect). Can some automatic coding cause this problem (I see
that 0A="\n" is sent, but 0A there should be a simple string of characters
"0" and "A".)

If I send EOF explicitly (with (process-send-eof h248-process)), it messes
up xxx: it start to write an error message in an infinite(?) loop, so I
have to delete-process then. Thus I think Emacs doesn't send EOF hiddenly
during the sending of the above, normal input (even if it is longer than
500 bytes) -- how can I check this?

Thanks!!
P

Re: asynchronous process management on WinWP; buffering problem?

[kgold]

The output buffering is a common problem.  For C and perhaps other 
languages, stdout is normally line buffered.  But when you pipe the 
output (like through emacs) it becomes fully buffered.  So you don't see 
the output until the buffer is full.

I don't know if there's a general solution.  If it's C and you have the 
source, this will fix it.

	setvbuf(stdout, _IONBF, 0);

Peter Tury wrote:
> I try to control program xxx on MS WinXP from EmacsW32 (cvs). xxx runs in
> DOS window normally and after a command it reads from stdin (until
> newline), handles the input and, after another command, writes result to
> stdout.
> 
> However it seems that some buffering (in Emacs? in MS Windows'
> stdin/stdout?) tries to kill me: I can't see the results (from stdout) in
> the associated buffer, only if it is long enough. At least it seems so: if
> I send many "\n"-s (to "generate" a lot of prompts (to fill up the
> buffer?)) then I get everything together after a while (of course, if I run
> the program normally (=in a DOS window) everything works fine)... I hope I
> could handle this through filter functions, but...

Re: asynchronous process management on WinWP; buffering problem?

[Lennart Borgman]

I do not understand this so I have some (perhaps stupid) questions:

- What determines how the buffering works?
- Can a program like Emacs change this behaviour?
- When does the buffering works differently? In a "console"? Is cmd.exe 
a "console"? But is it not just another program?


kgold wrote:
> The output buffering is a common problem.  For C and perhaps other 
> languages, stdout is normally line buffered.  But when you pipe the 
> output (like through emacs) it becomes fully buffered.  So you don't 
> see the output until the buffer is full.
>
> I don't know if there's a general solution.  If it's C and you have 
> the source, this will fix it.
>
>     setvbuf(stdout, _IONBF, 0);
>
> Peter Tury wrote:
>> I try to control program xxx on MS WinXP from EmacsW32 (cvs). xxx 
>> runs in
>> DOS window normally and after a command it reads from stdin (until
>> newline), handles the input and, after another command, writes result to
>> stdout.
>>
>> However it seems that some buffering (in Emacs? in MS Windows'
>> stdin/stdout?) tries to kill me: I can't see the results (from 
>> stdout) in
>> the associated buffer, only if it is long enough. At least it seems 
>> so: if
>> I send many "\n"-s (to "generate" a lot of prompts (to fill up the
>> buffer?)) then I get everything together after a while (of course, if 
>> I run
>> the program normally (=in a DOS window) everything works fine)... I 
>> hope I
>> could handle this through filter functions, but...

Re: asynchronous process management on WinWP; buffering problem?

[Miles Bader]

Lennart Borgman <lennart.borgman.073@student.lu.se> writes:
> I do not understand this so I have some (perhaps stupid) questions:
>
> - What determines how the buffering works?
> - Can a program like Emacs change this behaviour?
> - When does the buffering works differently? In a "console"? Is cmd.exe
> a "console"? But is it not just another program?

In unix-like systems, typically a program will test whether the output
stream (stdout) is a terminal or not using something like the
"isatty(1)" function (which is a POSIX standard function I believe).
If it's a terminal it will use line-buffering, if not, it will use a
larger fixed buffer size (which is more efficient).

-Miles
-- 
[|nurgle|]  ddt- demonic? so quake will have an evil kinda setting? one that
            will  make every christian in the world foamm at the mouth?
[iddt]      nurg, that's the goal

Re: asynchronous process management on WinWP; buffering problem?

[Lennart Borgman]

Miles Bader wrote:
> Lennart Borgman <lennart.borgman.073@student.lu.se> writes:
>   
>> I do not understand this so I have some (perhaps stupid) questions:
>>
>> - What determines how the buffering works?
>> - Can a program like Emacs change this behaviour?
>> - When does the buffering works differently? In a "console"? Is cmd.exe
>> a "console"? But is it not just another program?
>>     
>
> In unix-like systems, typically a program will test whether the output
> stream (stdout) is a terminal or not using something like the
> "isatty(1)" function (which is a POSIX standard function I believe).
> If it's a terminal it will use line-buffering, if not, it will use a
> larger fixed buffer size (which is more efficient).
>
> -Miles
>   

Jason wrote:

  I think Emacs can affect the buffering of programs that it invokes
  directly.  In my experience, the buffering problems normally start
  when a program is run through some other program, the classic example
  being ssh being run by cvs and the password prompt not appearing in
  Emacs until you kill cvs. I don't think anything Emacs does can fix
  this.

Thanks Miles and Jason!

Can Emacs then tell _isatty that it is a terminal? Or what does Emacs do 
to handle this?


Note: It looks like the POSIX name now is _isatty.

Re: asynchronous process management on WinWP; buffering problem?

[Jason Rumney]

Lennart Borgman <lennart.borgman.073@student.lu.se> writes:

> Can Emacs then tell _isatty that it is a terminal? Or what does Emacs
> do to handle this?
>
>
> Note: It looks like the POSIX name now is _isatty.

Probably not on Windows, since windows isn't POSIX and any POSIX layer
that MingW32 has is unlikely to add functionality that is not already
possible through other Windows APIs. On Windows stdout uses completely
different API calls than console I/O.

Re: asynchronous process management on WinWP; buffering problem?

[Lennart Borgman]

Jason Rumney wrote:
> Lennart Borgman <lennart.borgman.073@student.lu.se> writes:
>
>   
>> Can Emacs then tell _isatty that it is a terminal? Or what does Emacs
>> do to handle this?
>>
>>
>> Note: It looks like the POSIX name now is _isatty.
>>     
>
> Probably not on Windows, since windows isn't POSIX and any POSIX layer
> that MingW32 has is unlikely to add functionality that is not already
> possible through other Windows APIs. On Windows stdout uses completely
> different API calls than console I/O.
>   
It actually seems like is _isatty is defined in MinGW. I just tested a 
small program I found here:

    http://permalink.gmane.org/gmane.comp.gnu.mingw.msys/448

    #include <stdio.h>

    int main(int argc, char *argv[])
    {
      if (isatty(fileno(stdin)) && _isatty(fileno(stdout)))
        printf("interactive\n");
      else
        printf("batch\n");
      return 0;
    }

If I run this program in cmd.exe it says "interactive". I get the same 
result under MSYS and Cygwin. However running it in Emacs in shell:

    M-x shell

it instead says "batch". (This is in the CVS version of Emacs.) Is this 
what is expected?

Re: asynchronous process management on WinWP; buffering problem?

[Benjamin Riefenstahl]

Hi Lennart,


Lennart Borgman writes:
> It actually seems like is _isatty is defined in MinGW. I just tested
> a small program I found here:

The function isatty is defined in the Microsoft C runtime, and that is
what Mingw uses.  This version of isatty determines if a program is
running with stdin/stdout connected to an MS Windows console.

The way that consoles are implemented and used in MS Windows makes it
very complicated to use the console API to simulate a terminal in
Emacs.  On Unix-like systems this is done by so-called "pseudo-ttys",
"pty", but the MS Windows consoles are quite different beasts at the
OS API level.  Emacs would have to do a large amount of
reverse-engineering of the intent of the child process, to do with
consoles what Emacs does quite naturally on Unix with pseudo-ttys, and
there would still be features missing.  Therefore Emacs just uses
anonymous pipes instead which are a much better fit for the way Emacs
treats child processes, but that means that it can't make isatty in
child processes return true.

Cygwin can implement pseudo-ttys if it controls the parent as well as
the child, e.g. when you call a Cygwin program from a Cygwin compiled
Emacs.  It uses its own conventions to do that which are not
understood by non-Cygwin programs.


Sometimes you have a program that uses isatty to determine if it
should work in interactive mode or not, like the command shells of a
programming language or the some SQL interpreters.  In those cases you
will need to find a way put it into interactive mode without the help
of isatty.  Sometimes the shell will have a command-line option for
that, in other cases it can be configured after its start.


benny

Re: asynchronous process management on WinWP; buffering problem?

[Lennart Borgman]

Hi Benny,

Thanks for the good explanation. Indeed it looks difficult to fix this 
on the w32 platform. In my opinion w32 does not look very much POSIX 
conformant here. Or the POSIX specs are just too weak.


Benjamin Riefenstahl wrote:
> The function isatty is defined in the Microsoft C runtime, and that is
> what Mingw uses.  This version of isatty determines if a program is
> running with stdin/stdout connected to an MS Windows console.
>
> The way that consoles are implemented and used in MS Windows makes it
> very complicated to use the console API to simulate a terminal in
> Emacs.  On Unix-like systems this is done by so-called "pseudo-ttys",
> "pty", but the MS Windows consoles are quite different beasts at the
> OS API level.  Emacs would have to do a large amount of
> reverse-engineering of the intent of the child process, to do with
> consoles what Emacs does quite naturally on Unix with pseudo-ttys, and
> there would still be features missing.  Therefore Emacs just uses
> anonymous pipes instead which are a much better fit for the way Emacs
> treats child processes, but that means that it can't make isatty in
> child processes return true.
>
> Cygwin can implement pseudo-ttys if it controls the parent as well as
> the child, e.g. when you call a Cygwin program from a Cygwin compiled
> Emacs.  It uses its own conventions to do that which are not
> understood by non-Cygwin programs.
>
>
> Sometimes you have a program that uses isatty to determine if it
> should work in interactive mode or not, like the command shells of a
> programming language or the some SQL interpreters.  In those cases you
> will need to find a way put it into interactive mode without the help
> of isatty.  Sometimes the shell will have a command-line option for
> that, in other cases it can be configured after its start.
>
>
> benny
>   

Re: asynchronous process management on WinWP; buffering problem?

[Benjamin Riefenstahl]

Hi Lennart, 


Lennart Borgman writes:
> In my opinion w32 does not look very much POSIX conformant here. Or
> the POSIX specs are just too weak.

POSIX specifies behaviour of Unix-like systems.  MS Windows doesn't
care about POSIX and Microsoft never said that it wanted to be
POSIX-compatible.

Don't confuse this with NT's POSIX subsystem.  This is a separate OS
running on top of the same NT kernel in parallel to MS Windows.  While
this is nominally a complete POSIX system, it used to be lacking some
stuff that is usually considered essential by Unix hackers, but not
specified or (officially) optional in POSIX.  Also Microsoft's support
for this is wavering, e.g. it is not supported on XP.


benny

Re: asynchronous process management on WinWP; buffering problem?

[Lennart Borgman]

Benjamin Riefenstahl wrote:
> Hi Lennart, 
>
>
> Lennart Borgman writes:
>   
>> In my opinion w32 does not look very much POSIX conformant here. Or
>> the POSIX specs are just too weak.
>>     
>
> POSIX specifies behaviour of Unix-like systems.  MS Windows doesn't
> care about POSIX and Microsoft never said that it wanted to be
> POSIX-compatible.
>
> Don't confuse this with NT's POSIX subsystem.  This is a separate OS
> running on top of the same NT kernel in parallel to MS Windows.  While
> this is nominally a complete POSIX system, it used to be lacking some
> stuff that is usually considered essential by Unix hackers, but not
> specified or (officially) optional in POSIX.  Also Microsoft's support
> for this is wavering, e.g. it is not supported on XP.
>
>
> benny
>   
Hi Benny,

Thanks for the clarifications. What made me upset before was that MS 
claims that the _isatty API is POSIX compliant while MS does not give a 
way for programs (mostly shells I guess) to tell they are tty:s. (I 
guess my words here are much to inexact, but I hope you understand what 
I mean).

I mean that if they can get away with that then it seems like the 
specifications of _isatty is too weak.

Re: asynchronous process management on WinWP; buffering problem?

[Miles Bader]

Lennart Borgman <lennart.borgman.073@student.lu.se> writes:
> Thanks for the good explanation. Indeed it looks difficult to fix this
> on the w32 platform. In my opinion w32 does not look very much POSIX
> conformant here. Or the POSIX specs are just too weak.

I don't know if ptys are actually part of POSIX, but ... has w32 every
_claimed_ to be "POSIX conformant" (I mean, other than in the "deceive
the government" sense)?

-Miles
-- 
Run away!  Run away!

Re: asynchronous process management on WinWP; buffering problem?

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Sun, 16 Apr 2006 09:21:04 +0900
> 
> has w32 every _claimed_ to be "POSIX conformant"

No.

MS offers their own ``Posix compatibility layer'' on top of Windows
for those who need Posix conformance.

Re: asynchronous process management on WinWP; buffering problem?

[Miles Bader]

Lennart Borgman <lennart.borgman.073@student.lu.se> writes:
> If I run this program in cmd.exe it says "interactive". I get the same
> result under MSYS and Cygwin. However running it in Emacs in shell:
>
>    M-x shell
>
> it instead says "batch". (This is in the CVS version of Emacs.) Is this
> what is expected?

I don't use windows, so I can't give an informed answer, but my guess is
that perhaps the windows environment lacks ptys (pseudo-ttys)?

When Emacs in unix opens a shell session, it uses a pty to talk to the
inferior shell.  A pty appears to the shell to be a normal terminal, but
is actually controlled by Emacs.  When Emacs _can't_ create a pty for
some reason (e.g. in unix you can run out of ptys), it falls back to
using a normal pipe to talk to the subshell; this works mostly, but will
_not_ appear like a terminal to the shell.

-Miles
-- 
We live, as we dream -- alone....

Re: asynchronous process management on WinWP; buffering problem?

[Jason Rumney]

Lennart Borgman <lennart.borgman.073@student.lu.se> writes:

> I do not understand this so I have some (perhaps stupid) questions:
>
> - What determines how the buffering works?
> - Can a program like Emacs change this behaviour?
> - When does the buffering works differently? In a "console"? Is
> cmd.exe a "console"? But is it not just another program?

I think Emacs can affect the buffering of programs that it invokes
directly.  In my experience, the buffering problems normally start
when a program is run through some other program, the classic example
being ssh being run by cvs and the password prompt not appearing in
Emacs until you kill cvs. I don't think anything Emacs does can fix
this.

Re: asynchronous process management on WinWP; buffering problem?

[Peter Tury]

Hi,

thanks for all of your answers! They are pretty interesting and I am sure I
can learn a lot from them. Really.

However, unfortunately my real problem is not with buffering (I think)
(maybe my subject is not so good). "I hope I could handle this through
filter functions." Instead "my bigger problem is that it seems that some
(too long?) data cannot be sent correctly.
 
My question: do you know some easy "tricks" how to log/save/loopback/...
every bytes sent by Emacs to (the stdin for) an async. process?"

Please help me in this also. So: how can I check what bytes exactly arrive
to the stdin for a process. I don't have the sources of the DOS program in
question and I didn't want to start writing a C DOS program for it.

Thanks!!
P

On Fri, 14 Apr 2006 07:14:54 GMT, Peter Tury wrote:

> Hi,
> 
> I try to control program xxx on MS WinXP from EmacsW32 (cvs). xxx runs in
> DOS window normally and after a command it reads from stdin (until
> newline), handles the input and, after another command, writes result to
> stdout.
> 
> However it seems that some buffering (in Emacs? in MS Windows'
> stdin/stdout?) tries to kill me: I can't see the results (from stdout) in
> the associated buffer, only if it is long enough. At least it seems so: if
> I send many "\n"-s (to "generate" a lot of prompts (to fill up the
> buffer?)) then I get everything together after a while (of course, if I run
> the program normally (=in a DOS window) everything works fine)... I hope I
> could handle this through filter functions, but...
> 
> ...my bigger problem is that it seems that some (too long?) data cannot be
> sent correctly.
> 
> My question: do you know some easy "tricks" how to log/save/loopback/...
> every bytes sent by Emacs to (the stdin for) an async. process? xxx doesn't
> writes them back to stdout, so I can't see now what is sent exactly, but I
> see the error message -- what shouldn't happen, since the input I send from
> Emacs is good?
> 
> My problematic input e.g.: (process-send-string h248-process "43 6F 6E 74
> 65 78 74 20 3D 20 24 20 7B 20 50 72 69 6F 72 69 74 79 20 3D 20 31 35 2C 20
> 41 64 64 20 3D 20 24 20 7B 20 4D 65 64 69 61 20 7B 20 4C 6F 63 61 6C 43 6F
> 6E 74 72 6F 6C 20 7B 20 4D 6F 64 65 20 3D 20 53 65 6E 64 52 65 63 65 69 76
> 65 2C 20 52 65 73 65 72 76 65 64 56 61 6C 75 65 20 3D 20 4F 46 46 2C 20 52
> 65 73 65 72 76 65 64 47 72 6F 75 70 20 3D 20 4F 46 46 2C 20 62 63 70 2F 42
> 4E 43 43 68 61 72 20 3D 20 41 61 6C 32 20 2C 20 74 68 72 65 65 67 75 70 2F
> 6D 6F 64 65 20 3D 20 53 75 70 70 20 2C 20 74 68 72 65 65 67 75 70 2F 75 70
> 76 65 72 73 69 6F 6E 73 20 3D 20 32 20 2C 20 74 68 72 65 65 67 75 70 2F 64
> 65 6C 65 72 72 73 64 75 20 3D 20 59 65 73 20 2C 20 74 68 72 65 65 67 75 70
> 2F 69 6E 74 65 72 66 61 63 65 20 3D 20 43 4E 20 2C 20 74 68 72 65 65 67 75
> 70 2F 69 6E 69 74 64 69 72 20 3D 20 49 6E 20 20 7D 20 2C 20 4C 6F 63 61 6C
> 20 7B 20 6D 3D 61 75 64 69 6F 20 2D 20 2D 20 2D 63 3D 41 54 4D 20 4E 53 41
> 50 20 24 61 3D 76 73 65 6C 3A 55 4D 54 53 2D 41 4D 52 20 2D 20 2D 20 61 3D
> 63 6F 64 65 63 63 6F 6E 66 69 67 3A 30 32 30 35 41 35 41 35 30 34 61 3D 65
> 65 63 69 64 3A 24 0A 20 7D 20 20 7D 20 20 7D 20 7D 20 \n\n")
> 
> I tried both pipe and pty process-connection-type.
> 
> It seems as if somewhere (before the real end) a "\n" would be sent (what
> stops reading of the stdin in xxx, thus the got data (=the "first half") is
> obviuosly incorrect). Can some automatic coding cause this problem (I see
> that 0A="\n" is sent, but 0A there should be a simple string of characters
> "0" and "A".)
> 
> If I send EOF explicitly (with (process-send-eof h248-process)), it messes
> up xxx: it start to write an error message in an infinite(?) loop, so I
> have to delete-process then. Thus I think Emacs doesn't send EOF hiddenly
> during the sending of the above, normal input (even if it is longer than
> 500 bytes) -- how can I check this?
> 
> Thanks!!
> P

Re: asynchronous process management on WinWP; buffering problem?

[Peter Tury]

On Sat, 15 Apr 2006 20:56:11 GMT, Peter Tury wrote:

> Hi,
> 
> thanks for all of your answers! They are pretty interesting and I am sure I
> can learn a lot from them. Really.
> 
> However, unfortunately my real problem is not with buffering (I think)
> (maybe my subject is not so good). "I hope I could handle this through
> filter functions." Instead "my bigger problem is that it seems that some
> (too long?) data cannot be sent correctly.
>  
> My question: do you know some easy "tricks" how to log/save/loopback/...
> every bytes sent by Emacs to (the stdin for) an async. process?"
> 
> Please help me in this also. So: how can I check what bytes exactly arrive
> to the stdin for a process. I don't have the sources of the DOS program in
> question and I didn't want to start writing a C DOS program for it.

Well, finally I managed to get the sources of the DOS program + write my
own loopback.c for testing and found the bug in the DOS program -- it is
unrealted to Emacs ;-)

((The problem was it uses fgets to read a fixed amount of characters (from
stdin) what causes in some cases the reading of the "first half" of some
related data, and this "first half" is checked immediately, while its
"second half" is missing (=still not read by the next fgets)... and
executions stops when the incorrectness of the "first half" is
revealed...))

Thanks for all of your help,
P