How to communicate with a running external process with given PID?

How to communicate with a running external process with given PID?

[Thorsten Jolitz]

Hi List, 

say I want to call another program (more exactly, another Lisp program
that is not Emacs Lisp) from an Emacs Lisp program. 

My special requirement is that I don't want to start a new Emacs
subprocess, but want to communicate with an existing (running) process.
And there maybe several running instances of this program at the same
time, so I want to communicate with one existing (running) process with
a given PID.

Now there is splendid support in Emacs for communicating with other
programs, but always based on the assumption that Emacs starts and
controls a new subprocess. 

There is chapter 37.12 "Accessing Other Processes" in the Elisp manual,
and I can do successfully 

,-------------------------
| (process-attributes PID)
`-------------------------

to receive a lot of information about the process I want to communicate
with. 

But what then? Where are the (e.g.) `process-send-string' or
`process-send-region' functions for external processes I could use to
communicate with my external program?

I could run a server in the external lisp program and use a
network-connection-object to send http-requests (e.g. with the help of
emacs-request.el) via TCP - but that seems to be total overkill for my
requirements.

I just want to use a running external process with a given PID in a
similar way I would use (a)synchronous processes created from Emacs with
`call-process' or `start-process'. How can I do that?

Maybe there is an obvious answer to this question that I don't see.
Would be nice. 

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Hi List, 
>
> say I want to call another program (more exactly, another Lisp program
> that is not Emacs Lisp) from an Emacs Lisp program. 
>
> My special requirement is that I don't want to start a new Emacs
> subprocess, but want to communicate with an existing (running) process.
> And there maybe several running instances of this program at the same
> time, so I want to communicate with one existing (running) process with
> a given PID.
>
> Now there is splendid support in Emacs for communicating with other
> programs, but always based on the assumption that Emacs starts and
> controls a new subprocess. 
>
> There is chapter 37.12 "Accessing Other Processes" in the Elisp manual,
> and I can do successfully 
>
> ,-------------------------
> | (process-attributes PID)
> `-------------------------
>
> to receive a lot of information about the process I want to communicate
> with. 
>
> But what then? Where are the (e.g.) `process-send-string' or
> `process-send-region' functions for external processes I could use to
> communicate with my external program?
>
> I could run a server in the external lisp program and use a
> network-connection-object to send http-requests (e.g. with the help of
> emacs-request.el) via TCP - but that seems to be total overkill for my
> requirements.
>
> I just want to use a running external process with a given PID in a
> similar way I would use (a)synchronous processes created from Emacs with
> `call-process' or `start-process'. How can I do that?
>
> Maybe there is an obvious answer to this question that I don't see.
> Would be nice. 

You need to read UNP.
http://www.unpbook.com/

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Hi List, 
>
> say I want to call another program (more exactly, another Lisp program
> that is not Emacs Lisp) from an Emacs Lisp program. 
>
> My special requirement is that I don't want to start a new Emacs
> subprocess, but want to communicate with an existing (running) process.
> And there maybe several running instances of this program at the same
> time, so I want to communicate with one existing (running) process with
> a given PID.
>
> Now there is splendid support in Emacs for communicating with other
> programs, but always based on the assumption that Emacs starts and
> controls a new subprocess. 
>
> There is chapter 37.12 "Accessing Other Processes" in the Elisp manual,
> and I can do successfully 
>
> ,-------------------------
> | (process-attributes PID)
> `-------------------------
>
> to receive a lot of information about the process I want to communicate
> with. 
>
> But what then? Where are the (e.g.) `process-send-string' or
> `process-send-region' functions for external processes I could use to
> communicate with my external program?
>
> I could run a server in the external lisp program and use a
> network-connection-object to send http-requests (e.g. with the help of
> emacs-request.el) via TCP - but that seems to be total overkill for my
> requirements.
>
> I just want to use a running external process with a given PID in a
> similar way I would use (a)synchronous processes created from Emacs with
> `call-process' or `start-process'. How can I do that?
>
> Maybe there is an obvious answer to this question that I don't see.
> Would be nice. 

Ok, I'll bit the bullet: the only communication channel available to
you, a-priori, knowing only the process ID, is the kill(2) syscall.
And you're lucky, you can send more than one bit of information with
each kill call!

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Thorsten Jolitz]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> Hi List, 
>>
>> say I want to call another program (more exactly, another Lisp program
>> that is not Emacs Lisp) from an Emacs Lisp program. 

[..snip..]

>> I could run a server in the external lisp program and use a
>> network-connection-object to send http-requests (e.g. with the help of
>> emacs-request.el) via TCP - but that seems to be total overkill for my
>> requirements.

> Ok, I'll bit the bullet: the only communication channel available to
> you, a-priori, knowing only the process ID, is the kill(2) syscall.
> And you're lucky, you can send more than one bit of information with
> each kill call!

Yes, I should read UNP, but actually I read quite a lot of stuff about
this topic, but reading about it and starting to use it is not the same
thing. And I did not know that 'kill is so useful and universal, only
used it for its core task so far.

But in the light of your answer I think I go for the HTTP client/server
solution, expecially since the server stuff is all there anyway. 

Thanks for the tips. 

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> "Pascal J. Bourguignon" <pjb@informatimago.com> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> Hi List, 
>>>
>>> say I want to call another program (more exactly, another Lisp program
>>> that is not Emacs Lisp) from an Emacs Lisp program. 
>
> [..snip..]
>
>>> I could run a server in the external lisp program and use a
>>> network-connection-object to send http-requests (e.g. with the help of
>>> emacs-request.el) via TCP - but that seems to be total overkill for my
>>> requirements.
>
>> Ok, I'll bit the bullet: the only communication channel available to
>> you, a-priori, knowing only the process ID, is the kill(2) syscall.
>> And you're lucky, you can send more than one bit of information with
>> each kill call!
>
> Yes, I should read UNP, but actually I read quite a lot of stuff about
> this topic, but reading about it and starting to use it is not the same
> thing. And I did not know that 'kill is so useful and universal, only
> used it for its core task so far.
>
> But in the light of your answer I think I go for the HTTP client/server
> solution, expecially since the server stuff is all there anyway. 
>
> Thanks for the tips. 

Ok, that's a starting point.

The thing is that you cannot talk to a process that's not ready to talk,
with some protocol.

The summary of unp, is that there are various IPC (Inter Process
Communication) mechanisms:

- signals (kill(2), signal(2)),
- semaphores (semctl, semget, semop, semtimedop)
- messages (msgctl, msgget, msgrct, msgsnd)
- shared memory (shmat, shmctl, shmdt, shmget)
- sockets (listen, accept, read, write, close)
- pipes (pipe, fork, read, write, close)
- named pipes (open, read, write close)
- files (open, read, write close) 

and the only mechanism that does something by default in the receiver is
signals (cf. signal(2) and signal(7)).  Now, of course, most signals are
ignored, and a few others only kill the receiving process by default
(hence the name of kill(2)), which is not much of a communication, but
that's something, contrarily to the other mechanisms, when the receiving
process doesn't do anything about them.


But since you're saying that the process includes a HTTP server, that
means you can open a TCP socket on the port it's listening to, and you
can communicate with the HTTP protocol.

In emacs, you can do that with "network processes":
http://www.gnu.org/software/emacs/manual/html_node/elisp/Network-Processes.html
http://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Thorsten Jolitz]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> "Pascal J. Bourguignon" <pjb@informatimago.com> writes:
>>
>>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>>
>>>> Hi List,
>>>>
>>>> say I want to call another program (more exactly, another Lisp
>>>> program that is not Emacs Lisp) from an Emacs Lisp program.
>>
>> [..snip..]
>>
>>>> I could run a server in the external lisp program and use a
>>>> network-connection-object to send http-requests (e.g. with the help
>>>> of emacs-request.el) via TCP - but that seems to be total overkill
>>>> for my requirements.
>>
>>> Ok, I'll bit the bullet: the only communication channel available to
>>> you, a-priori, knowing only the process ID, is the kill(2) syscall.
>>> And you're lucky, you can send more than one bit of information with
>>> each kill call!
>>
>> Yes, I should read UNP, but actually I read quite a lot of stuff
>> about this topic, but reading about it and starting to use it is not
>> the same thing. And I did not know that 'kill is so useful and
>> universal, only used it for its core task so far.
>>
>> But in the light of your answer I think I go for the HTTP
>> client/server solution, expecially since the server stuff is all
>> there anyway.
>>
>> Thanks for the tips.
>
> Ok, that's a starting point.
>
> The thing is that you cannot talk to a process that's not ready to
> talk, with some protocol.
>
> The summary of unp, is that there are various IPC (Inter Process
> Communication) mechanisms:
>
> - signals (kill(2), signal(2)), - semaphores (semctl, semget, semop,
> semtimedop) - messages (msgctl, msgget, msgrct, msgsnd) - shared
> memory (shmat, shmctl, shmdt, shmget) - sockets (listen, accept, read,
> write, close) - pipes (pipe, fork, read, write, close) - named pipes
> (open, read, write close) - files (open, read, write close)
>
> and the only mechanism that does something by default in the receiver
> is signals (cf. signal(2) and signal(7)). Now, of course, most signals
> are ignored, and a few others only kill the receiving process by
> default (hence the name of kill(2)), which is not much of a
> communication, but that's something, contrarily to the other
> mechanisms, when the receiving process doesn't do anything about them.
>
>
> But since you're saying that the process includes a HTTP server, that
> means you can open a TCP socket on the port it's listening to, and you
> can communicate with the HTTP protocol.
>
> In emacs, you can do that with "network processes":
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Network-Processes.html
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html

Not that I have not heard about this UNP stuff, I probably hoped that
Emacs has some kind of 'magic' functionality for arranging things behind
the scene ...

But HTTP is a good and actually not that difficult solution in this
case. Thanks for the links.

-- 
cheers,
 Thorsten

Re: How to communicate with a running external process with given PID?

[Aurélien Aptel]

If you just want to read and write stuff, HTTP is probably overkill
too. Use plain sockets.

Re: How to communicate with a running external process with given PID?

[Thorsten Jolitz]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> If you just want to read and write stuff, HTTP is probably overkill
> too. Use plain sockets.

While I figured out that in my case using HTTP would be just fine, I
would still be interested in the answer to my (a tad naive) original
question: how would you establish (in an Emacs Lisp program) plain
socket communication between Emacs and an already running external
process (of a Lisp dialect that is not Emacs Lisp) with given PID?

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with given PID?

[Kevin Rodgers]

On 7/20/13 3:06 AM, Thorsten Jolitz wrote:
> say I want to call another program (more exactly, another Lisp program
> that is not Emacs Lisp) from an Emacs Lisp program.
>
> My special requirement is that I don't want to start a new Emacs
> subprocess, but want to communicate with an existing (running) process.
> And there maybe several running instances of this program at the same
> time, so I want to communicate with one existing (running) process with
> a given PID.
>
> Now there is splendid support in Emacs for communicating with other
> programs, but always based on the assumption that Emacs starts and
> controls a new subprocess.
>
> There is chapter 37.12 "Accessing Other Processes" in the Elisp manual,
> and I can do successfully
>
> ,-------------------------
> | (process-attributes PID)
> `-------------------------
>
> to receive a lot of information about the process I want to communicate
> with.
>
> But what then? Where are the (e.g.) `process-send-string' or
> `process-send-region' functions for external processes I could use to
> communicate with my external program?
>
> I could run a server in the external lisp program and use a
> network-connection-object to send http-requests (e.g. with the help of
> emacs-request.el) via TCP - but that seems to be total overkill for my
> requirements.
>
> I just want to use a running external process with a given PID in a
> similar way I would use (a)synchronous processes created from Emacs with
> `call-process' or `start-process'. How can I do that?
>
> Maybe there is an obvious answer to this question that I don't see.

Since Emacs did not start the external process, it does not have access to its 
standard input stream -- unless your OS provides a way to do that (e.g. via a
special /dev file).

How would you communicate with the external process, from any other program?

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: How to communicate with a running external process with given PID?

[Thorsten Jolitz]

Kevin Rodgers <kevin.d.rodgers@gmail.com> writes:

> On 7/20/13 3:06 AM, Thorsten Jolitz wrote:
>> say I want to call another program (more exactly, another Lisp program
>> that is not Emacs Lisp) from an Emacs Lisp program.

[snip]

>> I just want to use a running external process with a given PID in a
>> similar way I would use (a)synchronous processes created from Emacs
>> with `call-process' or `start-process'. How can I do that?
>
> Since Emacs did not start the external process, it does not have
> access to its standard input stream -- unless your OS provides a way
> to do that (e.g. via a special /dev file).
>
> How would you communicate with the external process, from any other
> program?

I don't know, and I interpreted the responses so far to my post as "it
is impossible, only 'kill' can access the running external process with
given PID" - but kill sends signals, no command-strings or so. 

So when suggestions arrive like 'use plain socket communication' I'm
still curious if there is a 'magic hack' to achieve that, given that the
external process is NO Emacs subprocess and already running.

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with given PID?

[Aurélien Aptel]

On Mon, Jul 22, 2013 at 11:43 AM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
> While I figured out that in my case using HTTP would be just fine, I
> would still be interested in the answer to my (a tad naive) original
> question: how would you establish (in an Emacs Lisp program) plain
> socket communication between Emacs and an already running external
> process (of a Lisp dialect that is not Emacs Lisp) with given PID?
>

I was a bored so I hacked this example together in an hour or so:

    https://bitbucket.org/knarf/emacs-ipc-example

The process is written in Perl but I'm sure the Lisp you're using has
similar functions to create, listen, accept, receive, send and close
sockets.

Re: How to communicate with a running external process with given PID?

[Aurélien Aptel]

On Mon, Jul 22, 2013 at 4:12 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>> Since Emacs did not start the external process, it does not have
>> access to its standard input stream -- unless your OS provides a way
>> to do that (e.g. via a special /dev file).
>>
>> How would you communicate with the external process, from any other
>> program?
>
> I don't know, and I interpreted the responses so far to my post as "it
> is impossible, only 'kill' can access the running external process with
> given PID" - but kill sends signals, no command-strings or so.

Kevin is right in that on Linux you can use /proc/$PID/fd/0 and 1 to
access the file descriptor of the stdin and stdout of the program
$PID. Actually, it's the input/output of the controlling terminal
which means you have to use some tricks [1] to communicate with the
process.

1: http://stackoverflow.com/questions/5374255/how-to-write-data-to-existing-processs-stdin-from-external-process?lq=1

Re: How to communicate with a running external process with given PID?

[Rustom Mody]

On Saturday, July 20, 2013 4:21:46 PM UTC+5:30, Pascal J. Bourguignon wrote:
> You need to read UNP.
> http://www.unpbook.com/

I believe a more modern approach may be to look at dbus

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>
>> If you just want to read and write stuff, HTTP is probably overkill
>> too. Use plain sockets.
>
> While I figured out that in my case using HTTP would be just fine, I
> would still be interested in the answer to my (a tad naive) original
> question: how would you establish (in an Emacs Lisp program) plain
> socket communication between Emacs and an already running external
> process (of a Lisp dialect that is not Emacs Lisp) with given PID?

Well, theorically, and subject to access rights, you could attach to a
different process (like gdb) and patch its program to add support for
communication.

But since you've not said that your emacs was running as root, or that
the other process runs as the same UID as your emacs, this won't be
possible.

Again, the problem is not how to do things with emacs, it is what the
other process lets you do to it!



-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Mon, Jul 22, 2013 at 4:12 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>>> Since Emacs did not start the external process, it does not have
>>> access to its standard input stream -- unless your OS provides a way
>>> to do that (e.g. via a special /dev file).
>>>
>>> How would you communicate with the external process, from any other
>>> program?
>>
>> I don't know, and I interpreted the responses so far to my post as "it
>> is impossible, only 'kill' can access the running external process with
>> given PID" - but kill sends signals, no command-strings or so.
>
> Kevin is right in that on Linux you can use /proc/$PID/fd/0 and 1 to
> access the file descriptor of the stdin and stdout of the program
> $PID. Actually, it's the input/output of the controlling terminal
> which means you have to use some tricks [1] to communicate with the
> process.

Which do you no good, if that process has closed fds 0, 1 and 2, as any
good daemon process would do.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Pascal J. Bourguignon]

Rustom Mody <rustompmody@gmail.com> writes:

> On Saturday, July 20, 2013 4:21:46 PM UTC+5:30, Pascal J. Bourguignon wrote:
>> You need to read UNP.
>> http://www.unpbook.com/
>
> I believe a more modern approach may be to look at dbus

What for?  Try to talk to processes created from this program with dbus:

------------------------------------------------------------------------
int main(){
    close(0);close(1);close(2);
    while(1)sleep(1);
    return 0;}
------------------------------------------------------------------------

Duh!


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with given PID?

[Rustom Mody]

On Monday, July 22, 2013 11:48:20 PM UTC+5:30, Pascal J. Bourguignon wrote:
> Rustom Mody writes:
> > I believe a more modern approach may be to look at dbus
> What for?  Try to talk to processes created from this program with dbus:
> 
> ------------------------------------------------------------------------
> 
> int main(){
>     close(0);close(1);close(2);
>     while(1)sleep(1);
>     return 0;}

If you dont want to talk to me I can hardly succeed in talking to you!!

If we do want to talk to each other, there may still be issues of rendezvous, finding a language we both understand etc. 
OPs question is not clear how much control he has on the internals of the other process -- the fact that it is in lisp suggests that it is not completely a black box to him.

And Dbus is a not too high or low level of IPC; ie it would work out more convenient than home-brewed IPC over sockets:
http://dbus.freedesktop.org/doc/dbus-specification.html#introduction

At the least it would save serialization headaches.

[If you are saying it will add its own share of headaches... maybe dunno... 
In particular I would not bet on the windows implementation.
But then unix domain sockets may not be so portable either]

> 
> ------------------------------------------------------------------------
> Duh!

Agreed!

Re: How to communicate with a running external process with given PID?

[Thorsten Jolitz]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Mon, Jul 22, 2013 at 11:43 AM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>> While I figured out that in my case using HTTP would be just fine, I
>> would still be interested in the answer to my (a tad naive) original
>> question: how would you establish (in an Emacs Lisp program) plain
>> socket communication between Emacs and an already running external
>> process (of a Lisp dialect that is not Emacs Lisp) with given PID?
>>
>
> I was a bored so I hacked this example together in an hour or so:
>
>     https://bitbucket.org/knarf/emacs-ipc-example
>
> The process is written in Perl but I'm sure the Lisp you're using has
> similar functions to create, listen, accept, receive, send and close
> sockets.

Thank you, thats indeed very helpfull!

-- 
cheers,
Thorsten