Re: How to communicate with a running external process with

Re: How to communicate with a running external process with

[Buchs, Kevin J.]

Thorsten,

Your questions and responses suggest that maybe you are not grasping some basics of interprocess communication. You have some other program running in another process, not started by emacs and you want to communicate with it. A critical factor in such communication will be knowing how that other program is set up to communicate. Some programs are developed to have standard communication ports and protocols, like HTTP. Unix/Linux supports signals (kill) which a program may be set up accept and respond appropriately toward. A program may support TCP/IP or Unix sockets. A program may read and write files. You might have shared memory space or semaphores. There are dozens of different mechanisms. Emacs supports some of them out of the box. It might be that you can extend emacs by writing code to support other mechanisms. Another alternative is for emacs to communicate with a subprocess it creates running a program designed to communicate with the other process. However, what does the other program support? This is the question you need to answer. We can't make up an answer for you.

As an analogy, if you and I wish to speak, we may need to speak face to face. If we don't speak the same language we have trouble. Maybe we can talk on the telephone, but if you don't have a telephone, we are not going to speak that way. Communication takes appropriate coordination for the sender and receiver. 

Kevin Buchs | Senior Engineer | SPPDG | 507-538-5459 | buchs.kevin@mayo.edu
Mayo Clinic | 200 First Street SW | Rochester, MN 55905 | http://www.mayo.edu

-----Original Message-----
> How would you communicate with the external process, from any other
> program?

I don't know, and I interpreted the responses so far to my post as "it
is impossible, only 'kill' can access the running external process with
given PID" - but kill sends signals, no command-strings or so. 

So when suggestions arrive like 'use plain socket communication' I'm
still curious if there is a 'magic hack' to achieve that, given that the
external process is NO Emacs subprocess and already running.

Re: How to communicate with a running external process with

[Thorsten Jolitz]

"Buchs, Kevin J." <buchs.kevin@mayo.edu> writes:

Hi Kevin,

> Your questions and responses suggest that maybe you are not grasping
> some basics of interprocess communication.

well, hmm, yes ... its indeed quite a difference between reading about
and learning this stuff and putting it into practice.

> You have some other program running in another process, not started by
> emacs and you want to communicate with it. A critical factor in such
> communication will be knowing how that other program is set up to
> communicate. 

Simply assume for the moment that the other lisp knows how to
communicate in a similar way like Emacs Lisp, thus many of the standard
UNP mechanisms are available (pipes, sockets, tcp/http...)

> Some programs are developed to have standard communication ports and
> protocols, like HTTP. Unix/Linux supports signals (kill) which a
> program may be set up accept and respond appropriately toward. A
> program may support TCP/IP or Unix sockets. A program may read and
> write files. You might have shared memory space or semaphores. There
> are dozens of different mechanisms. Emacs supports some of them out of
> the box. It might be that you can extend emacs by writing code to
> support other mechanisms. Another alternative is for emacs to
> communicate with a subprocess it creates running a program designed to
> communicate with the other process. However, what does the other
> program support? This is the question you need to answer. We can't
> make up an answer for you.

I figured out now that, in my special case, talking from Emacs via url
requests to the running http-server of the other Lisp works out just
fine, so my original problem is more or less solved. 

OTOH, this topic has become quite interesting, and I'm still curious
about my original question: how to use all these UNP mechanisms from
Emacs Lisp when the other (Lisp) process is not a subprocess but already
running (with a given PID). 

> As an analogy, if you and I wish to speak, we may need to speak face
> to face. If we don't speak the same language we have trouble. Maybe we
> can talk on the telephone, but if you don't have a telephone, we are
> not going to speak that way. Communication takes appropriate
> coordination for the sender and receiver.

Aurelien posted an example how to use plain TCP, I figured out how to
use HTTP, so there are possibilities. But wrt to pipes and sockets,
e.g., I still would not know how to address that other running (Lisp)
process with given PID from Emacs although both Lisp dialects are
definitely able to communicate via pipes and sockets.

This case is simply not covered in the manuals/books, its always about
(new) subprocesses started under the control of one of the two Lisps.
And from what I learnt from this discussion, it does not seem possible,
except sending signals with 'kill. 

Assume there are 2 otherLisp processes up and running and 2 Emacs
processes, and you want to communicate from otherLisp with PID 5555 to
Emacs with PID 6666.

Now, if you don't care about the specific process you talk to, just the
program, you could simply use

,---------------------------
| (in (list Emacs <<ARGS>>))
`---------------------------

or 

,-----------------------
| (call 'Emacs <<ARGS>>)
`-----------------------

from otherLisp to talk to Emacs, or use `call-process' or `start-process'
from Emacs to talk to otherLisp. 

But from an Emacs Lisp program, how would you establish pipe or socket
communication to that one specific otherLisp process with PID 5555
already running, and not under control of Emacs process with PID 6666?
From all I heard, all you can do is send 'kill signals to that process. 

> Kevin Buchs | Senior Engineer | SPPDG | 507-538-5459 | buchs.kevin@mayo.edu
> Mayo Clinic | 200 First Street SW | Rochester, MN 55905 | http://www.mayo.edu
>
> -----Original Message-----
>> How would you communicate with the external process, from any other
>> program?
>
> I don't know, and I interpreted the responses so far to my post as "it
> is impossible, only 'kill' can access the running external process with
> given PID" - but kill sends signals, no command-strings or so. 
>
> So when suggestions arrive like 'use plain socket communication' I'm
> still curious if there is a 'magic hack' to achieve that, given that the
> external process is NO Emacs subprocess and already running.

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with

[Yuri Khan]

On Tue, Jul 23, 2013 at 2:59 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Assume there are 2 otherLisp processes up and running and 2 Emacs
> processes, and you want to communicate from otherLisp with PID 5555 to
> Emacs with PID 6666.

First, decide which program is the server and which is the client. A
server, roughly speaking, is the process that first establishes the
channel of communication — creates a pipe, a Unix socket, or listens
on a TCP port. Clients then have to know the pipe path/filename, the
Unix socket path, or the IP address and TCP port, in order to connect
to the server. Once the connection by client is established, both
server and client have a file descriptor and can communicate just by
writing into it.

If you have two instances of the server, they have to listen on
different pipes, Unix sockets (by path/name), or, in case of TCP, the
<IP address:port> pairs have to be distinct. Thus, the clients will be
able to connect to the server they want, by knowing the specific
server pipe or Unix socket path or IP address and TCP port.

The process ID just never enters the picture. (Although, if you
control the server, you can make up a convention that, e.g., the
server always listens on the Unix socket with the path
/var/run/your_server_name.$PID.)

Re: How to communicate with a running external process with

[Thorsten Jolitz]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Tue, Jul 23, 2013 at 2:59 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>
>> Assume there are 2 otherLisp processes up and running and 2 Emacs
>> processes, and you want to communicate from otherLisp with PID 5555 to
>> Emacs with PID 6666.
>
> First, decide which program is the server and which is the client. A
> server, roughly speaking, is the process that first establishes the
> channel of communication — creates a pipe, a Unix socket, or listens
> on a TCP port. Clients then have to know the pipe path/filename, the
> Unix socket path, or the IP address and TCP port, in order to connect
> to the server. Once the connection by client is established, both
> server and client have a file descriptor and can communicate just by
> writing into it.
>
> If you have two instances of the server, they have to listen on
> different pipes, Unix sockets (by path/name), or, in case of TCP, the
> <IP address:port> pairs have to be distinct. Thus, the clients will be
> able to connect to the server they want, by knowing the specific
> server pipe or Unix socket path or IP address and TCP port.
>
> The process ID just never enters the picture. (Although, if you
> control the server, you can make up a convention that, e.g., the
> server always listens on the Unix socket with the path
> /var/run/your_server_name.$PID.)

ok, I think now I get it - just like starting a TCP server and then
accessing its <IP address:port> , the otherLisp (as a server) has to
open a pipe or socket first (which do have a path/filename), and knowing
the path/filename I can access that otherLisp process that acts as a
server from an Emacs Lisp program.

Vice versa it should be possible to open a pipe or socket from Emacs
Lisp (without starting a subprocess of otherLisp), and connect to it
from otherLisp using the specific path/filename.

I think I have to take a second look at the manuals wrt this idea. So in
both Lisp dialect there should be a way to open a pipe or socket, return
the path/filename of it, and keep listening for incoming connections
until it is closed. Thanks for the tips. 

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with

[Barry Margolin]

In article <mailman.1619.1374566396.12400.help-gnu-emacs@gnu.org>,
 Thorsten Jolitz <tjolitz@gmail.com> wrote:

> This case is simply not covered in the manuals/books, its always about
> (new) subprocesses started under the control of one of the two Lisps.

That's because those are the only cases where Emacs does anything 
particularly special. Otherwise, it's just generic networking or IPC: 
make a connection to some endpoint like a TCP port or Unix domain socket.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: How to communicate with a running external process with

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> OTOH, this topic has become quite interesting, and I'm still curious
> about my original question: how to use all these UNP mechanisms from
> Emacs Lisp when the other (Lisp) process is not a subprocess but already
> running (with a given PID). 

You started specifying only another "process" PID, but now you're saying
it's a "Lisp" process.  What does that mean?

If the PID is fixed, then the process is fixed, and therefore the
program of this process is fixed (even if you have the sources), the
process owner is fixed, etc, (unless some special cases where the
process is programmed to change those parameters, eg. a shell program or
an interpreter or indeed, a lisp implementation).

But you are not saying what the program of the other process is.  The
only thing you told us about it is that it has a HTTP server.  But by
what you're saying later, I've got the impression this HTTP server is
not actually running, is not actually listening to a TCP/IP port.  If it
is, why don't you tell us that you know what port it's listening to, and
how should you use emacs to connect to that port and perform the HTTP
requests you need?

It is trivial to obtain the map of listening ports and PID:

   sudo netstat -tnpl

(sudo is optional, it's to get the name of the program running in the
processes you don't own).





Now, you said lisp, but since you're building a history of imprecise
language, perhaps we should assume you mean Common Lisp?

There's a protocol that has been defined specifically to communicate
between an emacs and a Common Lisp process, with which a whole IDE has
been built in emacs, named slime (the backend, the library that
implements that protocol on the Common Lisp side is called swank (as is
called the protocol)).  With slime/swank you can connect a CL process
with an emacs, and from emacs, send CL expressions to evaluate to the CL
process and receive back results, and debug and develop CL programs.
But if you specifically meant that, you could easily read the manual of
slime which indicates how to create from emacs a new CL inferior process
running swank, and how to connect from emacs to an existing CL process
that runs a SWANK server (therefore that is listening to a specific
port, just like a HTTP server).



> Assume there are 2 otherLisp processes up and running and 2 Emacs
> processes, and you want to communicate from otherLisp with PID 5555 to
> Emacs with PID 6666.

As I told you, the only communication channels that use the PID to
identify the end point, is made of kill(2) and signal(2).

And as I told you, the receiver has better be ready to get the signal,
otherwise the default behavior for most signals is to kill the process.
Which is as good a reaction to a message as anything else.

So, for example, to send the USR1 message, I mean, the USR1 signal to
the process 5555 from emacs, you could do:

      (shell-command "kill -USR1 5555")



> But from an Emacs Lisp program, how would you establish pipe or socket
> communication to that one specific otherLisp process with PID 5555
> already running, and not under control of Emacs process with PID 6666?
> From all I heard, all you can do is send 'kill signals to that process. 

That's because you don't read the whole messages.  I've already
mentionned the emacs manual chapters you have to read to know that:

> In emacs, you can do that with "network processes":
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Network-Processes.html
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with

[Pascal J. Bourguignon]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> There's a protocol that has been defined specifically to communicate
> between an emacs and a Common Lisp process, with which a whole IDE has
> been built in emacs, named slime (the backend, the library that
> implements that protocol on the Common Lisp side is called swank (as is
> called the protocol)).

Oh, and if you're not using Common Lisp, you may want to consider
porting swank to it, so that you can benefit from slime with your other
lisp.  swank has already been ported to various other languages (scheme
and R come to mind), so it shouldn't be too hard to port it to another
lisp.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with

[Thorsten Jolitz]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> OTOH, this topic has become quite interesting, and I'm still curious
>> about my original question: how to use all these UNP mechanisms from
>> Emacs Lisp when the other (Lisp) process is not a subprocess but already
>> running (with a given PID). 
>
> You started specifying only another "process" PID, but now you're saying
> it's a "Lisp" process.  What does that mean?

[...]

> But you are not saying what the program of the other process is. 

[...]

Not really, this is the very beginning of my original post:

,----------------------------------------------------------------------------
| From: Thorsten Jolitz <tjolitz@gmail.com>
| Subject: How to communicate with a running external process with given PID?
| Newsgroups: gmane.emacs.help
| To: help-gnu-emacs@gnu.org
| Date: Sat, 20 Jul 2013 11:06:55 +0200 (3 days, 10 hours, 55 minutes ago)
| 
| 
| Hi List, 
| 
| say I want to call another program (more exactly, another Lisp program
| that is not Emacs Lisp) from an Emacs Lisp program. 
`----------------------------------------------------------------------------

> Now, you said lisp, but since you're building a history of imprecise
> language, perhaps we should assume you mean Common Lisp?

That wasn't my intention, I was just interested in the Emacs Lisp side
when starting the thread and thought the exact Lisp dialect of the other
side is irrelevant (its an uncommon lisp). After rereading that UNP
stuff I'm aware that the other side is important too.

> There's a protocol that has been defined specifically to communicate
> between an emacs and a Common Lisp process, with which a whole IDE has
> been built in emacs, named slime (the backend, the library that
> implements that protocol on the Common Lisp side is called swank (as is
> called the protocol)).  With slime/swank you can connect a CL process
> with an emacs, and from emacs, send CL expressions to evaluate to the CL
> process and receive back results, and debug and develop CL programs.
> But if you specifically meant that, you could easily read the manual of
> slime which indicates how to create from emacs a new CL inferior process
> running swank, and how to connect from emacs to an existing CL process
> that runs a SWANK server (therefore that is listening to a specific
> port, just like a HTTP server).

I did not mean Common Lisp, but there is a Swank implementation for my
Uncommon Lisp of interest, and it did not occur to me to consider it -
so thanks for this marvelous tip!

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with

[Thorsten Jolitz]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> "Pascal J. Bourguignon" <pjb@informatimago.com> writes:
>
>> There's a protocol that has been defined specifically to communicate
>> between an emacs and a Common Lisp process, with which a whole IDE has
>> been built in emacs, named slime (the backend, the library that
>> implements that protocol on the Common Lisp side is called swank (as is
>> called the protocol)).
>
> Oh, and if you're not using Common Lisp, you may want to consider
> porting swank to it, so that you can benefit from slime with your other
> lisp.  swank has already been ported to various other languages (scheme
> and R come to mind), so it shouldn't be too hard to port it to another
> lisp.

It has been done already by someone else, what opens new possibilities,
it just did not came to my mind when reading the process sections in the
Elisp manual.

-- 
cheers,
Thorsten

Re: How to communicate with a running external process with

[Pascal J. Bourguignon]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> I did not mean Common Lisp, but there is a Swank implementation for my
> Uncommon Lisp of interest, and it did not occur to me to consider it -
> so thanks for this marvelous tip!

So just loading swank in that other lisp, and starting a swank sever:

    (swank:create-server :port 4005) ; in CL

you can then connect to it from emacs with M-x slime-connect RET RET RET

You can even do some kind of "RPC" (Remote Procedure Call) in both
directions (trivial from slime, have a look at [1] to evaluate emacs lisp
expressions from the other lisp).

[1]: http://paste.lisp.org/display/22414

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.  
You know you've been lisping too long when you see a recent picture of George 
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

Re: How to communicate with a running external process with

[Thorsten Jolitz]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> I did not mean Common Lisp, but there is a Swank implementation for my
>> Uncommon Lisp of interest, and it did not occur to me to consider it -
>> so thanks for this marvelous tip!
>
> So just loading swank in that other lisp, and starting a swank sever:
>
>     (swank:create-server :port 4005) ; in CL
>
> you can then connect to it from emacs with M-x slime-connect RET RET RET
>
> You can even do some kind of "RPC" (Remote Procedure Call) in both
> directions (trivial from slime, have a look at [1] to evaluate emacs lisp
> expressions from the other lisp).
>
> [1]: http://paste.lisp.org/display/22414

Thats nice.

One headache is the security aspect, though.

On
[[http://stackoverflow.com/questions/309440/lisp-in-a-box-why-is-it-starting-a-server][stackoverflow]]
someone asked the questions that came immediately to my mind too when
thinking about starting a swank-server from a Lisp process that runs a
long-running production webserver:

,-------------------------------------------------------------------------
| Slime was meant to interface with a Swank server, which allows for
| debugging and patching a live, long-running process on-the-fly, correct?
| Isn't this dangerous? I certainly wouldn't want to just write and
| execute code on-the-fly, on a production web server, potentially causing
| it all to crash. – Cybis Nov 21 '08 at 22:18 1
|         
| Furthermore, isn't this a security issue? Anyone could connect to the
| process and execute arbitrary LISP code. Why wouldn't there by some sort
| of authentication/authorization mechanism for this, if that's what swank
| was meant for? – Cybis Nov 21 '08 at 22:20
`-------------------------------------------------------------------------

There seems to be authentication via a slime-secret file, but otherwise a
connected user could execute arbitrary LISP code on the swank-server, right?

-- 
cheers,
Thorsten