Executing org shell blocks on remote machine over ssh

Executing org shell blocks on remote machine over ssh

[David Bjergaard]

Hi All,

I'm sorry if this has been covered before, I've been searching for a
while and haven't found anything.

I use org mode as a lab notebook.  I write org-src blocks to keep track
of tasks I do at the command line, and then I copy paste them into the
terminal.  I would really like to hit "C-c C-c" on the source block and
have it executed on the remote machine.  I know that you can specify
the remote machine according to [1], however the software I use requires
a fairly complicated setup to get going.  

Ideally what I would have is my org-src blocks and another window with
the ssh terminal open.  Then hitting "C-c C-c" would execute the block
over a shared ssh connection and I could see the results in the other
window (quasi-interactively).  Is that possible?

Thanks for your time!

    Dave

Footnotes: 
[1]  http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html#sec-16

Re: Executing org shell blocks on remote machine over ssh

[Myles English]

Hi Dave,

David Bjergaard writes:

> Hi All,
>
> I'm sorry if this has been covered before, I've been searching for a
> while and haven't found anything.
>
> I use org mode as a lab notebook.  I write org-src blocks to keep track
> of tasks I do at the command line, and then I copy paste them into the
> terminal.  I would really like to hit "C-c C-c" on the source block and
> have it executed on the remote machine.  I know that you can specify
> the remote machine according to [1], however the software I use requires
> a fairly complicated setup to get going.  
>
> Ideally what I would have is my org-src blocks and another window with
> the ssh terminal open.  Then hitting "C-c C-c" would execute the block
> over a shared ssh connection and I could see the results in the other
> window (quasi-interactively).  Is that possible?
>
> Thanks for your time!
>
>     Dave
>
> Footnotes: 
> [1]  http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html#sec-16

I had a quick go at this but get tramp errors.  I think you need a
terminal multiplexer e.g. tmux or screen.

In a terminal, connect using ssh (or mosh) then type 'tmux' to start a
new session or 'tmux attach' to attach to an existing one.

I would have thought that this should have worked, i.e. that the output
would have showed up in the terminal and in the buffer:

#+BEGIN_SRC sh  :prologue "tmux attach" :dir /myles@myhost.biz
echo $PWD
echo $HOSTNAME
#+END_SRC

Hope this is on the right track..

Myles

Re: Executing org shell blocks on remote machine over ssh

[Andreas Leha]

Hi Myles,

Myles English <mylesenglish@gmail.com> writes:
> Hi Dave,
>
> David Bjergaard writes:
>
>> Hi All,
>>
>> I'm sorry if this has been covered before, I've been searching for a
>> while and haven't found anything.
>>
>> I use org mode as a lab notebook.  I write org-src blocks to keep track
>> of tasks I do at the command line, and then I copy paste them into the
>> terminal.  I would really like to hit "C-c C-c" on the source block and
>> have it executed on the remote machine.  I know that you can specify
>> the remote machine according to [1], however the software I use requires
>> a fairly complicated setup to get going.  
>>
>> Ideally what I would have is my org-src blocks and another window with
>> the ssh terminal open.  Then hitting "C-c C-c" would execute the block
>> over a shared ssh connection and I could see the results in the other
>> window (quasi-interactively).  Is that possible?
>>
>> Thanks for your time!
>>
>>     Dave
>>
>> Footnotes: 
>> [1]  http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html#sec-16
>
> I had a quick go at this but get tramp errors.  I think you need a
> terminal multiplexer e.g. tmux or screen.
>
> In a terminal, connect using ssh (or mosh) then type 'tmux' to start a
> new session or 'tmux attach' to attach to an existing one.
>
> I would have thought that this should have worked, i.e. that the output
> would have showed up in the terminal and in the buffer:
>
> #+BEGIN_SRC sh  :prologue "tmux attach" :dir /myles@myhost.biz
> echo $PWD
> echo $HOSTNAME
> #+END_SRC
>
> Hope this is on the right track..

I hope so, too, because I would be very interested in that, i.e. running
stuff from orgmode within a (remote or not remote) tmux session.

I do not have time to play with this right now, but I think, the
prologue should not be 'tmux attach' but rather something along 'tmux
paste-buffer'

Best,
Andreas

Re: Executing org shell blocks on remote machine over ssh

[Brett Viren]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

David Bjergaard <david.b@duke.edu> writes:

> I use org mode as a lab notebook.  I write org-src blocks to keep track
> of tasks I do at the command line, and then I copy paste them into the
> terminal.  I would really like to hit "C-c C-c" on the source block and
> have it executed on the remote machine.  I know that you can specify
> the remote machine according to [1], however the software I use requires
> a fairly complicated setup to get going.  

Is it just complicated, or is it also prohibitively long-running?

If just the former, you could maybe bundle the setup into some shell
script and source it in each of your sh source blocks.  Eg:

#+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren
  /bin/pwd
  echo $HOSTNAME
  ls -l foo.sh
  echo "---"
  cat foo.sh
  echo "---"
  source ./foo.sh
  echo $FOO
#+END_SRC

#+RESULTS:
: /home/bviren
: lycastus
: -rw-rw-r-- 1 bviren bviren 16 Nov 18 10:27 foo.sh
: ---
: export FOO=bar
: 
: ---
: bar



If the setup is purely environmental, and it takes a long time to
perform, maybe you could do the set up once and then cache the resulting
environment using the output of "env".


-Brett.

[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: Executing org shell blocks on remote machine over ssh

[Ista Zahn]

On Tue, Nov 18, 2014 at 10:43 AM, Brett Viren <bv@bnl.gov> wrote:
> David Bjergaard <david.b@duke.edu> writes:
>
>> I use org mode as a lab notebook.  I write org-src blocks to keep track
>> of tasks I do at the command line, and then I copy paste them into the
>> terminal.  I would really like to hit "C-c C-c" on the source block and
>> have it executed on the remote machine.  I know that you can specify
>> the remote machine according to [1], however the software I use requires
>> a fairly complicated setup to get going.
>
> Is it just complicated, or is it also prohibitively long-running?
>
> If just the former, you could maybe bundle the setup into some shell
> script and source it in each of your sh source blocks.  Eg:
>
> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren
>   /bin/pwd
>   echo $HOSTNAME
>   ls -l foo.sh
>   echo "---"
>   cat foo.sh
>   echo "---"
>   source ./foo.sh
>   echo $FOO
> #+END_SRC
>
> #+RESULTS:
> : /home/bviren
> : lycastus
> : -rw-rw-r-- 1 bviren bviren 16 Nov 18 10:27 foo.sh
> : ---
> : export FOO=bar
> :
> : ---
> : bar
>
>
>
> If the setup is purely environmental, and it takes a long time to
> perform, maybe you could do the set up once and then cache the resulting
> environment using the output of "env".

I guess I'm missing something (like why the OP want's to run a shell
in a separate window), but why not just

#+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session *shell*
  /bin/pwd
  echo $HOSTNAME
  ls -l foo.sh
  echo "---"
  cat foo.sh
  echo "---"
  source ./foo.sh
  echo $FOO
#+END_SRC

?

Best,
Ista
>
>
> -Brett.

Re: Executing org shell blocks on remote machine over ssh

[David Bjergaard]

Ista Zahn <istazahn@gmail.com> writes:

> On Tue, Nov 18, 2014 at 10:43 AM, Brett Viren <bv@bnl.gov> wrote:
>> David Bjergaard <david.b@duke.edu> writes:
>>
>>> I use org mode as a lab notebook.  I write org-src blocks to keep track
>>> of tasks I do at the command line, and then I copy paste them into the
>>> terminal.  I would really like to hit "C-c C-c" on the source block and
>>> have it executed on the remote machine.  I know that you can specify
>>> the remote machine according to [1], however the software I use requires
>>> a fairly complicated setup to get going.
>>
>> Is it just complicated, or is it also prohibitively long-running?
>>
>> If just the former, you could maybe bundle the setup into some shell
>> script and source it in each of your sh source blocks.  Eg:
>>
>> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren
>>   /bin/pwd
>>   echo $HOSTNAME
>>   ls -l foo.sh
>>   echo "---"
>>   cat foo.sh
>>   echo "---"
>>   source ./foo.sh
>>   echo $FOO
>> #+END_SRC
>>
>> #+RESULTS:
>> : /home/bviren
>> : lycastus
>> : -rw-rw-r-- 1 bviren bviren 16 Nov 18 10:27 foo.sh
>> : ---
>> : export FOO=bar
>> :
>> : ---
>> : bar
>>
>>
>>
>> If the setup is purely environmental, and it takes a long time to
>> perform, maybe you could do the set up once and then cache the resulting
>> environment using the output of "env".
>
> I guess I'm missing something (like why the OP want's to run a shell
> in a separate window), but why not just
>
> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session *shell*
>   /bin/pwd
>   echo $HOSTNAME
>   ls -l foo.sh
>   echo "---"
>   cat foo.sh
>   echo "---"
>   source ./foo.sh
>   echo $FOO
> #+END_SRC
>
> ?
>
> Best,
> Ista
>>
>>
>> -Brett.
Hi Ista, Brett,

Thanks for the hints, I'll try these. 
Some clarification:
>> If the setup is purely environmental, and it takes a long time to
>> perform, maybe you could do the set up once and then cache the resulting
>> environment using the output of "env".
Unfortunately it takes a long time to set up, and its complicated (I
have to initialize different versions of the software depending on
which project I'm working on.) It can take up to 10 seconds to restore a
saved environment, and it takes some setting up bootstrap the
restoration command.
> I guess I'm missing something (like why the OP want's to run a shell
> in a separate window), but why not just
I want the separate window because once the environment is set up, I
also jump around a lot.  I record the pieces of shell script that are
important for reproducing results in the notebook.  The snippets are
then recycled across many sessions, and different pieces are used at
different times (some are even used with different pieces of software).

What I would like is to log into the remote machine, set up the
software, and then have a way to tell emacs to send the org-src block to
the set up environment so that I can build up the parts I need at that
moment.  

I know this is working "against the grain" of the literate programming
paradigm where the document and the source code are coupled, and
tangling the document produces a program that can be executed.  I'm just
wondering if its possible.  If not that's fine.  Really I'm just trying
to save myself a copy-paste (and the associated issues with it getting
recorded in my .bash_history).

Cheers,

    Dave

Re: Executing org shell blocks on remote machine over ssh

[Ista Zahn]

On Tue, Nov 18, 2014 at 1:23 PM, David Bjergaard <david.b@duke.edu> wrote:
> Ista Zahn <istazahn@gmail.com> writes:
>
>> On Tue, Nov 18, 2014 at 10:43 AM, Brett Viren <bv@bnl.gov> wrote:
>>> David Bjergaard <david.b@duke.edu> writes:
>>>
>>>> I use org mode as a lab notebook.  I write org-src blocks to keep track
>>>> of tasks I do at the command line, and then I copy paste them into the
>>>> terminal.  I would really like to hit "C-c C-c" on the source block and
>>>> have it executed on the remote machine.  I know that you can specify
>>>> the remote machine according to [1], however the software I use requires
>>>> a fairly complicated setup to get going.
>>>
>>> Is it just complicated, or is it also prohibitively long-running?
>>>
>>> If just the former, you could maybe bundle the setup into some shell
>>> script and source it in each of your sh source blocks.  Eg:
>>>
>>> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren
>>>   /bin/pwd
>>>   echo $HOSTNAME
>>>   ls -l foo.sh
>>>   echo "---"
>>>   cat foo.sh
>>>   echo "---"
>>>   source ./foo.sh
>>>   echo $FOO
>>> #+END_SRC
>>>
>>> #+RESULTS:
>>> : /home/bviren
>>> : lycastus
>>> : -rw-rw-r-- 1 bviren bviren 16 Nov 18 10:27 foo.sh
>>> : ---
>>> : export FOO=bar
>>> :
>>> : ---
>>> : bar
>>>
>>>
>>>
>>> If the setup is purely environmental, and it takes a long time to
>>> perform, maybe you could do the set up once and then cache the resulting
>>> environment using the output of "env".
>>
>> I guess I'm missing something (like why the OP want's to run a shell
>> in a separate window), but why not just
>>
>> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session *shell*
>>   /bin/pwd
>>   echo $HOSTNAME
>>   ls -l foo.sh
>>   echo "---"
>>   cat foo.sh
>>   echo "---"
>>   source ./foo.sh
>>   echo $FOO
>> #+END_SRC
>>
>> ?
>>
>> Best,
>> Ista
>>>
>>>
>>> -Brett.
> Hi Ista, Brett,
>
> Thanks for the hints, I'll try these.
> Some clarification:
>>> If the setup is purely environmental, and it takes a long time to
>>> perform, maybe you could do the set up once and then cache the resulting
>>> environment using the output of "env".
> Unfortunately it takes a long time to set up, and its complicated (I
> have to initialize different versions of the software depending on
> which project I'm working on.) It can take up to 10 seconds to restore a
> saved environment, and it takes some setting up bootstrap the
> restoration command.
>> I guess I'm missing something (like why the OP want's to run a shell
>> in a separate window), but why not just
> I want the separate window because once the environment is set up, I
> also jump around a lot.

Emacs is great for jumping around a lot, another reason to run the
shell inside emacs instead of in a separate window.

I record the pieces of shell script that are
> important for reproducing results in the notebook.  The snippets are
> then recycled across many sessions, and different pieces are used at
> different times (some are even used with different pieces of software).
>
> What I would like is to log into the remote machine, set up the
> software, and then have a way to tell emacs to send the org-src block to
> the set up environment so that I can build up the parts I need at that
> moment.
>
> I know this is working "against the grain" of the literate programming
> paradigm where the document and the source code are coupled, and
> tangling the document produces a program that can be executed.  I'm just
> wondering if its possible.

It really does sound to me like you're just looking for the :session argument.

Best,
Ista

 If not that's fine.  Really I'm just trying
> to save myself a copy-paste (and the associated issues with it getting
> recorded in my .bash_history).
>
> Cheers,
>
>     Dave

Re: Executing org shell blocks on remote machine over ssh

[Brett Viren]

[-- Attachment #1: Type: text/plain, Size: 2034 bytes --]

David Bjergaard <david.b@duke.edu> writes:

> I know this is working "against the grain" of the literate programming
> paradigm where the document and the source code are coupled, and
> tangling the document produces a program that can be executed.  I'm just
> wondering if its possible.  If not that's fine.  Really I'm just trying
> to save myself a copy-paste (and the associated issues with it getting
> recorded in my .bash_history).

I usually come at it from the "reproducible research" angle which maybe
is more relaxed than literate programming.  In any case, I find it hard
to capture all the info needed to reproduce something and so I settle
for capturing as much as easily achievable - that is when I try at all
as capturing it in an RR org doc greatly increases the time I need to do
something.

Many of the software stacks I use also take significant time to
configure the end-user environment.  10 seconds is not unheard of and it
can be minutes if the stack lives on slow network disk.

I think the approach I suggested of caching the environment should work
for you.  Unfortunately, I do not know of a trivial, general way to do
this.  The "env" program comes close but does not spit out a format that
is immediately consumable by the shell.  In particular, spaces in
variable values confound it.  It also lacks the "export" keyword.  And,
in any case is only close to sh syntax.  Any exported functions also
have to be handled properly

In your shoes, I'd probably write a small Python script that dumps the
"os.environ" dictionary holding the environment of the caller into a
form suitable for consumption by your shell.  You can call this dumper
in a shell code block at the top of your org file and source the result
as the first line in each subsequent shell code block.

A starting point would be something like the following, but this does
not properly handle and sh functions defined.

#!/usr/bin/env python
import os
for k,v in os.environ.items():
    print 'export %s="%s"' % (k,v)



Good luck!
-Brett.



[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: Executing org shell blocks on remote machine over ssh

[Subhan Michael Tindall]

> -----Original Message-----
> From: emacs-orgmode-bounces+subhant=familycareinc.org@gnu.org
> [mailto:emacs-orgmode-bounces+subhant=familycareinc.org@gnu.org] On
> Behalf Of David Bjergaard
> Sent: Tuesday, November 18, 2014 10:23 AM
> To: Ista Zahn
> Cc: Brett Viren; emacs-orgmode Mailinglist
> Subject: Re: [O] Executing org shell blocks on remote machine over ssh
> 
> Ista Zahn <istazahn@gmail.com> writes:
> 
> > On Tue, Nov 18, 2014 at 10:43 AM, Brett Viren <bv@bnl.gov> wrote:
> >> David Bjergaard <david.b@duke.edu> writes:
> >>
> >>> I use org mode as a lab notebook.  I write org-src blocks to keep
> >>> track of tasks I do at the command line, and then I copy paste them
> >>> into the terminal.  I would really like to hit "C-c C-c" on the
> >>> source block and have it executed on the remote machine.  I know
> >>> that you can specify the remote machine according to [1], however
> >>> the software I use requires a fairly complicated setup to get going.
> >>
> >> Is it just complicated, or is it also prohibitively long-running?
> >>
> >> If just the former, you could maybe bundle the setup into some shell
> >> script and source it in each of your sh source blocks.  Eg:
> >>
> >> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren
> >>   /bin/pwd
> >>   echo $HOSTNAME
> >>   ls -l foo.sh
> >>   echo "---"
> >>   cat foo.sh
> >>   echo "---"
> >>   source ./foo.sh
> >>   echo $FOO
> >> #+END_SRC
> >>
> >> #+RESULTS:
> >> : /home/bviren
> >> : lycastus
> >> : -rw-rw-r-- 1 bviren bviren 16 Nov 18 10:27 foo.sh
> >> : ---
> >> : export FOO=bar
> >> :
> >> : ---
> >> : bar
> >>
> >>
> >>
> >> If the setup is purely environmental, and it takes a long time to
> >> perform, maybe you could do the set up once and then cache the
> >> resulting environment using the output of "env".
> >
> > I guess I'm missing something (like why the OP want's to run a shell
> > in a separate window), but why not just
> >
> > #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session
> *shell*
> >   /bin/pwd
> >   echo $HOSTNAME
> >   ls -l foo.sh
> >   echo "---"
> >   cat foo.sh
> >   echo "---"
> >   source ./foo.sh
> >   echo $FOO
> > #+END_SRC
> >
> > ?
> >
> > Best,
> > Ista
> >>
> >>
> >> -Brett.
> Hi Ista, Brett,
> 
> Thanks for the hints, I'll try these.
> Some clarification:
> >> If the setup is purely environmental, and it takes a long time to
> >> perform, maybe you could do the set up once and then cache the
> >> resulting environment using the output of "env".
> Unfortunately it takes a long time to set up, and its complicated (I have to
> initialize different versions of the software depending on which project I'm
> working on.) It can take up to 10 seconds to restore a saved environment,
> and it takes some setting up bootstrap the restoration command.
> > I guess I'm missing something (like why the OP want's to run a shell
> > in a separate window), but why not just
> I want the separate window because once the environment is set up, I also
> jump around a lot.  I record the pieces of shell script that are important for
> reproducing results in the notebook.  The snippets are then recycled across
> many sessions, and different pieces are used at different times (some are
> even used with different pieces of software).
> 
> What I would like is to log into the remote machine, set up the software, and
> then have a way to tell emacs to send the org-src block to the set up
> environment so that I can build up the parts I need at that moment.
> 
> I know this is working "against the grain" of the literate programming
> paradigm where the document and the source code are coupled, and
> tangling the document produces a program that can be executed.  I'm just
> wondering if its possible.  If not that's fine.  Really I'm just trying to save
> myself a copy-paste (and the associated issues with it getting recorded in my
> .bash_history).
> 
> Cheers,
> 
>     Dave

It's been quite a while since I used one (or even did much shell programming), so my details are a bit fuzzy. But, I think your solution is in Unix (I'm assuming that's what your using), not emacs itself.

Take a look at the concept of 'named pipes'

If you've got a shared file system or can remote mount, a process like this would probably work:

Create 2 named pipes
One you can write to on system A (eg. my_sys_a_pipe), with a small script on system B that reads my_sys_a_pipe and executes the commands it grabs
Second named pipe is created and stdout on system B is redirected to it, then system A listens to it & feeds what is received to stdout (my_sys_b_pipe)

Echo "my list of commands" > my_sys_a_pipe
echo < my_sys_b_pipe  (see below, probably want a loop here)

You probably also need to include a tag of some sort (echo "sys_b_done" or similar) as the last command and then run a read/echo loop on my_sys_b_pipe to terminate the driver script and make sure it returns to emacs.

Hopefully this isn't too confusing, unfortunately I don't have time to go back and look up my syntax, but I think conceptually it might get you down the road

This message is intended for the sole use of the individual and entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended addressee, nor authorized to receive for the intended addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone the message or any information contained in the message. If you have received this message in error, please immediately advise the sender by reply email and delete the message.  Thank you.

Re: Executing org shell blocks on remote machine over ssh

[Brady Trainor]

Ista Zahn <istazahn@gmail.com> writes:

[snip]
> I guess I'm missing something (like why the OP want's to run a shell
> in a separate window), but why not just
>
> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session *shell*
>   /bin/pwd
>   echo $HOSTNAME
>   ls -l foo.sh
>   echo "---"
>   cat foo.sh
>   echo "---"
>   source ./foo.sh
>   echo $FOO
> #+END_SRC

Tangentially related... 

*term* looks nice too (M-x term instead of M-x shell). This `term' is like a more literal terminal embedded in Emacs. Even key-bindings are hijacked. (Amusingly, readline utility gives bash some Emacs-like key-bindings.) In a stackoverflow question, it is pointed out that *term* versus *shell*, you miss out on features like isearch (C-s, C-r etc.). Then, switching around a lot is hindered, since C-x o (other-window) is hijacked (but the suggested windmove binding still works). 

--
Brady

Re: Executing org shell blocks on remote machine over ssh

[Andreas Leha]

Hi all,

Brady Trainor <algebrat@uw.edu> writes:
> Ista Zahn <istazahn@gmail.com> writes:
>
> [snip]
>> I guess I'm missing something (like why the OP want's to run a shell
>> in a separate window), but why not just
>>
>> #+BEGIN_SRC sh :results output :dir /ssh:lycastus:/home/bviren :session *shell*
>>   /bin/pwd
>>   echo $HOSTNAME
>>   ls -l foo.sh
>>   echo "---"
>>   cat foo.sh
>>   echo "---"
>>   source ./foo.sh
>>   echo $FOO
>> #+END_SRC
>
> Tangentially related... 
>
> *term* looks nice too (M-x term instead of M-x shell). This `term' is
> like a more literal terminal embedded in Emacs. Even key-bindings are
> hijacked. (Amusingly, readline utility gives bash some Emacs-like
> key-bindings.) In a stackoverflow question, it is pointed out that
> *term* versus *shell*, you miss out on features like isearch (C-s, C-r
> etc.). Then, switching around a lot is hindered, since C-x o
> (other-window) is hijacked (but the suggested windmove binding still
> works).

Well, if the code to be executed remotely is long running, I do not want
to assume the connection to last during the full execution of that code.
Hence, I am interested in sending code to a running tmux session.

One possibility is to have tmux attached from withing emacs' term.  This
might not work too well, unfortunately.  For instance, ess is currently
not able to talk properly to an R session running inside tmux inside
term (which is why I've given up on that idea for now).

Regards,
Andreas