remote plot with local output?

remote plot with local output?

[Benda Xu]

Dear All,

I am looking for a way to configure org-babel so that a program runs
remotely and outputs locally.  An example is

  #+NAME: line
   | 1 |
   | 2 |
   | 3 |

  #+BEGIN_SRC python :results file :var dt=line :dir /ipmuap02:/tmp
    from matplotlib import pylab as plt
    plt.plot(dt)
    plt.savefig("line.png")
    return "line.png"
  #+END_SRC

  #+RESULTS:
  [[file:/scp:ipmuap02:/tmp/line.png]]

I would like to embed this figure into my note.  The example above need
to fetch the figure from the remote host on each exportation, which is
very sensitive to the network environment.

I cannot make the plot locally, because (unlike the over-simplified
example) some potentially big data are only available remotely.

My solution is to cache the result (:cache yes), execute the code block,
copy the output file to localhost, update the #+RESULTS link to the
local one, manually.  Now I am facing many such tasks and feel like
automating that.


What is the recommended way for copying the file output from an remote
execution code block back to localhost?


BTW, my org-mode is 8.3.1 and emacs is 24.4.1.

Cheers,
Benda

Re: remote plot with local output?

[Suvayu Ali]

On Tue, Sep 15, 2015 at 12:11:03AM +0900, Benda Xu wrote:
> 
> My solution is to cache the result (:cache yes), execute the code block,
> copy the output file to localhost, update the #+RESULTS link to the
> local one, manually.  Now I am facing many such tasks and feel like
> automating that.

Maybe, you could do all that in your python source block?  You could use
the :file header to specify where the plot gets copied to on the local
filesystem.

WDYT?

-- 
Suvayu

Open source is the future. It sets us free.

Re: remote plot with local output?

[Benda Xu]

Hi Suvayu,

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Maybe, you could do all that in your python source block?  You could use
> the :file header to specify where the plot gets copied to on the local
> filesystem.
>
> WDYT?

I did think of putting the logic into python source block.  As I will
also have R block like this, having a language-neutral solution, like
:post, better suits.

BTW, how can one extract the string specified by :file inside a python
code block?

Benda

Re: remote plot with local output?

[Benda Xu]

Hi Suvayu,

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Maybe, you could do all that in your python source block?  You could use
> the :file header to specify where the plot gets copied to on the local
> filesystem.
>
> WDYT?

I did think of putting the logic into python source block.  As I will
also have R block like this, having a language-neutral solution, like
:post, better suits.

BTW, how can one extract the string specified by :file inside a python
code block?

Benda

Re: remote plot with local output?

[Charles C. Berry]

On Tue, 15 Sep 2015, Benda Xu wrote:

> Dear All,
>
> I am looking for a way to configure org-babel so that a program runs
> remotely and outputs locally.  An example is
>
>  #+NAME: line
>   | 1 |
>   | 2 |
>   | 3 |
>
>  #+BEGIN_SRC python :results file :var dt=line :dir /ipmuap02:/tmp
>    from matplotlib import pylab as plt
>    plt.plot(dt)
>    plt.savefig("line.png")
>    return "line.png"
>  #+END_SRC
>
>  #+RESULTS:
>  [[file:/scp:ipmuap02:/tmp/line.png]]
>
> I would like to embed this figure into my note.  The example above need
> to fetch the figure from the remote host on each exportation, which is
> very sensitive to the network environment.
>
> I cannot make the plot locally, because (unlike the over-simplified
> example) some potentially big data are only available remotely.
>
> My solution is to cache the result (:cache yes), execute the code block,
> copy the output file to localhost, update the #+RESULTS link to the
> local one, manually.  Now I am facing many such tasks and feel like
> automating that.
>
>
> What is the recommended way for copying the file output from an remote
> execution code block back to localhost?
>

Look at the :post header arg

 	(info "(org) post")

You write a src block that extracts the remote file name from *this*,
creates a local file name from it, copies the remote file to the local 
host, then substitutes the local file name in *this* and uses it as the 
return value.

Use the name of that src block as the argument to :post

HTH,

Chuck

Re: remote plot with local output?

[Benda Xu]

Hi Charles,

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> Look at the :post header arg
>
> 	(info "(org) post")
>
> You write a src block that extracts the remote file name from *this*,
> creates a local file name from it, copies the remote file to the local
> host, then substitutes the local file name in *this* and uses it as
> the return value.
>
> Use the name of that src block as the argument to :post

Thanks for your hint.  I come up with the following example:

   #+NAME: line
   | 1 |
   | 2 |
   | 3 |

   #+name: localize
   #+BEGIN_SRC emacs-lisp :var file="" dir=""
     (let ((rfile (concat (file-name-as-directory dir) file)))
       (let ((lfile (car (last (split-string rfile ":")))))
         (copy-file rfile lfile 1)
         lfile))
   #+END_SRC
   
   #+HEADER: :post localize(*this*, "/ipmuap02:/tmp")
   #+BEGIN_SRC python :results file :var dt=line :dir /ipmuap02:/tmp
     from matplotlib import pylab as plt
     plt.plot(dt)
     plt.savefig("line.png")
     return "line.png"
   #+END_SRC

   #+RESULTS:
   [[file:/tmp/line.png]]

*this* only returns the resulting file name, without :dir.  I have to
set the same remote directory again in the :post call.  Is there a
smarter way to achieve it without duplication?

Cheers,
Benda

Re: remote plot with local output?

[Charles C. Berry]

On Tue, 15 Sep 2015, Benda Xu wrote:

> Hi Charles,
>
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> Look at the :post header arg
>>
>> 	(info "(org) post")
>>
>> You write a src block that extracts the remote file name from *this*,
>> creates a local file name from it, copies the remote file to the local
>> host, then substitutes the local file name in *this* and uses it as
>> the return value.
>>
>> Use the name of that src block as the argument to :post
>
> Thanks for your hint.  I come up with the following example:
>
>   #+NAME: line
>   | 1 |
>   | 2 |
>   | 3 |
>
>   #+name: localize
>   #+BEGIN_SRC emacs-lisp :var file="" dir=""
>     (let ((rfile (concat (file-name-as-directory dir) file)))
>       (let ((lfile (car (last (split-string rfile ":")))))
>         (copy-file rfile lfile 1)
>         lfile))
>   #+END_SRC
>
>   #+HEADER: :post localize(*this*, "/ipmuap02:/tmp")
>   #+BEGIN_SRC python :results file :var dt=line :dir /ipmuap02:/tmp
>     from matplotlib import pylab as plt
>     plt.plot(dt)
>     plt.savefig("line.png")
>     return "line.png"
>   #+END_SRC
>
>   #+RESULTS:
>   [[file:/tmp/line.png]]
>
> *this* only returns the resulting file name, without :dir.  I have to
> set the same remote directory again in the :post call.  Is there a
> smarter way to achieve it without duplication?
>


Untested, but try this :

#+name: localize
#+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
   (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
          (rfile (concat (file-name-as-directory dir) file))
          (lfile (car (last (split-string rfile ":")))))
     (copy-file rfile lfile 1)
     lfile)
#+END_SRC

then use

#+HEADER: :post localize(*this*)

in your python src block.

HTH,

Chuck

Re: remote plot with local output?

[Benda Xu]

Hi Charles,

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> Untested, but try this :
>
> #+name: localize
> #+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
>    (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
>           (rfile (concat (file-name-as-directory dir) file))
>           (lfile (car (last (split-string rfile ":")))))
>      (copy-file rfile lfile 1)
>      lfile)
> #+END_SRC
>
> then use
>
> #+HEADER: :post localize(*this*)
>
> in your python src block.

It successfully extracts the :dir field.  Thanks!

Cheers,
Benda

Re: remote plot with local output?

[Benda Xu]

Hi,

Benda Xu <heroxbd@gentoo.org> writes:

> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> Untested, but try this :
>>
>> #+name: localize
>> #+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
>>    (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
>>           (rfile (concat (file-name-as-directory dir) file))
>>           (lfile (car (last (split-string rfile ":")))))
>>      (copy-file rfile lfile 1)
>>      lfile)
>> #+END_SRC
>>
>> then use
>>
>> #+HEADER: :post localize(*this*)
>>
>> in your python src block.
>
> It successfully extracts the :dir field.  Thanks!

Python works this way.  But babel R has a completely different *this*
value.  Consider the following example:

  #+BEGIN_SRC R :results output graphics :file line.png :dir /ipmuap02:/tmp
  plot(c(1,2,3))
  #+END_SRC

  #+RESULTS:
  [[file:/scp:ipmuap02:/tmp/line.png]]

*this* equals "[[file:/scp:ipmuap02:/tmp/line.png]]", but in babel
python *this* equals "line.png".

So I come up with

  #+name: localize
  #+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
    (let ((lang (car srcinfo)))
      (cond ((string= lang "python")
             (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
                    (rfile (concat (file-name-as-directory dir) file))
                    (lfile (car (last (split-string rfile ":")))))
               (copy-file rfile lfile 1)
               lfile))
            ((string= lang "R")
             (let* ((rfile (substring file 7 -2))
                    (lfile (car (last (split-string rfile ":")))))
               (copy-file rfile lfile 1)
               (concat "[[file:" lfile "]]")))))
  #+END_SRC

But the result is a string rather than a file link:

  #+HEADER: :post localize(*this*)
  #+BEGIN_SRC R :results output graphics :file line.png :dir /ipmuap02:/tmp
  plot(c(1,2,3))
  #+END_SRC

  #+RESULTS:
  : [[file:/tmp/line.png]]

Any hints?

Yours,
Benda

Re: remote plot with local output?

[Charles C. Berry]

On Fri, 18 Sep 2015, Benda Xu wrote:

> Hi,
>
> Benda Xu <heroxbd@gentoo.org> writes:
>
>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>>
>>> Untested, but try this :
>>>
>>> #+name: localize
>>> #+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
>>>    (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
>>>           (rfile (concat (file-name-as-directory dir) file))
>>>           (lfile (car (last (split-string rfile ":")))))
>>>      (copy-file rfile lfile 1)
>>>      lfile)
>>> #+END_SRC
>>>
>>> then use
>>>
>>> #+HEADER: :post localize(*this*)
>>>
>>> in your python src block.
>>
>> It successfully extracts the :dir field.  Thanks!
>
> Python works this way.  But babel R has a completely different *this*
> value.  Consider the following example:
>
>  #+BEGIN_SRC R :results output graphics :file line.png :dir /ipmuap02:/tmp
>  plot(c(1,2,3))
>  #+END_SRC
>
>  #+RESULTS:
>  [[file:/scp:ipmuap02:/tmp/line.png]]
>
> *this* equals "[[file:/scp:ipmuap02:/tmp/line.png]]", but in babel
> python *this* equals "line.png".
>
> So I come up with
>
>  #+name: localize
>  #+BEGIN_SRC emacs-lisp :var file="" srcinfo=(org-babel-get-src-block-info)
>    (let ((lang (car srcinfo)))
>      (cond ((string= lang "python")
>             (let* ((dir (cdr (assoc :dir (nth 2 srcinfo))))
>                    (rfile (concat (file-name-as-directory dir) file))
>                    (lfile (car (last (split-string rfile ":")))))
>               (copy-file rfile lfile 1)
>               lfile))
>            ((string= lang "R")
>             (let* ((rfile (substring file 7 -2))
>                    (lfile (car (last (split-string rfile ":")))))
>               (copy-file rfile lfile 1)
>               (concat "[[file:" lfile "]]")))))
>  #+END_SRC
>
> But the result is a string rather than a file link:
>
>  #+HEADER: :post localize(*this*)
>  #+BEGIN_SRC R :results output graphics :file line.png :dir /ipmuap02:/tmp
>  plot(c(1,2,3))
>  #+END_SRC
>
>  #+RESULTS:
>  : [[file:/tmp/line.png]]
>
> Any hints?
>

Forget *this* and use the

 	(cdr (assoc :file (nth 2 srcinfo))

to get the file name.  That should work the same way regardless of 
language.

Chuck