remote execution in heterogeneous environment

remote execution in heterogeneous environment

[Neil Best]

I run Macport's Emacs and ELPA's Org on my Mac but my data lives in a
Red Hat environment.  I thought I could edit locally and compute
remotely but there seems to be a problem with temp files for executing
source blocks.

#+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
in `pwd`" #+END_SRC

Results are empty and in the message buffers appears this:

executing Sh code block...  Tramp: Opening connection for midway using
ssh...

Opening connection for midway using ssh... \ Tramp: Opening connection
for midway using ssh...done Tramp: Encoding region using function
`base64-encode-region'...done Tramp: Decoding region into remote file
/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
byte-code: Couldn't write region to
`/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN',
decode using `base64 -d -i >%s' failed

I suspect this has something to do with how Emacs asks OSX for temp
file names.  Can anyone confirm this?  Is there a work-around?

Ultimately I want to inspect the environment where the remote
execution is happening.  We use a system called Modules [1] to
customize $PATH, etc. When I try this:

#+BEGIN_SRC sh :dir /midway:~ module list #+END_SRC

I get "sh: line 1: module: command not found" which makes me think
that the command is running on the local system where Modules is not
installed.  How does this square with the discussion at [2]?


[1] http://modules.sourceforge.net/

[2]
http://lists.gnu.org/archive/html/emacs-orgmode/2010-03/msg00071.html

Re: remote execution in heterogeneous environment

[Nick Dokos]

Neil Best <nbest@ci.uchicago.edu> wrote:

> I run Macport's Emacs and ELPA's Org on my Mac but my data lives in a
> Red Hat environment.  I thought I could edit locally and compute
> remotely but there seems to be a problem with temp files for executing
> source blocks.
> 
> #+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
> in `pwd`" #+END_SRC
> 
> Results are empty and in the message buffers appears this:
> 
> executing Sh code block...  Tramp: Opening connection for midway using
> ssh...
> 
> Opening connection for midway using ssh... \ Tramp: Opening connection
> for midway using ssh...done Tramp: Encoding region using function
> `base64-encode-region'...done Tramp: Decoding region into remote file
> /ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
> byte-code: Couldn't write region to
> `/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN',
> decode using `base64 -d -i >%s' failed
> 
> I suspect this has something to do with how Emacs asks OSX for temp
> file names.  Can anyone confirm this?  Is there a work-around?
> 

Not sure about this.

> Ultimately I want to inspect the environment where the remote
> execution is happening.  We use a system called Modules [1] to
> customize $PATH, etc. When I try this:
> 
> #+BEGIN_SRC sh :dir /midway:~ module list #+END_SRC
> 
> I get "sh: line 1: module: command not found" which makes me think
> that the command is running on the local system where Modules is not
> installed.  How does this square with the discussion at [2]?
> 

But here you are probably right: see the thread at

    http://thread.gmane.org/gmane.emacs.orgmode/60102

and particularly this post:

    http://thread.gmane.org/gmane.emacs.orgmode/60102/focus=62587

It might be worth trying the fix in that mail (the funcall part),
although it is known to be incomplete (at the very least: there are a
few more things that seem to go wrong).

Nick

> 
> [1] http://modules.sourceforge.net/
> 
> [2]
> http://lists.gnu.org/archive/html/emacs-orgmode/2010-03/msg00071.html
> 

Re: remote execution in heterogeneous environment

[Nick Dokos]

Neil Best <nbest@ci.uchicago.edu> wrote:

> I run Macport's Emacs and ELPA's Org on my Mac but my data lives in a
> Red Hat environment.  I thought I could edit locally and compute
> remotely but there seems to be a problem with temp files for executing
> source blocks.
> 
> #+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
> in `pwd`" #+END_SRC
> 
> Results are empty and in the message buffers appears this:
> 
> executing Sh code block...  Tramp: Opening connection for midway using
> ssh...
> 
> Opening connection for midway using ssh... \ Tramp: Opening connection
> for midway using ssh...done Tramp: Encoding region using function
> `base64-encode-region'...done Tramp: Decoding region into remote file
> /ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
> byte-code: Couldn't write region to
> `/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN',
> decode using `base64 -d -i >%s' failed
> 
> I suspect this has something to do with how Emacs asks OSX for temp
> file names.  Can anyone confirm this?  Is there a work-around?
> 

It may be that the fix I pointed to in the previous email will change
this (who knows? it may even fix it :-) ) But there is one additional
point that you might want to check.

It looks as if you are creating temp files under /var/folders/2y on the
remote: are you sure that a) the structure exists and b) permissions are
set correctly?

Do you have the /var/folders/2y structure on the local host or only on
the remote? If only on the remote and the source block is running
locally (because of the funcall bug), that would sort-of explain this
behavior.

Nick

Re: remote execution in heterogeneous environment

[Neil Best]

On Thu, Nov 29, 2012 at 11:16 AM, Nick Dokos <nicholas.dokos@hp.com> wrote:

> It looks as if you are creating temp files under /var/folders/2y on the
> remote: are you sure that a) the structure exists and b) permissions are
> set correctly?
>
> Do you have the /var/folders/2y structure on the local host or only on
> the remote? If only on the remote and the source block is running
> locally (because of the funcall bug), that would sort-of explain this
> behavior.
>

/var/folders/2y is peculiar to the local host, my Mac.  I am pretty
sure that this is a Mac (BSD?) idiom for temp files.  /var on the
remote is for root only and I have no influence over its permissions.

By the way I have also tried this with ":session *ssh-midway*" in the
header pointing to a session started with M-x ssh -- same results.

In case more precision about my (org-version) is helpful:

Org-mode version 7.9.2 (7.9.2-82-g2aeb28-elpa @
/Users/nbest/.emacs.d/elpa/org-20121112/)

Re: remote execution in heterogeneous environment

[Neil Best]

On Thu, Nov 29, 2012 at 1:25 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:

>
>> But your earlier mail says that it is trying to create the temp file on the remote
>>
>> ,----
>> | > Tramp: Decoding region into remote file
>> | > /ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
>> | > byte-code: Couldn't write region to
>> | > `/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN'
>> `----
>>
>> Not sure what causes tramp to use /var/folders/2y for remote temp files
>> but that seems to be the main problem here.
>>
>
> If you evaluate
>
>    (tramp-compat-temporary-file-directory)
>
> on your local system, what do you get?
>

> tramp may be assuming that whatever temp directory you are using
> on your local system should work on the remote as well, an
> assumption that's not true in your case. You may have to provide
> your own function to get around this - or use /tmp just like the
> rest of the world.
>

It's exactly as you suspected, Nick.  That crazy /var/folders/...
value was apparently hard-coded in the value of
temporary-file-directory out of the box.  That must be something that
happens in Macports.  When I changed it to "/tmp" things appear to
work better:

#+BEGIN_SRC sh :dir /midway:~
echo "Executed by `whoami` on `hostname` in `pwd`"
#+END_SRC

#+RESULTS:
: Executed by nbest on lep in /Users/nbest

The temp file problem was preventing any results from appearing, but
now we see that the code is being executed locally, as suspected.
There is a work-around for this, however:  M-x ssh.

#+BEGIN_SRC sh :session *ssh-midway*
echo "Executed by `whoami` on `hostname` in `pwd`"
#+END_SRC

#+RESULTS:
: Executed by nbest on midway-login1 in /pwd/from/that/session

This opens up some possibilities but probably does not address the
shortcomings detailed in the thread that Nick points to.  The elisp
hacking going on there is over my head so I will play with this some
more and watch for updates.

Re: remote execution in heterogeneous environment

[Michael Albinus]

Neil Best <nbest@ci.uchicago.edu> writes:

Hi,

> I run Macport's Emacs and ELPA's Org on my Mac but my data lives in a
> Red Hat environment.  I thought I could edit locally and compute
> remotely but there seems to be a problem with temp files for executing
> source blocks.
>
> #+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
> in `pwd`" #+END_SRC
>
> Results are empty and in the message buffers appears this:
>
> executing Sh code block...  Tramp: Opening connection for midway using
> ssh...
>
> Opening connection for midway using ssh... \ Tramp: Opening connection
> for midway using ssh...done Tramp: Encoding region using function
> `base64-encode-region'...done Tramp: Decoding region into remote file
> /ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
> byte-code: Couldn't write region to
> `/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN',
> decode using `base64 -d -i >%s' failed

Which version of Tramp are you using? You find it by "C-h v tramp-version".
I remember a problem with remote temporary directories, fixed in Tramp 2.2.6.

Btw, I don't use org. For local tests, what shall I apply in order to
execute the code in #+BEGIN_SRC  ... #+END_SRC?

Best regards, Michael (Tramp maintainer).

Re: remote execution in heterogeneous environment

[Nick Dokos]

Michael Albinus <michael.albinus@gmx.de> wrote:

> Neil Best <nbest@ci.uchicago.edu> writes:
> 
> Hi,
> 
> > I run Macport's Emacs and ELPA's Org on my Mac but my data lives in a
> > Red Hat environment.  I thought I could edit locally and compute
> > remotely but there seems to be a problem with temp files for executing
> > source blocks.
> >
> > #+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
> > in `pwd`" #+END_SRC
> >
> > Results are empty and in the message buffers appears this:
> >
> > executing Sh code block...  Tramp: Opening connection for midway using
> > ssh...
> >
> > Opening connection for midway using ssh... \ Tramp: Opening connection
> > for midway using ssh...done Tramp: Encoding region using function
> > `base64-encode-region'...done Tramp: Decoding region into remote file
> > /ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN...done
> > byte-code: Couldn't write region to
> > `/ssh:midway:/var/folders/2y/qrw3hhkx0dlb9sbl51mjy5wm0000gn/T/sh-510999vN',
> > decode using `base64 -d -i >%s' failed
> 
> Which version of Tramp are you using? You find it by "C-h v tramp-version".
> I remember a problem with remote temporary directories, fixed in Tramp 2.2.6.
> 
> Btw, I don't use org. For local tests, what shall I apply in order to
> execute the code in #+BEGIN_SRC  ... #+END_SRC?
> 

See

	http://thread.gmane.org/gmane.emacs.orgmode/62963

It was a weird setting of the variable ``temporary-file-directory'' in
Neil's Macport setup, plus the assumption by tramp that that setting was
valid for the remote - that turned out to be a bad assumption. Not sure
it's worth it for tramp to provide a way to customize that, but that's
your decision :-) AFAIAC, /tmp should be a universal standard.

Nick

Re: remote execution in heterogeneous environment

[Michael Albinus]

Nick Dokos <nicholas.dokos@hp.com> writes:

> See
>
> 	http://thread.gmane.org/gmane.emacs.orgmode/62963

I've seen that.

> It was a weird setting of the variable ``temporary-file-directory'' in
> Neil's Macport setup, plus the assumption by tramp that that setting was
> valid for the remote - that turned out to be a bad assumption. Not sure
> it's worth it for tramp to provide a way to customize that, but that's
> your decision :-) AFAIAC, /tmp should be a universal standard.

I don't believe it is a weird setting of `temporary-file-directory'. It
is a valid value for OS X, AFAIK.

And "/tmp" isn't a universal standard. For example, it doesn't work,
when your local machine runs MS Windows.

The patch in Tramp I have mentioned tries to distinguish between local
and remote temporary directories. I would like to know, whether this
works correctly for org-mode, or whether there is still an issue.

> Nick

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Neil Best]

Thanks for taking a look, Michael.

On Fri, Nov 30, 2012 at 1:37 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Neil Best <nbest@ci.uchicago.edu> writes:
>>
>> #+BEGIN_SRC sh :dir /midway:~ echo "Executed by `whoami` on `hostname`
>> in `pwd`" #+END_SRC
>>
>
> Which version of Tramp are you using? You find it by "C-h v tramp-version".
> I remember a problem with remote temporary directories, fixed in Tramp 2.2.6.
>

tramp-version
"2.2.3-24.1"

emacs-version
"24.2.1"

Looks like I should upgrade tramp.  I will look into that.  I think if
I restart emacs it will be upgraded to some minor, minor release, so
maybe that took care of it already.  Fat chance?

> Btw, I don't use org. For local tests, what shall I apply in order to
> execute the code in #+BEGIN_SRC  ... #+END_SRC?

I should mention that my source code block got mangled when I tried to
post through Gmane.  It should look like this:

#+BEGIN_SRC sh :dir /midway:~
echo "Executed by `whoami` on `hostname` in `pwd`"
#+END_SRC

Just go into Org mode, move the point within the block, and hit C-c C-c.

Re: remote execution in heterogeneous environment

[Nick Dokos]

Michael Albinus <michael.albinus@gmx.de> wrote:

> Nick Dokos <nicholas.dokos@hp.com> writes:
> 
> > See
> >
> > 	http://thread.gmane.org/gmane.emacs.orgmode/62963
> 
> I've seen that.
> 
> > It was a weird setting of the variable ``temporary-file-directory'' in
> > Neil's Macport setup, plus the assumption by tramp that that setting was
> > valid for the remote - that turned out to be a bad assumption. Not sure
> > it's worth it for tramp to provide a way to customize that, but that's
> > your decision :-) AFAIAC, /tmp should be a universal standard.
> 
> I don't believe it is a weird setting of `temporary-file-directory'. It
> is a valid value for OS X, AFAIK.
> 
> And "/tmp" isn't a universal standard. For example, it doesn't work,
> when your local machine runs MS Windows.
> 
> The patch in Tramp I have mentioned tries to distinguish between local
> and remote temporary directories. I would like to know, whether this
> works correctly for org-mode, or whether there is still an issue.
> 

OK - thanks. Sorry for the noise.

Trying-to-get-foot-out-of-mouth-ly yours,
Nick

Re: remote execution in heterogeneous environment

[Michael Albinus]

Nick Dokos <nicholas.dokos@hp.com> writes:

> OK - thanks. Sorry for the noise.
>
> Trying-to-get-foot-out-of-mouth-ly yours,

Come on, there's nothing you have to sorry. You don't know Tramp, that's
all. And this is not exclusive by you :-)

> Nick

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Michael Albinus]

Neil Best <nbest@ci.uchicago.edu> writes:

> tramp-version
> "2.2.3-24.1"
>
> emacs-version
> "24.2.1"
>
> Looks like I should upgrade tramp.  I will look into that.

Emacs 24.3 is in pretest, it contains Tramp 2.2.6. This might be another
option. See <http://thread.gmane.org/gmane.emacs.devel/155058>.

> I think if I restart emacs it will be upgraded to some minor, minor
> release, so maybe that took care of it already.  Fat chance?

Sorry, this I don't understand ...

>> Btw, I don't use org. For local tests, what shall I apply in order to
>> execute the code in #+BEGIN_SRC  ... #+END_SRC?
>
> I should mention that my source code block got mangled when I tried to
> post through Gmane.  It should look like this:
>
> #+BEGIN_SRC sh :dir /midway:~
> echo "Executed by `whoami` on `hostname` in `pwd`"
> #+END_SRC
>
> Just go into Org mode, move the point within the block, and hit C-c C-c.

Thanks, I will try to reproduce your problem locally.

Best regards, Michael.

PS: it's on my wishlist for a long time to learn org-mode. Maybe I have
a chance now.

Re: remote execution in heterogeneous environment

[Neil Best]

On Fri, Nov 30, 2012 at 1:22 PM, Michael Albinus <michael.albinus@gmx.de> wrote:
> Neil Best <nbest@ci.uchicago.edu> writes:
>
>> tramp-version
>> "2.2.3-24.1"
>>
>> emacs-version
>> "24.2.1"
>>
>> Looks like I should upgrade tramp.  I will look into that.
>
> Emacs 24.3 is in pretest, it contains Tramp 2.2.6. This might be another
> option. See <http://thread.gmane.org/gmane.emacs.devel/155058>.
>
>> I think if I restart emacs it will be upgraded to some minor, minor
>> release, so maybe that took care of it already.  Fat chance?
>
> Sorry, this I don't understand ...
>

I had just recently upgraded Emacs through Macports but had not since
restarted it, that's all.  No such luck on the Tramp upgrade.  I will
just wait for the next release to propagate through Macports unless
some intolerable behavior crops up.  I do have a shiny new version of
Org to play with thanks to ELPA.  :)

>
> PS: it's on my wishlist for a long time to learn org-mode. Maybe I have
> a chance now.

You won't regret it.  Have fun.  What's your use case?  How did you
happen to get on this thread?  Was there a cross-post that I missed?

Thanks for pitching in, everyone.

Re: remote execution in heterogeneous environment

[Michael Albinus]

Neil Best <nbest@ci.uchicago.edu> writes:

> I had just recently upgraded Emacs through Macports but had not since
> restarted it, that's all.  No such luck on the Tramp upgrade.  I will
> just wait for the next release to propagate through Macports unless
> some intolerable behavior crops up.  I do have a shiny new version of
> Org to play with thanks to ELPA.  :)

There are still several problems, even with Tramp 2.2.6. I use Org
20121126 from ELPA.

1. test
=======

My local and remote hosts use "/tmp", both run Linux. So I have applied
prior to the test

--8<---------------cut here---------------start------------->8---
(setq temporary-file-directory "~/tmp/")
--8<---------------cut here---------------end--------------->8---

This directory is known to exist locally, but not remotely. When I eval
the source code block, I don't run into this infloop as you have seen
(this is fixed with Tramp 2.2.6, indeed). But there is another error

--8<---------------cut here---------------start------------->8---
Tramp: Decoding region into remote file /scpc:ford:/home/albinus/tmp/sh-205735cU...done
cond: Couldn't write region to `/scpc:ford:/home/albinus/tmp/sh-205735cU', decode using `base64 -d -i >%s' failed
--8<---------------cut here---------------end--------------->8---

Somewhere in the code, the remote temporary directory is created based
on `temporary-file-directory', which is wrong. I believe this is in
Org's code, must be debugged further.

2. test
=======

I keep `temporary-file-directory' as "/tmp". The source code block
evaluates fine, but the result is:

--8<---------------cut here---------------start------------->8---
#+BEGIN_SRC sh :dir /ford:~
echo "Executed by `whoami` on `hostname` in `pwd`"
#+END_SRC

#+RESULTS:
: Executed by albinus on detlef in /home/albinus
--8<---------------cut here---------------end--------------->8---

Obviously, the code is executed on the local host. As far as I could
debug it, this is because `org-babel-shell-command-on-region' calls
`call-process-region', which is not aware of remote execution.

>> PS: it's on my wishlist for a long time to learn org-mode. Maybe I have
>> a chance now.
>
> You won't regret it.  Have fun.  What's your use case?

Accessing lists on different devices, under Linux, Android etc.

> How did you happen to get on this thread?  Was there a cross-post that
> I missed?

Nothing special. I have looked into the ML archive for some other
reason, and I have seen this thread by accident.

> Thanks for pitching in, everyone.

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Michael Albinus]

[Cc emacs-orgmode@gnu.org]

George Jones <eludom@gmail.com> writes:

Hi George,

> Did you ever get resolution on these?  I think I'm hitting the same
> exact problems.

In my last message, I gave some hints what to do in org's code. Since I
don't use org myself (yet), I haven't followed that.

Looks, like nobody did take the ball :-(

Maybe I'll check it again over XMas days, and maybe I can propose a
concrete patch then.

> Thanks,
> ---George Jones

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

> Looks, like nobody did take the ball :-(
>
> Maybe I'll check it again over XMas days, and maybe I can propose a
> concrete patch then.

The following patch I've committed to Emacs' trunk, it fixes the problem
as far as I could test. Maybe somebody from the org maintainers could
merge it into the org repository.

--8<---------------cut here---------------start------------->8---
~/src/emacs/lisp/org> bzr diff >123
=== modified file 'lisp/org/ChangeLog'
--- lisp/org/ChangeLog	2012-12-13 05:29:15 +0000
+++ lisp/org/ChangeLog	2012-12-20 13:09:40 +0000
@@ -1,3 +1,12 @@
+2012-12-20  Michael Albinus  <michael.albinus@gmx.de>
+
+	* ob.el (org-babel-temp-file): Fix setting of
+	`temporary-file-directory' on remote hosts.
+
+	* ob-eval.el (org-babel-shell-command-on-region): Use
+	`process-file' instead of `call-process-region'.  The latter one
+	does not work on remote hosts.
+
 2012-12-13  Bastien Guerry  <bzg@gnu.org>

 	* org-latex.el (org-export-latex-links): Escape raw path when

=== modified file 'lisp/org/ob-eval.el'
--- lisp/org/ob-eval.el	2012-09-30 15:14:59 +0000
+++ lisp/org/ob-eval.el	2012-12-20 13:06:27 +0000
@@ -134,14 +134,13 @@
 		       current-prefix-arg
 		       shell-command-default-error-buffer
 		       t)))
-  (let ((error-file
-	 (if error-buffer
-	     (make-temp-file
-	      (expand-file-name "scor"
-                                (if (featurep 'xemacs)
-                                    (temp-directory)
-                                  temporary-file-directory)))
-	   nil))
+  (let ((input-file (org-babel-temp-file "input-"))
+	(error-file (if error-buffer (org-babel-temp-file "scor-") nil))
+	(shell-file-name
+	 (if (file-executable-p
+	      (concat (file-remote-p default-directory) shell-file-name))
+	     shell-file-name
+	   "/bin/sh"))
 	exit-status)
     (if (or replace
 	    (and output-buffer
@@ -151,12 +150,14 @@
 	  ;; Don't muck with mark unless REPLACE says we should.
 	  (goto-char start)
 	  (and replace (push-mark (point) 'nomsg))
+	  (write-region start end input-file)
+	  (delete-region start end)
 	  (setq exit-status
-		(call-process-region start end shell-file-name t
-				     (if error-file
-					 (list output-buffer error-file)
-				       t)
-				     nil shell-command-switch command))
+		(process-file shell-file-name input-file
+			      (if error-file
+				  (list output-buffer error-file)
+				t)
+			      nil shell-command-switch command))
 	  ;; It is rude to delete a buffer which the command is not using.
 	  ;; (let ((shell-buffer (get-buffer "*Shell Command Output*")))
 	  ;;   (and shell-buffer (not (eq shell-buffer (current-buffer)))
@@ -175,14 +176,14 @@
 		(progn (setq buffer-read-only nil)
 		       (delete-region (max start end) (point-max))
 		       (delete-region (point-min) (min start end))
+		       (write-region (point-min) (point-max) input-file)
+		       (delete-region (point-min) (point-max))
 		       (setq exit-status
-			     (call-process-region (point-min) (point-max)
-						  shell-file-name t
-						  (if error-file
-						      (list t error-file)
-						    t)
-						  nil shell-command-switch
-						  command)))
+			     (process-file shell-file-name input-file
+					   (if error-file
+					       (list t error-file)
+					     t)
+					   nil shell-command-switch command)))
 	      ;; Clear the output buffer, then run the command with
 	      ;; output there.
 	      (let ((directory default-directory))
@@ -192,11 +193,11 @@
 		      (setq default-directory directory))
 		  (erase-buffer)))
 	      (setq exit-status
-		    (call-process-region start end shell-file-name nil
-					 (if error-file
-					     (list buffer error-file)
-					   buffer)
-					 nil shell-command-switch command)))
+		    (process-file shell-file-name nil
+				  (if error-file
+				      (list buffer error-file)
+				    buffer)
+				  nil shell-command-switch command)))
 	  ;; Report the output.
 	  (with-current-buffer buffer
 	    (setq mode-line-process
@@ -230,6 +231,9 @@
 	    ;; (kill-buffer buffer)
 	    ))))

+    (when (and input-file (file-exists-p input-file))
+      (delete-file input-file))
+
     (when (and error-file (file-exists-p error-file))
       (if (< 0 (nth 7 (file-attributes error-file)))
 	  (with-current-buffer (get-buffer-create error-buffer)

=== modified file 'lisp/org/ob.el'
--- lisp/org/ob.el	2012-10-26 14:42:05 +0000
+++ lisp/org/ob.el	2012-12-20 13:05:00 +0000
@@ -2547,18 +2547,14 @@
 Passes PREFIX and SUFFIX directly to `make-temp-file' with the
 value of `temporary-file-directory' temporarily set to the value
 of `org-babel-temporary-directory'."
-  (if (file-remote-p default-directory)
-      (make-temp-file
-       (concat (file-remote-p default-directory)
-	       (expand-file-name
-		prefix temporary-file-directory)
-	       nil suffix))
-    (let ((temporary-file-directory
+  (let ((temporary-file-directory
+	 (if (file-remote-p default-directory)
+	     (concat (file-remote-p default-directory) "/tmp")
 	   (or (and (boundp 'org-babel-temporary-directory)
 		    (file-exists-p org-babel-temporary-directory)
 		    org-babel-temporary-directory)
-	       temporary-file-directory)))
-      (make-temp-file prefix nil suffix))))
+	       temporary-file-directory))))
+      (make-temp-file prefix nil suffix)))

 (defun org-babel-remove-temporary-directory ()
   "Remove `org-babel-temporary-directory' on Emacs shutdown."
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Bastien]

Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> The following patch I've committed to Emacs' trunk, it fixes the problem
> as far as I could test. Maybe somebody from the org maintainers could
> merge it into the org repository.

I've already merged it in maint and master.

Thanks a lot!

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Achim Gratz]

Bastien writes:
> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> The following patch I've committed to Emacs' trunk, it fixes the problem
>> as far as I could test. Maybe somebody from the org maintainers could
>> merge it into the org repository.
>
> I've already merged it in maint and master.

Unfortunately it breaks testing, at least in batch mode.  See my other
posting about it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: remote execution in heterogeneous environment

[Nick Dokos]

Achim Gratz <Stromeko@nexgo.de> wrote:

> Bastien writes:
> > Michael Albinus <michael.albinus@gmx.de> writes:
> >
> >> The following patch I've committed to Emacs' trunk, it fixes the problem
> >> as far as I could test. Maybe somebody from the org maintainers could
> >> merge it into the org repository.
> >
> > I've already merged it in maint and master.
> 
> Unfortunately it breaks testing, at least in batch mode.  See my other
> posting about it.
> 
> 

Isn't ``make test'' standard procedure before a commit? If not, it should be.
Otherwise, why have a test suite?

Nick

Re: remote execution in heterogeneous environment

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> Unfortunately it breaks testing, at least in batch mode.  See my other
> posting about it.

Not sure what other posting I should read -- sorry, I can't read and
process on the threads at once.

If you know how to fix the tests, please go ahead. 

Thanks!

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Bastien]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Isn't ``make test'' standard procedure before a commit? If not, it should be.
> Otherwise, why have a test suite?

"Errare humanum est... <snip>"

Yes, I should have tested.

But note that this is already committed in Emacs trunk.
My assumption by looking at the code was that if tests 
breaks, then we need to fix the tests, not the patch.

Again, if someone can help fixing the tests, that'd
help me a lot!  

Thanks,

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Michael Albinus]

Achim Gratz <Stromeko@nexgo.de> writes:

>>> The following patch I've committed to Emacs' trunk, it fixes the problem
>>> as far as I could test. Maybe somebody from the org maintainers could
>>> merge it into the org repository.
>>
>> I've already merged it in maint and master.
>
> Unfortunately it breaks testing, at least in batch mode.  See my other
> posting about it.

Could you, please, give me a short pointer how to run the test suite? I
will check then what happened.

Note, that I don't use the org repository (yet). Maybe you must explain
from the beginning ...

Or you give me a short test program I could run for this dedicated test.

> Regards,
> Achim.

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Bastien]

Hi Michael,

thanks for offering your help!

The first step is to clone the repository:

~$ git clone git://orgmode.org/org-mode.git

You are in the master branch, the one we use for latest developments.
The other branch is "maint", the one we use for bug fixing.  The next
version of Org, 7.9.3 will be released from maint, the next major 
version from master.

From the org-mode repo, simply run "make test": it will compile the
files and run the test suite.  

To debug the tests, go to the "testing/" directory, load org-*.el 
files, then find the failing tests in testing/lisp/.

Since there are many tests affected, my guess is that some top-level
testing function (either in testing/ or testing/lisp) is not up to date.

If you have any other question, let us know.

Also, if you can send patches here first next time, that'd of 
*great* help!  

Best,

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Achim Gratz]

Bastien writes:
>> Unfortunately it breaks testing, at least in batch mode.  See my other
>> posting about it.
>
> Not sure what other posting I should read -- sorry, I can't read and
> process on the threads at once.

The other posting is "27bcf9a70b Backport revno 111277 from Emacs trunk."

> If you know how to fix the tests, please go ahead. 

I don't.  What I found out is that at least for the first four of these
it only fils when the test is run in batch mode.  The interactive test
works.  The error message indicates that it seems to attempt to create
error-file twice, but I don't see how this might happen — the file is
created with a unique name, so something must later use the bound
variable with that name to again attempt to create the file, perhaps
when it creates a buffer for it (the test used to be in a buffer that
then gets written to file IIRC).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: remote execution in heterogeneous environment

[George Jones]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Hi.  Patch seems to work.  Tested on fresh emacs-snapshot (24.3.50.1)
install on ubuntu with org-mode freshly pulled from git (7.9.2).

#+NAME: hobbits
| bilbo | frodo |

#+name: execOnExample.com
#+BEGIN_SRC sh :var foo=hobbits  :dir /gmj@example.com: :results output
hostname

echo executed on `hostname` at `date`

echo two $foo

#+END_SRC

Similar procedure on a mac still yeilds:

FYI/Thanks,
---George Jones

Couldn't write region to
`/scpc:gmj@example.com:/var/folders/j6/j6w17khs3vl9s2_yg4rb5zsm0000gn/T/input-61683xyS',
decode using `mimencode -u -b >%s' f\
ailed

On Thu, Dec 20, 2012 at 11:39 AM, Bastien <bzg@altern.org> wrote:

> Hi Michael,
>
> Michael Albinus <michael.albinus@gmx.de> writes:
>
> > The following patch I've committed to Emacs' trunk, it fixes the problem
> > as far as I could test. Maybe somebody from the org maintainers could
> > merge it into the org repository.
>
> I've already merged it in maint and master.
>
> Thanks a lot!
>

[-- Attachment #2: Type: text/html, Size: 2494 bytes --]

Re: remote execution in heterogeneous environment

[Michael Albinus]

Bastien <bzg@altern.org> writes:

> Hi Michael,

Hi everybody,

> From the org-mode repo, simply run "make test": it will compile the
> files and run the test suite.  

I could reproduce the problem.

Finally, it is an error in `process-file', which wasn't detected until
now. Thanks to triggering this!

I've fixed it already in Emacs' trunk. For org-mode, I have used the
following workaround:

--8<---------------cut here---------------start------------->8---
~/src/org-mode> git diff
diff --git a/lisp/ob-eval.el b/lisp/ob-eval.el
index 23e7143..df95d0a 100644
--- a/lisp/ob-eval.el
+++ b/lisp/ob-eval.el
@@ -143,6 +143,11 @@ specifies the value of ERROR-BUFFER."
             shell-file-name
           "/bin/sh"))
        exit-status)
+    ;; There is an error in `process-file', when `error-file' exists.
+    ;; Fixed already in Emacs trunk; for the time being we apply a
+    ;; workaround.
+    (unless (file-remote-p default-directory)
+      (delete-file error-file))
     (if (or replace
            (and output-buffer
                 (not (or (bufferp output-buffer) (stringp output-buffer)))))
--8<---------------cut here---------------end--------------->8---

With that workaround, "make test" results in

--8<---------------cut here---------------start------------->8---
Ran 320 tests, 319 results as expected, 1 unexpected (2012-12-21 09:13:44+0100)
7 expected failures

1 unexpected results:
   FAILED  ob-exp/exports-inline
--8<---------------cut here---------------end--------------->8---

> Best,

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Michael Albinus]

George Jones <eludom@gmail.com> writes:

> Hi.

Hi George,

> Similar procedure on a mac still yeilds:
>
> Couldn't write region to
> `/scpc:gmj@example.com:/var/folders/j6/j6w17khs3vl9s2_yg4rb5zsm0000gn/T/input-
> 61683xyS', decode using `mimencode -u -b >%s' f\
> ailed

Could you, please, show the exact test case? And please apply the
workaround first (see my other email).

> FYI/Thanks,
> ---George Jones

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Bastien]

Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> Finally, it is an error in `process-file', which wasn't detected until
> now. Thanks to triggering this!
>
> I've fixed it already in Emacs' trunk. For org-mode, I have used the
> following workaround:

great, thanks for fixing this so fast, I applied your change in Org.

All tests in org's git repo now run fine, both for maint and for master.

Best,

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Nick Dokos]

Michael Albinus <michael.albinus@gmx.de> wrote:

> Bastien <bzg@altern.org> writes:
> 
> > Hi Michael,
> 
> Hi everybody,
> 
> > From the org-mode repo, simply run "make test": it will compile the
> > files and run the test suite.  
> 
> I could reproduce the problem.
> 
> Finally, it is an error in `process-file', which wasn't detected until
> now. Thanks to triggering this!
> 
> I've fixed it already in Emacs' trunk. For org-mode, I have used the
> following workaround:
> 
> ~/src/org-mode> git diff
> diff --git a/lisp/ob-eval.el b/lisp/ob-eval.el
> index 23e7143..df95d0a 100644
> --- a/lisp/ob-eval.el
> +++ b/lisp/ob-eval.el
> @@ -143,6 +143,11 @@ specifies the value of ERROR-BUFFER."
>              shell-file-name
>            "/bin/sh"))
>         exit-status)
> +    ;; There is an error in `process-file', when `error-file' exists.
> +    ;; Fixed already in Emacs trunk; for the time being we apply a
> +    ;; workaround.
> +    (unless (file-remote-p default-directory)
> +      (delete-file error-file))
>      (if (or replace
>             (and output-buffer
>                  (not (or (bufferp output-buffer) (stringp output-buffer)))))
> 

I can confirm that this fixes the local dir case in the simple test I
posted previously (and the remote dir case was fixed by the process-file
fix previously).

> With that workaround, "make test" results in
> 
> Ran 320 tests, 319 results as expected, 1 unexpected (2012-12-21 09:13:44+0100)
> 7 expected failures
> 
> 1 unexpected results:
>    FAILED  ob-exp/exports-inline
> 

I don't get that failure with the current code, so maybe it has been fixed already (?):

,----
| Ran 334 tests, 334 results as expected (2012-12-21 12:27:17-0500)
| 5 expected failures
`----

Not sure why I have a different number of tests/expected failures either.

But in any case, the main problem is resolved.

Thanks for the fixes!
Nick

Re: remote execution in heterogeneous environment

[George]

Michael Albinus <michael.albinus@gmx.de> writes:

>> Similar procedure on a mac still yeilds:
>>
>> Couldn't write region to
>> `/scpc:gmj@example.com:/var/folders/j6/j6w17khs3vl9s2_yg4rb5zsm0000gn/T/input-
>> 61683xyS', decode using `mimencode -u -b >%s' f\
>> ailed
>
> Could you, please, show the exact test case? And please apply the
> workaround first (see my other email).
>

Everything works if I do, as you suggested:

  (setq temporary-file-directory "~/tmp/")

-------------------------------cut here-------------------------------
#+NAME: hobbits
| bilbo | frodo |

#+name: executeOnLocalVM
#+BEGIN_SRC sh :var foo=hobbits  :dir /george@localhost#2222: :results output
echo executed on `hostname` at `date`                                                                                                                 echo two $foo                                                                                                                                         #+END_SRC                                                                                                                                              

#+RESULTS: executeOnLocalVM
: executed on garlic at Sat Dec 22 13:25:35 UTC 2012
: two bilbo frodo

#+name: relevantVars
#+BEGIN_SRC emacs-lisp
(concat
  "temporary-file-directory " temporary-file-directory
)
#+END_SRC

#+RESULTS: relevantVars
: temporary-file-directory /tmp/

#+name: versions
#+BEGIN_SRC emacs-lisp
(concat (emacs-version)
 "
tramp version: " tramp-version
"
org version " org-version)
#+END_SRC

#+RESULTS: versions
: GNU Emacs 24.2.1 (x86_64-apple-darwin11.4.2)
:  of 2012-11-19 on tenseven-slave.macports.org
: tramp version: 2.2.6
: org version 7.9.2

-------------------------------cut here-------------------------------

Thanks,
---George Jones

Re: remote execution in heterogeneous environment

[George]

Michael Albinus <michael.albinus@gmx.de> writes:

>> Similar procedure on a mac still yeilds:
>>
>> Couldn't write region to
>> `/scpc:gmj@example.com:/var/folders/j6/j6w17khs3vl9s2_yg4rb5zsm0000gn/T/input-
>> 61683xyS', decode using `mimencode -u -b >%s' f\
>> ailed
>
> Could you, please, show the exact test case? And please apply the
> workaround first (see my other email).

Make everything works with:

  (setq temporary-file-directory "/tmp/")

Thanks,
---George Jones

Re: remote execution in heterogeneous environment

[Michael Albinus]

George <eludom@gmail.com> writes:

Hi George,

> Make everything works with:
>
>   (setq temporary-file-directory "/tmp/")

That's a misunderstanding. I meant the workaround I have proposed
yesterday on the org-mode ML, and which has been committed by Bastien.

Meanwhile, everything shall work out-of-the box. Does it for you?

> Thanks,
> ---George Jones

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Michael Albinus]

George Jones <eludom@gmail.com> writes:

> Still need to 
> (setq temporary-file-directory "/tmp/") 
>
> for things to work, else I get:
>
> Couldn't write region to
> `/scpc:george@localhost#2222:/var/folders/j6/j6w17khs3vl9s2_yg4rb5zsm0000gn/T/input-
> 11249kNp', decode using `base64 -d -i >%s\
> ' failed 
>
> Test script and output, without explicitly setting
> temporary-file-directory, attached:

Indeed. I've published a patch to ob.el (as of Emacs trunk). Meanwhile,
ob.el has been renamed to ob-core.el; likely that patch was lost :-(

Bastien, could you please apply this patch (for `org-babel-temp-file')?

> Thanks,
> ---George Jones

Best regards, Michael.

Re: remote execution in heterogeneous environment

[Bastien]

Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> Indeed. I've published a patch to ob.el (as of Emacs trunk). Meanwhile,
> ob.el has been renamed to ob-core.el; likely that patch was lost :-(
>
> Bastien, could you please apply this patch (for `org-babel-temp-file')?

Mhh... not sure what patch you refer to -- can you send me the patch
itself and/or a link to it?

Thanks!

-- 
 Bastien

Re: remote execution in heterogeneous environment

[Michael Albinus]

Bastien <bzg@altern.org> writes:

> Hi Michael,

Hi Bastien,

>> Indeed. I've published a patch to ob.el (as of Emacs trunk). Meanwhile,
>> ob.el has been renamed to ob-core.el; likely that patch was lost :-(
>>
>> Bastien, could you please apply this patch (for `org-babel-temp-file')?
>
> Mhh... not sure what patch you refer to -- can you send me the patch
> itself and/or a link to it?

<http://thread.gmane.org/gmane.emacs.orgmode/63586/focus=63709>

The last patch in this message. Again, the function seems to be moved to
ob-core.el now.

> Thanks!

Re: remote execution in heterogeneous environment

[Bastien]

Hi Michael,

Michael Albinus <michael.albinus@gmx.de> writes:

> <http://thread.gmane.org/gmane.emacs.orgmode/63586/focus=63709>
>
> The last patch in this message. Again, the function seems to be moved to
> ob-core.el now.

Ah okay, applied, thanks!

-- 
 Bastien

Re: remote execution in heterogeneous environment

[George Jones]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

Should a git pull of development version of org pull the right patch?

Thanks,
---george jones
On Dec 23, 2012 11:47 AM, "Bastien" <bzg@altern.org> wrote:

> Hi Michael,
>
> Michael Albinus <michael.albinus@gmx.de> writes:
>
> > <http://thread.gmane.org/gmane.emacs.orgmode/63586/focus=63709>
> >
> > The last patch in this message. Again, the function seems to be moved to
> > ob-core.el now.
>
> Ah okay, applied, thanks!
>
> --
>  Bastien
>

[-- Attachment #2: Type: text/html, Size: 908 bytes --]

Re: remote execution in heterogeneous environment

[Bastien]

George Jones <eludom@gmail.com> writes:

> Should a git pull of development version of org pull the right patch?

Yes.  

Simply ~$ git clone git://orgmode.org/org-mode.git (our master branch 
is the development branch.)

HTH,

-- 
 Bastien

Re: remote execution in heterogeneous environment

[George Jones]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

Success.   No setting of temporary-file-directory needed.

Thanks,
---George



On Sun, Dec 23, 2012 at 2:54 PM, Bastien <bzg@altern.org> wrote:

> George Jones <eludom@gmail.com> writes:
>
> > Should a git pull of development version of org pull the right patch?
>
> Yes.
>
> Simply ~$ git clone git://orgmode.org/org-mode.git (our master branch
> is the development branch.)
>
> HTH,
>
> --
>  Bastien
>

[-- Attachment #2: Type: text/html, Size: 1001 bytes --]

Re: remote execution in heterogeneous environment

[Michael Albinus]

George Jones <eludom@gmail.com> writes:

> Success. No setting of temporary-file-directory needed.

Great! Thanks for your inisistent testing.

> Thanks,
> ---George

Best regards, Micvhael.