[BUG][babel] ":result output table" doesn't work for python code blocks

[BUG][babel] ":result output table" doesn't work for python code blocks

[Ethan Ligon]

The results from python code blocks aren't correctly returned as tables
when they should be.  To wit:

#+begin_src python :results output table
print "A,B,C"
#+end_src

#+results:
: A,B,C

where I'd expect to see

#+results:
| A | B | C |

Eric Fraga seems to have noticed the same problem, but with octave code
blocks in 

http://thread.gmane.org/gmane.emacs.orgmode/36091

but so far there's been no useful response to his post.

Thanks for any help!
-Ethan Ligon

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric Schulte]

Ethan Ligon <ligon@are.berkeley.edu> writes:

> The results from python code blocks aren't correctly returned as tables
> when they should be.  To wit:
>
> #+begin_src python :results output table
> print "A,B,C"
> #+end_src
>
> #+results:
> : A,B,C
>
> where I'd expect to see
>
> #+results:
> | A | B | C |
>

In many languages ":results output" will always return a scalar, the
thinking being, that what was printed to STDOUT was a series of strings
and thus should be inserted into the buffer as a string, rather than for
example

#+begin_src python
  return [1, 2, 3]
#+end_src

#+results:
| 1 | 2 | 3 |

in which case the value returned is a list, and is thus inserted into
the buffer as a list.

There are currently options for printing to standard out and having the
results inserted as a list.  For example,

#+begin_src python :results output raw
  print "| 1 | 2 | 3 |"
#+end_src

#+results:
| 1 | 2 | 3 |

That said, I agree that in examples like yours above the returned value
should be a table given that the ":results table" is explicitly stated.
I've just pushed up a patch after which the following is possible.

#+begin_src python :results output table
  print "[A, B, C]"
#+end_src

#+results:
| A | B | C |

Note that your exact example above would still not return a list as the
printed value does not look like a python list.

Cheers -- Eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

[...]

> That said, I agree that in examples like yours above the returned value
> should be a table given that the ":results table" is explicitly stated.
> I've just pushed up a patch after which the following is possible.

Eric,

It would appear that this change you've made is only for python)?  Is
there any chance of having the same for octave, please?  But only if it
is easy to do as =:results output raw= with carefully formatted output
does the job for me for the moment!

Thanks!

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.181.g60864)

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
> [...]
>
>> That said, I agree that in examples like yours above the returned value
>> should be a table given that the ":results table" is explicitly stated.
>> I've just pushed up a patch after which the following is possible.
>
> Eric,
>
> It would appear that this change you've made is only for python)?  Is
> there any chance of having the same for octave, please?  But only if it
> is easy to do as =:results output raw= with carefully formatted output
> does the job for me for the moment!
>

Hi Eric,

I do not have a local copy of octave, so I'm less confident making
changes to that file, but the attached patch attempts to make the same
changes in ob-octave which were made in ob-python.

Could you please test this patch for both external and session based
evaluation and let me know if it works (I'm more hopeful that the
external evaluation will work as expected than the session evaluation).

Once this is working I'll commit it to the core.

Thanks -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-octave-allow-collecting-tabular-results-when-resu.patch --]
[-- Type: text/x-diff, Size: 3905 bytes --]

From 7477c3ac10d28342ccf993ea36b41ebfcab015ac Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 15 Apr 2011 10:11:18 -0600
Subject: [PATCH] ob-octave: allow collecting tabular results when ":results output"

* lisp/ob-octave.el (org-babel-octave-evaluate-external-process): Dump
  output to temp file, and read results from that file as with ":results value".
  (org-babel-octave-evaluate-session): Read results from temporary
  file with both ":results output" and ":results value".
---
 lisp/ob-octave.el |   55 ++++++++++++++++++++++++++++------------------------
 1 files changed, 30 insertions(+), 25 deletions(-)

diff --git a/lisp/ob-octave.el b/lisp/ob-octave.el
index 3430dea..064677e 100644
--- a/lisp/ob-octave.el
+++ b/lisp/ob-octave.el
@@ -164,18 +164,18 @@ value of the last statement in BODY, as elisp."
 
 (defun org-babel-octave-evaluate-external-process (body result-type matlabp)
   "Evaluate BODY in an external octave process."
-  (let ((cmd (if matlabp
+  (let ((tmp-file (org-babel-temp-file "octave-"))
+	(cmd (if matlabp
 		 org-babel-matlab-shell-command
 	       org-babel-octave-shell-command)))
     (case result-type
-      (output (org-babel-eval cmd body))
-      (value (let ((tmp-file (org-babel-temp-file "octave-")))
-	       (org-babel-eval
-		cmd
-		(format org-babel-octave-wrapper-method body
-			(org-babel-process-file-name tmp-file 'noquote)
-			(org-babel-process-file-name tmp-file 'noquote)))
-	       (org-babel-octave-import-elisp-from-file tmp-file))))))
+      (output (with-temp-file tmp-file (insert (org-babel-eval cmd body))))
+      (value (org-babel-eval
+	      cmd
+	      (format org-babel-octave-wrapper-method body
+		      (org-babel-process-file-name tmp-file 'noquote)
+		      (org-babel-process-file-name tmp-file 'noquote)))))
+    (org-babel-octave-import-elisp-from-file tmp-file)))
 
 (defun org-babel-octave-evaluate-session
   (session body result-type &optional matlabp)
@@ -194,7 +194,8 @@ value of the last statement in BODY, as elisp."
 		  (format org-babel-matlab-emacs-link-wrapper-method
 			  body
 			  (org-babel-process-file-name tmp-file 'noquote)
-			  (org-babel-process-file-name tmp-file 'noquote) wait-file) "\n")
+			  (org-babel-process-file-name tmp-file 'noquote)
+			  wait-file) "\n")
 	       (mapconcat
 		#'org-babel-chomp
 		(list (format org-babel-octave-wrapper-method
@@ -221,21 +222,25 @@ value of the last statement in BODY, as elisp."
 		       org-babel-octave-eoe-output)
 		     t full-body)
 		  (insert full-body) (comint-send-input nil t)))) results)
-    (case result-type
-      (value
-       (org-babel-octave-import-elisp-from-file tmp-file))
-      (output
-       (progn
-	 (setq results
-	       (if matlabp
-		   (cdr (reverse (delq "" (mapcar
-					   #'org-babel-octave-read-string
-					   (mapcar #'org-babel-trim raw)))))
-		 (cdr (member org-babel-octave-eoe-output
-			      (reverse (mapcar
-					#'org-babel-octave-read-string
-					(mapcar #'org-babel-trim raw)))))))
-	 (mapconcat #'identity (reverse results) "\n"))))))
+    (if (or (member "code" result-params)
+	    (member "pp" result-params)
+	    (member "scalar" result-params)
+	    (and (member "output" result-params)
+		 (not (member "table" result-params))))
+	(progn
+	  (setq results
+		(if matlabp
+		    (cdr (reverse (delq "" (mapcar
+					    #'org-babel-octave-read-string
+					    (mapcar #'org-babel-trim raw)))))
+		  (cdr (member org-babel-octave-eoe-output
+			       (reverse (mapcar
+					 #'org-babel-octave-read-string
+					 (mapcar #'org-babel-trim raw)))))))
+	  (mapconcat #'identity (reverse results) "\n"))
+      (when (equal results-type 'output)
+	(with-temp-file tmp-file (insert raw)))
+      (org-babel-octave-import-elisp-from-file tmp-file))))
 
 (defun org-babel-octave-import-elisp-from-file (file-name)
   "Import data from FILE-NAME.
-- 
1.7.1


[-- Attachment #3: Type: text/plain, Size: 47 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>
>> [...]
>>
>>> That said, I agree that in examples like yours above the returned value
>>> should be a table given that the ":results table" is explicitly stated.
>>> I've just pushed up a patch after which the following is possible.
>>
>> Eric,
>>
>> It would appear that this change you've made is only for python)?  Is
>> there any chance of having the same for octave, please?  But only if it
>> is easy to do as =:results output raw= with carefully formatted output
>> does the job for me for the moment!
>>
>
> Hi Eric,
>
> I do not have a local copy of octave, so I'm less confident making
> changes to that file, but the attached patch attempts to make the same
> changes in ob-octave which were made in ob-python.
>
> Could you please test this patch for both external and session based
> evaluation and let me know if it works (I'm more hopeful that the
> external evaluation will work as expected than the session
> evaluation).

Eric,

thanks for this.

I haven't tried the session based evaluation but the external one,
unfortunately, doesn't work as I would expect.  In the following, I
would expect both source code blocks to give me the same output with the
first, value based, being the correct output.

--8<---------------cut here---------------start------------->8---

  #+srcname: valueresult
  #+begin_src octave :results value table :cache yes
values = [];
for i=1:15
  values(end+1,:) = [i, i^3];
endfor
ans = values
  #+end_src

  #+results[7a87a711b7814ecf23ec5d3741a5fec3ffff3e44]: valueresult
  |  1 |    1 |
  |  2 |    8 |
  |  3 |   27 |
  |  4 |   64 |
  |  5 |  125 |
  |  6 |  216 |
  |  7 |  343 |
  |  8 |  512 |
  |  9 |  729 |
  | 10 | 1000 |
  | 11 | 1331 |
  | 12 | 1728 |
  | 13 | 2197 |
  | 14 | 2744 |
  | 15 | 3375 |

  
  #+srcname: outputresult
  #+begin_src octave :results output table :cache yes
values = [];
for i=1:15
  values(end+1,:) = [i, i^3];
endfor
disp(values)
  #+end_src

  #+results[3104573dcf8ec30de1a26aae7051a9e8fce6a92a]: outputresult
  | 1      1  |
  | 2      8  |
  | 3     27  |
  | 4     64  |
  | 5    125  |
  | 6    216  |
  | 7    343  |
  | 8    512  |
  | 9    729  |
  | 10   1000 |
  | 11   1331 |
  | 12   1728 |
  | 13   2197 |
  | 14   2744 |
  | 15   3375 |
--8<---------------cut here---------------end--------------->8---

Thanks,
eric
-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.183.g1997.dirty)

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric Schulte]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> "Eric Schulte" <schulte.eric@gmail.com> writes:
>>>
>>> [...]
>>>
>>>> That said, I agree that in examples like yours above the returned value
>>>> should be a table given that the ":results table" is explicitly stated.
>>>> I've just pushed up a patch after which the following is possible.
>>>
>>> Eric,
>>>
>>> It would appear that this change you've made is only for python)?  Is
>>> there any chance of having the same for octave, please?  But only if it
>>> is easy to do as =:results output raw= with carefully formatted output
>>> does the job for me for the moment!
>>>
>>
>> Hi Eric,
>>
>> I do not have a local copy of octave, so I'm less confident making
>> changes to that file, but the attached patch attempts to make the same
>> changes in ob-octave which were made in ob-python.
>>
>> Could you please test this patch for both external and session based
>> evaluation and let me know if it works (I'm more hopeful that the
>> external evaluation will work as expected than the session
>> evaluation).
>
> Eric,
>
> thanks for this.
>
> I haven't tried the session based evaluation but the external one,
> unfortunately, doesn't work as I would expect.  In the following, I
> would expect both source code blocks to give me the same output with the
> first, value based, being the correct output.
>

Hi Eric,

Does the matlab/octave `disp' function display tabular data in the same
manner as it is written literally in source code?

This is part of the issue with the "output table" :results combination,
namely what constitutes a table for printed output.  In source code the
answer is obvious, namely whatever the language's interpreter would read
as a literal table, however with printed output there are many possible
ways to represent tabular data, but none *is* tabular in the way that
source code can *be* tabular.

If the printed output is exactly the same as how a table would be
written in matlab/octave source code, then does my patched version work?
If not, then rather than writing another table parser, perhaps the data
could be printed as an Org-mode table, and then the "output raw"
:results combination could be used, or the output could be sent through
another code block to convert the string to a table.

Hope this helps, Best -- Eric

>
>
>   #+srcname: valueresult
>   #+begin_src octave :results value table :cache yes
> values = [];
> for i=1:15
>   values(end+1,:) = [i, i^3];
> endfor
> ans = values
>   #+end_src
>
>   #+results[7a87a711b7814ecf23ec5d3741a5fec3ffff3e44]: valueresult
>   |  1 |    1 |
>   |  2 |    8 |
>   |  3 |   27 |
>   |  4 |   64 |
>   |  5 |  125 |
>   |  6 |  216 |
>   |  7 |  343 |
>   |  8 |  512 |
>   |  9 |  729 |
>   | 10 | 1000 |
>   | 11 | 1331 |
>   | 12 | 1728 |
>   | 13 | 2197 |
>   | 14 | 2744 |
>   | 15 | 3375 |
>
>   
>   #+srcname: outputresult
>   #+begin_src octave :results output table :cache yes
> values = [];
> for i=1:15
>   values(end+1,:) = [i, i^3];
> endfor
> disp(values)
>   #+end_src
>
>   #+results[3104573dcf8ec30de1a26aae7051a9e8fce6a92a]: outputresult
>   | 1      1  |
>   | 2      8  |
>   | 3     27  |
>   | 4     64  |
>   | 5    125  |
>   | 6    216  |
>   | 7    343  |
>   | 8    512  |
>   | 9    729  |
>   | 10   1000 |
>   | 11   1331 |
>   | 12   1728 |
>   | 13   2197 |
>   | 14   2744 |
>   | 15   3375 |
>
> Thanks,
> eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

[...]

>
> Hi Eric,
>
> Does the matlab/octave `disp' function display tabular data in the same
> manner as it is written literally in source code?

Hi Eric!

No, unfortunately not.  Arrays (tabular data) are written out using only
whitespace for formatting:

--8<---------------cut here---------------start------------->8---
*** simple example
    #+srcname: octave-simple
    #+begin_src octave :results output 
A = [1, 2; 3, 4];
disp(A)
    #+end_src

    #+results: octave-simple
    :    1   2
    :    3   4
--8<---------------cut here---------------end--------------->8---

> This is part of the issue with the "output table" :results combination,
> namely what constitutes a table for printed output.  In source code the
> answer is obvious, namely whatever the language's interpreter would read
> as a literal table, however with printed output there are many possible
> ways to represent tabular data, but none *is* tabular in the way that
> source code can *be* tabular.

Yes, that makes sense.

> If the printed output is exactly the same as how a table would be
> written in matlab/octave source code, then does my patched version work?
> If not, then rather than writing another table parser, perhaps the data
> could be printed as an Org-mode table, and then the "output raw"
> :results combination could be used, or the output could be sent through
> another code block to convert the string to a table.

That's what I do at the moment, actually.

Mind you, org is able to take the output and convert it to a table
easily enough with =C-c |= (org-table-create-or-convert-from-region)
operating on the output if selected as a region.  Would it be possible
to post-process the output from babel automatically using this method
(maybe with a hook?)  with the output selected if ":results output
table", say, were specified?  That alone would be sufficient for 90% of
the cases...

Apologies for the sporadic nature of my responses lately: I have been on
holiday (much needed!) and purposely without email most of the time!

Thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.209.g1a687)

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric Schulte]

>> perhaps the data could be printed as an Org-mode table, and then the
>> "output raw" :results combination could be used, or the output could
>> be sent through another code block to convert the string to a table.
>
> That's what I do at the moment, actually.
>
> Mind you, org is able to take the output and convert it to a table
> easily enough with =C-c |= (org-table-create-or-convert-from-region)
> operating on the output if selected as a region.  Would it be possible
> to post-process the output from babel automatically using this method
> (maybe with a hook?)  with the output selected if ":results output
> table", say, were specified?  That alone would be sufficient for 90% of
> the cases...
>

Ah, that sounds like a much more satisfying and general solution than
the previous approach of parsing output in language-specific manners.
Although I don't have time at the moment, I will put automatic
conversion of org type results onto my todo list, so that at some point
in the future, code like the following will be possible...

#+source: org-results
#+begin_src sh :results output org
  echo "| 1 |"
  echo "| 2 |"
#+end_src

#+begin_src emacs-lisp :var in=org-results
  (listp in)
#+end_src

#+results:
: t

>
> Apologies for the sporadic nature of my responses lately: I have been on
> holiday (much needed!) and purposely without email most of the time!
>

No need to apologize!  I try to do the same when on vacation.

Best -- Eric

>
> Thanks,
> eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG][babel] ":result output table" doesn't work for python code blocks

[Eric S Fraga]

"Eric Schulte" <schulte.eric@gmail.com> writes:

>>> perhaps the data could be printed as an Org-mode table, and then the
>>> "output raw" :results combination could be used, or the output could
>
>>> be sent through another code block to convert the string to a table.
>>
>> That's what I do at the moment, actually.
>>
>> Mind you, org is able to take the output and convert it to a table
>> easily enough with =C-c |= (org-table-create-or-convert-from-region)
>> operating on the output if selected as a region.  Would it be possible
>> to post-process the output from babel automatically using this method
>> (maybe with a hook?)  with the output selected if ":results output
>> table", say, were specified?  That alone would be sufficient for 90% of
>> the cases...
>>
>
> Ah, that sounds like a much more satisfying and general solution than
> the previous approach of parsing output in language-specific manners.
> Although I don't have time at the moment, I will put automatic
> conversion of org type results onto my todo list, so that at some point
> in the future, code like the following will be possible...
>
> #+source: org-results
> #+begin_src sh :results output org
>   echo "| 1 |"
>   echo "| 2 |"
> #+end_src
>
> #+begin_src emacs-lisp :var in=org-results
>   (listp in)
> #+end_src
> #+results:
> : t

Eric,

I am not sure I explained myself clearly above.  What I meant is that
the output of, say, this octave code:

--8<---------------cut here---------------start------------->8---
    #+srcname: octave-formatted
    #+begin_src octave :results output raw
A = [1, 2; 3, 4];
disp(A)
    #+end_src

    #+results: octave-formatted
       1   2
       3   4
--8<---------------cut here---------------end--------------->8---

can subsequently be selected and converted to a table by applying the
org-table-create-or-convert-from-region command, which gives:

--8<---------------cut here---------------start------------->8---
| 1 | 2 |
| 3 | 4 |
--8<---------------cut here---------------end--------------->8---

It's not about identifying /org-like/ output but simply applying that
function automatically to the output, if "output table" has been
requested.

I hope this makes more sense?

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.209.g1a687)