org-mode and python pandas

org-mode and python pandas

[Dov Grobgeld]

Has anyone used org-mode with the python pandas package? Pandas is in
a certain way an alternative to R, but with the (for me) familiar
syntax of python. See: http://pandas.pydata.org/

Pandas is very much built to be used interactively, and it outputs its
data in space separated tabular format. E.g. in ipython:

In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
Out[3]:
          A         B         C
0  0.628365  0.424279  0.619791
1  0.799666  0.527572  0.132928
2  0.837255  0.138906  0.408233
3  0.388080  0.146212  0.575346

Unfortunately this doesn't output as nicely when used from org-mode:

#+BEGIN_SRC python
import pandas as pd
import numpy as np

return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
#+END_SRC

#+RESULTS:
: A         B         C
: 0  0.827817  0.664009  0.089161
: 1  0.170031  0.729214  0.110918
: 2  0.575918  0.863924  0.757536
: 3  0.682722  0.774445  0.992041

while I would like to have:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

The question is how to get this? Here are a few ideas:

1. Write a general filter in the org-mode elisp than uses heuristics
to recognize ascii aligned tables and change these to org-tables.
2. Add to pandas the option of globally influencing the text
formatting so that it outputs something more parsable by org-mode.
3. Create a special language "pandas" that recognize the ascii aligned
tables and saves the need to import pandas and np?
4. And the obvious approach of writing a python function that writes a
org-mode parsable table and always call it as part of the return.

Which is the preferable approach? Any other ideas?

Regards,
Dov

Re: org-mode and python pandas

[Eric Schulte]

Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Has anyone used org-mode with the python pandas package? Pandas is in
> a certain way an alternative to R, but with the (for me) familiar
> syntax of python. See: http://pandas.pydata.org/
>
> Pandas is very much built to be used interactively, and it outputs its
> data in space separated tabular format. E.g. in ipython:
>
> In [1]: import pandas as pd
> In [2]: import numpy as np
>
> In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
> Out[3]:
>           A         B         C
> 0  0.628365  0.424279  0.619791
> 1  0.799666  0.527572  0.132928
> 2  0.837255  0.138906  0.408233
> 3  0.388080  0.146212  0.575346
>
> Unfortunately this doesn't output as nicely when used from org-mode:
>
> #+BEGIN_SRC python
> import pandas as pd
> import numpy as np
>
> return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
> #+END_SRC
>
> #+RESULTS:
> : A         B         C
> : 0  0.827817  0.664009  0.089161
> : 1  0.170031  0.729214  0.110918
> : 2  0.575918  0.863924  0.757536
> : 3  0.682722  0.774445  0.992041
>
> while I would like to have:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>

What happens if you add ":results table" to your code block?  Would that
be sufficient?

>
> The question is how to get this? Here are a few ideas:
>
> 1. Write a general filter in the org-mode elisp than uses heuristics
> to recognize ascii aligned tables and change these to org-tables.

The default value should be to convert multi-line output to tables, the
":results table" option above will force this conversion in case it is
currently not taking place due to the default header arguments in use.

> 
> 2. Add to pandas the option of globally influencing the text
> formatting so that it outputs something more parsable by org-mode.

This sounds promising, if pandas support csv output that will be
correctly parsed by Org-mode.

> 
> 3. Create a special language "pandas" that recognize the ascii aligned
> tables and saves the need to import pandas and np?  4. And the obvious
> approach of writing a python function that writes a org-mode parsable
> table and always call it as part of the return.
>
> Which is the preferable approach? Any other ideas?
>

I think a header-argument-based approach would be ideal, I'd look at the
value of org-babel-default-header-args:python, and read the portion of
the manual related to the "results" header arguments.

I don't understand multi-line strings in python, but I get the following
behavior from simple shell script blocks.

#+begin_src sh
  cat <<EOF
            A         B         C
  0  0.628365  0.424279  0.619791
  1  0.799666  0.527572  0.132928
  2  0.837255  0.138906  0.408233
  3  0.388080  0.146212  0.575346
  EOF
#+end_src

#+RESULTS:
| A |        B |        C |          |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |
| 2 | 0.837255 | 0.138906 | 0.408233 |
| 3 |  0.38808 | 0.146212 | 0.575346 |

#+begin_src sh
  cat <<EOF
  ,A,B,C
  0,0.628365,0.424279,0.619791
  1,0.799666,0.527572,0.132928
  2,0.837255,0.138906,0.408233
  3,0.388080,0.146212,0.575346
  EOF
#+end_src

#+RESULTS:
|   |        A |        B |        C |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |
| 2 | 0.837255 | 0.138906 | 0.408233 |
| 3 |  0.38808 | 0.146212 | 0.575346 |

Hope this helps,

>
> Regards,
> Dov
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: org-mode and python pandas

[Achim Gratz]

Eric Schulte writes:
>> |   |        A |        B |        C |
>> |---+----------+----------+----------|
>> | 0 | 0.827817 | 0.664009 | 0.089161 |
>> | 1 | 0.170031 | 0.729214 | 0.110918 |
>> | 2 | 0.575918 | 0.863924 | 0.757536 |
>> | 3 | 0.682722 | 0.774445 | 0.992041 |
>>
>
> What happens if you add ":results table" to your code block?  Would that
> be sufficient?

The problem is that he's trying to consume human-readable output.  The
alignment in this output is done with spaces, not tabs (as Org would
expect).  There are more problems considering that the numbers likely
lose precision in this format (nota bene: there are other Babel
languages that have that same problem).

> The default value should be to convert multi-line output to tables, the
> ":results table" option above will force this conversion in case it is
> currently not taking place due to the default header arguments in use.

Since there are no tabs in there, it is impossible to skip empty cells
like the first one (they'd have to be written as "" instead, as would be
cells that should contain literal spaces).

>> 2. Add to pandas the option of globally influencing the text
>> formatting so that it outputs something more parsable by org-mode.
>
> This sounds promising, if pandas support csv output that will be
> correctly parsed by Org-mode.

The package already has CSV export, so one could use that.  I don't know
if you could echo the result directly to the output, all examples
revolve around putting the CSV into a file.  For Org, TSV output would
be more natural.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: org-mode and python pandas

[Rasmus]

Achim Gratz <Stromeko@nexgo.de> writes:

>>> 2. Add to pandas the option of globally influencing the text
>>> formatting so that it outputs something more parsable by org-mode.
>>
>> This sounds promising, if pandas support csv output that will be
>> correctly parsed by Org-mode.
>
> The package already has CSV export, so one could use that.  I don't know
> if you could echo the result directly to the output, all examples
> revolve around putting the CSV into a file.  For Org, TSV output would
> be more natural.

Something like:

from pandas import DataFrame
from numpy.random import rand
from sys import stdout
df = DataFrame(rand(10,3), columns = list('abc'))
df
df.to_csv(stdout, sep="\t", header = True, cols=(1,2))

I was completely unable to get ob-python working this morning, so I
haven't tested it.  I'm using python3, build in python mode and elpy.

In any case, the csv route might be better, as Pandas doesn't print
the table if it's too big (try changing 10 to 1000 above).

-- 
Powered by magic pixies!

Re: org-mode and python pandas

[Dov Grobgeld]

Thanks for the answers, but there is still something missing in order
to get it to work. Part of it seems to be connected to the python
parsing. E.g. the following translation of Eric's sh example doesn't
output correctly with python:


#+BEGIN_SRC python :results output
print """,A,B,C
0,0.628365,0.424279,0.619791
1,0.799666,0.527572,0.132928
2,0.837255,0.138906,0.408233
3,0.388080,0.146212,0.575346
"""
#+END_SRC

#+RESULTS:
: ,A,B,C
: 0,0.628365,0.424279,0.619791
: 1,0.799666,0.527572,0.132928
: 2,0.837255,0.138906,0.408233
: 3,0.388080,0.146212,0.575346


#+BEGIN_SRC python :results table
return """,A,B,C
0,0.628365,0.424279,0.619791
1,0.799666,0.527572,0.132928
2,0.837255,0.138906,0.408233
3,0.388080,0.146212,0.575346
"""
#+END_SRC

#+RESULTS:
| ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
|

It seems that the only way to get a table from python is by outputting
a two dimensional python structure:

#+BEGIN_SRC python
return [[0,0.628365,0.424279,0.619791],
        [1,0.799666,0.527572,0.132928]]
#+END_SRC

#+RESULTS:
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

This seems quite limiting....

Another related question is if there is any support for header tables?
I.e. instead of this:

|   |        A |        B |        C |
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

I want this:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

I guess that if I start playing around with the python ob module, it
should be possible to get this working?

Regards,
Dov

On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>>>> 2. Add to pandas the option of globally influencing the text
>>>> formatting so that it outputs something more parsable by org-mode.
>>>
>>> This sounds promising, if pandas support csv output that will be
>>> correctly parsed by Org-mode.
>>
>> The package already has CSV export, so one could use that.  I don't know
>> if you could echo the result directly to the output, all examples
>> revolve around putting the CSV into a file.  For Org, TSV output would
>> be more natural.
>
> Something like:
>
> from pandas import DataFrame
> from numpy.random import rand
> from sys import stdout
> df = DataFrame(rand(10,3), columns = list('abc'))
> df
> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
>
> I was completely unable to get ob-python working this morning, so I
> haven't tested it.  I'm using python3, build in python mode and elpy.
>
> In any case, the csv route might be better, as Pandas doesn't print
> the table if it's too big (try changing 10 to 1000 above).
>
> --
> Powered by magic pixies!
>
>

Re: org-mode and python pandas

[Rasmus]

Hi Dov,

> Another related question is if there is any support for header tables?
> I.e. instead of this:
>
> |   |        A |        B |        C |
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I want this:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |

This is an issue in R as well, e.g. often I might want to have 
|A|
|-|
|1|
|2|
|-|
|3|

where the last is a summary statistic.  The last hline can be gotten
as 

#+BEGIN_SRC emacs-lisp :var X=mytabel :exports results
    (let
        ((L (- (length X) 1)))
      (append
       (subseq X 0 L)
       (list 'hline)
       (subseq X L)))
#+END_SRC

Perhaps a better way exists?

–Rasmus

-- 
. . . It begins of course with The Internet.  A Net of Peers.

Re: org-mode and python pandas

[Eric Schulte]

Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Thanks for the answers, but there is still something missing in order
> to get it to work. Part of it seems to be connected to the python
> parsing. E.g. the following translation of Eric's sh example doesn't
> output correctly with python:
>
>
> #+BEGIN_SRC python :results output
> print """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> : ,A,B,C
> : 0,0.628365,0.424279,0.619791
> : 1,0.799666,0.527572,0.132928
> : 2,0.837255,0.138906,0.408233
> : 3,0.388080,0.146212,0.575346
>
>
> #+BEGIN_SRC python :results table
> return """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
> |
>
> It seems that the only way to get a table from python is by outputting
> a two dimensional python structure:
>
> #+BEGIN_SRC python
> return [[0,0.628365,0.424279,0.619791],
>         [1,0.799666,0.527572,0.132928]]
> #+END_SRC
>
> #+RESULTS:
> | 0 | 0.628365 | 0.424279 | 0.619791 |
> | 1 | 0.799666 | 0.527572 | 0.132928 |
>
> This seems quite limiting....
>

In most cases this is what one wants when returning data from python
code.  The following elisp defined a "panda" code block, which is just
like python, only it assumes that the results will be these sort of
human readable strings instead of python code.

    ;; -*- emacs-lisp -*-
    (defun org-babel-execute:panda (body params)
      (let ((results
             (org-babel-execute:python
              body (org-babel-merge-params '((:results . "scalar")) params))))
        (org-babel-result-cond (cdr (assoc :result-params params))
          results
          (let ((tmp-file (org-babel-temp-file "sh-")))
            (with-temp-file tmp-file (insert results))
            (org-babel-import-elisp-from-file tmp-file)))))

With the above evaluated the following works

    #+BEGIN_SRC panda
    return """,A,B,C
    0,0.628365,0.424279,0.619791
    1,0.799666,0.527572,0.132928
    2,0.837255,0.138906,0.408233
    3,0.388080,0.146212,0.575346
    """
    #+END_SRC

    #+RESULTS:
    |   |        A |        B |        C |
    | 0 | 0.628365 | 0.424279 | 0.619791 |
    | 1 | 0.799666 | 0.527572 | 0.132928 |
    | 2 | 0.837255 | 0.138906 | 0.408233 |
    | 3 |  0.38808 | 0.146212 | 0.575346 |

>
> Another related question is if there is any support for header tables?
> I.e. instead of this:
>
> |   |        A |        B |        C |
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I want this:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I guess that if I start playing around with the python ob module, it
> should be possible to get this working?
>

See the :colnames header argument in the manual.

Best,

>
> Regards,
> Dov
>
> On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
>> Achim Gratz <Stromeko@nexgo.de> writes:
>>
>>>>> 2. Add to pandas the option of globally influencing the text
>>>>> formatting so that it outputs something more parsable by org-mode.
>>>>
>>>> This sounds promising, if pandas support csv output that will be
>>>> correctly parsed by Org-mode.
>>>
>>> The package already has CSV export, so one could use that.  I don't know
>>> if you could echo the result directly to the output, all examples
>>> revolve around putting the CSV into a file.  For Org, TSV output would
>>> be more natural.
>>
>> Something like:
>>
>> from pandas import DataFrame
>> from numpy.random import rand
>> from sys import stdout
>> df = DataFrame(rand(10,3), columns = list('abc'))
>> df
>> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
>>
>> I was completely unable to get ob-python working this morning, so I
>> haven't tested it.  I'm using python3, build in python mode and elpy.
>>
>> In any case, the csv route might be better, as Pandas doesn't print
>> the table if it's too big (try changing 10 to 1000 above).
>>
>> --
>> Powered by magic pixies!
>>
>>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: org-mode and python pandas

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 5615 bytes --]

I returned to this issue recently and tried to get the ob-python to output
a table with a header, but didn't manage. Here is the code:

#+BEGIN_SRC python :colnames true
return [['','A','B','C'],
        [0,0.628365,0.424279,0.619791],
        [1,0.799666,0.527572,0.132928]]
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

What I want is:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

Is there any way to do that besides using the :results raw option?

Thanks!
Dov


On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte <schulte.eric@gmail.com> wrote:

> Dov Grobgeld <dov.grobgeld@gmail.com> writes:
>
> > Thanks for the answers, but there is still something missing in order
> > to get it to work. Part of it seems to be connected to the python
> > parsing. E.g. the following translation of Eric's sh example doesn't
> > output correctly with python:
> >
> >
> > #+BEGIN_SRC python :results output
> > print """,A,B,C
> > 0,0.628365,0.424279,0.619791
> > 1,0.799666,0.527572,0.132928
> > 2,0.837255,0.138906,0.408233
> > 3,0.388080,0.146212,0.575346
> > """
> > #+END_SRC
> >
> > #+RESULTS:
> > : ,A,B,C
> > : 0,0.628365,0.424279,0.619791
> > : 1,0.799666,0.527572,0.132928
> > : 2,0.837255,0.138906,0.408233
> > : 3,0.388080,0.146212,0.575346
> >
> >
> > #+BEGIN_SRC python :results table
> > return """,A,B,C
> > 0,0.628365,0.424279,0.619791
> > 1,0.799666,0.527572,0.132928
> > 2,0.837255,0.138906,0.408233
> > 3,0.388080,0.146212,0.575346
> > """
> > #+END_SRC
> >
> > #+RESULTS:
> > |
> ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
> > |
> >
> > It seems that the only way to get a table from python is by outputting
> > a two dimensional python structure:
> >
> > #+BEGIN_SRC python
> > return [[0,0.628365,0.424279,0.619791],
> >         [1,0.799666,0.527572,0.132928]]
> > #+END_SRC
> >
> > #+RESULTS:
> > | 0 | 0.628365 | 0.424279 | 0.619791 |
> > | 1 | 0.799666 | 0.527572 | 0.132928 |
> >
> > This seems quite limiting....
> >
>
> In most cases this is what one wants when returning data from python
> code.  The following elisp defined a "panda" code block, which is just
> like python, only it assumes that the results will be these sort of
> human readable strings instead of python code.
>
>     ;; -*- emacs-lisp -*-
>     (defun org-babel-execute:panda (body params)
>       (let ((results
>              (org-babel-execute:python
>               body (org-babel-merge-params '((:results . "scalar"))
> params))))
>         (org-babel-result-cond (cdr (assoc :result-params params))
>           results
>           (let ((tmp-file (org-babel-temp-file "sh-")))
>             (with-temp-file tmp-file (insert results))
>             (org-babel-import-elisp-from-file tmp-file)))))
>
> With the above evaluated the following works
>
>     #+BEGIN_SRC panda
>     return """,A,B,C
>     0,0.628365,0.424279,0.619791
>     1,0.799666,0.527572,0.132928
>     2,0.837255,0.138906,0.408233
>     3,0.388080,0.146212,0.575346
>     """
>     #+END_SRC
>
>     #+RESULTS:
>     |   |        A |        B |        C |
>     | 0 | 0.628365 | 0.424279 | 0.619791 |
>     | 1 | 0.799666 | 0.527572 | 0.132928 |
>     | 2 | 0.837255 | 0.138906 | 0.408233 |
>     | 3 |  0.38808 | 0.146212 | 0.575346 |
>
> >
> > Another related question is if there is any support for header tables?
> > I.e. instead of this:
> >
> > |   |        A |        B |        C |
> > | 0 | 0.827817 | 0.664009 | 0.089161 |
> > | 1 | 0.170031 | 0.729214 | 0.110918 |
> > | 2 | 0.575918 | 0.863924 | 0.757536 |
> > | 3 | 0.682722 | 0.774445 | 0.992041 |
> >
> > I want this:
> >
> > |   |        A |        B |        C |
> > |---+----------+----------+----------|
> > | 0 | 0.827817 | 0.664009 | 0.089161 |
> > | 1 | 0.170031 | 0.729214 | 0.110918 |
> > | 2 | 0.575918 | 0.863924 | 0.757536 |
> > | 3 | 0.682722 | 0.774445 | 0.992041 |
> >
> > I guess that if I start playing around with the python ob module, it
> > should be possible to get this working?
> >
>
> See the :colnames header argument in the manual.
>
> Best,
>
> >
> > Regards,
> > Dov
> >
> > On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
> >> Achim Gratz <Stromeko@nexgo.de> writes:
> >>
> >>>>> 2. Add to pandas the option of globally influencing the text
> >>>>> formatting so that it outputs something more parsable by org-mode.
> >>>>
> >>>> This sounds promising, if pandas support csv output that will be
> >>>> correctly parsed by Org-mode.
> >>>
> >>> The package already has CSV export, so one could use that.  I don't
> know
> >>> if you could echo the result directly to the output, all examples
> >>> revolve around putting the CSV into a file.  For Org, TSV output would
> >>> be more natural.
> >>
> >> Something like:
> >>
> >> from pandas import DataFrame
> >> from numpy.random import rand
> >> from sys import stdout
> >> df = DataFrame(rand(10,3), columns = list('abc'))
> >> df
> >> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
> >>
> >> I was completely unable to get ob-python working this morning, so I
> >> haven't tested it.  I'm using python3, build in python mode and elpy.
> >>
> >> In any case, the csv route might be better, as Pandas doesn't print
> >> the table if it's too big (try changing 10 to 1000 above).
> >>
> >> --
> >> Powered by magic pixies!
> >>
> >>
> >
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte
>

[-- Attachment #2: Type: text/html, Size: 8155 bytes --]

Re: org-mode and python pandas

[Dror Atariah]

[-- Attachment #1: Type: text/plain, Size: 4554 bytes --]

Using the tabulate python module it is possible to have the following
"inline" workaround:

------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------
#+BEGIN_SRC python :results raw
import pandas as pd
import numpy as np
from tabulate import tabulate

df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
print("foo")
return(tabulate(df, headers="keys", tablefmt="orgtbl"))
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.754799 | 0.492722 | 0.144595 |
| 1 | 0.198475 | 0.417721 | 0.083459 |
| 2 | 0.645011 | 0.444857 | 0.278874 |
| 3 | 0.314883 |   0.7521 | 0.789418 |
------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------

However, this is not optimal, mainly because it pollutes the code block.
Note, that the :results is set to `raw`.

A brief discussion with Dov, yielded the following better workaround:
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(setq org-babel-python-wrapper-method "
def main():
%s

res = main()
if 'pandas.core.frame.DataFrame' in str(type(res)):
    from tabulate import tabulate
    out = tabulate(res, headers='keys', tablefmt='orgtbl')
else:
    out = str(res)

open('%s', 'w').write(out)")
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This allows a nice output of pandas.DataFrame (again when using :results
raw). Unfortunately, this wrapper has no influence in the case when
:session is used. To that end, it is possible to hack

------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(defun org-babel-python-evaluate-session
    (session body &optional result-type result-params)
  "Pass BODY to the Python process in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
 (dump-last-value
  (lambda
    (tmp-file pp)
    (mapc
     (lambda (statement) (insert statement) (funcall send-wait))
     (if pp
 (list
  "import pprint"
  (format "open('%s', 'w').write(pprint.pformat(_))"
  (org-babel-process-file-name tmp-file 'noquote)))
       (list (format "_org_tmp = _;

if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)):
    from tabulate import tabulate
    _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl')
else:
    _org_out = _org_tmp

open('%s', 'w').write(str(_org_out))"
     (org-babel-process-file-name tmp-file
                                                          'noquote)))))))
 (input-body (lambda (body)
       (mapc (lambda (line) (insert line) (funcall send-wait))
     (split-string body "[\r\n]"))
       (funcall send-wait)))
         (results
          (case result-type
            (output
             (mapconcat
              #'org-babel-trim
              (butlast
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator t body)
                 (funcall input-body body)
                 (funcall send-wait) (funcall send-wait)
                 (insert org-babel-python-eoe-indicator)
                 (funcall send-wait))
               2) "\n"))
            (value
             (let ((tmp-file (org-babel-temp-file "python-")))
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator nil body)
                 (let ((comint-process-echoes nil))
                   (funcall input-body body)
                   (funcall dump-last-value tmp-file
                            (member "pp" result-params))
                   (funcall send-wait) (funcall send-wait)
                   (insert org-babel-python-eoe-indicator)
                   (funcall send-wait)))
               (org-babel-eval-read-file tmp-file))))))
    (unless (string= (substring org-babel-python-eoe-indicator 1 -1)
results)
      (org-babel-result-cond result-params
results
        (org-babel-python-table-or-string results)))))
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This works, but I would be surprised if this hack meets org-mode's
standards. Nevertheless, maybe someone would find it useful.

What do you think? How can it be improved?

Best,
Dror

[-- Attachment #2: Type: text/html, Size: 12485 bytes --]