org-mode, python and unicode

[Daniele Pizzolli <dan@toel.it>]

Hello,

I found some problems and workarounds in the use of python (version
2) and org-mode.  This applies to:

#+BEGIN_EXAMPLE
Org-mode version 8.3.4 (8.3.4-60-g19cf68-elpa @ /home/vagrant/.emacs.d/elpa/org-20160516/)
#+END_EXAMPLE

Some workaround are not complete, and suggestions are welcome!

Best,
Daniele

* python

** load python support for babel

#+BEGIN_SRC emacs-lisp
(org-babel-do-load-languages
 'org-babel-load-languages
 '((python . t)))
#+END_SRC

#+RESULTS:
: ((python . t))

** passing utf-8 strings to python

#+NAME: unicode_str
#+BEGIN_EXAMPLE
“this string is not ascii!”
#+END_EXAMPLE

#+NAME: error-in-passing-var
#+BEGIN_SRC python :var data=unicode_str
return data
#+END_SRC

#+RESULTS: error-in-passing-var

Will result in the following message in the buffer =*Org-Babel Error Output*=:

#+BEGIN_EXAMPLE
  File "<stdin>", line 3
SyntaxError: Non-ASCII character '\xe2' in file <stdin> on line 3, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
#+END_EXAMPLE

** passing utf-8 strings to python with workaround

A workaround is to use =:preamble= with wthe value =# -*- coding:utf-8 -*-=

#+NAME: ok-in-passing-var
#+BEGIN_SRC python :preamble "# -*- coding: utf-8 -*-" :var data=unicode_str
return data
#+END_SRC

#+RESULTS: ok-in-passing-var
: “this string is not ascii!”

** passing utf-8 tables to python

#+NAME: unicode_table
| key | value                       |
|-----+-----------------------------|
|   1 | “this string is not ascii!” |

#+NAME: error-in-passing-table
#+BEGIN_SRC python :var data=unicode_table
return data
#+END_SRC

Will result in the following message in the buffer =*Org-Babel Error Output*=:

#+BEGIN_EXAMPLE
  File "<stdin>", line 3
SyntaxError: Non-ASCII character '\xe2' in file <stdin> on line 3, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
#+END_EXAMPLE

** returning utf-8 tables from python

#+NAME: error-in-returnig-table
#+BEGIN_SRC python :preamble "# -*- coding: utf-8 -*-" :var data=unicode_table :colnames yes
return data
#+END_SRC

Will output the ascii text with escape sequence for unicode
characters.  I still do not know the best workaround

#+RESULTS: error-in-returnig-table
| key | value                                             |
|-----+---------------------------------------------------|
|   1 | \xe2\x80\x9cthis string is not ascii!\xe2\x80\x9d |

Please note that the single cell is correctly returned:

#+NAME: returnig-table-cell
#+BEGIN_SRC python :preamble "# -*- coding: utf-8 -*-" :var data=unicode_table
return data[0][1]
#+END_SRC

#+RESULTS: returnig-table-cell
: “this string is not ascii!”

** Undocumented parts

I did not found out:

- how to get the table headers in python

- in general, the magic that org-mode does to pass and retrieve data
  structures to various languages is lacking documentation