[PATCH] Expanded ob-python results handling and plotting

[PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

The attached patch adds ob-python value results handling for the
following types of results:

- Dictionaries
- Numpy arrays
- Pandas dataframes
- Matplotlib figures

This is a bigger commit than I'm used to, so I thought I better send it
out before merging, in case someone notices obvious problems I missed.

Overview of changes:

Dictionaries are now transformed into alists before being converted to
lisp. Previously, they had been getting mangled, like so:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
| a | : | 1 | b | : | 2 |

But now they appear like so:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
| a | 1 |
| b | 2 |

Numpy arrays and pandas dataframes are also converted to tables
automatically now. Tables converted from Pandas dataframes have row and
column names.

To avoid conversion, you can specify "raw", "verbatim", "scalar", or
"output" in the ":results" header argument.

For plotting, you can specify "graphics" in the ":results"
header. You'll also need to provide a ":file" argument. The behavior
depends on whether using output or value results. For output results,
the current figure (pyplot.gcf) is cleared before evaluating, then the
result saved. For value results, the block is expected to return a
matplotlib Figure, which is saved. To set the figure size, do it from
within Python.

Here is an example of how to plot:

#+begin_src python :results output graphics file :file boxplot.svg
  import matplotlib.pyplot as plt
  import seaborn as sns
  plt.figure(figsize=(5, 5))
  tips = sns.load_dataset("tips")
  sns.boxplot(x="day", y="tip", data=tips)
#+end_src


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Add-results-handling-for-dicts-dataframes-.patch --]
[-- Type: text/x-patch, Size: 11849 bytes --]

From 09f9c42bb629a356e1c36f04f69c8baf795b411b Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Tue, 25 Aug 2020 21:57:24 -0700
Subject: [PATCH] ob-python: Add results handling for dicts, dataframes,
 arrays, plots

* lisp/ob-python.el (org-babel-execute:python): Parse graphics-file
from params.
(org-babel-python--def-format-value): Python code for formatting
value results before returning.
(org-babel-python--output-graphics-wrapper): Python code for handling
output graphics results.
(org-babel-python--nonsession-value-wrapper): Replaces
org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method.
(org-babel-python--session-output-wrapper): Renamed from
org-babel-python--exec-tmpfile.
(org-babel-python--session-value-wrapper): Renamed and modified from
org-babel-python--eval-ast.
(org-babel-python-evaluate-external-process): New parameter for
graphics file.
(org-babel-python-evaluate-session): New parameter for graphics file.

Added results handling for dictionaries, Pandas and numpy tables, and
matplotlib plots.
---
 etc/ORG-NEWS      |  16 +++++-
 lisp/ob-python.el | 122 +++++++++++++++++++++++++++++++---------------
 2 files changed, 96 insertions(+), 42 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 10658a970..4f9863a5b 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -66,8 +66,8 @@ to switch to the new signature.
 *** Python session return values must be top-level expression statements
 
 Python blocks with ~:session :results value~ header arguments now only
-return a value if the last line is a top-level expression statement.
-Also, when a None value is returned, "None" will be printed under
+return a value if the last line is a top-level expression statement,
+otherwise the result is None. Also, None will now show up under
 "#+RESULTS:", as it already did with ~:results value~ for non-session
 blocks.
 
@@ -235,6 +235,18 @@ Screen blocks now recognize the =:screenrc= header argument and pass
 its value to the screen command via the "-c" option. The default
 remains =/dev/null= (i.e. a clean screen session)
 
+*** =ob-python.el=: Support for more result types and plotting
+
+=ob-python= now recognizes dictionaries, numpy arrays, and pandas
+dataframes, and will convert them to org-mode tables when appropriate.
+
+When the header argument =:results graphic= is set, =ob-python= will
+use matplotlib to save graphics. The behavior depends on whether value
+or output results are used. For value results, the last line should
+return a matplotlib Figure object to plot. For output results, the
+current figure (as returned by =pyplot.gcf()=) is cleared and then
+plotted.
+
 *** =RET= and =C-j= now obey ~electric-indent-mode~
 
 Since Emacs 24.4, ~electric-indent-mode~ is enabled by default.  In
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 44e1b63e0..92ca82625 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params)
 	      org-babel-python-command))
 	 (session (org-babel-python-initiate-session
 		   (cdr (assq :session params))))
+	 (graphics-file (and (member "graphics" (assq :result-params params))
+			     (org-babel-graphical-output-file params)))
          (result-params (cdr (assq :result-params params)))
          (result-type (cdr (assq :result-type params)))
 	 (return-val (when (and (eq result-type 'value) (not session))
@@ -89,7 +91,8 @@ (defun org-babel-execute:python (body params)
 	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
 	   params (org-babel-variable-assignments:python params)))
          (result (org-babel-python-evaluate
-		  session full-body result-type result-params preamble)))
+		  session full-body result-type result-params preamble
+		  graphics-file)))
     (org-babel-reassemble-table
      result
      (org-babel-pick-name (cdr (assq :colname-names params))
@@ -225,67 +228,102 @@ (defun org-babel-python-initiate-session (&optional session _params)
     (org-babel-python-session-buffer
      (org-babel-python-initiate-session-by-key session))))
 
-(defconst org-babel-python-wrapper-method
-  "
-def main():
+(defconst org-babel-python--def-format-value "\
+def __org_babel_python_format_value(result, result_file, result_params):
+    with open(result_file, 'w') as f:
+        if 'graphics' in result_params:
+            result.savefig(result_file)
+        elif 'pp' in result_params:
+            import pprint
+            f.write(pprint.pformat(result))
+        else:
+            if not set(result_params).intersection(\
+['scalar', 'verbatim', 'raw']):
+                def dict2alist(res):
+                    if isinstance(res, dict):
+                        return [(k, dict2alist(v)) for k, v in res.items()]
+                    elif isinstance(res, list) or isinstance(res, tuple):
+                        return [dict2alist(x) for x in res]
+                    else:
+                        return res
+                result = dict2alist(result)
+                try:
+                    import pandas as pd
+                except ModuleNotFoundError:
+                    pass
+                else:
+                    if isinstance(result, pd.DataFrame):
+                        result = [[''] + list(result.columns), None] + \
+[[i] + list(row) for i, row in result.iterrows()]
+                try:
+                    import numpy as np
+                except ModuleNotFoundError:
+                    pass
+                else:
+                    if isinstance(result, np.ndarray):
+                        result = result.tolist()
+            f.write(str(result))")
+
+(defun org-babel-python--output-graphics-wrapper
+    (body graphics-file)
+  "Wrap BODY to plot to GRAPHICS-FILE if it is non-nil."
+  (if graphics-file
+      (format "\
+import matplotlib.pyplot as __org_babel_python_plt
+__org_babel_python_plt.gcf().clear()
 %s
+__org_babel_python_plt.savefig('%s')" body graphics-file)
+    body))
 
-open('%s', 'w').write( str(main()) )")
-(defconst org-babel-python-pp-wrapper-method
-  "
-import pprint
+(defconst org-babel-python--nonsession-value-wrapper
+  (concat org-babel-python--def-format-value "
 def main():
 %s
 
-open('%s', 'w').write( pprint.pformat(main()) )")
+__org_babel_python_format_value(main(), '%s', %s)")
+  "TODO")
 
-(defconst org-babel-python--exec-tmpfile "\
+(defconst org-babel-python--session-output-wrapper "\
 with open('%s') as f:
     exec(compile(f.read(), f.name, 'exec'))"
-  "Template for Python session command with output results.
+  "Wrapper for session block with output results.
 
 Has a single %s escape, the tempfile containing the source code
 to evaluate.")
 
-(defconst org-babel-python--eval-ast "\
+(defconst org-babel-python--session-value-wrapper
+  (concat org-babel-python--def-format-value "
 import ast
-
 with open('%s') as f:
     __org_babel_python_ast = ast.parse(f.read())
 __org_babel_python_final = __org_babel_python_ast.body[-1]
-
 if isinstance(__org_babel_python_final, ast.Expr):
     __org_babel_python_ast.body = __org_babel_python_ast.body[:-1]
     exec(compile(__org_babel_python_ast, '<string>', 'exec'))
     __org_babel_python_final = eval(compile(ast.Expression(
         __org_babel_python_final.value), '<string>', 'eval'))
-    with open('%s', 'w') as f:
-        if %s:
-            import pprint
-            f.write(pprint.pformat(__org_babel_python_final))
-        else:
-            f.write(str(__org_babel_python_final))
 else:
     exec(compile(__org_babel_python_ast, '<string>', 'exec'))
-    __org_babel_python_final = None"
-  "Template for Python session command with value results.
+    __org_babel_python_final = None
+__org_babel_python_format_value(__org_babel_python_final, '%s', %s)")
+  "Wrapper for session block with value results.
 
 Has three %s escapes to be filled in:
 1. Tempfile containing source to evaluate.
 2. Tempfile to write results to.
-3. Whether to pretty print, \"True\" or \"False\".")
+3. result-params, converted from lisp to Python list.")
 
 (defun org-babel-python-evaluate
-  (session body &optional result-type result-params preamble)
+  (session body &optional result-type result-params preamble graphics-file)
   "Evaluate BODY as Python code."
   (if session
       (org-babel-python-evaluate-session
-       session body result-type result-params)
+       session body result-type result-params graphics-file)
     (org-babel-python-evaluate-external-process
-     body result-type result-params preamble)))
+     body result-type result-params preamble graphics-file)))
 
 (defun org-babel-python-evaluate-external-process
-    (body &optional result-type result-params preamble)
+    (body &optional result-type result-params preamble graphics-file)
   "Evaluate BODY in external python process.
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
@@ -294,16 +332,16 @@ (defun org-babel-python-evaluate-external-process
          (pcase result-type
            (`output (org-babel-eval org-babel-python-command
 				    (concat preamble (and preamble "\n")
-					    body)))
-           (`value (let ((tmp-file (org-babel-temp-file "python-")))
+					    (org-babel-python--output-graphics-wrapper
+					     body graphics-file))))
+           (`value (let ((results-file (or graphics-file
+				       (org-babel-temp-file "python-"))))
 		     (org-babel-eval
 		      org-babel-python-command
 		      (concat
 		       preamble (and preamble "\n")
 		       (format
-			(if (member "pp" result-params)
-			    org-babel-python-pp-wrapper-method
-			  org-babel-python-wrapper-method)
+			org-babel-python--nonsession-value-wrapper
 			(with-temp-buffer
 			  (python-mode)
 			  (insert body)
@@ -314,14 +352,15 @@ (defun org-babel-python-evaluate-external-process
 							 (line-end-position)))
 			   (forward-line 1))
 			 (buffer-string))
-			(org-babel-process-file-name tmp-file 'noquote))))
-		     (org-babel-eval-read-file tmp-file))))))
+			(org-babel-process-file-name results-file 'noquote)
+			(org-babel-python-var-to-python result-params))))
+		     (org-babel-eval-read-file results-file))))))
     (org-babel-result-cond result-params
       raw
       (org-babel-python-table-or-string (org-trim raw)))))
 
 (defun org-babel-python-evaluate-session
-    (session body &optional result-type result-params)
+    (session body &optional result-type result-params graphics-file)
   "Pass BODY to the Python process in SESSION.
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
@@ -334,17 +373,20 @@ (defun org-babel-python-evaluate-session
 	    (with-temp-file tmp-src-file (insert body))
 	    (pcase result-type
 	      (`output
-	       (let ((src-str (format org-babel-python--exec-tmpfile
-				      (org-babel-process-file-name
-				       tmp-src-file 'noquote))))
+	       (let ((src-str (org-babel-python--output-graphics-wrapper
+			       (format org-babel-python--session-output-wrapper
+				       (org-babel-process-file-name
+					tmp-src-file 'noquote))
+			       graphics-file)))
 		 (if (eq 'python-mode org-babel-python-mode)
 		     (py-send-string-no-output
 		      src-str (get-buffer-process session) session)
 		   (python-shell-send-string-no-output src-str))))
 	      (`value
-	       (let* ((results-file (org-babel-temp-file "python-"))
+	       (let* ((results-file (or graphics-file
+					(org-babel-temp-file "python-")))
 		      (src-str (format
-				org-babel-python--eval-ast
+				org-babel-python--session-value-wrapper
 				(org-babel-process-file-name tmp-src-file 'noquote)
 				(org-babel-process-file-name results-file 'noquote)
 				(org-babel-python-var-to-python result-params))))
-- 
2.28.0

Re: [PATCH] Expanded ob-python results handling and plotting

[Kyle Meyer]

Jack Kamm writes:

> The attached patch adds ob-python value results handling for the
> following types of results:
>
> - Dictionaries
> - Numpy arrays
> - Pandas dataframes
> - Matplotlib figures

Thanks.  Just a couple of drive-by comments...

> +                try:
> +                    import pandas as pd
> +                except ModuleNotFoundError:

ModuleNotFoundError wasn't added until Python 3.6, so I think it'd be
better to use its parent class, ImportError.

> +                    pass
> +                else:
> +                    if isinstance(result, pd.DataFrame):
> +                        result = [[''] + list(result.columns), None] + \
> +[[i] + list(row) for i, row in result.iterrows()]

Should handling of Series also be added?

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

Hi Kyle,

Thanks for the comments, I'm attaching an updated patch.

Kyle Meyer <kyle@kyleam.com> writes:

> ModuleNotFoundError wasn't added until Python 3.6, so I think it'd be
> better to use its parent class, ImportError.

I did not know this, thanks for the tip.

> Should handling of Series also be added?

Yes, I've done so now. I'm not sure whether it's better to treat it like
a row or column vector, but since it has an "index", which are the row
names in a DataFrame, I decided to treat it as a column.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Add-results-handling-for-dicts-dataframes-.patch --]
[-- Type: text/x-patch, Size: 12292 bytes --]

From 40db6b5497de78a9e69de219f4686b405db10c81 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Tue, 25 Aug 2020 21:57:24 -0700
Subject: [PATCH] ob-python: Add results handling for dicts, dataframes,
 arrays, plots

* lisp/ob-python.el (org-babel-execute:python): Parse graphics-file
from params.
(org-babel-python--def-format-value): Python code for formatting
value results before returning.
(org-babel-python--output-graphics-wrapper): Python code for handling
output graphics results.
(org-babel-python--nonsession-value-wrapper): Replaces
org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method.
(org-babel-python--session-output-wrapper): Renamed from
org-babel-python--exec-tmpfile.
(org-babel-python--session-value-wrapper): Renamed and modified from
org-babel-python--eval-ast.
(org-babel-python-evaluate-external-process): New parameter for
graphics file.
(org-babel-python-evaluate-session): New parameter for graphics file.

Added results handling for dictionaries, Pandas and numpy tables, and
matplotlib plots.
---
 etc/ORG-NEWS      |  17 ++++++-
 lisp/ob-python.el | 126 +++++++++++++++++++++++++++++++---------------
 2 files changed, 100 insertions(+), 43 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 10658a970..75c945572 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -66,8 +66,8 @@ to switch to the new signature.
 *** Python session return values must be top-level expression statements
 
 Python blocks with ~:session :results value~ header arguments now only
-return a value if the last line is a top-level expression statement.
-Also, when a None value is returned, "None" will be printed under
+return a value if the last line is a top-level expression statement,
+otherwise the result is None. Also, None will now show up under
 "#+RESULTS:", as it already did with ~:results value~ for non-session
 blocks.
 
@@ -235,6 +235,19 @@ Screen blocks now recognize the =:screenrc= header argument and pass
 its value to the screen command via the "-c" option. The default
 remains =/dev/null= (i.e. a clean screen session)
 
+*** =ob-python.el=: Support for more result types and plotting
+
+=ob-python= now recognizes dictionaries, numpy arrays, and pandas
+dataframes/series, and will convert them to org-mode tables when
+appropriate.
+
+When the header argument =:results graphics= is set, =ob-python= will
+use matplotlib to save graphics. The behavior depends on whether value
+or output results are used. For value results, the last line should
+return a matplotlib Figure object to plot. For output results, the
+current figure (as returned by =pyplot.gcf()=) is cleared before
+evaluation, and then plotted afterwards.
+
 *** =RET= and =C-j= now obey ~electric-indent-mode~
 
 Since Emacs 24.4, ~electric-indent-mode~ is enabled by default.  In
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 44e1b63e0..fb8fe380e 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params)
 	      org-babel-python-command))
 	 (session (org-babel-python-initiate-session
 		   (cdr (assq :session params))))
+	 (graphics-file (and (member "graphics" (assq :result-params params))
+			     (org-babel-graphical-output-file params)))
          (result-params (cdr (assq :result-params params)))
          (result-type (cdr (assq :result-type params)))
 	 (return-val (when (and (eq result-type 'value) (not session))
@@ -89,7 +91,8 @@ (defun org-babel-execute:python (body params)
 	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
 	   params (org-babel-variable-assignments:python params)))
          (result (org-babel-python-evaluate
-		  session full-body result-type result-params preamble)))
+		  session full-body result-type result-params preamble
+		  graphics-file)))
     (org-babel-reassemble-table
      result
      (org-babel-pick-name (cdr (assq :colname-names params))
@@ -225,67 +228,104 @@ (defun org-babel-python-initiate-session (&optional session _params)
     (org-babel-python-session-buffer
      (org-babel-python-initiate-session-by-key session))))
 
-(defconst org-babel-python-wrapper-method
-  "
-def main():
+(defconst org-babel-python--def-format-value "\
+def __org_babel_python_format_value(result, result_file, result_params):
+    with open(result_file, 'w') as f:
+        if 'graphics' in result_params:
+            result.savefig(result_file)
+        elif 'pp' in result_params:
+            import pprint
+            f.write(pprint.pformat(result))
+        else:
+            if not set(result_params).intersection(\
+['scalar', 'verbatim', 'raw']):
+                def dict2alist(res):
+                    if isinstance(res, dict):
+                        return [(k, dict2alist(v)) for k, v in res.items()]
+                    elif isinstance(res, list) or isinstance(res, tuple):
+                        return [dict2alist(x) for x in res]
+                    else:
+                        return res
+                result = dict2alist(result)
+                try:
+                    import pandas as pd
+                except ImportError:
+                    pass
+                else:
+                    if isinstance(result, pd.DataFrame):
+                        result = [[''] + list(result.columns), None] + \
+[[i] + list(row) for i, row in result.iterrows()]
+                    elif isinstance(result, pd.Series):
+                        result = list(result.items())
+                try:
+                    import numpy as np
+                except ImportError:
+                    pass
+                else:
+                    if isinstance(result, np.ndarray):
+                        result = result.tolist()
+            f.write(str(result))")
+
+(defun org-babel-python--output-graphics-wrapper
+    (body graphics-file)
+  "Wrap BODY to plot to GRAPHICS-FILE if it is non-nil."
+  (if graphics-file
+      (format "\
+import matplotlib.pyplot as __org_babel_python_plt
+__org_babel_python_plt.gcf().clear()
 %s
+__org_babel_python_plt.savefig('%s')" body graphics-file)
+    body))
 
-open('%s', 'w').write( str(main()) )")
-(defconst org-babel-python-pp-wrapper-method
-  "
-import pprint
+(defconst org-babel-python--nonsession-value-wrapper
+  (concat org-babel-python--def-format-value "
 def main():
 %s
 
-open('%s', 'w').write( pprint.pformat(main()) )")
+__org_babel_python_format_value(main(), '%s', %s)")
+  "TODO")
 
-(defconst org-babel-python--exec-tmpfile "\
+(defconst org-babel-python--session-output-wrapper "\
 with open('%s') as f:
     exec(compile(f.read(), f.name, 'exec'))"
-  "Template for Python session command with output results.
+  "Wrapper for session block with output results.
 
 Has a single %s escape, the tempfile containing the source code
 to evaluate.")
 
-(defconst org-babel-python--eval-ast "\
+(defconst org-babel-python--session-value-wrapper
+  (concat org-babel-python--def-format-value "
 import ast
-
 with open('%s') as f:
     __org_babel_python_ast = ast.parse(f.read())
 __org_babel_python_final = __org_babel_python_ast.body[-1]
-
 if isinstance(__org_babel_python_final, ast.Expr):
     __org_babel_python_ast.body = __org_babel_python_ast.body[:-1]
     exec(compile(__org_babel_python_ast, '<string>', 'exec'))
     __org_babel_python_final = eval(compile(ast.Expression(
         __org_babel_python_final.value), '<string>', 'eval'))
-    with open('%s', 'w') as f:
-        if %s:
-            import pprint
-            f.write(pprint.pformat(__org_babel_python_final))
-        else:
-            f.write(str(__org_babel_python_final))
 else:
     exec(compile(__org_babel_python_ast, '<string>', 'exec'))
-    __org_babel_python_final = None"
-  "Template for Python session command with value results.
+    __org_babel_python_final = None
+__org_babel_python_format_value(__org_babel_python_final, '%s', %s)")
+  "Wrapper for session block with value results.
 
 Has three %s escapes to be filled in:
 1. Tempfile containing source to evaluate.
 2. Tempfile to write results to.
-3. Whether to pretty print, \"True\" or \"False\".")
+3. result-params, converted from lisp to Python list.")
 
 (defun org-babel-python-evaluate
-  (session body &optional result-type result-params preamble)
+  (session body &optional result-type result-params preamble graphics-file)
   "Evaluate BODY as Python code."
   (if session
       (org-babel-python-evaluate-session
-       session body result-type result-params)
+       session body result-type result-params graphics-file)
     (org-babel-python-evaluate-external-process
-     body result-type result-params preamble)))
+     body result-type result-params preamble graphics-file)))
 
 (defun org-babel-python-evaluate-external-process
-    (body &optional result-type result-params preamble)
+    (body &optional result-type result-params preamble graphics-file)
   "Evaluate BODY in external python process.
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
@@ -294,16 +334,16 @@ (defun org-babel-python-evaluate-external-process
          (pcase result-type
            (`output (org-babel-eval org-babel-python-command
 				    (concat preamble (and preamble "\n")
-					    body)))
-           (`value (let ((tmp-file (org-babel-temp-file "python-")))
+					    (org-babel-python--output-graphics-wrapper
+					     body graphics-file))))
+           (`value (let ((results-file (or graphics-file
+				       (org-babel-temp-file "python-"))))
 		     (org-babel-eval
 		      org-babel-python-command
 		      (concat
 		       preamble (and preamble "\n")
 		       (format
-			(if (member "pp" result-params)
-			    org-babel-python-pp-wrapper-method
-			  org-babel-python-wrapper-method)
+			org-babel-python--nonsession-value-wrapper
 			(with-temp-buffer
 			  (python-mode)
 			  (insert body)
@@ -314,14 +354,15 @@ (defun org-babel-python-evaluate-external-process
 							 (line-end-position)))
 			   (forward-line 1))
 			 (buffer-string))
-			(org-babel-process-file-name tmp-file 'noquote))))
-		     (org-babel-eval-read-file tmp-file))))))
+			(org-babel-process-file-name results-file 'noquote)
+			(org-babel-python-var-to-python result-params))))
+		     (org-babel-eval-read-file results-file))))))
     (org-babel-result-cond result-params
       raw
       (org-babel-python-table-or-string (org-trim raw)))))
 
 (defun org-babel-python-evaluate-session
-    (session body &optional result-type result-params)
+    (session body &optional result-type result-params graphics-file)
   "Pass BODY to the Python process in SESSION.
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
@@ -334,24 +375,27 @@ (defun org-babel-python-evaluate-session
 	    (with-temp-file tmp-src-file (insert body))
 	    (pcase result-type
 	      (`output
-	       (let ((src-str (format org-babel-python--exec-tmpfile
-				      (org-babel-process-file-name
-				       tmp-src-file 'noquote))))
+	       (let ((src-str (org-babel-python--output-graphics-wrapper
+			       (format org-babel-python--session-output-wrapper
+				       (org-babel-process-file-name
+					tmp-src-file 'noquote))
+			       graphics-file)))
 		 (if (eq 'python-mode org-babel-python-mode)
 		     (py-send-string-no-output
 		      src-str (get-buffer-process session) session)
 		   (python-shell-send-string-no-output src-str))))
 	      (`value
-	       (let* ((results-file (org-babel-temp-file "python-"))
+	       (let* ((results-file (or graphics-file
+					(org-babel-temp-file "python-")))
 		      (src-str (format
-				org-babel-python--eval-ast
+				org-babel-python--session-value-wrapper
 				(org-babel-process-file-name tmp-src-file 'noquote)
 				(org-babel-process-file-name results-file 'noquote)
 				(org-babel-python-var-to-python result-params))))
 		 (if (eq 'python-mode org-babel-python-mode)
 		     (py-shell-send-string src-str (get-buffer-process session))
 		   (python-shell-send-string src-str))
-		 (sleep-for 0 5)
+		 (sleep-for 0 10)
 		 (org-babel-eval-read-file results-file)))))))
     (org-babel-result-cond result-params
       results
-- 
2.28.0

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

After taking another look at my patch, I realized that I was not quite
converting dictionaries to proper alists.

Attached is a tweak to do this properly. The printing of dictionaries is
not quite as pretty, in particular it's not a table anymore:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
: ((a . 1) (b . 2))

But, it feels like the right thing to do, since the result handling code
works by converting the result to an elisp value, before passing it to
org-mode to decide how to render it. And the proper elisp conversion of
a dict should be an alist or a plist.

Ideally I wouldn't have to do this from the Python code, and could let
org-babel-script-escape convert the dict objects. It would also be
useful for other languages with similar dictionaries, like
javascript. But it seems fairly complex to implement this from the elisp
side, and I'm not sure I'm up for it.

I also noticed that I had left a couple docstrings as TODOs -- I'll fix
those before finalizing the patch over the next couple weeks.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0002-Convert-dictionary-output-to-a-proper-alist.patch --]
[-- Type: text/x-patch, Size: 1364 bytes --]

From 76a1ad4d50e6638244d9aa17e45895b8b38b3cd0 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 30 Aug 2020 08:51:04 -0700
Subject: [PATCH 2/2] Convert dictionary output to a proper alist

Note: to be squashed with the previous patch before merging
---
 lisp/ob-python.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index fb8fe380e..08c1c48e9 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -239,9 +239,14 @@ (defconst org-babel-python--def-format-value "\
         else:
             if not set(result_params).intersection(\
 ['scalar', 'verbatim', 'raw']):
+                class alist(dict):
+                    def __str__(self):
+                        return '({})'.format(' '.join(['({} . {})'.format(repr(k), repr(v)) for k, v in self.items()]))
+                    def __repr__(self):
+                        return self.__str__()
                 def dict2alist(res):
                     if isinstance(res, dict):
-                        return [(k, dict2alist(v)) for k, v in res.items()]
+                        return alist({k: dict2alist(v) for k, v in res.items()})
                     elif isinstance(res, list) or isinstance(res, tuple):
                         return [dict2alist(x) for x in res]
                     else:
-- 
2.28.0

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

After letting it sit, I'm not sure that my patch above is a good idea
anymore. While it would be useful, it also adds substantial complexity
to ob-python.

For now, I think I prefer to keep ob-python leaner, so am going to hold
off on this.

An alternative approach is to have the user handle graphics and
dataframes via noweb or header arguments.

I've added a couple examples on worg, demonstrating how to use noweb
to insert boilerplate code for handling matplotlib figures and pandas
dataframes [0,1].

Additionally, I'm attaching a small patch to make it easier to handle
graphics/dataframes via the :return header argument, as an alternative
to noweb. The patch includes a couple examples in ORG-NEWS
illustrating this.

I'll wait a week or so for comments before merging this new, more
limited patch into master.

[0] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-python.html
[1] worg commit 59e320ad

Cheers,
Jack


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Improvements-to-return-header-argument.patch --]
[-- Type: text/x-patch, Size: 3643 bytes --]

From 118d8b5eb817e9a21e9d84f2f942fcc841ddc51f Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sat, 19 Sep 2020 08:44:30 -0700
Subject: [PATCH] ob-python: Improvements to :return header argument

* lisp/ob-python.el (org-babel-execute:python): Allow return-val to be
non-nil in sessions, and concatenate it after the expanded body.
---
 etc/ORG-NEWS      | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 lisp/ob-python.el | 11 ++++++----
 2 files changed, 60 insertions(+), 4 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 0ed626fb7..50a455ad5 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -11,6 +11,59 @@ See the end of the file for license conditions.
 Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.5 (not yet released)
+** New features
+*** =ob-python= improvements to =:return= header argument 
+
+The =:return= header argument in =ob-python= now works for session
+blocks as well as non-session blocks.  Also, it now works with the
+=:epilogue= header argument -- previously, setting the =:return=
+header would cause the =:epilogue= to be ignored.
+
+This change allows more easily moving boilerplate out of the main code
+block and into the header. For example, for plotting, we need to add
+boilerplate to save the figure to a file and return the
+filename. Instead of doing this within the code block, we can now
+handle it through the header arguments as follows:
+
+#+BEGIN_SRC org
+,#+header: :var fname="/home/jack/tmp/plot.svg"
+,#+header: :epilogue plt.savefig(fname)
+,#+header: :return fname
+,#+begin_src python :results value file
+  import matplotlib, numpy
+  import matplotlib.pyplot as plt
+  fig=plt.figure(figsize=(4,2))
+  x=numpy.linspace(-15,15)
+  plt.plot(numpy.sin(x)/x)
+  fig.tight_layout()
+,#+end_src
+
+,#+RESULTS:
+[[file:/home/jack/tmp/plot.svg]]
+#+END_SRC
+
+As another example, we can use =:return= with the external [[https://pypi.org/project/tabulate/][tabulate]]
+package, to convert pandas Dataframes into orgmode tables:
+
+#+begin_src org
+,#+header: :prologue from tabulate import tabulate
+,#+header: :return tabulate(table, headers=table.columns, tablefmt="orgtbl")
+,#+begin_src python :results value raw :session
+  import pandas as pd
+  table = pd.DataFrame({
+      "a": [1,2,3],
+      "b": [4,5,6]
+  })
+,#+end_src
+
+,#+RESULTS:
+|   | a | b |
+|---+---+---|
+| 0 | 1 | 4 |
+| 1 | 2 | 5 |
+| 2 | 3 | 6 |
+#+end_src
+
 * Version 9.4
 ** Incompatible changes
 *** Possibly broken internal file links: please check and fix
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 00a7c1a2d..785b9191b 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -81,13 +81,16 @@ (defun org-babel-execute:python (body params)
 		   (cdr (assq :session params))))
          (result-params (cdr (assq :result-params params)))
          (result-type (cdr (assq :result-type params)))
-	 (return-val (when (and (eq result-type 'value) (not session))
+	 (return-val (when (eq result-type 'value)
 		       (cdr (assq :return params))))
 	 (preamble (cdr (assq :preamble params)))
          (full-body
-	  (org-babel-expand-body:generic
-	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
-	   params (org-babel-variable-assignments:python params)))
+	  (concat
+	   (org-babel-expand-body:generic
+	    body params
+	    (org-babel-variable-assignments:python params))
+	   (when return-val
+	     (format (if session "\n%s" "\nreturn %s") return-val))))
          (result (org-babel-python-evaluate
 		  session full-body result-type result-params preamble)))
     (org-babel-reassemble-table
-- 
2.28.0

Re: [PATCH] Expanded ob-python results handling and plotting

[Bastien]

Hi Jack,

Jack Kamm <jackkamm@gmail.com> writes:

> For now, I think I prefer to keep ob-python leaner, so am going to hold
> off on this.

The leaner the less maintainance ahead :)

> I'll wait a week or so for comments before merging this new, more
> limited patch into master.

LGTM, thanks!

-- 
 Bastien

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

Thanks -- I've pushed this to master now.

Jack

Bastien <bzg@gnu.org> writes:

> Hi Jack,
>
> Jack Kamm <jackkamm@gmail.com> writes:
>
>> For now, I think I prefer to keep ob-python leaner, so am going to hold
>> off on this.
>
> The leaner the less maintainance ahead :)
>
>> I'll wait a week or so for comments before merging this new, more
>> limited patch into master.
>
> LGTM, thanks!
>
> -- 
>  Bastien