ipython integration fails

ipython integration fails

[Prof. Dr. Johanna May]

Hi everybody,

I am creating lecture notes including some python snippets. Now, in
order to teach jupyter notebook use I would like to include just the
same code as in a python snippet in an ipython snippet.

However ipython blocks don't work in my set up and I could not figure
out which dependency or config line I might need to add in order to do
this properly (and scimax also creates errors, but since I have my own
config running I thought ipython might  already do the job).

The ipython snippet I used is:
#+BEGIN_SRC ipython :session :results raw drawer
  %matplotlib inline
  import matplotlib.pyplot as plt
  import numpy as np
#+END_SRC

The error message is:
Traceback (most recent call last):
  File "/home/username/.emacs.d/elpa/ob-ipython-20180224.953/client.py", line 60, in <module>
    c = create_client(args.conn_file)
  File "/home/username/.emacs.d/elpa/ob-ipython-20180224.953/client.py", line 43, in create_client
    cf = find_connection_file('emacs-' + name)
  File "/usr/lib/python3.6/site-packages/jupyter_client/connect.py", line 218, in find_connection_file
    raise IOError("Could not find %r in %r" % (filename, path))
OSError: Could not find 'emacs-default' in ['.', '/run/user/1000/jupyter']

Maybe I do not need ipython anyway. What works are python code snippets
like the following one. Minted exports both, code and results and they
are both displayed in the pdf slides/manuscript (using lualatex,
beamerarticle etc.)

#+begin_src python :results output :session :exports both
import time
time.sleep(3)
#+end_src

However, then, what does not work properly is several code blocks in one
session, e.g.

(1)
#+begin_src python :results output :session :exports both
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import datetime as dt
import seaborn as sns
sns.set(style="darkgrid")
#+end_src

(2)
#+begin_src python :results output :session :exports both
df = pd.read_csv('./household_data_1min_singleindex.csv')
#+end_src

(3)
#+begin_src python :results output :session :exports both
zeit=df.utc_timestamp.apply(lambda x: dt.datetime.strptime(x, "%Y-%m-%dT%H:%M:%SZ"))
#+end_src

(4)
#+begin_src python :results file :session :var matplot_lib_filename=(org-babel-temp-file "figu" ".png"),fontsize=fs :exports both
plt.rcParams.update({'font.size':fontsize})
rcParams.update({'figure.autolayout': True})
plt.plot(zeit,df.DE_KN_residential2_washing_machine)
plt.savefig(matplot_lib_filename)
matplot_lib_filename
#+end_src

I do not like the fact that plt.rcParams.update and other lines that
jupyter does not need get exported - is there a simple way to exclude
certain lines from export?

And I do get in serious trouble when I add a second data set (another csv) and do
further analysis steps - all graphics then look the same and bad and do
not represent the actual result of the code block. It seems that all get
the same wrong name and then at each graphics location this same png
gets displayed. 

So, maybe there is a good tutorial on using sessions in ob-python or
some other hint that I could use?

Is there a way to restart the (python) kernel with some header message
in the src block?

Thank you very much,

Cheers,

J. May

Re: ipython integration fails

[John Kitchin]

I guess you are having a problem with the version of jupyter, and maybe
jupyter_client. Maybe upgrading thises will fix the ipython problem.

See notes below for python sessions.
Prof. Dr. Johanna May <johanna.may@th-koeln.de> writes:

> Hi everybody,
>
> I am creating lecture notes including some python snippets. Now, in
> order to teach jupyter notebook use I would like to include just the
> same code as in a python snippet in an ipython snippet.

If you get ipython working, then you can write your lectures in
org-mode, and use ox-ipynb from scimax to export a jupyter notebook from
the org-file.
>
> However ipython blocks don't work in my set up and I could not figure
> out which dependency or config line I might need to add in order to do
> this properly (and scimax also creates errors, but since I have my own
> config running I thought ipython might  already do the job).
>
> The ipython snippet I used is:
> #+BEGIN_SRC ipython :session :results raw drawer
>   %matplotlib inline
>   import matplotlib.pyplot as plt
>   import numpy as np
> #+END_SRC
>
> The error message is:
> Traceback (most recent call last):
>   File "/home/username/.emacs.d/elpa/ob-ipython-20180224.953/client.py", line 60, in <module>
>     c = create_client(args.conn_file)
>   File "/home/username/.emacs.d/elpa/ob-ipython-20180224.953/client.py", line 43, in create_client
>     cf = find_connection_file('emacs-' + name)
>   File "/usr/lib/python3.6/site-packages/jupyter_client/connect.py", line 218, in find_connection_file
>     raise IOError("Could not find %r in %r" % (filename, path))
> OSError: Could not find 'emacs-default' in ['.', '/run/user/1000/jupyter']
>
> Maybe I do not need ipython anyway. What works are python code snippets
> like the following one. Minted exports both, code and results and they
> are both displayed in the pdf slides/manuscript (using lualatex,
> beamerarticle etc.)
>
> #+begin_src python :results output :session :exports both
> import time
> time.sleep(3)
> #+end_src
>
> However, then, what does not work properly is several code blocks in one
> session, e.g.
>
> (1)
> #+begin_src python :results output :session :exports both
> import pandas as pd
> import numpy as np
> import matplotlib.pyplot as plt
> import datetime as dt
> import seaborn as sns
> sns.set(style="darkgrid")
> #+end_src
>
> (2)
> #+begin_src python :results output :session :exports both
> df = pd.read_csv('./household_data_1min_singleindex.csv')
> #+end_src
>
> (3)
> #+begin_src python :results output :session :exports both
> zeit=df.utc_timestamp.apply(lambda x: dt.datetime.strptime(x, "%Y-%m-%dT%H:%M:%SZ"))
> #+end_src
>
> (4)
> #+begin_src python :results file :session :var matplot_lib_filename=(org-babel-temp-file "figu" ".png"),fontsize=fs :exports both
> plt.rcParams.update({'font.size':fontsize})
> rcParams.update({'figure.autolayout': True})
> plt.plot(zeit,df.DE_KN_residential2_washing_machine)
> plt.savefig(matplot_lib_filename)
> matplot_lib_filename
> #+end_src

To get this to work, I used

#+begin_src python :results output :session :exports both
import time
time.sleep(3)
#+end_src

#+RESULTS:
: Python 3.6.0 |Anaconda custom (x86_64)| (default, Dec 23 2016, 13:19:00)
: [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
: Type "help", "copyright", "credits" or "license" for more information.
: >>> python.el: native completion setup loaded

However, then, what does not work properly is several code blocks in one
session, e.g.

(1)
#+begin_src python :results output :session :exports both
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import datetime as dt
import seaborn as sns
sns.set(style="darkgrid")
#+end_src

#+RESULTS:

(2)
#+begin_src python :results output :session :exports both
x = np.linspace(0, 1)
y = x**2
#+end_src

#+RESULTS:

(3)
#+begin_src python :results output :session :exports both
fontsize=12
#+end_src


(4)
#+begin_src python :results file :session :var matplot_lib_filename=(org-babel-temp-file "figu" ".png") :exports both
plt.rcParams.update({'font.size':fontsize})
plt.rcParams.update({'figure.autolayout': True})
plt.plot(x, y)
plt.savefig(matplot_lib_filename)
print(matplot_lib_filename)
#+end_src

5)

#+begin_src python :results file :session :var matplot_lib_filename=(org-babel-temp-file "figu" ".png") :exports both
plt.figure()
plt.rcParams.update({'font.size':fontsize})
plt.rcParams.update({'figure.autolayout': True})
plt.plot(x, 2 * x)
plt.savefig(matplot_lib_filename)
print(matplot_lib_filename)
#+end_src

which for me worked like I think you expect, which is your run each
block sequentially. There were some errors in your last block related to
the fontsize and rcParams that are different in my block. Probably I
have different defaults for the results (I always use output) so I
printed the results instead of using the returned value.
>
> I do not like the fact that plt.rcParams.update and other lines that
> jupyter does not need get exported - is there a simple way to exclude
> certain lines from export?

I don't think so. If you want the Jupyter plots to look like these, it
seems like you need to include them. Otherwise, you have to put the
lines in a separate block, with :exports none in the header.
>
> And I do get in serious trouble when I add a second data set (another csv) and do
> further analysis steps - all graphics then look the same and bad and do
> not represent the actual result of the code block. It seems that all get
> the same wrong name and then at each graphics location this same png
> gets displayed.

This isn't something I was able to reproduce. I get different temp file
names with the code above.
>
> So, maybe there is a good tutorial on using sessions in ob-python or
> some other hint that I could use?
>
> Is there a way to restart the (python) kernel with some header message
> in the src block?

In scimax you can do this on an ipython src block with :restart in the
header. You can restart the Python session by killing the *Python*
buffer.

I think you can get something like this in a header with:
#+header: :var restart=(when (get-buffer "*Python*") (kill-buffer "*Python*") 0)

>
> Thank you very much,
>
> Cheers,
>
> J. May


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu