Pyenv and Jupyter integration with emacs.

Pyenv and Jupyter integration with emacs.

[Hongyi Zhao]

On Ubuntu 20.04, I use pyenv, <https://github.com/pyenv/pyenv>, as the
python version management tool, and Jupyter (formerly known as
ipython) to write notebook. Currently, I consider migrating to Emacs
as a python development environment. I've searched and found the
following relevant projects on GitHub:

https://github.com/millejoh/emacs-ipython-notebook
https://github.com/nnicandro/emacs-jupyter
https://github.com/twlz0ne/shim.el

But I myself haven't any experience on this filed. Could you please
kindly give me some hints/suggestions?

Regards,
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Pyenv and Jupyter integration with emacs.

[Skip Montanaro]

>
> On Ubuntu 20.04, I use pyenv, <https://github.com/pyenv/pyenv>, as the
> python version management tool, and Jupyter (formerly known as
> ipython) to write notebook. Currently, I consider migrating to Emacs
> as a python development environment. I've searched and found the
> following relevant projects on GitHub:
>
> https://github.com/millejoh/emacs-ipython-notebook
> https://github.com/nnicandro/emacs-jupyter
> https://github.com/twlz0ne/shim.el
>
> But I myself haven't any experience on this filed. Could you please
> kindly give me some hints/suggestions?
>

I don't have any experience with these packages either. I installed
emacs-jupyter, but was initially stymied trying to get started. I
eventually got jupyter-connect-repl to work by tracking down one of the
kernel-mumble-mumble.json files in ~/.local/share/jupyter/runtime, e.g.
kernel-eed594bc-6480-40b4-b8e2-4c56d2f6bf29.json. Here's the one I
referenced:

{
  "shell_port": 37041,
  "iopub_port": 58707,
  "stdin_port": 41637,
  "control_port": 42543,
  "hb_port": 44903,
  "ip": "127.0.0.1",
  "key": "69964a9c-9cc1c1cd6bb7b925c4649648",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Note the empty kernel_name. I have two notebooks open in my browser at the
moment and see no way to name them from that interface. I had to try
evaluating existing objects in the kernel to figure out which one I was
talking to. Not sure whose problem that is, but it seems a bit clunky.

This has to be (at least marginally) better than interacting with the
kernels through a web browser, just because you have Emacs functionality at
your keyboard.

Skip

Re: Pyenv and Jupyter integration with emacs.

[Hongyi Zhao]

On Thu, Mar 11, 2021 at 9:02 PM Skip Montanaro <skip.montanaro@gmail.com> wrote:
>>
>> On Ubuntu 20.04, I use pyenv, <https://github.com/pyenv/pyenv>, as the
>> python version management tool, and Jupyter (formerly known as
>> ipython) to write notebook. Currently, I consider migrating to Emacs
>> as a python development environment. I've searched and found the
>> following relevant projects on GitHub:
>>
>> https://github.com/millejoh/emacs-ipython-notebook
>> https://github.com/nnicandro/emacs-jupyter
>> https://github.com/twlz0ne/shim.el
>>
>> But I myself haven't any experience on this filed. Could you please
>> kindly give me some hints/suggestions?
>
>
> I don't have any experience with these packages either. I installed emacs-jupyter, but was initially stymied trying to get started. I eventually got jupyter-connect-repl to work by tracking down one of the kernel-mumble-mumble.json files in ~/.local/share/jupyter/runtime, e.g. kernel-eed594bc-6480-40b4-b8e2-4c56d2f6bf29.json. Here's the one I referenced:
>
> {
>   "shell_port": 37041,
>   "iopub_port": 58707,
>   "stdin_port": 41637,
>   "control_port": 42543,
>   "hb_port": 44903,
>   "ip": "127.0.0.1",
>   "key": "69964a9c-9cc1c1cd6bb7b925c4649648",
>   "transport": "tcp",
>   "signature_scheme": "hmac-sha256",
>   "kernel_name": ""
> }
>
> Note the empty kernel_name. I have two notebooks open in my browser at the moment and see no way to name them from that interface. I had to try evaluating existing objects in the kernel to figure out which one I was talking to. Not sure whose problem that is, but it seems a bit clunky.
>
> This has to be (at least marginally) better than interacting with the kernels through a web browser, just because you have Emacs functionality at your keyboard.

I still don't quite understand what you wrote above. OTOH, from my
intuitive point of view, it seems that emacs-ipython-notebook is more
powerful then emacs-jupyter.

HY


> Skip
>


-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Pyenv and Jupyter integration with emacs.

[Skip Montanaro]

>
> I still don't quite understand what you wrote above. OTOH, from my
> intuitive point of view, it seems that emacs-ipython-notebook is more
> powerful then emacs-jupyter.
>

I just picked one and tried it. I chose it in large part because "jupyter"
is a newer name than "ipython." I found the README file unhelpful and just
stumbled my way to connecting to an existing kernel. As for the file, when
you run "jupyter notebook" it creates a number of files in
~/.local/share/jupyter/runtime and prints a bit of output to the terminal,
for example:

   To access the notebook, open this file in a browser:

file:///home/skip/.local/share/jupyter/runtime/nbserver-533322-open.html
    Or copy and paste one of these URLs:

http://localhost:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0
     or
http://127.0.0.1:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0

None of those URLs is actually what jupyter-connect-repl requires. Here are
the files just created when I restarted my notebook server and the two
kernels just now:

% ls -ltr ~/.local/share/jupyter/runtime/
total 120
...
-rw-r--r-- 1 skip skip 287 Mar 11 09:19 nbserver-533322.json
-rw-r--r-- 1 skip skip 673 Mar 11 09:19 nbserver-533322-open.html
-rw------T 1 skip skip 263 Mar 11 09:21
kernel-9e8ca820-639b-44e8-9322-6570db2a9a21.json
-rw------T 1 skip skip 263 Mar 11 09:21
kernel-2b4f57f1-466c-44c6-9871-c9bcb007e3ca.json

The first two files are the ones the notebook command created. The second
two files are the result of restarting the two kernels. Those are the files
jupyter-connect-repl needs. As I indicated, this is not clear from the
README file. There is no good way to guess which of the kernel-*.json files
you want to use for the connect operation because the kernel_name field is
empty. If you are only running a single kernel that's okay I suppose. It
would be nice if there was an obvious way to name kernels when opening an
existing ipynb file or when (re)starting the associated kernel. It doesn't
appear there is, however.

Rereading your original note, I didn't see any indication that you'd tried
any of the three options or had a perceived preference. Had you expressed a
preference at that point, I would likely have given it a try.

Skip

Re: Pyenv and Jupyter integration with emacs.

[Hongyi Zhao]

On Thu, Mar 11, 2021 at 11:27 PM Skip Montanaro
<skip.montanaro@gmail.com> wrote:
>>
>> I still don't quite understand what you wrote above. OTOH, from my
>> intuitive point of view, it seems that emacs-ipython-notebook is more
>> powerful then emacs-jupyter.
>
>
> I just picked one and tried it. I chose it in large part because "jupyter" is a newer name than "ipython."

I think that the emacs-ipython-notebook project is created a bit
longer ago when the ipython is the prevailing name.

> I found the README file unhelpful and just stumbled my way to connecting to an existing kernel. As for the file, when you run "jupyter   notebook" it creates a number of files in ~/.local/share/jupyter/runtime and prints a bit of output to the terminal, for example:
>
>    To access the notebook, open this file in a browser:
>         file:///home/skip/.local/share/jupyter/runtime/nbserver-533322-open.html
>     Or copy and paste one of these URLs:
>         http://localhost:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0
>      or http://127.0.0.1:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0
>
> None of those URLs is actually what jupyter-connect-repl requires. Here are the files just created when I restarted my notebook server and the two kernels just now:
>
> % ls -ltr ~/.local/share/jupyter/runtime/
> total 120
> ...
> -rw-r--r-- 1 skip skip 287 Mar 11 09:19 nbserver-533322.json
> -rw-r--r-- 1 skip skip 673 Mar 11 09:19 nbserver-533322-open.html
> -rw------T 1 skip skip 263 Mar 11 09:21 kernel-9e8ca820-639b-44e8-9322-6570db2a9a21.json
> -rw------T 1 skip skip 263 Mar 11 09:21 kernel-2b4f57f1-466c-44c6-9871-c9bcb007e3ca.json
>
> The first two files are the ones the notebook command created. The second two files are the result of restarting the two kernels. Those are the files jupyter-connect-repl needs. As I indicated, this is not clear from the README file. There is no good way to guess which of the kernel-*.json files you want to use for the connect operation because the kernel_name field is empty. If you are only running a single kernel that's okay I suppose. It would be nice if there was an obvious way to name kernels when opening an existing ipynb file or when (re)starting the associated kernel. It doesn't appear there is, however.
>
> Rereading your original note, I didn't see any indication that you'd tried any of the three options or had a perceived preference. Had you expressed a preference at that point, I would likely have given it a try.
>

I've just simply given a try to emacs-ipython-notebook and opened a
.ipynb file in Emacs.

And searching google with "emacs ipython notebook tutorial" directs me
to the following introduction represented by the author of the
project:

https://www.youtube.com/watch?v=OB9vFu9Za8w

> Skip
>


-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Pyenv and Jupyter integration with emacs.

[Hongyi Zhao]

On Fri, Mar 12, 2021 at 9:04 AM Hongyi Zhao <hongyi.zhao@gmail.com> wrote:
>
> On Thu, Mar 11, 2021 at 11:27 PM Skip Montanaro
> <skip.montanaro@gmail.com> wrote:
> >>
> >> I still don't quite understand what you wrote above. OTOH, from my
> >> intuitive point of view, it seems that emacs-ipython-notebook is more
> >> powerful then emacs-jupyter.
> >
> >
> > I just picked one and tried it. I chose it in large part because "jupyter" is a newer name than "ipython."
>
> I think that the emacs-ipython-notebook project is created a bit
> longer ago when the ipython is the prevailing name.
>
> > I found the README file unhelpful and just stumbled my way to connecting to an existing kernel. As for the file, when you run "jupyter   notebook" it creates a number of files in ~/.local/share/jupyter/runtime and prints a bit of output to the terminal, for example:
> >
> >    To access the notebook, open this file in a browser:
> >         file:///home/skip/.local/share/jupyter/runtime/nbserver-533322-open.html
> >     Or copy and paste one of these URLs:
> >         http://localhost:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0
> >      or http://127.0.0.1:8888/?token=5de6661699de63f03da52dd1bdf6a4f18f8cb2119373d6e0
> >
> > None of those URLs is actually what jupyter-connect-repl requires. Here are the files just created when I restarted my notebook server and the two kernels just now:
> >
> > % ls -ltr ~/.local/share/jupyter/runtime/
> > total 120
> > ...
> > -rw-r--r-- 1 skip skip 287 Mar 11 09:19 nbserver-533322.json
> > -rw-r--r-- 1 skip skip 673 Mar 11 09:19 nbserver-533322-open.html
> > -rw------T 1 skip skip 263 Mar 11 09:21 kernel-9e8ca820-639b-44e8-9322-6570db2a9a21.json
> > -rw------T 1 skip skip 263 Mar 11 09:21 kernel-2b4f57f1-466c-44c6-9871-c9bcb007e3ca.json
> >
> > The first two files are the ones the notebook command created. The second two files are the result of restarting the two kernels. Those are the files jupyter-connect-repl needs. As I indicated, this is not clear from the README file. There is no good way to guess which of the kernel-*.json files you want to use for the connect operation because the kernel_name field is empty. If you are only running a single kernel that's okay I suppose. It would be nice if there was an obvious way to name kernels when opening an existing ipynb file or when (re)starting the associated kernel. It doesn't appear there is, however.
> >
> > Rereading your original note, I didn't see any indication that you'd tried any of the three options or had a perceived preference. Had you expressed a preference at that point, I would likely have given it a try.
> >
>
> I've just simply given a try to emacs-ipython-notebook and opened a
> .ipynb file in Emacs.
>
> And searching google with "emacs ipython notebook tutorial" directs me
> to the following introduction represented by the author of the
> project:
>
> https://www.youtube.com/watch?v=OB9vFu9Za8w

Some new discoveries that I stumbled upon:

https://ddavis.io/posts/emacs-python-lsp/
https://realpython.com/emacs-the-best-python-editor/

Based on the experience/guidance shared on the above instructions, it
seems the following combination is a better setup for the problem
discussed here:

https://github.com/jorgenschaefer/elpy
https://github.com/jorgenschaefer/pyvenv
https://github.com/millejoh/emacs-ipython-notebook

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Pyenv and Jupyter integration with emacs.

[Doug Davis]

Hongyi Zhao <hongyi.zhao@gmail.com> writes:

> Some new discoveries that I stumbled upon:
>
> https://ddavis.io/posts/emacs-python-lsp/

Hey, that's my post! glad it was discovered :)

It targets a work flow that I think is on the opposite side of the
Python spectrum with respect to the Jupyter project, that is, regular
Python development (not so much experimentation, which is what I do with
Jupyter, playing around with prototypes and just getting some thoughts
typed out).

I use Emacs + lsp-mode (I still tinker with Eglot even though the blog
post is focused on lsp-mode) for pretty standard Python development,
especially when I want to navigate around a project.

That said, here's a bit of a rundown of my Jupyter usage as an Emacs
devotee: I have used EIN before (with help from pyenv + pyvenv, which I
walk through in the blog post). If you have jupyter installed in a
particular virtual environment, in Emacs execute `M-x pyvenv-workon' to
activate that environment, then `M-x ein:jupyter-server-start' to start
EIN. As much as I love to stay in Emacs, I'm not a heavy
notebooks-in-Emacs person. I typically use Emacs to work on code living
in a Python module and use:

%load_ext autoreload
%autoreload 2

In a web browser hosted Jupyter notebook to get changes from code I'm
editing in Emacs. More info about autoreload here:
https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html

This has given me a bit of motivation to perhaps give the
notebook-in-Emacs workflow a shot again.

Cheers,
Doug

Re: Pyenv and Jupyter integration with emacs.

[Joost Kremers]

On Thu, Mar 11 2021, Hongyi Zhao wrote:
> On Ubuntu 20.04, I use pyenv, <https://github.com/pyenv/pyenv>, as the
> python version management tool, and Jupyter (formerly known as
> ipython) to write notebook. Currently, I consider migrating to Emacs
> as a python development environment. I've searched and found the
> following relevant projects on GitHub:
>
> https://github.com/millejoh/emacs-ipython-notebook
> https://github.com/nnicandro/emacs-jupyter
> https://github.com/twlz0ne/shim.el

I don't know shim.el or xenv, so I can't say anything about that.

I have used emacs-ipython-notebook (EIN for short) a year or two back, and
although I didn't use it for very long, my impressions were positive. It dates
from the time that Jupyter was still IPython, hence the name, but it is still
updated regularly, so I wouldn't be put off by the name.

EIN is a package that aims to mimic the Jupyter Notebook experience in Emacs. It
reads and writes .ipynb files, so it allows you to switch between Emacs and a
browser for working on your project.

I have never used emacs-jupyter, but from what I've read on the website, it has
a different aim: it's is a package to manage Jupyter kernels from Emacs. It is
essentially a library (i.e., a package intended to be used by other packages).

It does offer two front-ends (meant for users): a REPL and ob-jupyter, to
integrate it with org-mode. With `ob-jupyter`, you can recreate the user
experience of a Jupyter notebook (in fact, org-mode is much more powerful than
a Jupyter notebook), but the files you create aren't in `.ipynb` format, so
there is no direct interoperability with Jupyter Notebooks.

You may be able to convert .ipynb files to Org with Pandoc, I don't know, but if
interoperability is a concern, you should probably stick to EIN.

HTH

-- 
Joost Kremers
Life has its moments