ob-python: import local package into a session

ob-python: import local package into a session

[Joost Kremers]

Hi all,

If I have an Org file with Python source blocks, I can run them in a session
with the `:session` header arg. That way, I can include packages installed in
`site-packages` and have them available in all code blocks. But is there a way
to import my own packages into a session? In particular, packages I haven't
installed system-wide?

What I'm trying to do is to import a Python file with a bunch of utility
functions into the ob-python session. I thought this might be possible if I'd
structure my code as a regular Python package, because that works if I want to
import my utility functions into another Python file. But it doesn't seem to
work for the ob-python session.

Is there a way to achieve this? I don't *have* to structure my utility functions
as a Python package, so if there's another way of doing this, I'd be interested
as well.

TIA

Joost


-- 
Joost Kremers
Life has its moments

Re: ob-python: import local package into a session

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]

I usually do that by either having the py file in the working directory of
the session, in which case you simply import it.

Alternatively something like this in python:

import sys
sys.path.insert(0, '/path/to/dir/with/myfile.py')
import myfile

you could also append to sys.path if that matters.

John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Mon, Nov 23, 2020 at 10:49 AM Joost Kremers <joostkremers@fastmail.fm>
wrote:

> Hi all,
>
> If I have an Org file with Python source blocks, I can run them in a
> session
> with the `:session` header arg. That way, I can include packages installed
> in
> `site-packages` and have them available in all code blocks. But is there a
> way
> to import my own packages into a session? In particular, packages I haven't
> installed system-wide?
>
> What I'm trying to do is to import a Python file with a bunch of utility
> functions into the ob-python session. I thought this might be possible if
> I'd
> structure my code as a regular Python package, because that works if I
> want to
> import my utility functions into another Python file. But it doesn't seem
> to
> work for the ob-python session.
>
> Is there a way to achieve this? I don't *have* to structure my utility
> functions
> as a Python package, so if there's another way of doing this, I'd be
> interested
> as well.
>
> TIA
>
> Joost
>
>
> --
> Joost Kremers
> Life has its moments
>
>

[-- Attachment #2: Type: text/html, Size: 2355 bytes --]

Re: ob-python: import local package into a session

[Jack Kamm]

Hi Joost,

> What I'm trying to do is to import a Python file with a bunch of utility
> functions into the ob-python session. I thought this might be possible if I'd
> structure my code as a regular Python package, because that works if I want to
> import my utility functions into another Python file. But it doesn't seem to
> work for the ob-python session.

You need to make sure your module is either in the working directory you
started the Python session in, or in your PYTHONPATH, for example by
adjusting os.env["PYTHONPATH"] before attempting to import the module.

This shouldn't be ob-python or even Emacs specific. You can test whether
things work by typing "python" in the terminal and attempting to import
your module.

By the way, are you using IPython or vanilla Python? I recently
encountered an issue trying to import modules through a symlink in
IPython, whereas it worked perfectly fine in a vanilla Python session.

Jack

Re: ob-python: import local package into a session

[Jack Kamm]

Jack Kamm <jackkamm@gmail.com> writes:

> You need to make sure your module is either in the working directory you
> started the Python session in, or in your PYTHONPATH, for example by
> adjusting os.env["PYTHONPATH"] before attempting to import the module.

Sorry, this was incorrect, you need to set PYTHONPATH before starting
Python. The correct way to do it from within Python is to use sys.path,
as John points out.

Re: ob-python: import local package into a session

[Joost Kremers]

Hi Jack,

On Mon, Nov 23 2020, Jack Kamm wrote:
> This shouldn't be ob-python or even Emacs specific. You can test whether
> things work by typing "python" in the terminal and attempting to import
> your module.

It all seems to depend on the exact directories, though, and other than
modifying =sys.path= there doesn't seem to be a way to make Org, Python and me
happy at the same time. :-)

I wanted to create a few small, related projects that would share the utility
functions, so I thought I'd put them all in separate subpackages of a single
package. That works with `M-x run-python` and with an LSP server, but when I
then put an Org file in its own subdirectory inside the package, I couldn't
import the utils subpackage.

I guess putting a =:dir= header arg might resolve that, but the Org manual says
that =:dir= should not be used with =:exports both=, which I was also using.

Anyway, thanks to you and to John. I think I have a better idea now how it all
works and what my options are.

> By the way, are you using IPython or vanilla Python? I recently
> encountered an issue trying to import modules through a symlink in
> IPython, whereas it worked perfectly fine in a vanilla Python session.

I'm using IPython in `M-x run-python`, but vanilla Python for Org. There are no
symlinks involved, so I guess it shouldn't matter.

-- 
Joost Kremers
Life has its moments

Re: ob-python: import local package into a session

[Maxim Nikulin]

2020-11-23 Joost Kremers wrote:
> 
> I can include packages installed in
> `site-packages` and have them available in all code blocks. But is there a way
> to import my own packages into a session? In particular, packages I haven't
> installed system-wide?

I do not have a recipe ready for using with org mode, and maybe you have 
already tried the following instruments:

1. Besides system-wide site-packages directory, there is similar ones 
for users, used by e.g. `pip install` (unless it is disabled accordingly 
to some recommendations).

2. It seems that *recommended* and more flexible way is per-project 
(per-version) virtual environments: venv in python3, similar thing were 
called virtualenv in python2: 
https://docs.python.org/3/tutorial/venv.html Maybe there is a convenient 
way to choose and switch venv's directly from emacs. In simple cases 
venv could be activated before starting emacs.

3. The Hitchhiker's Guide to Python have a couple of related sections. 
They might be a bit outdated but could be a source for convincing 
arguments to use venv's or for some usage patterns.
https://docs.python-guide.org/dev/virtualenvs/
https://docs.python-guide.org/dev/pip-virtualenv/

Re: ob-python: import local package into a session

[Joost Kremers]

On Tue, Nov 24 2020, Maxim Nikulin wrote:
> 2. It seems that *recommended* and more flexible way is per-project 
> (per-version) virtual environments: venv in python3, similar thing were 
> called virtualenv in python2: 
> https://docs.python.org/3/tutorial/venv.html Maybe there is a convenient 
> way to choose and switch venv's directly from emacs. In simple cases 
> venv could be activated before starting emacs.

Yes, I'm using virtual environments. Took me a while to get that figured out,
though. Python-the-language is nice enough, but Python-the-ecosystem is quite a
different thing... (Who said there should only be one way to do something?)

I haven't really considered the option to install the utility functions as a
package in the virtual environment, because I expect to change and develop those
functions together with the rest of the project. If it were a separate package,
I'd need to reinstall it every time I make changes to it, which will probably
happen often.

-- 
Joost Kremers
Life has its moments

Re: ob-python: import local package into a session

[Jack Kamm]

Joost Kremers <joostkremers@fastmail.fm> writes:

> I haven't really considered the option to install the utility functions as a
> package in the virtual environment, because I expect to change and develop those
> functions together with the rest of the project. If it were a separate package,
> I'd need to reinstall it every time I make changes to it, which will probably
> happen often.

If you install the package using either "python setup.py develop", or
"pip install -e", then Python will install your code via symlinks
instead of copying, so then you don't have to worry about reinstalling
every time you make an edit.

To switch between venv's in emacs, I use pyvenv:
https://github.com/jorgenschaefer/pyvenv

Re: ob-python: import local package into a session

[Tom Gillespie]

I have also been dissatisfied with the current options for making
local python libraries accessible in certain org files. The amount of
setup that is required outside the org file itself was too large,
especially if you want someone else who is not intimately familiar
with python to be able to use it. My old solution was to modify the
PYTHONPATH environment variable for the whole Emacs process. However,
after a bit of digging inspired by this thread I now have a solution
that is entirely local: advise ~org-babel-execute:python~ to set a new
PYTHONPATH via the ~process-environment~ dynamic variable and set that
from a buffer local variable for the local additions to PYTHONPATH
along with getenv PYTHONPATH. A working example below. Best!
Tom

#+name: orgstrap
#+begin_src elisp :results none :noweb noexport
(defvar-local local-python-path nil)

(defun advise--obe-python-path (command &rest args)
  (let ((process-environment
         (or (and local-python-path
                  (cons
                   (format
                    "PYTHONPATH=%s"
                    (concat local-python-path (getenv "PYTHONPATH")))
                   process-environment))
             process-environment)))
    (apply command args)))

(advice-add #'org-babel-execute:python :around
            #'advise--obe-python-path)

(setq-local local-python-path (concat default-directory "code:"))
#+end_src

On Tue, Nov 24, 2020 at 9:26 AM Jack Kamm <jackkamm@gmail.com> wrote:
>
> Joost Kremers <joostkremers@fastmail.fm> writes:
>
> > I haven't really considered the option to install the utility functions as a
> > package in the virtual environment, because I expect to change and develop those
> > functions together with the rest of the project. If it were a separate package,
> > I'd need to reinstall it every time I make changes to it, which will probably
> > happen often.
>
> If you install the package using either "python setup.py develop", or
> "pip install -e", then Python will install your code via symlinks
> instead of copying, so then you don't have to worry about reinstalling
> every time you make an edit.
>
> To switch between venv's in emacs, I use pyvenv:
> https://github.com/jorgenschaefer/pyvenv
>

Re: ob-python: import local package into a session

[Joost Kremers]

On Tue, Nov 24 2020, Jack Kamm wrote:
> If you install the package using either "python setup.py develop", or
> "pip install -e", then Python will install your code via symlinks
> instead of copying, so then you don't have to worry about reinstalling
> every time you make an edit.

Hey, that's good to know, thanks.

> To switch between venv's in emacs, I use pyvenv:
> https://github.com/jorgenschaefer/pyvenv

Yes, that's what I use, as well.

-- 
Joost Kremers
Life has its moments