bug#24464: 25.1; python.el virtualenv

bug#24464: 25.1; python.el virtualenv

[Pim]

After setting the variable python-shell-virtualenv-root to the path of a
virtual environment and running the run-python command the virtualenv
doesn't get used.



In GNU Emacs 25.1.1 (x86_64-w64-mingw32)
  of 2016-09-17 built on KAEL
Windowing system distributor 'Microsoft Corp.', version 10.0.14926
Configured using:
  'configure --prefix=/tmp/emacs --without-imagemagick 'CFLAGS=-O2
  -fomit-frame-pointer -g0''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
   value of $LANG: ENU
   locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
   shell-dirtrack-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
You can run the command ‘eval-buffer’ with M-x ev-b RET
You can run the command ‘run-python’ with M-x r-py RET
Shell native completion is disabled, using fallback
Quit

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired rfc822 mml mml-sec epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
kmacro compile warnings tramp-cache python tramp-sh tramp tramp-compat
auth-source cl-seq eieio eieio-core cl-macs gnus-util mm-util help-fns
mail-prsvr password-cache tramp-loaddefs trampver ucs-normalize shell
pcomplete format-spec advice json map seq byte-opt gv bytecomp
byte-compile cl-extra help-mode easymenu cconv comint ring cl-loaddefs
pcase cl-lib ansi-color time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify dbusbind w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 225628 10804)
  (symbols 56 23171 0)
  (miscs 48 53 150)
  (strings 32 27484 7701)
  (string-bytes 1 879903)
  (vectors 16 38112)
  (vector-slots 8 711739 6301)
  (floats 8 237 36)
  (intervals 56 257 15)
  (buffers 976 20))

bug#24464: 25.1; python.el virtualenv

[Justin Timmons]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]


It looks like this was caused by an incorrect assumption on the
directory structure of the virtualenv directory. According to the
virtualenv docs only POSIX systems follow the structure
"/path/to/venv/bin/", while windows systems use "/path/to/venv/Scripts"
for the location of the binary files, most importantly including the
python interpreter (see:
https://virtualenv.pypa.io/en/stable/userguide/#windows-notes).

This meant that the wrong path was added to `exec-path` and so we were
starting the python shell installed on the system, rather than the one
bundled inside of the virtualenv.

This should be addressed in the attached patch file - it just adds a
special case for Windows and prepends that path to the system PATH.

I also updated the docs for the `python-shell-calculate-exec-path`
function to make it more clear that using the python interpreter bundled
in the virtualenv will automatically set the PYTHONPATH to point at the
libraries inside of the virtualenv - something that was not immediately
obvious to me.



Steps to verify that patch is working correctly:
  1. (On Windows)
  2. Create virtualenv (eg. 'virtualenv ~/venv')
  3. Install a library NOT currently installed on the your system inside
  the virtualenv. ex:
       ~/venv/Scripts/activate.bat
       pip install sphinx
       deactivate.bat
  4. Start emacs
  5. Set `python-shell-virtualenv-root` to the root directory of your
  virtualenv (eg. "/home/jtimmons/venv")
  6. M-x run-python
  7. Make sure that you can import the library installed in the
  virtualenv in step 3 (eg. 'import sphinx')


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: A patch to fix Windows venv usage in python shells --]
[-- Type: text/x-patch, Size: 1666 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 3701812..86e99f4 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -2099,20 +2099,25 @@ remote host, the returned value is intended for
 (defun python-shell-calculate-exec-path ()
   "Calculate `exec-path'.
 Prepends `python-shell-exec-path' and adds the binary directory
-for virtualenv if `python-shell-virtualenv-root' is set.  If
-`default-directory' points to a remote host, the returned value
-appends `python-shell-remote-exec-path' instead of `exec-path'."
+for virtualenv if `python-shell-virtualenv-root' is set - this
+will use the python interpreter from inside the virtualenv when
+starting the shell.  If `default-directory' points to a remote host,
+the returned value appends `python-shell-remote-exec-path' instead
+of `exec-path'."
   (let ((new-path (copy-sequence
                    (if (file-remote-p default-directory)
                        python-shell-remote-exec-path
-                     exec-path))))
+                     exec-path)))
+
+        ;; Windows and POSIX systems use different venv directory structures
+        (virtualenv-bin-dir (if (eq system-type 'windows-nt) "Scripts" "bin")))
     (python-shell--add-to-path-with-priority
      new-path python-shell-exec-path)
     (if (not python-shell-virtualenv-root)
         new-path
       (python-shell--add-to-path-with-priority
        new-path
-       (list (expand-file-name "bin" python-shell-virtualenv-root)))
+       (list (expand-file-name virtualenv-bin-dir python-shell-virtualenv-root)))
       new-path)))
 
 (defun python-shell-tramp-refresh-remote-path (vec paths)

bug#24464: 25.1; python.el virtualenv

[Noam Postavsky]

retitle 24464 python virtualenv support doesn't work on w32
quit

Justin Timmons <justinmtimmons@gmail.com> writes:

> It looks like this was caused by an incorrect assumption on the
> directory structure of the virtualenv directory. According to the
> virtualenv docs only POSIX systems follow the structure
> "/path/to/venv/bin/", while windows systems use "/path/to/venv/Scripts"
> for the location of the binary files, most importantly including the
> python interpreter (see:
> https://virtualenv.pypa.io/en/stable/userguide/#windows-notes).
>
> This meant that the wrong path was added to `exec-path` and so we were
> starting the python shell installed on the system, rather than the one
> bundled inside of the virtualenv.
>
> This should be addressed in the attached patch file - it just adds a
> special case for Windows and prepends that path to the system PATH.
>
> I also updated the docs for the `python-shell-calculate-exec-path`
> function to make it more clear that using the python interpreter bundled
> in the virtualenv will automatically set the PYTHONPATH to point at the
> libraries inside of the virtualenv - something that was not immediately
> obvious to me.


>  (defun python-shell-calculate-exec-path ()
>    "Calculate `exec-path'.
>  Prepends `python-shell-exec-path' and adds the binary directory
> -for virtualenv if `python-shell-virtualenv-root' is set.  If
> -`default-directory' points to a remote host, the returned value
> -appends `python-shell-remote-exec-path' instead of `exec-path'."
> +for virtualenv if `python-shell-virtualenv-root' is set - this
> +will use the python interpreter from inside the virtualenv when
> +starting the shell.  If `default-directory' points to a remote host,
> +the returned value appends `python-shell-remote-exec-path' instead
> +of `exec-path'."
>    (let ((new-path (copy-sequence
>                     (if (file-remote-p default-directory)
>                         python-shell-remote-exec-path
> -                     exec-path))))
> +                     exec-path)))
> +
> +        ;; Windows and POSIX systems use different venv directory structures
> +        (virtualenv-bin-dir (if (eq system-type 'windows-nt) "Scripts" "bin")))
>      (python-shell--add-to-path-with-priority
>       new-path python-shell-exec-path)
>      (if (not python-shell-virtualenv-root)
>          new-path
>        (python-shell--add-to-path-with-priority
>         new-path
> -       (list (expand-file-name "bin" python-shell-virtualenv-root)))
> +       (list (expand-file-name virtualenv-bin-dir python-shell-virtualenv-root)))
>        new-path)))

Looks good to me.  Have you assigned copyright for Emacs?  (The patch is
small enough to go in regardless, I only ask to know if we should mark
it.)

bug#24464: 25.1; python.el virtualenv

[Noam Postavsky]

tags 24464 fixed
close 24464 26.1
quit

Justin Timmons <justinmtimmons@gmail.com> writes:

> This should be addressed in the attached patch file - it just adds a
> special case for Windows and prepends that path to the system PATH.

I've pushed this to emacs-26.

[1: 18af404ef3]: 2017-11-05 11:39:06 -0500
  Support python virtualenv on w32 (Bug#24464)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=18af404ef33d8efcbb9446945e543251ab33aa3c