bug#28051: 25.2; Multiline Statement Indentation Error in Inferior Python Mode

bug#28051: 25.2; Multiline Statement Indentation Error in Inferior Python Mode

[evgeniy.sharapov]

Having simple init.el file like this

(require 'package)
(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
                    (not (gnutls-available-p))))
       (url (concat (if no-ssl "http" "https") "://melpa.org/packages/")))
  (add-to-list 'package-archives (cons "melpa" url) t))
(package-initialize)

(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

(eval-when-compile
  (require 'use-package))

(use-package company
  :ensure t
  :init
  (setq company-idle-delay 0.1
	company-tooltip-limit 20
	company-show-numbers t
	company-selection-wrap-around t
	company-minimum-prefix-length 2
	company-tooltip-align-annotations t
	company-echo-delay 0)
  :config
  ;; default `company-backends'
  (setq company-backends '(company-capf
			   (company-dabbrev-code
			    company-gtags
			    company-etags
			    company-keywords)
			   company-files
			   company-dabbrev))

  (defun ffe-add-company-backends (&rest backends)
    "Adds BACKENDS to the beginning of the buffer-local version of `company-backends' list"
    (set (make-local-variable 'company-backends)
	 (append backends company-backends)))

  (global-company-mode 1))

(use-package python
  :defer t
  :commands python-mode
  :config
  (add-hook 'python-mode-hook #'eldoc-mode))

(use-package anaconda-mode
  :ensure t
  :init (progn
          (add-hook 'python-mode-hook #'anaconda-mode)
          (add-hook 'python-mode-hook #'anaconda-eldoc-mode)))

(use-package company-anaconda
  :defer t
  :ensure t
  :init (with-eval-after-load 'company
          (add-hook 'python-mode-hook (lambda () (ffe-add-company-backends 'company-anaconda)))))


leads to IndentationError once I try to enter multi-line statements in
the Inferior python mode. 

Steps:

1. Start Emacs
2. M-x run-python
3. try to enter something simple like

   for x in range(10):<RET>
   <TAB>print x<RET>

4. The following error occurs:

File "<stdin>", line 1
    print x
    ^
IndentationError: unexpected indent

If `company-mode' is disabled in the Inferior Python Mode via M-x company-mode
then entering multi-line statements work as expected. Using C-M-i or
`complete-symbol' works. 

Also at the start of the Inferior Pyhon Mode following Warnings have
been shown


Warning (python): Python shell prompts cannot be detected.
If your emacs session hangs when starting python shells
recover with ‘keyboard-quit’ and then try fixing the
interactive flag for your interpreter by adjusting the
‘python-shell-interpreter-interactive-arg’ or add regexps
matching shell prompts in the directory-local friendly vars:
  + ‘python-shell-prompt-regexp’
  + ‘python-shell-prompt-block-regexp’
  + ‘python-shell-prompt-output-regexp’
Or alternatively in:
  + ‘python-shell-prompt-input-regexps’
  + ‘python-shell-prompt-output-regexps’
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. 

Howver at the start it said:

Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded


Corresponding issue for the `company-mode' on github.com 
https://github.com/company-mode/company-mode/issues/603


In GNU Emacs 25.2.1 (x86_64-w64-mingw32)
 of 2017-04-24 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 10.0.14393
Configured using:
 'configure --without-dbus --without-compress-install 'CFLAGS=-O2
 -static -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Inferior Python

Minor modes in effect:
  shell-dirtrack-mode: t
  global-company-mode: t
  override-global-mode: t
  compilation-shell-minor-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Saving file c:/Users/esharapov/.emacs.d/init.el...
Wrote c:/Users/esharapov/.emacs.d/init.el [2 times]
For information about GNU Emacs and the GNU system, type C-h C-a.
(New file)
Can’t guess python-indent-offset, using defaults: 4
Shell native completion is disabled, using fallback
Company mode disabled in current buffer
Mark set
Making completion list...
Type "q" in help window to restore previous buffer.

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail eieio-opt speedbar sb-image
ezimage dframe tramp-cache company-anaconda pp cus-edit cus-start
cus-load wid-edit anaconda-mode derived pythonic f python tramp-sh tramp
tramp-compat tramp-loaddefs trampver ucs-normalize shell pcomplete
advice json map dash s company-files company-keywords company-etags
company-gtags company-dabbrev-code company-capf tempo warnings ispell
etags xref project find-func company-dabbrev company-template company
edmacro kmacro tar-mode use-package diminish bind-key easy-mmode cl
compile comint ansi-color ring autoload lisp-mnt mm-archive message
dired format-spec rfc822 mml mml-sec mailabbrev gmm-utils mailheader
mm-decode mm-bodies mm-encode mail-utils network-stream nsm starttls
url-http tls gnutls mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw
url-cache url-auth url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs gnus-util mm-util
help-fns mail-prsvr password-cache url-vars epg finder-inf package
epg-config seq byte-opt gv bytecomp byte-compile cl-extra help-mode
easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 353292 42268)
 (symbols 56 34067 0)
 (miscs 48 165 297)
 (strings 32 68207 11339)
 (string-bytes 1 1877375)
 (vectors 16 55331)
 (vector-slots 8 1659539 79071)
 (floats 8 320 326)
 (intervals 56 1115 2875)
 (buffers 976 70))

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[Evgeniy Sharapov]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

Reproducing this bug with plain Emacs

1. Start emacs without any .emacs or .emacs.d/init.el files
2. Start Inferior python mode by M-x run-python
3. Enter first line of multi-line statement, press Enter and then start
entering second line

>>> for x in range(10):<RET>
...<TAB>pri

4. Now press C-M-i, it will complete it to `print`, then enter x so the
second line looks like

...     print x

5. Now press Enter

instead of a ... prompt you should get an error


>>> for x in range(10):
...     print x
  File "<stdin>", line 1
    print x
    ^
IndentationError: unexpected indent
>>>



==========================================
  Evgeniy N. Sharapov
  Phone: (630) - 779 - 3208

[-- Attachment #2: Type: text/html, Size: 1182 bytes --]

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[Glenn Morris]

Evgeniy Sharapov wrote:

> 1. Start emacs without any .emacs or .emacs.d/init.el files

emacs -Q is better

> 2. Start Inferior python mode by M-x run-python
> 3. Enter first line of multi-line statement, press Enter and then start
> entering second line
>
>>>> for x in range(10):<RET>
> ...<TAB>pri
>
> 4. Now press C-M-i, it will complete it to `print`, then enter x so the
> second line looks like
>
> ...     print x
>
> 5. Now press Enter
>
> instead of a ... prompt you should get an error

Works fine for me. Python 2.7.13 on Debian, Python 2.7.5 on RHEL 7.

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

retitle 28051 [w32 / python "legacy" completion] Multiline Statement Indentation Error
tags 28051 + confirmed
quit

Glenn Morris <rgm@gnu.org> writes:

> Works fine for me. Python 2.7.13 on Debian, Python 2.7.5 on RHEL 7.

I can reproduce this on Windows.  And on GNU/Linux after doing (setq
python-shell-completion-native-enable nil).  Perhaps this is one of the
"limitations" referred to in the comments of python.el:

    ;; ...the "fallback" or "legacy" mechanism works by executing Python
    ;; code in the background and enables auto-completion for shells
    ;; that do not support receiving escape sequences (with some
    ;; limitations, i.e. completion in blocks does not work).

bug#28051: 25.2; Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

evgeniy.sharapov@gmail.com writes:

> leads to IndentationError once I try to enter multi-line statements in
> the Inferior python mode. 
>
> Steps:
>
> 1. Start Emacs
> 2. M-x run-python
> 3. try to enter something simple like
>
>    for x in range(10):<RET>
>    <TAB>print x<RET>
>
> 4. The following error occurs:
>
> File "<stdin>", line 1
>     print x
>     ^
> IndentationError: unexpected indent

Does it help if you install pyreadline?  From comments in python.el:

    ;; If your
    ;; Python installation lacks readline (like CPython for Windows),
    ;; installing pyreadline (URL `http://ipython.org/pyreadline.html')
    ;; should suffice.  To troubleshoot why you are not getting any
    ;; completions, you can try the following in your Python shell:

    ;; >>> import readline, rlcompleter

    ;; If you see an error, then you need to either install pyreadline or
    ;; setup custom code that avoids that dependency.

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

tags 28051 + patch
quit

npostavs@users.sourceforge.net writes:

> I can reproduce this on Windows.  And on GNU/Linux after doing (setq
> python-shell-completion-native-enable nil).  Perhaps this is one of the
> "limitations" referred to in the comments of python.el:
>
>     ;; ...the "fallback" or "legacy" mechanism works by executing Python
>     ;; code in the background and enables auto-completion for shells
>     ;; that do not support receiving escape sequences (with some
>     ;; limitations, i.e. completion in blocks does not work).

I think the best we can do is disable completion with an error message
when the prompt is "...", see attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 3155 bytes --]

From 20693abb64a342255a076645b758590373308a52 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sat, 19 Aug 2017 11:45:07 -0400
Subject: [PATCH] Disable completion while entering python multiline statements

The "legacy" completion mechanism sends newlines to the running python
process to get the list of completions, which confuses things if the
user is in the middle of entering a multiline statement (Bug#28051).
It's better to disable completion in this case.
* lisp/progmodes/python.el (python-shell--block-prompt): New variable.
(python-shell-prompt-set-calculated-regexps): Set it.
(python-shell-completion-at-point): Bail out with error message if
current prompt is a block prompt.
---
 lisp/progmodes/python.el | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b8f8ed2a06..897ee856a5 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -2212,6 +2212,11 @@ python-shell--prompt-calculated-input-regexp
 Do not set this variable directly, instead use
 `python-shell-prompt-set-calculated-regexps'.")
 
+(defvar python-shell--block-prompt nil
+  "Input block prompt for inferior python shell.
+Do not set this variable directly, instead use
+`python-shell-prompt-set-calculated-regexps'.")
+
 (defvar python-shell--prompt-calculated-output-regexp nil
   "Calculated output prompt regexp for inferior python shell.
 Do not set this variable directly, instead use
@@ -2366,6 +2371,7 @@ python-shell-prompt-set-calculated-regexps
         (dolist (prompt (butlast detected-prompts))
           (setq prompt (regexp-quote prompt))
           (cl-pushnew prompt input-prompts :test #'string=))
+        (setq python-shell--block-prompt (nth 1 detected-prompts))
         (cl-pushnew (regexp-quote
                      (car (last detected-prompts)))
                     output-prompts :test #'string=))
@@ -2726,6 +2732,7 @@ inferior-python-mode
   (set (make-local-variable 'python-shell-interpreter-args)
        (or python-shell--interpreter-args python-shell-interpreter-args))
   (set (make-local-variable 'python-shell--prompt-calculated-input-regexp) nil)
+  (set (make-local-variable 'python-shell--block-prompt) nil)
   (set (make-local-variable 'python-shell--prompt-calculated-output-regexp) nil)
   (python-shell-prompt-set-calculated-regexps)
   (setq comint-prompt-regexp python-shell--prompt-calculated-input-regexp)
@@ -3632,7 +3639,9 @@ python-shell-completion-at-point
                        ;; Also, since pdb interaction is single-line
                        ;; based, this is enough.
                        (string-match-p python-shell-prompt-pdb-regexp prompt))
-                   #'python-shell-completion-get-completions)
+                   (if (equal python-shell--block-prompt prompt)
+                       (user-error "Cannot use standard completion in multiline statement")
+                     #'python-shell-completion-get-completions))
                   (t #'python-shell-completion-native-get-completions)))))
     (list start end
           (completion-table-dynamic
-- 
2.14.1

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

npostavs@users.sourceforge.net writes:

>> I can reproduce this on Windows.  And on GNU/Linux after doing (setq
>> python-shell-completion-native-enable nil).  Perhaps this is one of the
>> "limitations" referred to in the comments of python.el:
>>
>>     ;; ...the "fallback" or "legacy" mechanism works by executing Python
>>     ;; code in the background and enables auto-completion for shells
>>     ;; that do not support receiving escape sequences (with some
>>     ;; limitations, i.e. completion in blocks does not work).
>
> I think the best we can do is disable completion with an error message
> when the prompt is "..."

> @@ -3632,7 +3639,9 @@ python-shell-completion-at-point
>                         ;; Also, since pdb interaction is single-line
>                         ;; based, this is enough.
>                         (string-match-p python-shell-prompt-pdb-regexp prompt))
> -                   #'python-shell-completion-get-completions)
> +                   (if (equal python-shell--block-prompt prompt)
> +                       (user-error "Cannot use standard completion in multiline statement")
> +                     #'python-shell-completion-get-completions))
>                    (t #'python-shell-completion-native-get-completions)))))

Hmm, when using company-mode the error message is sent repeatedly, which
seems pretty annoying.  But silently returning no matches could be
confusing.  Dimitri, any advice?

bug#28051: 25.2; Multiline Statement Indentation Error in Inferior Python Mode

[Evgeniy Sharapov]

[-- Attachment #1: Type: text/plain, Size: 2556 bytes --]

I have pyreadline installed. It does help in a sense that completion works
then in the inferior python mode, although I get following *Warning* buffer

Warning (python): Python shell prompts cannot be detected.
If your emacs session hangs when starting python shells
recover with ‘keyboard-quit’ and then try fixing the
interactive flag for your interpreter by adjusting the
‘python-shell-interpreter-interactive-arg’ or add regexps
matching shell prompts in the directory-local friendly vars:
  + ‘python-shell-prompt-regexp’
  + ‘python-shell-prompt-block-regexp’
  + ‘python-shell-prompt-output-regexp’
Or alternatively in:
  + ‘python-shell-prompt-input-regexps’
  + ‘python-shell-prompt-output-regexps’
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support
readline, yet ‘python-shell-completion-native-enable’ was t and "python" is
not part of the ‘python-shell-completion-native-disabled-interpreters’
list.  Native completions have been disabled locally.



But the *Python* buffer says everythng has been loaded ok


Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>>




==========================================
  Evgeniy N. Sharapov
  Phone: (630) - 779 - 3208


On Sat, Aug 19, 2017 at 10:45 AM, <npostavs@users.sourceforge.net> wrote:

> evgeniy.sharapov@gmail.com writes:
>
> > leads to IndentationError once I try to enter multi-line statements in
> > the Inferior python mode.
> >
> > Steps:
> >
> > 1. Start Emacs
> > 2. M-x run-python
> > 3. try to enter something simple like
> >
> >    for x in range(10):<RET>
> >    <TAB>print x<RET>
> >
> > 4. The following error occurs:
> >
> > File "<stdin>", line 1
> >     print x
> >     ^
> > IndentationError: unexpected indent
>
> Does it help if you install pyreadline?  From comments in python.el:
>
>     ;; If your
>     ;; Python installation lacks readline (like CPython for Windows),
>     ;; installing pyreadline (URL `http://ipython.org/pyreadline.html')
>     ;; should suffice.  To troubleshoot why you are not getting any
>     ;; completions, you can try the following in your Python shell:
>
>     ;; >>> import readline, rlcompleter
>
>     ;; If you see an error, then you need to either install pyreadline or
>     ;; setup custom code that avoids that dependency.
>

[-- Attachment #2: Type: text/html, Size: 3670 bytes --]

bug#28051: 25.2; Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

Evgeniy Sharapov <evgeniy.sharapov@gmail.com> writes:

> I have pyreadline installed. It does help in a sense that completion
> works then in the inferior python mode, although I get following
> *Warning* buffer
>
> Warning (python): Python shell prompts cannot be detected.

Ah, this is Bug#21376, you could try applying the patch at
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21376#11.

> Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64

Alternatively, since that bug appers to only occur with Python 2, you
could try using Python 3 instead.

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[Dmitry Gutov]

On 8/22/17 4:36 AM, npostavs@users.sourceforge.net wrote:

> Hmm, when using company-mode the error message is sent repeatedly, which
> seems pretty annoying.  But silently returning no matches could be
> confusing.  Dimitri, any advice?

Use a python-specific global variable and only show the warning one 
time? Or at most once in XX seconds?

There's no existing diagnostic UI for this in company-mode. And the idea 
for completion-at-point-functions is to not signal errors during normal 
operation (an error means there's a bug to be fixed, and bugging the 
user in that case seems okay).

In this particular case, though, I'd simply return no completions. We do 
the same in inf-ruby anyway (no completion on continuation lines). Add a 
warning message once a session, if you prefer.

bug#28051: Multiline Statement Indentation Error in Inferior Python Mode

[npostavs]

close 28051 26.1
quit

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 8/22/17 4:36 AM, npostavs@users.sourceforge.net wrote:
>
>> Hmm, when using company-mode the error message is sent repeatedly, which
>> seems pretty annoying.  But silently returning no matches could be
>> confusing.  Dimitri, any advice?
>
> Use a python-specific global variable and only show the warning one
> time? Or at most once in XX seconds?
>
> There's no existing diagnostic UI for this in company-mode. And the
> idea for completion-at-point-functions is to not signal errors during
> normal operation (an error means there's a bug to be fixed, and
> bugging the user in that case seems okay).
>
> In this particular case, though, I'd simply return no completions. We
> do the same in inf-ruby anyway (no completion on continuation
> lines). Add a warning message once a session, if you prefer.

If it's okay for inf-ruby it's probably okay for python.

[1: 5440b238b1]: 2017-08-27 13:46:41 -0400
  Disable completion while entering python multiline statements
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=5440b238b1ec4175dd32bc14b4098f6570b2ca85