bug#32390: 26.1; python.el: cleanup font lock buffer after input is sent

[Carlos Pita <carlosjosepita@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3812 bytes --]

Hi Noam,

I've been carefully studying this issue and below I explain in detail
what is really happening and what different cases are being (sometimes
wrongly) covered. There is a patch attached, I hope you consider it
worth of being applied.

The condition for cleaning up of the font lock buffer is:

-------------
  (if (and (not (string= "" output))                             (1)
           ;; Is end of output and is not just a prompt.
           (not (member
                 (python-shell-comint-end-of-output-p
                  (ansi-color-filter-apply output))
                 '(nil
         (2)
                   0))))
        (3)
-------------

First nota that (1) is really irrelevant since
(python-shell-comint-end-of-output-p "") returns nil anyway.

Now, the problem originating this report was:

-------------
In [15]: "
  File "<ipython-input-15-3b7a06bb1102>", line 1
    "
     ^
SyntaxError: EOL while scanning string literal


In [16]:   string face still here"
-------------

This happens because
python-shell-font-lock-comint-output-filter-function is called twice,
first for the error output and then for the "In [16]: " part (I assume
this is because one part is coming from stderr and the other from
stdout, but that's just a guess). The first time (2) applies since
we're *not* at the end of an input prompt. The second time (3) applies
since we're at the end of *just* and input prompt. So in any case the
buffer is cleaned up.

Now, my first reaction was to ignore the *just* part: what damage
could it do to just check if we're at the end of an input prompt
disregarding the fact that it could be the only thing there? Well, the
problem is with multiline input, of course. Nevertheless the current
code is relying in a very weak rule: it considers "just an input
prompt" to be a continuation prompt. Another unreliable aspect of the
current rule is that sometimes (python-shell-comint-end-of-output-p
(ansi-color-filter-apply output)) returns 1 and not 0 for continuation
prompts. In short, the rule does a very poor job identifying
continuations.

So, all in all I had rewritten (in a previous post) the condition above as:

-------------
  (if (and (python-shell-comint-end-of-output-p
            (ansi-color-filter-apply output))
           (not (string-match "\\.\\.\\.: $" output)))        (4)
-------------

Where:
- Clause (1) is disregarded because it's redundant.
- Clause (2) is taken into account.
- Clause (3) is disregarded because it's unreliable.
- Clause (4) was added to address the multiline input case
(continuation prompt).

Now, it's a sad fact that python-shell-prompt-input-regexps doesn't
distinguish between initial and continuation input prompts, so I
explicitly had to put that particular regexp there. I assume this is
the main reason while my fix was not yet accepted, isn't it?

At this point we have at least two alternatives:

a) Consider that an input prompt that includes the pattern "\\.\\.\\."
is a continuation prompt. This is a heuristic but I think it's robust
enough. I favor this solution and the attached patch implements it.

b) Add a new customization option with a list of continuation prompts.
I believe this would be too much.

So the attached patch implements this new rule:

-------------
    (if (let ((output (ansi-color-filter-apply output)))
        (and (python-shell-comint-end-of-output-p output)
             (not (string-match "\\.\\.\\." output))))       (5)
-----------

The difference between (4) and (5) is that (5) relaxes the match to
just include three sequential dots (because we already know we have an
input prompt at the end of the output!). I've been more careful by
matching on the filtered string instead of the raw one also.

Please let me know if you prefer option (b) instead.

Best regards
--
Carlos

[-- Attachment #2: font-lock-cleanup-filter.diff --]
[-- Type: text/x-patch, Size: 1181 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index c7bb2d9..218c6e4 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -2556,14 +2556,11 @@ python-shell-font-lock-cleanup-buffer
 
 (defun python-shell-font-lock-comint-output-filter-function (output)
   "Clean up the font-lock buffer after any OUTPUT."
-  (if (and (not (string= "" output))
-           ;; Is end of output and is not just a prompt.
-           (not (member
-                 (python-shell-comint-end-of-output-p
-                  (ansi-color-filter-apply output))
-                 '(nil 0))))
-      ;; If output is other than an input prompt then "real" output has
-      ;; been received and the font-lock buffer must be cleaned up.
+  (if (let ((output (ansi-color-filter-apply output)))
+        (and (python-shell-comint-end-of-output-p output)
+             (not (string-match "\\.\\.\\." output))))
+      ;; If output ends with an initial (not continuation) input prompt
+      ;; then the font-lock buffer must be cleaned up.
       (python-shell-font-lock-cleanup-buffer)
     ;; Otherwise just add a newline.
     (python-shell-font-lock-with-font-lock-buffer