bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 9231 bytes --]

	
`python-shell-font-lock-post-command-hook’ makes the presumption that only a single line of input can appear at the prompt.  It includes this preparatory “clean-up” of its font lock buffer after first having copied all input lines after the prompt:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))
           ...
             (replacement
              (python-shell-font-lock-with-font-lock-buffer
                (delete-region (line-beginning-position)
                               (point-max))
                (setq font-lock-buffer-pos (point))
                (insert input)


This means that using, e.g., C-c SPC (`comint-accumulate’) to insert a newline at the shell prompt leads to runaway growth of the font-lock buffer (“ *Python-font-lock”, by default).  Any command, like an insert, produces a new copy of all but the final line of input in this buffer.  Since this entire buffer is font-locked from scratch after every single command, this becomes very inefficient.  

Find an example, below.

The simple fix would be to delete the entire font-lock buffer file contents:

	(delete-region (point-min) (point-max))

A perhaps smarter and more efficient fix would be to enable font locking in the inferior Python buffer itself, but constrain it to the text after the process mark.  This would also prevent front-lock from constantly having to re-fontify the entire input text after each command. 

++++

(In *Python* shell buffer):

In [2]: a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I’ 

(In " *Python-font-lock"; this can become arbitrarily long):

a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I' 

[-- Attachment #2: Type: text/html, Size: 49654 bytes --]

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[Lars Ingebrigtsen]

JD Smith <jdtsmith@gmail.com> writes:

> `python-shell-font-lock-post-command-hook’ makes the presumption that only a
> single line of input can appear at the prompt.  It includes this preparatory
> “clean-up” of its font lock buffer after first having copied all input lines after the
> prompt:
>
>        (let* ((input (buffer-substring-no-properties
>                       prompt-end (point-max)))
>             ...
>               (replacement
>                (python-shell-font-lock-with-font-lock-buffer
>                  (delete-region (line-beginning-position)
>                                 (point-max))
>                  (setq font-lock-buffer-pos (point))
>                  (insert input)

I think the point here is that if we're in a multi-line expression, then
the font locking may be dependent on the previous lines, too, so we
can't just delete the contents of the buffer.

> Find an example, below.

[...]

> (In *Python* shell buffer):
>
> In [2]: a= 'now is the time'
> for i in range(12):
> a = 12; print(i+4-4) 
> b='one two one two and through and through'
> d='and I’ 
>
> (In " *Python-font-lock"; this can become arbitrarily long):
>
> a= 'now is the time'
> a= 'now is the time'

I was unable to reproduce this (in Emacs 28).  I pasted this into the
*Python* buffer:

a= 'now is the time'
for i in range(12):
    a = 12; print(i+4-4) 
    b='one two one two and through and through'
    d='and I'

and the font-lock buffer never became very big.

Do you have a step-by-step recipe, starting from "emacs -Q", to
reproduce this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]



> On Apr 12, 2021, at 5:32 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> I was unable to reproduce this (in Emacs 28).  I pasted this into the
> *Python* buffer:

Pasting multiple lines constitutes a single command for post-command-hook. In that case you would not expect to see any repeated lines.

> Do you have a step-by-step recipe, starting from "emacs -Q", to
> reproduce this problem?

emacs -Q
M-x run-python
C-x 5 b “ *Python-font-lock*”
In inferior python shell: type any line.
C-c SPC (`comint-accumulate’) to continue.
Type another line.  Notice the first line is repeated. 
Repeat steps 5 & 6 several times. 

The issue is how input is gathered from the prompt to the end of shell buffer in the post-command-hook:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))

Line-at-a-time also won’t be able to handle editing lines before the last.  On emacs-devel Stefan has been helpful with some ideas to perform font lock in-place in the shell buffer, which is working well in my tests.  I’m using that as part of a package to implement true multi-line editing (ala iPython).

[-- Attachment #2: Type: text/html, Size: 2206 bytes --]

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[Lars Ingebrigtsen]

JD Smith <jdtsmith@gmail.com> writes:

>  Do you have a step-by-step recipe, starting from "emacs -Q", to
>  reproduce this problem?
>
> 1 emacs -Q
> 2 M-x run-python
> 3 C-x 5 b “ *Python-font-lock*”
> 4 In inferior python shell: type any line.
> 5 C-c SPC (`comint-accumulate’) to continue.
> 6 Type another line.  Notice the first line is repeated. 
> 7 Repeat steps 5 & 6 several times. 

You didn't say what Emacs version you're using -- I tested this in Emacs
28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
never contains more than a single non-blank line using this recipe.
(The "any line" I used to test was "5".)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]



> On Apr 13, 2021, at 4:05 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> Do you have a step-by-step recipe, starting from "emacs -Q", to
>> reproduce this problem?
>> 
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line.  Notice the first line is repeated. 
>> 7 Repeat steps 5 & 6 several times. 
> 
> You didn't say what Emacs version you're using -- I tested this in Emacs
> 28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
> never contains more than a single non-blank line using this recipe.
> (The "any line" I used to test was "5”.)

Mac port, v27.2, python-mode v0.26.1.  I have a hard time understanding how this would not reproduce, as (buffer-substring-no-properties  prompt-end (point-max)) clearly takes all of the text (multi-lines included) from the prompt onward.  I took a look here <https://github.com/emacs-mirror/emacs/blob/b064ddd3f600ed28e62b09d556ecced5f80d9883/lisp/progmodes/python.el#L2710> (python-mode v0.27.1) and the input is gathered in the same manner in the post-command-hook.  The “repeated” lines in #6 are in the *Python-font-lock* buffer, btw. 

[-- Attachment #2: Type: text/html, Size: 2341 bytes --]

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

JD Smith <jdtsmith@gmail.com> writes:

>  1 emacs -Q
>  2 M-x run-python
>  3 C-x 5 b “ *Python-font-lock*”
>  4 In inferior python shell: type any line.
>  5 C-c SPC (`comint-accumulate’) to continue.
>  6 Type another line.  Notice the first line is repeated. 
>  7 Repeat steps 5 & 6 several times. 
>
>  You didn't say what Emacs version you're using -- I tested this in Emacs
>  28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
>  never contains more than a single non-blank line using this recipe.
>  (The "any line" I used to test was "5”.)
>
> Mac port, v27.2, python-mode v0.26.1.  I have a hard time
> understanding how this would not reproduce, as
> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
> all of the text (multi-lines included) from the prompt onward.  I took
> a look here (python-mode v0.27.1) and the input is gathered in the
> same manner in the post-command-hook.  The “repeated” lines in #6 are
> in the *Python-font-lock* buffer, btw.

After typing "1" in 4) and "7" in 6), the buffer looks like:


[-- Attachment #2: Type: image/png, Size: 59560 bytes --]

[-- Attachment #3: Type: text/plain, Size: 215 bytes --]


So just the "7", and not the first line, which was "1".

Perhaps there's some missing element in your recipe?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

Very odd, I get :

1
1
7

in *Python-font-lock* after precisely these same steps.  What if you use C-j in step #5 instead of C-c SPC?


> On Apr 25, 2021, at 1:34 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line.  Notice the first line is repeated. 
>> 7 Repeat steps 5 & 6 several times. 
>> 
>> You didn't say what Emacs version you're using -- I tested this in Emacs
>> 28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
>> never contains more than a single non-blank line using this recipe.
>> (The "any line" I used to test was "5”.)
>> 
>> Mac port, v27.2, python-mode v0.26.1.  I have a hard time
>> understanding how this would not reproduce, as
>> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
>> all of the text (multi-lines included) from the prompt onward.  I took
>> a look here (python-mode v0.27.1) and the input is gathered in the
>> same manner in the post-command-hook.  The “repeated” lines in #6 are
>> in the *Python-font-lock* buffer, btw.
> 
> After typing "1" in 4) and "7" in 6), the buffer looks like:
> 
> <Mail Attachment.png>
> So just the "7", and not the first line, which was "1".
> 
> Perhaps there's some missing element in your recipe?
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no


[-- Attachment #2: Type: text/html, Size: 2873 bytes --]

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[Lars Ingebrigtsen]

JD Smith <jdtsmith@gmail.com> writes:

> Very odd, I get :
>
>  1
>  1
>  7
>
> in *Python-font-lock* after precisely these same steps.  What if you use C-j in step
> #5 instead of C-c SPC?

I don't see any difference.

(This is with Emacs 27.2 on Debian/Bullseye.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> JD Smith <jdtsmith@gmail.com> writes:
>
>> Very odd, I get :
>>
>>  1
>>  1
>>  7
>>
>> in *Python-font-lock* after precisely these same steps.  What if you
>> use C-j in step
>> #5 instead of C-c SPC?
>
> I don't see any difference.
>
> (This is with Emacs 27.2 on Debian/Bullseye.)

I was finally able to reproduce this -- I'm not sure what I was doing
differently before.  This is now fixed in Emacs 29 (I'm not putting the
fix into emacs-28, because I'm not 100% sure about it).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no