* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
@ 2021-04-08 15:48 JD Smith
2021-04-12 9:32 ` Lars Ingebrigtsen
0 siblings, 1 reply; 9+ messages in thread
From: JD Smith @ 2021-04-08 15:48 UTC (permalink / raw)
To: 47657
[-- Attachment #1: Type: text/plain, Size: 9231 bytes --]
`python-shell-font-lock-post-command-hook’ makes the presumption that only a single line of input can appear at the prompt. It includes this preparatory “clean-up” of its font lock buffer after first having copied all input lines after the prompt:
(let* ((input (buffer-substring-no-properties
prompt-end (point-max)))
...
(replacement
(python-shell-font-lock-with-font-lock-buffer
(delete-region (line-beginning-position)
(point-max))
(setq font-lock-buffer-pos (point))
(insert input)
This means that using, e.g., C-c SPC (`comint-accumulate’) to insert a newline at the shell prompt leads to runaway growth of the font-lock buffer (“ *Python-font-lock”, by default). Any command, like an insert, produces a new copy of all but the final line of input in this buffer. Since this entire buffer is font-locked from scratch after every single command, this becomes very inefficient.
Find an example, below.
The simple fix would be to delete the entire font-lock buffer file contents:
(delete-region (point-min) (point-max))
A perhaps smarter and more efficient fix would be to enable font locking in the inferior Python buffer itself, but constrain it to the text after the process mark. This would also prevent front-lock from constantly having to re-fontify the entire input text after each command.
++++
(In *Python* shell buffer):
In [2]: a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
d='and I’
(In " *Python-font-lock"; this can become arbitrarily long):
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
d='and I'
[-- Attachment #2: Type: text/html, Size: 49654 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-08 15:48 bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length JD Smith
@ 2021-04-12 9:32 ` Lars Ingebrigtsen
2021-04-12 13:18 ` JD Smith
0 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-12 9:32 UTC (permalink / raw)
To: JD Smith; +Cc: 47657
JD Smith <jdtsmith@gmail.com> writes:
> `python-shell-font-lock-post-command-hook’ makes the presumption that only a
> single line of input can appear at the prompt. It includes this preparatory
> “clean-up” of its font lock buffer after first having copied all input lines after the
> prompt:
>
> (let* ((input (buffer-substring-no-properties
> prompt-end (point-max)))
> ...
> (replacement
> (python-shell-font-lock-with-font-lock-buffer
> (delete-region (line-beginning-position)
> (point-max))
> (setq font-lock-buffer-pos (point))
> (insert input)
I think the point here is that if we're in a multi-line expression, then
the font locking may be dependent on the previous lines, too, so we
can't just delete the contents of the buffer.
> Find an example, below.
[...]
> (In *Python* shell buffer):
>
> In [2]: a= 'now is the time'
> for i in range(12):
> a = 12; print(i+4-4)
> b='one two one two and through and through'
> d='and I’
>
> (In " *Python-font-lock"; this can become arbitrarily long):
>
> a= 'now is the time'
> a= 'now is the time'
I was unable to reproduce this (in Emacs 28). I pasted this into the
*Python* buffer:
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4)
b='one two one two and through and through'
d='and I'
and the font-lock buffer never became very big.
Do you have a step-by-step recipe, starting from "emacs -Q", to
reproduce this problem?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-12 9:32 ` Lars Ingebrigtsen
@ 2021-04-12 13:18 ` JD Smith
2021-04-13 8:05 ` Lars Ingebrigtsen
0 siblings, 1 reply; 9+ messages in thread
From: JD Smith @ 2021-04-12 13:18 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: 47657
[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]
> On Apr 12, 2021, at 5:32 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> I was unable to reproduce this (in Emacs 28). I pasted this into the
> *Python* buffer:
Pasting multiple lines constitutes a single command for post-command-hook. In that case you would not expect to see any repeated lines.
> Do you have a step-by-step recipe, starting from "emacs -Q", to
> reproduce this problem?
emacs -Q
M-x run-python
C-x 5 b “ *Python-font-lock*”
In inferior python shell: type any line.
C-c SPC (`comint-accumulate’) to continue.
Type another line. Notice the first line is repeated.
Repeat steps 5 & 6 several times.
The issue is how input is gathered from the prompt to the end of shell buffer in the post-command-hook:
(let* ((input (buffer-substring-no-properties
prompt-end (point-max)))
Line-at-a-time also won’t be able to handle editing lines before the last. On emacs-devel Stefan has been helpful with some ideas to perform font lock in-place in the shell buffer, which is working well in my tests. I’m using that as part of a package to implement true multi-line editing (ala iPython).
[-- Attachment #2: Type: text/html, Size: 2206 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-12 13:18 ` JD Smith
@ 2021-04-13 8:05 ` Lars Ingebrigtsen
2021-04-13 13:14 ` JD Smith
0 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-13 8:05 UTC (permalink / raw)
To: JD Smith; +Cc: 47657
JD Smith <jdtsmith@gmail.com> writes:
> Do you have a step-by-step recipe, starting from "emacs -Q", to
> reproduce this problem?
>
> 1 emacs -Q
> 2 M-x run-python
> 3 C-x 5 b “ *Python-font-lock*”
> 4 In inferior python shell: type any line.
> 5 C-c SPC (`comint-accumulate’) to continue.
> 6 Type another line. Notice the first line is repeated.
> 7 Repeat steps 5 & 6 several times.
You didn't say what Emacs version you're using -- I tested this in Emacs
28, and was unable to reproduce this. The " *Python-font-lock*" buffer
never contains more than a single non-blank line using this recipe.
(The "any line" I used to test was "5".)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-13 8:05 ` Lars Ingebrigtsen
@ 2021-04-13 13:14 ` JD Smith
2021-04-25 17:34 ` Lars Ingebrigtsen
0 siblings, 1 reply; 9+ messages in thread
From: JD Smith @ 2021-04-13 13:14 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: 47657
[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]
> On Apr 13, 2021, at 4:05 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> JD Smith <jdtsmith@gmail.com> writes:
>
>> Do you have a step-by-step recipe, starting from "emacs -Q", to
>> reproduce this problem?
>>
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line. Notice the first line is repeated.
>> 7 Repeat steps 5 & 6 several times.
>
> You didn't say what Emacs version you're using -- I tested this in Emacs
> 28, and was unable to reproduce this. The " *Python-font-lock*" buffer
> never contains more than a single non-blank line using this recipe.
> (The "any line" I used to test was "5”.)
Mac port, v27.2, python-mode v0.26.1. I have a hard time understanding how this would not reproduce, as (buffer-substring-no-properties prompt-end (point-max)) clearly takes all of the text (multi-lines included) from the prompt onward. I took a look here <https://github.com/emacs-mirror/emacs/blob/b064ddd3f600ed28e62b09d556ecced5f80d9883/lisp/progmodes/python.el#L2710> (python-mode v0.27.1) and the input is gathered in the same manner in the post-command-hook. The “repeated” lines in #6 are in the *Python-font-lock* buffer, btw.
[-- Attachment #2: Type: text/html, Size: 2341 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-13 13:14 ` JD Smith
@ 2021-04-25 17:34 ` Lars Ingebrigtsen
2021-04-25 17:53 ` JD Smith
0 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-25 17:34 UTC (permalink / raw)
To: JD Smith; +Cc: 47657
[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]
JD Smith <jdtsmith@gmail.com> writes:
> 1 emacs -Q
> 2 M-x run-python
> 3 C-x 5 b “ *Python-font-lock*”
> 4 In inferior python shell: type any line.
> 5 C-c SPC (`comint-accumulate’) to continue.
> 6 Type another line. Notice the first line is repeated.
> 7 Repeat steps 5 & 6 several times.
>
> You didn't say what Emacs version you're using -- I tested this in Emacs
> 28, and was unable to reproduce this. The " *Python-font-lock*" buffer
> never contains more than a single non-blank line using this recipe.
> (The "any line" I used to test was "5”.)
>
> Mac port, v27.2, python-mode v0.26.1. I have a hard time
> understanding how this would not reproduce, as
> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
> all of the text (multi-lines included) from the prompt onward. I took
> a look here (python-mode v0.27.1) and the input is gathered in the
> same manner in the post-command-hook. The “repeated” lines in #6 are
> in the *Python-font-lock* buffer, btw.
After typing "1" in 4) and "7" in 6), the buffer looks like:
[-- Attachment #2: Type: image/png, Size: 59560 bytes --]
[-- Attachment #3: Type: text/plain, Size: 215 bytes --]
So just the "7", and not the first line, which was "1".
Perhaps there's some missing element in your recipe?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-25 17:34 ` Lars Ingebrigtsen
@ 2021-04-25 17:53 ` JD Smith
2021-04-25 19:19 ` Lars Ingebrigtsen
0 siblings, 1 reply; 9+ messages in thread
From: JD Smith @ 2021-04-25 17:53 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: 47657
[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]
Very odd, I get :
1
1
7
in *Python-font-lock* after precisely these same steps. What if you use C-j in step #5 instead of C-c SPC?
> On Apr 25, 2021, at 1:34 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> JD Smith <jdtsmith@gmail.com> writes:
>
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line. Notice the first line is repeated.
>> 7 Repeat steps 5 & 6 several times.
>>
>> You didn't say what Emacs version you're using -- I tested this in Emacs
>> 28, and was unable to reproduce this. The " *Python-font-lock*" buffer
>> never contains more than a single non-blank line using this recipe.
>> (The "any line" I used to test was "5”.)
>>
>> Mac port, v27.2, python-mode v0.26.1. I have a hard time
>> understanding how this would not reproduce, as
>> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
>> all of the text (multi-lines included) from the prompt onward. I took
>> a look here (python-mode v0.27.1) and the input is gathered in the
>> same manner in the post-command-hook. The “repeated” lines in #6 are
>> in the *Python-font-lock* buffer, btw.
>
> After typing "1" in 4) and "7" in 6), the buffer looks like:
>
> <Mail Attachment.png>
> So just the "7", and not the first line, which was "1".
>
> Perhaps there's some missing element in your recipe?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
> bloggy blog: http://lars.ingebrigtsen.no
[-- Attachment #2: Type: text/html, Size: 2873 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-25 17:53 ` JD Smith
@ 2021-04-25 19:19 ` Lars Ingebrigtsen
2021-10-13 16:45 ` Lars Ingebrigtsen
0 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-25 19:19 UTC (permalink / raw)
To: JD Smith; +Cc: 47657
JD Smith <jdtsmith@gmail.com> writes:
> Very odd, I get :
>
> 1
> 1
> 7
>
> in *Python-font-lock* after precisely these same steps. What if you use C-j in step
> #5 instead of C-c SPC?
I don't see any difference.
(This is with Emacs 27.2 on Debian/Bullseye.)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
2021-04-25 19:19 ` Lars Ingebrigtsen
@ 2021-10-13 16:45 ` Lars Ingebrigtsen
0 siblings, 0 replies; 9+ messages in thread
From: Lars Ingebrigtsen @ 2021-10-13 16:45 UTC (permalink / raw)
To: JD Smith; +Cc: 47657
Lars Ingebrigtsen <larsi@gnus.org> writes:
> JD Smith <jdtsmith@gmail.com> writes:
>
>> Very odd, I get :
>>
>> 1
>> 1
>> 7
>>
>> in *Python-font-lock* after precisely these same steps. What if you
>> use C-j in step
>> #5 instead of C-c SPC?
>
> I don't see any difference.
>
> (This is with Emacs 27.2 on Debian/Bullseye.)
I was finally able to reproduce this -- I'm not sure what I was doing
differently before. This is now fixed in Emacs 29 (I'm not putting the
fix into emacs-28, because I'm not 100% sure about it).
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-13 16:45 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-08 15:48 bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length JD Smith
2021-04-12 9:32 ` Lars Ingebrigtsen
2021-04-12 13:18 ` JD Smith
2021-04-13 8:05 ` Lars Ingebrigtsen
2021-04-13 13:14 ` JD Smith
2021-04-25 17:34 ` Lars Ingebrigtsen
2021-04-25 17:53 ` JD Smith
2021-04-25 19:19 ` Lars Ingebrigtsen
2021-10-13 16:45 ` Lars Ingebrigtsen
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).