unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
@ 2021-04-08 15:48 JD Smith
  2021-04-12  9:32 ` Lars Ingebrigtsen
  0 siblings, 1 reply; 5+ messages in thread
From: JD Smith @ 2021-04-08 15:48 UTC (permalink / raw)
  To: 47657

[-- Attachment #1: Type: text/plain, Size: 9231 bytes --]

	
`python-shell-font-lock-post-command-hook’ makes the presumption that only a single line of input can appear at the prompt.  It includes this preparatory “clean-up” of its font lock buffer after first having copied all input lines after the prompt:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))
           ...
             (replacement
              (python-shell-font-lock-with-font-lock-buffer
                (delete-region (line-beginning-position)
                               (point-max))
                (setq font-lock-buffer-pos (point))
                (insert input)


This means that using, e.g., C-c SPC (`comint-accumulate’) to insert a newline at the shell prompt leads to runaway growth of the font-lock buffer (“ *Python-font-lock”, by default).  Any command, like an insert, produces a new copy of all but the final line of input in this buffer.  Since this entire buffer is font-locked from scratch after every single command, this becomes very inefficient.  

Find an example, below.

The simple fix would be to delete the entire font-lock buffer file contents:

	(delete-region (point-min) (point-max))

A perhaps smarter and more efficient fix would be to enable font locking in the inferior Python buffer itself, but constrain it to the text after the process mark.  This would also prevent front-lock from constantly having to re-fontify the entire input text after each command. 

++++

(In *Python* shell buffer):

In [2]: a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I’ 

(In " *Python-font-lock"; this can become arbitrarily long):

a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I' 

[-- Attachment #2: Type: text/html, Size: 49654 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
  2021-04-08 15:48 bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length JD Smith
@ 2021-04-12  9:32 ` Lars Ingebrigtsen
  2021-04-12 13:18   ` JD Smith
  0 siblings, 1 reply; 5+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-12  9:32 UTC (permalink / raw)
  To: JD Smith; +Cc: 47657

JD Smith <jdtsmith@gmail.com> writes:

> `python-shell-font-lock-post-command-hook’ makes the presumption that only a
> single line of input can appear at the prompt.  It includes this preparatory
> “clean-up” of its font lock buffer after first having copied all input lines after the
> prompt:
>
>        (let* ((input (buffer-substring-no-properties
>                       prompt-end (point-max)))
>             ...
>               (replacement
>                (python-shell-font-lock-with-font-lock-buffer
>                  (delete-region (line-beginning-position)
>                                 (point-max))
>                  (setq font-lock-buffer-pos (point))
>                  (insert input)

I think the point here is that if we're in a multi-line expression, then
the font locking may be dependent on the previous lines, too, so we
can't just delete the contents of the buffer.

> Find an example, below.

[...]

> (In *Python* shell buffer):
>
> In [2]: a= 'now is the time'
> for i in range(12):
> a = 12; print(i+4-4) 
> b='one two one two and through and through'
> d='and I’ 
>
> (In " *Python-font-lock"; this can become arbitrarily long):
>
> a= 'now is the time'
> a= 'now is the time'

I was unable to reproduce this (in Emacs 28).  I pasted this into the
*Python* buffer:

a= 'now is the time'
for i in range(12):
    a = 12; print(i+4-4) 
    b='one two one two and through and through'
    d='and I'

and the font-lock buffer never became very big.

Do you have a step-by-step recipe, starting from "emacs -Q", to
reproduce this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 5+ messages in thread

* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
  2021-04-12  9:32 ` Lars Ingebrigtsen
@ 2021-04-12 13:18   ` JD Smith
  2021-04-13  8:05     ` Lars Ingebrigtsen
  0 siblings, 1 reply; 5+ messages in thread
From: JD Smith @ 2021-04-12 13:18 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: 47657

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]



> On Apr 12, 2021, at 5:32 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> I was unable to reproduce this (in Emacs 28).  I pasted this into the
> *Python* buffer:

Pasting multiple lines constitutes a single command for post-command-hook. In that case you would not expect to see any repeated lines.

> Do you have a step-by-step recipe, starting from "emacs -Q", to
> reproduce this problem?

emacs -Q
M-x run-python
C-x 5 b “ *Python-font-lock*”
In inferior python shell: type any line.
C-c SPC (`comint-accumulate’) to continue.
Type another line.  Notice the first line is repeated. 
Repeat steps 5 & 6 several times. 

The issue is how input is gathered from the prompt to the end of shell buffer in the post-command-hook:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))

Line-at-a-time also won’t be able to handle editing lines before the last.  On emacs-devel Stefan has been helpful with some ideas to perform font lock in-place in the shell buffer, which is working well in my tests.  I’m using that as part of a package to implement true multi-line editing (ala iPython).

[-- Attachment #2: Type: text/html, Size: 2206 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
  2021-04-12 13:18   ` JD Smith
@ 2021-04-13  8:05     ` Lars Ingebrigtsen
  2021-04-13 13:14       ` JD Smith
  0 siblings, 1 reply; 5+ messages in thread
From: Lars Ingebrigtsen @ 2021-04-13  8:05 UTC (permalink / raw)
  To: JD Smith; +Cc: 47657

JD Smith <jdtsmith@gmail.com> writes:

>  Do you have a step-by-step recipe, starting from "emacs -Q", to
>  reproduce this problem?
>
> 1 emacs -Q
> 2 M-x run-python
> 3 C-x 5 b “ *Python-font-lock*”
> 4 In inferior python shell: type any line.
> 5 C-c SPC (`comint-accumulate’) to continue.
> 6 Type another line.  Notice the first line is repeated. 
> 7 Repeat steps 5 & 6 several times. 

You didn't say what Emacs version you're using -- I tested this in Emacs
28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
never contains more than a single non-blank line using this recipe.
(The "any line" I used to test was "5".)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 5+ messages in thread

* bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length
  2021-04-13  8:05     ` Lars Ingebrigtsen
@ 2021-04-13 13:14       ` JD Smith
  0 siblings, 0 replies; 5+ messages in thread
From: JD Smith @ 2021-04-13 13:14 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: 47657

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]



> On Apr 13, 2021, at 4:05 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> JD Smith <jdtsmith@gmail.com> writes:
> 
>> Do you have a step-by-step recipe, starting from "emacs -Q", to
>> reproduce this problem?
>> 
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line.  Notice the first line is repeated. 
>> 7 Repeat steps 5 & 6 several times. 
> 
> You didn't say what Emacs version you're using -- I tested this in Emacs
> 28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
> never contains more than a single non-blank line using this recipe.
> (The "any line" I used to test was "5”.)

Mac port, v27.2, python-mode v0.26.1.  I have a hard time understanding how this would not reproduce, as (buffer-substring-no-properties  prompt-end (point-max)) clearly takes all of the text (multi-lines included) from the prompt onward.  I took a look here <https://github.com/emacs-mirror/emacs/blob/b064ddd3f600ed28e62b09d556ecced5f80d9883/lisp/progmodes/python.el#L2710> (python-mode v0.27.1) and the input is gathered in the same manner in the post-command-hook.  The “repeated” lines in #6 are in the *Python-font-lock* buffer, btw. 

[-- Attachment #2: Type: text/html, Size: 2341 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-04-13 13:14 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-08 15:48 bug#47657: python-shell font-lock with multi-line input: runaway fontification buffer length JD Smith
2021-04-12  9:32 ` Lars Ingebrigtsen
2021-04-12 13:18   ` JD Smith
2021-04-13  8:05     ` Lars Ingebrigtsen
2021-04-13 13:14       ` JD Smith

unofficial mirror of bug-gnu-emacs@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/emacs-bugs/0 emacs-bugs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 emacs-bugs emacs-bugs/ https://yhetil.org/emacs-bugs \
		bug-gnu-emacs@gnu.org
	public-inbox-index emacs-bugs

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.emacs.bugs
	nntp://news.gmane.io/gmane.emacs.bugs


code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs.git

AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git