* Selective font-locking?
@ 2021-04-11 15:27 JD Smith
2021-04-11 16:31 ` Stefan Monnier
0 siblings, 1 reply; 8+ messages in thread
From: JD Smith @ 2021-04-11 15:27 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2358 bytes --]
What is the current state of applying font-lock to only portions of a buffer? I’ve seen font-lock+ which allows adding a ‘font-lock-ignore’ property, but it redefines font-lock functions and so may not be reliable long term.
To make this concrete, here’s a usage case. I’m currently extending python mode to include support for multiline input. Here is how python mode currently fontifies the text being input at a shell prompt:
In a post-command-hook, after every change, copies the entire input after the prompt, sans properties, to a hidden “font-lock” buffer with python-mode enabled.
Calls font-lock-ensure, which refontifies this entire buffer.
Copies all the newly updated text properties back into the shell input.
So: super inefficient. For single lines of input, this is "fast enough". Once you have multi-line input with hundreds of lines or more, this incurs 50-100ms latency for each and every insertion, deletion, etc. For a good approximation of how typing with this amount of latency feels, eval the following in *scratch*:
(add-hook 'post-self-insert-hook (lambda () (sleep-for 0.1) ) nil t)
#1 is readily fixed by using an after-change-function which only updates the relevant text from the shell to the hidden buffer. But #2 is the real killer, taking 70ms or more to completely re-fontify a decent sized block of input. Adding region beg/end to font-lock-ensure doesn’t work; how do you know if a change occurred in a long string, for example?
But then, why bother round-tripping text out to a special-use buffer anyway, vs. just letting font-lock operate in-situ in the shell buffer itself using python-mode’s fairly simple font-lock-defaults. The only thing needed to make this work is asking font-lock to ignore all the text with ‘field of ‘output?
It seems what would be ideal is tying font-lock-defaults to specific ‘field properties, so that only text with a given ‘field (or not matching a given ‘field) is fontified according to the matching set of font-lock rules (with no field specifier matching all text). This would make mixed multi-mode buffer fontification fairly straightforward.
I’m sure this is simple-minded given the complexities font-lock has to solve, but there has to be a better solution than re-fontifying everything after each character is typed!
[-- Attachment #2: Type: text/html, Size: 3220 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-11 15:27 Selective font-locking? JD Smith
@ 2021-04-11 16:31 ` Stefan Monnier
2021-04-11 20:54 ` JD Smith
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Monnier @ 2021-04-11 16:31 UTC (permalink / raw)
To: JD Smith; +Cc: emacs-devel
> But then, why bother round-tripping text out to a special-use buffer anyway,
> vs. just letting font-lock operate in-situ in the shell buffer itself using
> python-mode’s fairly simple font-lock-defaults. The only thing needed to
> make this work is asking font-lock to ignore all the text with ‘field of
> ‘output?
Maybe you can try something like the following?
(defvar python--font-lock-keywords ...)
(defvar python-font-lock-keywords
'(python--apply-font-lock))
(defun python--apply-font-lock (limit)
(while (< (point) limit)
(let ((next-boundary (find-next-boundary limit)))
(if (we-should-skip-this-block)
(goto-char next-boundary)
(let ((font-lock-keywords python--font-lock-keywords))
(font-lock-ensure (point) limit))))))
-- Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-11 16:31 ` Stefan Monnier
@ 2021-04-11 20:54 ` JD Smith
2021-04-11 21:10 ` Stefan Monnier
0 siblings, 1 reply; 8+ messages in thread
From: JD Smith @ 2021-04-11 20:54 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]
Definitely worth trying, thanks. I came up with:
(defun python-shell-multiline--apply-font-lock (limit)
(let ((end (cdr-safe comint-last-prompt)))
(if (and end (> limit end))
(let ((font-lock-keywords python-font-lock-keywords)
(font-lock-syntactic-face-function
#'python-font-lock-syntactic-face-function)
(start (max end (point))))
(font-lock-flush start limit)
(font-lock-ensure start limit)))))
(set (make-local-variable 'font-lock-keywords)
'(python-shell-multiline--apply-font-lock)))
I can verify that font-lock-ensure is being called on an appropriate region (lots of times). With either font-lock-flush or font-lock-ensure, no actual fontification occurs. With both of these together (as above), this error is signaled:
Error during redisplay: (jit-lock-function 179) signaled (void-function python-font-lock-keywords-level-1)
Note that python-font-lock-keywords is a *list* beginning with this symbol:
Python-font-lock-keywords is a variable defined in ‘python.el’.
Its value is
(python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)
Not sure why jit-lock-function would be evaluating it like an sexp.
> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>> But then, why bother round-tripping text out to a special-use buffer anyway,
>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>> make this work is asking font-lock to ignore all the text with ‘field of
>> ‘output?
>
> Maybe you can try something like the following?
>
> (defvar python--font-lock-keywords ...)
> (defvar python-font-lock-keywords
> '(python--apply-font-lock))
> (defun python--apply-font-lock (limit)
> (while (< (point) limit)
> (let ((next-boundary (find-next-boundary limit)))
> (if (we-should-skip-this-block)
> (goto-char next-boundary)
> (let ((font-lock-keywords python--font-lock-keywords))
> (font-lock-ensure (point) limit))))))
>
>
> -- Stefan
>
[-- Attachment #2: Type: text/html, Size: 5199 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-11 20:54 ` JD Smith
@ 2021-04-11 21:10 ` Stefan Monnier
2021-04-13 1:51 ` JD Smith
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Monnier @ 2021-04-11 21:10 UTC (permalink / raw)
To: JD Smith; +Cc: emacs-devel
> Python-font-lock-keywords is a variable defined in ‘python.el’.
> Its value is
> (python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)
>
> Not sure why jit-lock-function would be evaluating it like an sexp.
`python-font-lock-keywords` does not hold a valid value for use on
`font-lock-keywords` but a value to be used as the first element of
`font-lock-defaults`. This "first element" is used to initialize
`font-lock-keywords` but it depends on things like the
`font-lock-maximum-decoration`.
Stefan
>> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>>> But then, why bother round-tripping text out to a special-use buffer anyway,
>>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>>> make this work is asking font-lock to ignore all the text with ‘field of
>>> ‘output?
>>
>> Maybe you can try something like the following?
>>
>> (defvar python--font-lock-keywords ...)
>> (defvar python-font-lock-keywords
>> '(python--apply-font-lock))
>> (defun python--apply-font-lock (limit)
>> (while (< (point) limit)
>> (let ((next-boundary (find-next-boundary limit)))
>> (if (we-should-skip-this-block)
>> (goto-char next-boundary)
>> (let ((font-lock-keywords python--font-lock-keywords))
>> (font-lock-ensure (point) limit))))))
>>
>>
>> -- Stefan
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-11 21:10 ` Stefan Monnier
@ 2021-04-13 1:51 ` JD Smith
2021-04-13 2:07 ` Stefan Monnier
0 siblings, 1 reply; 8+ messages in thread
From: JD Smith @ 2021-04-13 1:51 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 3752 bytes --]
Thank you! This method works surprisingly well to fontify input at the end of the buffer with another major-mode’s keywords/syntax. It seems to be fairly efficient even for long multi-line input text:
(defun python-shell-multiline--apply-font-lock (limit)
(if-let ((process (get-buffer-process (current-buffer)))
(pmark (process-mark)))
(if (> limit pmark)
(let ((font-lock-keywords python-shell-multiline-font-lock-keywords)
(font-lock-syntactic-face-function
#'python-font-lock-syntactic-face-function)
(start (max pmark (point))))
(with-syntax-table python-mode-syntax-table
(font-lock-flush start limit)
(font-lock-ensure start limit))))))
(setq-local font-lock-keywords '(python-shell-multiline--apply-font-lock)
font-lock-keywords-only nil
syntax-propertize-function python-syntax-propertize-function)
(setq python-shell-multiline-font-lock-keywords
(symbol-value
(font-lock-choose-keywords python-font-lock-keywords
(font-lock-value-in-major-mode
font-lock-maximum-decoration))))
The one curious thing: it definitely requires calls to both font-lock-flush and font-lock-ensure. Otherwise the input isn’t fontified. Tracing through, it seems when 'font-lock-mode is enabled, 'font-lock-flush is set to `jit-lock-refontify`, which simply clears the ‘fontified property, presumably expecting it to be noticed and re-fontified. Not sure if this is the correct/most efficient approach.
It seems to me this general technique could be useful for lots of different “mixed fontification buffers”, simply rebinding font-lock-keywords/syntax table/etc. as needed for the region under consideration. It’s much simpler than the round-trip copy + full buffer re-fontify that many modes use, with no extra buffers to manage, post-command-hooks, etc. There may be race conditions for highly dynamic buffers, but it’s working well for this usage case.
Thanks again.
> On Apr 11, 2021, at 5:10 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>> Python-font-lock-keywords is a variable defined in ‘python.el’.
>> Its value is
>> (python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)
>>
>> Not sure why jit-lock-function would be evaluating it like an sexp.
>
> `python-font-lock-keywords` does not hold a valid value for use on
> `font-lock-keywords` but a value to be used as the first element of
> `font-lock-defaults`. This "first element" is used to initialize
> `font-lock-keywords` but it depends on things like the
> `font-lock-maximum-decoration`.
>
>
> Stefan
>
>
>>> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>
>>>> But then, why bother round-tripping text out to a special-use buffer anyway,
>>>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>>>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>>>> make this work is asking font-lock to ignore all the text with ‘field of
>>>> ‘output?
>>>
>>> Maybe you can try something like the following?
>>>
>>> (defvar python--font-lock-keywords ...)
>>> (defvar python-font-lock-keywords
>>> '(python--apply-font-lock))
>>> (defun python--apply-font-lock (limit)
>>> (while (< (point) limit)
>>> (let ((next-boundary (find-next-boundary limit)))
>>> (if (we-should-skip-this-block)
>>> (goto-char next-boundary)
>>> (let ((font-lock-keywords python--font-lock-keywords))
>>> (font-lock-ensure (point) limit))))))
>>>
>>>
>>> -- Stefan
>>>
>
[-- Attachment #2: Type: text/html, Size: 8747 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-13 1:51 ` JD Smith
@ 2021-04-13 2:07 ` Stefan Monnier
2021-04-13 3:33 ` JD Smith
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Monnier @ 2021-04-13 2:07 UTC (permalink / raw)
To: JD Smith; +Cc: emacs-devel
> (defun python-shell-multiline--apply-font-lock (limit)
> (if-let ((process (get-buffer-process (current-buffer)))
> (pmark (process-mark)))
> (if (> limit pmark)
> (let ((font-lock-keywords python-shell-multiline-font-lock-keywords)
> (font-lock-syntactic-face-function
> #'python-font-lock-syntactic-face-function)
> (start (max pmark (point))))
> (with-syntax-table python-mode-syntax-table
> (font-lock-flush start limit)
> (font-lock-ensure start limit))))))
Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
is quite odd. I'd call something like `font-lock-fontify-region` instead.
> It seems to me this general technique could be useful for lots of
> different “mixed fontification buffers”, simply rebinding
> font-lock-keywords/syntax table/etc. as needed for the region under
> consideration. It’s much simpler than the round-trip copy + full
> buffer re-fontify that many modes use, with no extra buffers to
> manage, post-command-hooks, etc.
Indeed that matches my intuition.
There's one thing with which you might want to be careful, tho, which is
the `syntax-ppss` state. You might want to `narrow-to-region` around
the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).
This is because in a shell buffer, some of the past interactions may
have been truncated (e.g. by `comint-truncate-buffer`), so you may end
up with (point-min) being in the middle of a string or something.
[ Similar problems can occur if the prompt itself contains funny characters
like unmatched quotes. or if past interactions include output which
is not lexically valid Python code. ]
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-13 2:07 ` Stefan Monnier
@ 2021-04-13 3:33 ` JD Smith
2021-04-13 4:04 ` Stefan Monnier
0 siblings, 1 reply; 8+ messages in thread
From: JD Smith @ 2021-04-13 3:33 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]
> On Apr 12, 2021, at 10:07 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
> is quite odd. I'd call something like `font-lock-fontify-region` instead.
Using `font-lock-fontify-region` instead causes Emacs to become mostly unresponsive. Sending a USR2 reveals:
Debugger entered--entering a function:
* #f(compiled-function () #<bytecode 0x1fed77d1f329>)()
font-lock-default-fontify-region(188 189 nil)
font-lock-fontify-region(188 189)
#f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>)(font-lock-fontify-region)
run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>) font-lock-fontify-region)
jit-lock--run-functions(188 189)
jit-lock-fontify-now(188 688)
jit-lock-function(188)
redisplay_internal\ \(C\ function\)()
> There's one thing with which you might want to be careful, tho, which is
> the `syntax-ppss` state. You might want to `narrow-to-region` around
> the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).
>
> This is because in a shell buffer, some of the past interactions may
> have been truncated (e.g. by `comint-truncate-buffer`), so you may end
> up with (point-min) being in the middle of a string or something.
> [ Similar problems can occur if the prompt itself contains funny characters
> like unmatched quotes. or if past interactions include output which
> is not lexically valid Python code. ]
Well hmm, this is a bummer. l tested for this issue by inserting an entirely unmatched quote:
In [12]: print(chr(39))
'
and this does affect the syntax (everything is a string). But unfortunately narrowing as follows doesn’t seem to fix this:
(save-restriction
(narrow-to-region pmark (point-max))
(with-syntax-table python-mode-syntax-table
(font-lock-flush start limit)
(font-lock-ensure start limit)))))))
I’m not sure it’s the same thing, but I found a related issue with `indent-for-tab-command'. In attempting to ignore the prompt for computing indentation, I narrowed to a region which excluded it, but indent.el calls `indent--funcall-widened’, which undoes my narrowing!
Is there any way to specify "narrow to this region and don’t let anybody widen it(!)"?
[-- Attachment #2: Type: text/html, Size: 4196 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Selective font-locking?
2021-04-13 3:33 ` JD Smith
@ 2021-04-13 4:04 ` Stefan Monnier
0 siblings, 0 replies; 8+ messages in thread
From: Stefan Monnier @ 2021-04-13 4:04 UTC (permalink / raw)
To: JD Smith; +Cc: emacs-devel
>> On Apr 12, 2021, at 10:07 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>> Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
>> is quite odd. I'd call something like `font-lock-fontify-region` instead.
>
> Using `font-lock-fontify-region` instead causes Emacs to become mostly
> unresponsive. Sending a USR2 reveals:
>
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode 0x1fed77d1f329>)()
> font-lock-default-fontify-region(188 189 nil)
> font-lock-fontify-region(188 189)
> #f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>)(font-lock-fontify-region)
> run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>) font-lock-fontify-region)
> jit-lock--run-functions(188 189)
> jit-lock-fontify-now(188 688)
> jit-lock-function(188)
> redisplay_internal\ \(C\ function\)()
That's not giving enough info to figure out what's going on, sorry.
My crystal ball suggests that maybe your matching function simply
forgets to return nil, so font-lock calls it again and again thinking
we're keeping matching "more stuff" without ever reaching limit.
But it does remind me that rather than mess with `font-lock-keywords`,
you could do a similar dance with `font-lock-fontify-region-function`
(and internally call `font-lock-default-fontify-region`).
It might prove simpler and more reliable.
>> There's one thing with which you might want to be careful, tho, which is
>> the `syntax-ppss` state. You might want to `narrow-to-region` around
>> the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).
>>
>> This is because in a shell buffer, some of the past interactions may
>> have been truncated (e.g. by `comint-truncate-buffer`), so you may end
>> up with (point-min) being in the middle of a string or something.
>> [ Similar problems can occur if the prompt itself contains funny characters
>> like unmatched quotes. or if past interactions include output which
>> is not lexically valid Python code. ]
>
> Well hmm, this is a bummer. l tested for this issue by inserting an entirely unmatched quote:
>
> In [12]: print(chr(39))
> '
>
> and this does affect the syntax (everything is a string). But unfortunately narrowing as follows doesn’t seem to fix this:
>
> (save-restriction
> (narrow-to-region pmark (point-max))
> (with-syntax-table python-mode-syntax-table
> (font-lock-flush start limit)
> (font-lock-ensure start limit)))))))
>
> I’m not sure it’s the same thing, but I found a related issue with `indent-for-tab-command'. In attempting to ignore the prompt for computing indentation, I narrowed to a region which excluded it, but indent.el calls `indent--funcall-widened’, which undoes my narrowing!
>
> Is there any way to specify "narrow to this region and don’t let anybody widen it(!)"?
It's called `font-lock-dont-widen`.
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-04-13 4:04 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-04-11 15:27 Selective font-locking? JD Smith
2021-04-11 16:31 ` Stefan Monnier
2021-04-11 20:54 ` JD Smith
2021-04-11 21:10 ` Stefan Monnier
2021-04-13 1:51 ` JD Smith
2021-04-13 2:07 ` Stefan Monnier
2021-04-13 3:33 ` JD Smith
2021-04-13 4:04 ` Stefan Monnier
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.