Selective font-locking?

Selective font-locking?

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2358 bytes --]

What is the current state of applying font-lock to only portions of a buffer? I’ve seen font-lock+ which allows adding a ‘font-lock-ignore’ property, but it redefines font-lock functions and so may not be reliable long term.  

To make this concrete, here’s a usage case.  I’m currently extending python mode to include support for multiline input.  Here is how python mode currently fontifies the text being input at a shell prompt:
In a post-command-hook, after every change, copies the entire input after the prompt, sans properties, to a hidden “font-lock” buffer with python-mode enabled.
Calls font-lock-ensure, which refontifies this entire buffer.
Copies all the newly updated text properties back into the shell input.
So: super inefficient.  For single lines of input, this is "fast enough".  Once you have multi-line input with hundreds of lines or more, this incurs 50-100ms latency for each and every insertion, deletion, etc.   For a good approximation of how typing with this amount of latency feels, eval the following in *scratch*:

	(add-hook 'post-self-insert-hook (lambda () (sleep-for 0.1) ) nil t)

#1 is readily fixed by using an after-change-function which only updates the relevant text from the shell to the hidden buffer.  But #2 is the real killer, taking 70ms or more to completely re-fontify a decent sized block of input.  Adding region beg/end to font-lock-ensure doesn’t work; how do you know if a change occurred in a long string, for example?

But then, why bother round-tripping text out to a special-use buffer anyway, vs. just letting font-lock operate in-situ in the shell buffer itself using python-mode’s fairly simple font-lock-defaults. The only thing needed to make this work is asking font-lock to ignore all the text with ‘field of ‘output?  

It seems what would be ideal is tying font-lock-defaults to specific ‘field properties, so that only text with a given ‘field (or not matching a given ‘field) is fontified according to the matching set of font-lock rules (with no field specifier matching all text). This would make mixed multi-mode buffer fontification fairly straightforward. 

I’m sure this is simple-minded given the complexities font-lock has to solve, but there has to be a better solution than re-fontifying everything after each character is typed!

[-- Attachment #2: Type: text/html, Size: 3220 bytes --]

Re: Selective font-locking?

[Stefan Monnier]

> But then, why bother round-tripping text out to a special-use buffer anyway,
> vs. just letting font-lock operate in-situ in the shell buffer itself using
> python-mode’s fairly simple font-lock-defaults. The only thing needed to
> make this work is asking font-lock to ignore all the text with ‘field of
> ‘output?  

Maybe you can try something like the following?

    (defvar python--font-lock-keywords ...)
    (defvar python-font-lock-keywords
      '(python--apply-font-lock))
    (defun python--apply-font-lock (limit)
      (while (< (point) limit)
        (let ((next-boundary (find-next-boundary limit)))
          (if (we-should-skip-this-block)
              (goto-char next-boundary)
            (let ((font-lock-keywords python--font-lock-keywords))
              (font-lock-ensure (point) limit))))))


-- Stefan

Re: Selective font-locking?

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

Definitely worth trying, thanks.  I came up with:

(defun python-shell-multiline--apply-font-lock (limit)
  (let ((end (cdr-safe comint-last-prompt)))
    (if (and end (> limit end))
	(let ((font-lock-keywords python-font-lock-keywords)
	      (font-lock-syntactic-face-function
	       #'python-font-lock-syntactic-face-function)
	      (start (max end (point))))
	  (font-lock-flush start limit)
	  (font-lock-ensure start limit)))))

(set (make-local-variable 'font-lock-keywords)
       '(python-shell-multiline--apply-font-lock)))

I can verify that font-lock-ensure is being called on an appropriate region (lots of times).  With either font-lock-flush or font-lock-ensure, no actual fontification occurs.  With both of these together (as above), this error is signaled:

Error during redisplay: (jit-lock-function 179) signaled (void-function python-font-lock-keywords-level-1)

Note that python-font-lock-keywords is a *list* beginning with this symbol:

Python-font-lock-keywords is a variable defined in ‘python.el’.
Its value is
(python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)

Not sure why jit-lock-function would be evaluating it like an sexp.

> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> But then, why bother round-tripping text out to a special-use buffer anyway,
>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>> make this work is asking font-lock to ignore all the text with ‘field of
>> ‘output?  
> 
> Maybe you can try something like the following?
> 
>    (defvar python--font-lock-keywords ...)
>    (defvar python-font-lock-keywords
>      '(python--apply-font-lock))
>    (defun python--apply-font-lock (limit)
>      (while (< (point) limit)
>        (let ((next-boundary (find-next-boundary limit)))
>          (if (we-should-skip-this-block)
>              (goto-char next-boundary)
>            (let ((font-lock-keywords python--font-lock-keywords))
>              (font-lock-ensure (point) limit))))))
> 
> 
> -- Stefan
> 


[-- Attachment #2: Type: text/html, Size: 5199 bytes --]

Re: Selective font-locking?

[Stefan Monnier]

> Python-font-lock-keywords is a variable defined in ‘python.el’.
> Its value is
> (python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)
>
> Not sure why jit-lock-function would be evaluating it like an sexp.

`python-font-lock-keywords` does not hold a valid value for use on
`font-lock-keywords` but a value to be used as the first element of
`font-lock-defaults`.  This "first element" is used to initialize
`font-lock-keywords` but it depends on things like the
`font-lock-maximum-decoration`.


        Stefan


>> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>>> But then, why bother round-tripping text out to a special-use buffer anyway,
>>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>>> make this work is asking font-lock to ignore all the text with ‘field of
>>> ‘output?  
>> 
>> Maybe you can try something like the following?
>> 
>>    (defvar python--font-lock-keywords ...)
>>    (defvar python-font-lock-keywords
>>      '(python--apply-font-lock))
>>    (defun python--apply-font-lock (limit)
>>      (while (< (point) limit)
>>        (let ((next-boundary (find-next-boundary limit)))
>>          (if (we-should-skip-this-block)
>>              (goto-char next-boundary)
>>            (let ((font-lock-keywords python--font-lock-keywords))
>>              (font-lock-ensure (point) limit))))))
>> 
>> 
>> -- Stefan
>> 

Re: Selective font-locking?

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3752 bytes --]

Thank you!  This method works surprisingly well to fontify input at the end of the buffer with another major-mode’s keywords/syntax.  It seems to be fairly efficient even for long multi-line input text:

(defun python-shell-multiline--apply-font-lock (limit)
  (if-let ((process (get-buffer-process (current-buffer)))
	   (pmark (process-mark)))
      (if (> limit pmark)
	  (let ((font-lock-keywords python-shell-multiline-font-lock-keywords)
		(font-lock-syntactic-face-function
		 #'python-font-lock-syntactic-face-function)
		(start (max pmark (point))))
	    (with-syntax-table python-mode-syntax-table
	      (font-lock-flush start limit)
	      (font-lock-ensure start limit))))))

(setq-local font-lock-keywords '(python-shell-multiline--apply-font-lock)
	    font-lock-keywords-only nil
	    syntax-propertize-function python-syntax-propertize-function)
(setq python-shell-multiline-font-lock-keywords
      (symbol-value
       (font-lock-choose-keywords python-font-lock-keywords
				  (font-lock-value-in-major-mode
				   font-lock-maximum-decoration))))

The one curious thing: it definitely requires calls to both font-lock-flush and font-lock-ensure.  Otherwise the input isn’t fontified.  Tracing through, it seems when 'font-lock-mode is enabled, 'font-lock-flush is set to `jit-lock-refontify`, which simply clears the ‘fontified property, presumably expecting it to be noticed and re-fontified.  Not sure if this is the correct/most efficient approach.

It seems to me this general technique could be useful for lots of different “mixed fontification buffers”, simply rebinding font-lock-keywords/syntax table/etc. as needed for the region under consideration. It’s much simpler than the round-trip copy + full buffer re-fontify that many modes use, with no extra buffers to manage, post-command-hooks, etc.  There may be race conditions for highly dynamic buffers, but it’s working well for this usage case. 

Thanks again.

> On Apr 11, 2021, at 5:10 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> Python-font-lock-keywords is a variable defined in ‘python.el’.
>> Its value is
>> (python-font-lock-keywords-level-1 python-font-lock-keywords-level-1 python-font-lock-keywords-level-2 python-font-lock-keywords-maximum-decoration)
>> 
>> Not sure why jit-lock-function would be evaluating it like an sexp.
> 
> `python-font-lock-keywords` does not hold a valid value for use on
> `font-lock-keywords` but a value to be used as the first element of
> `font-lock-defaults`.  This "first element" is used to initialize
> `font-lock-keywords` but it depends on things like the
> `font-lock-maximum-decoration`.
> 
> 
>        Stefan
> 
> 
>>> On Apr 11, 2021, at 12:31 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> 
>>>> But then, why bother round-tripping text out to a special-use buffer anyway,
>>>> vs. just letting font-lock operate in-situ in the shell buffer itself using
>>>> python-mode’s fairly simple font-lock-defaults. The only thing needed to
>>>> make this work is asking font-lock to ignore all the text with ‘field of
>>>> ‘output?  
>>> 
>>> Maybe you can try something like the following?
>>> 
>>>   (defvar python--font-lock-keywords ...)
>>>   (defvar python-font-lock-keywords
>>>     '(python--apply-font-lock))
>>>   (defun python--apply-font-lock (limit)
>>>     (while (< (point) limit)
>>>       (let ((next-boundary (find-next-boundary limit)))
>>>         (if (we-should-skip-this-block)
>>>             (goto-char next-boundary)
>>>           (let ((font-lock-keywords python--font-lock-keywords))
>>>             (font-lock-ensure (point) limit))))))
>>> 
>>> 
>>> -- Stefan
>>> 
> 


[-- Attachment #2: Type: text/html, Size: 8747 bytes --]

Re: Selective font-locking?

[Stefan Monnier]

> (defun python-shell-multiline--apply-font-lock (limit)
>   (if-let ((process (get-buffer-process (current-buffer)))
> 	   (pmark (process-mark)))
>       (if (> limit pmark)
> 	  (let ((font-lock-keywords python-shell-multiline-font-lock-keywords)
> 		(font-lock-syntactic-face-function
> 		 #'python-font-lock-syntactic-face-function)
> 		(start (max pmark (point))))
> 	    (with-syntax-table python-mode-syntax-table
> 	      (font-lock-flush start limit)
> 	      (font-lock-ensure start limit))))))

Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
is quite odd.  I'd call something like `font-lock-fontify-region` instead.

> It seems to me this general technique could be useful for lots of
> different “mixed fontification buffers”, simply rebinding
> font-lock-keywords/syntax table/etc. as needed for the region under
> consideration. It’s much simpler than the round-trip copy + full
> buffer re-fontify that many modes use, with no extra buffers to
> manage, post-command-hooks, etc.

Indeed that matches my intuition.
There's one thing with which you might want to be careful, tho, which is
the `syntax-ppss` state.  You might want to `narrow-to-region` around
the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).

This is because in a shell buffer, some of the past interactions may
have been truncated (e.g. by `comint-truncate-buffer`), so you may end
up with (point-min) being in the middle of a string or something.
[ Similar problems can occur if the prompt itself contains funny characters
  like unmatched quotes.  or if past interactions include output which
  is not lexically valid Python code.  ]


        Stefan

Re: Selective font-locking?

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]



> On Apr 12, 2021, at 10:07 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
> is quite odd.  I'd call something like `font-lock-fontify-region` instead.

Using `font-lock-fontify-region` instead causes Emacs to become mostly unresponsive.  Sending a USR2 reveals:

Debugger entered--entering a function:
* #f(compiled-function () #<bytecode 0x1fed77d1f329>)()
  font-lock-default-fontify-region(188 189 nil)
  font-lock-fontify-region(188 189)
  #f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>)(font-lock-fontify-region)
  run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>) font-lock-fontify-region)
  jit-lock--run-functions(188 189)
  jit-lock-fontify-now(188 688)
  jit-lock-function(188)
  redisplay_internal\ \(C\ function\)()  

> There's one thing with which you might want to be careful, tho, which is
> the `syntax-ppss` state.  You might want to `narrow-to-region` around
> the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).
> 
> This is because in a shell buffer, some of the past interactions may
> have been truncated (e.g. by `comint-truncate-buffer`), so you may end
> up with (point-min) being in the middle of a string or something.
> [ Similar problems can occur if the prompt itself contains funny characters
>  like unmatched quotes.  or if past interactions include output which
>  is not lexically valid Python code.  ]

Well hmm, this is a bummer.  l tested for this issue by inserting an entirely unmatched quote:

In [12]: print(chr(39))
'

and this does affect the syntax (everything is a string).  But unfortunately narrowing as follows doesn’t seem to fix this:

	    (save-restriction
	      (narrow-to-region pmark (point-max))
	      (with-syntax-table python-mode-syntax-table
		(font-lock-flush start limit)
		(font-lock-ensure start limit)))))))

I’m not sure it’s the same thing, but I found a related issue with `indent-for-tab-command'.  In attempting to ignore the prompt for computing indentation, I narrowed to a region which excluded it, but indent.el calls `indent--funcall-widened’, which undoes my narrowing!  

Is there any way to specify "narrow to this region and don’t let anybody widen it(!)"?

[-- Attachment #2: Type: text/html, Size: 4196 bytes --]

Re: Selective font-locking?

[Stefan Monnier]

>> On Apr 12, 2021, at 10:07 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>> Calling `font-lock-flush` or `font-lock-ensure` from font-lock-keywords
>> is quite odd.  I'd call something like `font-lock-fontify-region` instead.
>
> Using `font-lock-fontify-region` instead causes Emacs to become mostly
> unresponsive.  Sending a USR2 reveals:
>
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode 0x1fed77d1f329>)()
>   font-lock-default-fontify-region(188 189 nil)
>   font-lock-fontify-region(188 189)
>   #f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>)(font-lock-fontify-region)
>   run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x1fed77d1f2f9>) font-lock-fontify-region)
>   jit-lock--run-functions(188 189)
>   jit-lock-fontify-now(188 688)
>   jit-lock-function(188)
>   redisplay_internal\ \(C\ function\)()  

That's not giving enough info to figure out what's going on, sorry.
My crystal ball suggests that maybe your matching function simply
forgets to return nil, so font-lock calls it again and again thinking
we're keeping matching "more stuff" without ever reaching limit.

But it does remind me that rather than mess with `font-lock-keywords`,
you could do a similar dance with `font-lock-fontify-region-function`
(and internally call `font-lock-default-fontify-region`).
It might prove simpler and more reliable.

>> There's one thing with which you might want to be careful, tho, which is
>> the `syntax-ppss` state.  You might want to `narrow-to-region` around
>> the call to `font-lock-fontify-region` (maybe narrow to pmark...(point-max)?).
>> 
>> This is because in a shell buffer, some of the past interactions may
>> have been truncated (e.g. by `comint-truncate-buffer`), so you may end
>> up with (point-min) being in the middle of a string or something.
>> [ Similar problems can occur if the prompt itself contains funny characters
>>  like unmatched quotes.  or if past interactions include output which
>>  is not lexically valid Python code.  ]
>
> Well hmm, this is a bummer.  l tested for this issue by inserting an entirely unmatched quote:
>
> In [12]: print(chr(39))
> '
>
> and this does affect the syntax (everything is a string).  But unfortunately narrowing as follows doesn’t seem to fix this:
>
> 	    (save-restriction
> 	      (narrow-to-region pmark (point-max))
> 	      (with-syntax-table python-mode-syntax-table
> 		(font-lock-flush start limit)
> 		(font-lock-ensure start limit)))))))
>
> I’m not sure it’s the same thing, but I found a related issue with `indent-for-tab-command'.  In attempting to ignore the prompt for computing indentation, I narrowed to a region which excluded it, but indent.el calls `indent--funcall-widened’, which undoes my narrowing!  
>
> Is there any way to specify "narrow to this region and don’t let anybody widen it(!)"?

It's called `font-lock-dont-widen`.


        Stefan