Any expert on font-lock machinery able to provide some insight

Any expert on font-lock machinery able to provide some insight

[Harald Kirsch]

Hi font-lock experts?

I am working on a font-lock function and see it called by the jit-lock
timer at weird times.

I see a normal font-lock function call after a change, for example when
inserting a character. But then I see

- calls when moving the cursor (no scroll)
- when setting the mark
- when moving the mouse into **another** frame
- when moving the mouse into yet **another** frame

I understand that when the function does not return

   (jit-lock-bounds beg . end)

jit-lock assumes the region was fontified just fine. For completeness I
also added the an explicit (jit-lock-bounds beg . end) return to my
font-lock function. Same behavior.

But it seems I am missing another channel of information which triggers
font-locking too often.

With describe-char I do see

There are text properties here:
   fontified            defer

not going away. Can this point to the problem?

Hints appreciated.

Harald


P.S.: I am not showing code on purpose as this would be a lot. I'd
rather just hear from anyone who might have had a similar issue to get
ideas where to look.

Re: Any expert on font-lock machinery able to provide some insight

[Eli Zaretskii]

> Date: Fri, 3 Jan 2025 11:26:11 +0100
> From: Harald Kirsch <pifpafpuf@gmx.de>
> 
> Hi font-lock experts?
> 
> I am working on a font-lock function and see it called by the jit-lock
> timer at weird times.
> 
> I see a normal font-lock function call after a change, for example when
> inserting a character. But then I see
> 
> - calls when moving the cursor (no scroll)
> - when setting the mark
> - when moving the mouse into **another** frame
> - when moving the mouse into yet **another** frame

This is expected.  Lisp programs sometimes need to make layout
calculations for some reason.  For example, when you move the cursor
with C-f, Emacs needs to know the buffer position that corresponds to
the character to the right of the previous one.  Since you can have
several different faces and fonts in a buffer, the answer to that
question needs to execute some of the layout code that is part of the
display engine, and that will call jit-lock if the corresponding
region of the buffer text was not yet fontified.

> I understand that when the function does not return
> 
>    (jit-lock-bounds beg . end)
> 
> jit-lock assumes the region was fontified just fine. For completeness I
> also added the an explicit (jit-lock-bounds beg . end) return to my
> font-lock function. Same behavior.
> 
> But it seems I am missing another channel of information which triggers
> font-locking too often.

Why does it bother you that it happens too often?

> With describe-char I do see
> 
> There are text properties here:
>    fontified            defer
> 
> not going away. Can this point to the problem?

This should only happen with buffer positions that were not yet
fontified.  If the buffer position was already fontified, the value
should be t.

Re: Any expert on font-lock machinery able to provide some insight

[Eli Zaretskii]

[Please use Reply All, to keep the list CC'ed.]

> Date: Fri, 3 Jan 2025 13:44:04 +0100
> From: Harald Kirsch <pifpafpuf@gmx.de>
> 
> Hi Eli,
> 
> thanks for the explanation.
> 
> On 03.01.25 12:58, Eli Zaretskii wrote:
> ...
> >> But it seems I am missing another channel of information which triggers
> >> font-locking too often.
> >
> > Why does it bother you that it happens too often?
> 
> 1. I compare with elisp font-locking which is much less frequent.
> 
> 2. It is eglot-semtok, which does an LSP server call to get font-lock
> information. It is quick enough and I wouldn't have noticed without the
> logging, but it seems a waste nevertheless.
> 
> >> With describe-char I do see
> >>
> >> There are text properties here:
> >>     fontified            defer
> >>
> >> not going away. Can this point to the problem?
> >
> > This should only happen with buffer positions that were not yet
> > fontified.  If the buffer position was already fontified, the value
> > should be t.
> 
> The buffer position was already fontified, so I should not see this. I
> might be doing something wrong so that the font-lock machinery thinks,
> font-locking did not happen. The actual fontification happens
> asynchronously (due to the server roundtrip), but I thought I had given
> the engine enough info pretending all is done. I don't fully understand,
> how the decision is made to fontify again.
> 
> Cheers
> Harald
> 

Re: Any expert on font-lock machinery able to provide some insight

[Harald Kirsch]

For better research I made a trivial example font lock function and call
it, to simulate the async server round-trip, on a timer like so:

;;; -*- lexical-binding: t; -*-

(defun eglot-semtok-request-fontification (&optional beg end loudly)
   (message "front %s-%s" beg end)
   (run-with-timer
    0.1 nil
    (lambda () (t-eglot-semtok-request-fontification beg end loudly)) ) )

(defun t-eglot-semtok-request-fontification (&optional beg end loudly)
   (message "timer %s-%s" beg end)
   (save-excursion
     (font-lock-unfontify-region beg end)
     (goto-char beg)
     (while-let ((mend (re-search-forward "[a-z]+\\(-[a-z]+\\)+" end t))
                 (mstart (car (match-data))) )
       (add-face-text-property mstart mend 'bold) ) ))

(setq font-lock-defaults
       '(nil nil nil nil
             (font-lock-fontify-region-function
              . eglot-semtok-request-fontification )
             (font-lock-fontify-buffer-function
              . eglot-semtok-request-fontification ) ) )


- put into text file (easier to override font-lock)
- M-x eval-buffer
- M-x font-lock-mode
- M-x font-lock-mode
- insert a space near the top
- move cursor

See how for each cursor move the same front and timer messages are
shown. When configuring the t-... function directly in setq and starting
over, the extra font-lock calls are not seen.

How would eglot-semtok-request-fontification need to change, short of
actually fontifying everything, to lure font-locking into thinking all
is fine, no need to run again shortly after?

Harald.


On 03.01.25 14:32, Eli Zaretskii wrote:
> [Please use Reply All, to keep the list CC'ed.]
>
>> Date: Fri, 3 Jan 2025 13:44:04 +0100
>> From: Harald Kirsch <pifpafpuf@gmx.de>
>>
>> Hi Eli,
>>
>> thanks for the explanation.
>>
>> On 03.01.25 12:58, Eli Zaretskii wrote:
>> ...
>>>> But it seems I am missing another channel of information which triggers
>>>> font-locking too often.
>>>
>>> Why does it bother you that it happens too often?
>>
>> 1. I compare with elisp font-locking which is much less frequent.
>>
>> 2. It is eglot-semtok, which does an LSP server call to get font-lock
>> information. It is quick enough and I wouldn't have noticed without the
>> logging, but it seems a waste nevertheless.
>>
>>>> With describe-char I do see
>>>>
>>>> There are text properties here:
>>>>      fontified            defer
>>>>
>>>> not going away. Can this point to the problem?
>>>
>>> This should only happen with buffer positions that were not yet
>>> fontified.  If the buffer position was already fontified, the value
>>> should be t.
>>
>> The buffer position was already fontified, so I should not see this. I
>> might be doing something wrong so that the font-lock machinery thinks,
>> font-locking did not happen. The actual fontification happens
>> asynchronously (due to the server roundtrip), but I thought I had given
>> the engine enough info pretending all is done. I don't fully understand,
>> how the decision is made to fontify again.
>>
>> Cheers
>> Harald
>>

Re: Any expert on font-lock machinery able to provide some insight

[John ff]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]



⁣​

On 3 Jan 2025, 13:34, at 13:34, Eli Zaretskii <eliz@gnu.org> wrote:
>[Please use Reply All, to keep the list CC'ed.]
>
>> Date: Fri, 3 Jan 2025 13:44:04 +0100
>> From: Harald Kirsch <pifpafpuf@gmx.de>
>>
>> Hi Eli,
>>
>> thanks for the explanation.
>>
>> On 03.01.25 12:58, Eli Zaretskii wrote:
>> ...
>> >> But it seems I am missing another channel of information which
>triggers
>> >> font-locking too often.
>> >
>> > Why does it bother you that it happens too often?
>>
>> 1. I compare with elisp font-locking which is much less frequent.
>>
>> 2. It is eglot-semtok, which does an LSP server call to get font-lock
>> information. It is quick enough and I wouldn't have noticed without
>the
>> logging, but it seems a waste nevertheless.
>>
>> >> With describe-char I do see
>> >>
>> >> There are text properties here:
>> >>     fontified            defer
>> >>
>> >> not going away. Can this point to the problem?
>> >
>> > This should only happen with buffer positions that were not yet
>> > fontified.  If the buffer position was already fontified, the value
>> > should be t.
>>
>> The buffer position was already fontified, so I should not see this.
>I
>> might be doing something wrong so that the font-lock machinery
>thinks,
>> font-locking did not happen. The actual fontification happens
>> asynchronously (due to the server roundtrip), but I thought I had
>given
>> the engine enough info pretending all is done. I don't fully
>understand,
>> how the decision is made to fontify again.
>>
>> Cheers
>> Harald
>>

[-- Attachment #2: Type: text/html, Size: 2541 bytes --]

Re: Any expert on font-lock machinery able to provide some insight --- problem solved

[Harald Kirsch]

I think I found the problem.

If I wrap the face-adding code into with-silent-modifications, things
start to behave. Interpretation: without it the timer function doing the
font-locking is not under control of jit-lock so it sees the buffer
changes as a trigger to re-evaluate font-locking.

The confusing part was that I have with-silent-modifications in my
original code already. But what I have in fact is:

  (combine-change-calls start end
     ...
     (with-silent-modifications

which is (in hindsight obviously) the wrong order :-)

Thanks for your patience.

Harald


On 03.01.25 14:57, Harald Kirsch wrote:
> For better research I made a trivial example font lock function and call
> it, to simulate the async server round-trip, on a timer like so:
>
> ;;; -*- lexical-binding: t; -*-
>
> (defun eglot-semtok-request-fontification (&optional beg end loudly)
>    (message "front %s-%s" beg end)
>    (run-with-timer
>     0.1 nil
>     (lambda () (t-eglot-semtok-request-fontification beg end loudly)) ) )
>
> (defun t-eglot-semtok-request-fontification (&optional beg end loudly)
>    (message "timer %s-%s" beg end)
>    (save-excursion
>      (font-lock-unfontify-region beg end)
>      (goto-char beg)
>      (while-let ((mend (re-search-forward "[a-z]+\\(-[a-z]+\\)+" end t))
>                  (mstart (car (match-data))) )
>        (add-face-text-property mstart mend 'bold) ) ))
>
> (setq font-lock-defaults
>        '(nil nil nil nil
>              (font-lock-fontify-region-function
>               . eglot-semtok-request-fontification )
>              (font-lock-fontify-buffer-function
>               . eglot-semtok-request-fontification ) ) )
>
>
> - put into text file (easier to override font-lock)
> - M-x eval-buffer
> - M-x font-lock-mode
> - M-x font-lock-mode
> - insert a space near the top
> - move cursor
>
> See how for each cursor move the same front and timer messages are
> shown. When configuring the t-... function directly in setq and starting
> over, the extra font-lock calls are not seen.
>
> How would eglot-semtok-request-fontification need to change, short of
> actually fontifying everything, to lure font-locking into thinking all
> is fine, no need to run again shortly after?
>
> Harald.
>
>
> On 03.01.25 14:32, Eli Zaretskii wrote:
>> [Please use Reply All, to keep the list CC'ed.]
>>
>>> Date: Fri, 3 Jan 2025 13:44:04 +0100
>>> From: Harald Kirsch <pifpafpuf@gmx.de>
>>>
>>> Hi Eli,
>>>
>>> thanks for the explanation.
>>>
>>> On 03.01.25 12:58, Eli Zaretskii wrote:
>>> ...
>>>>> But it seems I am missing another channel of information which
>>>>> triggers
>>>>> font-locking too often.
>>>>
>>>> Why does it bother you that it happens too often?
>>>
>>> 1. I compare with elisp font-locking which is much less frequent.
>>>
>>> 2. It is eglot-semtok, which does an LSP server call to get font-lock
>>> information. It is quick enough and I wouldn't have noticed without the
>>> logging, but it seems a waste nevertheless.
>>>
>>>>> With describe-char I do see
>>>>>
>>>>> There are text properties here:
>>>>>      fontified            defer
>>>>>
>>>>> not going away. Can this point to the problem?
>>>>
>>>> This should only happen with buffer positions that were not yet
>>>> fontified.  If the buffer position was already fontified, the value
>>>> should be t.
>>>
>>> The buffer position was already fontified, so I should not see this. I
>>> might be doing something wrong so that the font-lock machinery thinks,
>>> font-locking did not happen. The actual fontification happens
>>> asynchronously (due to the server roundtrip), but I thought I had given
>>> the engine enough info pretending all is done. I don't fully understand,
>>> how the decision is made to fontify again.
>>>
>>> Cheers
>>> Harald
>>>