Partly deferred font-locking?

Partly deferred font-locking?

[Michael Heerdegen]

Hello,

Is it somehow possible to defer (only) the more expensive parts of
font-lock using a timer?  A use case would be the symlink-target
checking fontification rules in dired: the need to check the target file
properties makes scrolling large dired buffers sluggish.

It seems there are only global settings available that also always
influence font-locking completely as a whole (`jit-lock-stealth-time',
`jit-lock-defer-time').

Are there other ways?  Or would it be worth to implement?


TIA,

Michael.

RE: [External] : Partly deferred font-locking?

[Drew Adams]

> Is it somehow possible to defer (only) the more expensive parts of
> font-lock using a timer?  A use case would be the symlink-target
> checking fontification rules in dired: the need to check the target file
> properties makes scrolling large dired buffers sluggish.
> 
> It seems there are only global settings available that also always
> influence font-locking completely as a whole (`jit-lock-stealth-time',
> `jit-lock-defer-time').
> 
> Are there other ways?  Or would it be worth to implement?

There's option `font-lock-maximum-decoration'.
However, it's up to a major mode whether to
provide multiple levels, and how many to provide.

Beyond levels, we could imagine being able to
specify a list of particular kinds of fontifying
per mode.  That's not possible with this option
today.  No doubt there's room for improvement.

But AFAIK, hardly any major modes, even those
from vanilla Emacs itself, provide multiple
levels.  So it's not clear whether improvement
in the possibilities would actually be put to
use by modes, to give users more control.

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Wed, 11 Jan 2023 18:15:19 +0100
> 
> Is it somehow possible to defer (only) the more expensive parts of
> font-lock using a timer?  A use case would be the symlink-target
> checking fontification rules in dired: the need to check the target file
> properties makes scrolling large dired buffers sluggish.
> 
> It seems there are only global settings available that also always
> influence font-locking completely as a whole (`jit-lock-stealth-time',
> `jit-lock-defer-time').

Yes.

> Are there other ways?  Or would it be worth to implement?

I think it's basically impossible to know up front which parts of
fontifications will be expensive and which won't.  For starters, it
depends on the text around the window.

Re: [External] : Partly deferred font-locking?

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Beyond levels, we could imagine being able to
> specify a list of particular kinds of fontifying
> per mode.  That's not possible with this option
> today.  No doubt there's room for improvement.

But those levels only let the user turn off some parts of font-locking,
but not to defer those parts, right?

Michael.

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think it's basically impossible to know up front which parts of
> fontifications will be expensive and which won't.  For starters, it
> depends on the text around the window.

Probably. OTOH, for some parts it's clear that they are always expensive
(like for the nearly context insensitive dired buffer fontification).

Michael.

RE: [External] : Partly deferred font-locking?

[Drew Adams]

> > Beyond levels, we could imagine being able to
> > specify a list of particular kinds of fontifying
> > per mode.  That's not possible with this option
> > today.  No doubt there's room for improvement.
> 
> But those levels only let the user turn off some parts of font-locking,
> but not to defer those parts, right?

Yes.  I was trying to say that what we offer could
be improved. ;-)  And that even what we do offer
(the "levels") doesn't seem to be used (supported
by modes, so usable by users).

RE: [External] : Partly deferred font-locking?

[Drew Adams]

> > > Beyond levels, we could imagine being able to
> > > specify a list of particular kinds of fontifying
> > > per mode.  That's not possible with this option
> > > today.  No doubt there's room for improvement.
> >
> > But those levels only let the user turn off some parts
> > of font-locking, but not to defer those parts, right?
> 
> Yes.  I was trying to say that what we offer could
> be improved. ;-)  And that even what we do offer
> (the "levels") doesn't seem to be used (supported
> by modes, so usable by users).

And your example of fontifying symlinks in Dired
is an example of _what_ gets fontified, and not
how, no?

If that symlink font-lock rule is in a higher level
then that already should provide a way to avoid its
cost.  IOW, IIUC, just choosing not to fontify
symlinks should take care of that particular cost.
(But maybe I've misunderstood you.)

Re: [External] : Partly deferred font-locking?

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> If that symlink font-lock rule is in a higher level
> then that already should provide a way to avoid its
> cost.  IOW, IIUC, just choosing not to fontify
> symlinks should take care of that particular cost.
> (But maybe I've misunderstood you.)

"What" and "how" are coupled: dired now indicates the type of symlink
targets by color (attached to the symlink).  Also broken symlinks are
highlighted specially.  To do that font-lock has to consult the file
system.  The rest of dired fontification is purely text based AFAIK.

But this was only an example to demonstrate the purpose of my question.

Michael.

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Wed, 11 Jan 2023 19:36:05 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think it's basically impossible to know up front which parts of
> > fontifications will be expensive and which won't.  For starters, it
> > depends on the text around the window.
> 
> Probably. OTOH, for some parts it's clear that they are always expensive
> (like for the nearly context insensitive dired buffer fontification).

My point is that your example is the exception rather than the rule.
Fontifications almost never access files or do any other fancy stuff
like that.  At least AFAIK.

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> My point is that your example is the exception rather than the rule.
> Fontifications almost never access files or do any other fancy stuff
> like that.  At least AFAIK.

Yes yes, all good.

But I missed a feature like this for several times.  One application is
syntax-based highlighting of el-search matches, for example.  Or a
hi-lock like thing but not restricted to simple regexps.

I implemented the el-search hi-lock library more or less without
font-lock, but were reinventing parts of it.

Maybe there would be more use cases if Emacs would support this kind of
highlighting, I don't know.


If you wanted to add some more expensive on-the-fly highlighting, how
would you do it?

Currently, one can't use font-lock directly: when the additional deferred
highlighting doesn't happen for a buffer chunk but normal font-locking
was performed, the deferred highlighting would not be triggered again
when that buffer area gets visible again.
So I think I would have to remember which parts of the buffer where
treated by the deferred highlighting by myself, I would have to keep
that information in an additional variable.  That feels a bit odd
since this is one of the things that font-lock naturally does.
I need to reinvent parts of font-lock.


Michael.

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 12 Jan 2023 14:14:15 +0100
> 
> If you wanted to add some more expensive on-the-fly highlighting, how
> would you do it?

AFAIU, it would require changes in jit-lock.el and perhaps also in the
display engine.

But I'm not sure we are on the same page regarding what you'd like to
see, exactly.  In particular, what do you mean by "deferred
highlighting"? when would the "deferred" part be done?

Both jit-lock-defer-time and jit-lock-stealth-time defer highlighting
to when enough idle time has passed since last command.  Is that what
you want -- to be able to complement the highlighting by more
highlighting?  If so, you'd need to change the 'fontified' property to
have more than just a boolean value, and you'd need a special timer
which examined the 'fontified' property and applied more highlighting
if the value says it wasn't applied yet.

> Currently, one can't use font-lock directly: when the additional deferred
> highlighting doesn't happen for a buffer chunk but normal font-locking
> was performed, the deferred highlighting would not be triggered again
> when that buffer area gets visible again.
> So I think I would have to remember which parts of the buffer where
> treated by the deferred highlighting by myself, I would have to keep
> that information in an additional variable.  That feels a bit odd
> since this is one of the things that font-lock naturally does.
> I need to reinvent parts of font-lock.

See above: no need to reinvent, but the property needs to record the
fact of "partial" highlighting, and you need a new timer to apply the
"expensive" highlighting under whatever conditions you decide are
suitable.

Please note that expensive highlighting might make Emacs less
responsive.

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Both jit-lock-defer-time and jit-lock-stealth-time defer highlighting
> to when enough idle time has passed since last command.  Is that what
> you want -- to be able to complement the highlighting by more
> highlighting?

Exactly that.

> If so, you'd need to change the 'fontified' property to
> have more than just a boolean value, and you'd need a special timer
> which examined the 'fontified' property and applied more highlighting
> if the value says it wasn't applied yet.

That would have to be done in Emacs, not in my code, right?

> Please note that expensive highlighting might make Emacs less
> responsive.

So far I'm always using `while-no-input' and code that doesn't cause
trouble when it gets interrupted.  With that approach I had been able to
avoid this issue nearly completely.

Michael.

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 12 Jan 2023 15:00:43 +0100
> 
> > If so, you'd need to change the 'fontified' property to
> > have more than just a boolean value, and you'd need a special timer
> > which examined the 'fontified' property and applied more highlighting
> > if the value says it wasn't applied yet.
> 
> That would have to be done in Emacs, not in my code, right?

Depends on whether the additional highlighting will be part of
jit-lock.el or not.  The display engine only cares whether the value
of 'fontified' is nil or not.

> > Please note that expensive highlighting might make Emacs less
> > responsive.
> 
> So far I'm always using `while-no-input' and code that doesn't cause
> trouble when it gets interrupted.  With that approach I had been able to
> avoid this issue nearly completely.

while-no-input is not a way to interrupt an on-going calculation,
because it requires Emacs to check whether any input arrived, and
Emacs only does that when it's idle.  IOW, if some fontification is
expensive and doesn't itself test quit-flag, you cannot interrupt it
in the middle by wrapping it with while-no-input.  You are
specifically talking about accessing the filesystem, which means you
will call some file I/O C API, and those won't be interrupted.

Re: Partly deferred font-locking?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> So far I'm always using `while-no-input' and code that doesn't cause
>> trouble when it gets interrupted.  With that approach I had been able to
>> avoid this issue nearly completely.
>
> while-no-input is not a way to interrupt an on-going calculation,
> because it requires Emacs to check whether any input arrived, and
> Emacs only does that when it's idle.

Yet, there is observable difference.
AFAIK, helm does use `while-no-input' and I've seen helm being
non-blocking on functions that are not designed with interrupts in mind.
I suspect that you are missing something.

Also, on the subject of deferred font-locking:

More complex fontification may stumble on _editing_ long lines - if
fontification is performed line-by-line, long lines can slow down the
performance significantly (which is a known problem). It is often
a reasonable trade-off to defer re-fontification while editing a long
line and re-fontify it later. I have stumbled upon this issue myself in
my parser-based fontification for Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Partly deferred font-locking?

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> while-no-input is not a way to interrupt an on-going calculation,
>> because it requires Emacs to check whether any input arrived, and
>> Emacs only does that when it's idle.
>
> Yet, there is observable difference.
> AFAIK, helm does use `while-no-input' and I've seen helm being
> non-blocking on functions that are not designed with interrupts in mind.
> I suspect that you are missing something.

I just tried

(while-no-input (let ((x 0)) (while t (cl-incf x) (when (> 3 (random 1000000)) (message "%d" x)))))

and it does not block.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel@gnu.org
> Date: Thu, 12 Jan 2023 14:26:13 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So far I'm always using `while-no-input' and code that doesn't cause
> >> trouble when it gets interrupted.  With that approach I had been able to
> >> avoid this issue nearly completely.
> >
> > while-no-input is not a way to interrupt an on-going calculation,
> > because it requires Emacs to check whether any input arrived, and
> > Emacs only does that when it's idle.
> 
> Yet, there is observable difference.
> AFAIK, helm does use `while-no-input' and I've seen helm being
> non-blocking on functions that are not designed with interrupts in mind.
> I suspect that you are missing something.

Maybe you are missing what I wrote about testing quit-flag?

> More complex fontification may stumble on _editing_ long lines - if
> fontification is performed line-by-line, long lines can slow down the
> performance significantly (which is a known problem). It is often
> a reasonable trade-off to defer re-fontification while editing a long
> line and re-fontify it later. I have stumbled upon this issue myself in
> my parser-based fontification for Org.

It's a trade-off: you assume that users don't want to see the
fontifications change in near real-time when editing long lines, but
that is just an assumption, not necessarily true.

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > That would have to be done in Emacs, not in my code, right?
>
> Depends on whether the additional highlighting will be part of
> jit-lock.el or not.  The display engine only cares whether the value
> of 'fontified' is nil or not.

I can't image a solution that is not part of jit-lock.

> while-no-input is not a way to interrupt an on-going calculation,
> because it requires Emacs to check whether any input arrived, and
> Emacs only does that when it's idle.

Only when idle?  Why is

  (while-no-input (while t (+ 1 1)))

interruptable?  I don't think Emacs is idle while evaluating the loop...?

> You are specifically talking about accessing the filesystem, which
> means you will call some file I/O C API, and those won't be
> interrupted.

Right, but there are dozens of them per window size, and I think we can
interrupt the thing between the individual request.

Michael.

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel@gnu.org
> Date: Thu, 12 Jan 2023 14:30:32 +0000
> 
> I just tried
> 
> (while-no-input (let ((x 0)) (while t (cl-incf x) (when (> 3 (random 1000000)) (message "%d" x)))))
> 
> and it does not block.

Do you understand why?  Specifically: which code in Emacs caused us to
throw on input in this case?  And is that going to happen when Emacs
calls a C function that hits the disk?

Re: Partly deferred font-locking?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> More complex fontification may stumble on _editing_ long lines - if
>> fontification is performed line-by-line, long lines can slow down the
>> performance significantly (which is a known problem). It is often
>> a reasonable trade-off to defer re-fontification while editing a long
>> line and re-fontify it later. I have stumbled upon this issue myself in
>> my parser-based fontification for Org.
>
> It's a trade-off: you assume that users don't want to see the
> fontifications change in near real-time when editing long lines, but
> that is just an assumption, not necessarily true.

You are right. However, I am pretty sure that many users will prefer
having font-lock stop ongoing fontification when they try to enter key
sequence. Or, at least, when they try to enter a key and fontification
is taking more than something like `jit-lock-non-stealth-time' variable
(defaulting to ~0.1 sec).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> It's a trade-off: you assume that users don't want to see the
> fontifications change in near real-time when editing long lines, but
> that is just an assumption, not necessarily true.

Yes, it really depends on the use case.

My cases so far were all of the category: "expensive" fontification
still happens nearly instantly, hardly noticeable for the human eye, but
slow enough to make scrolling feeling a bit sluggish.

Michael.

Re: Partly deferred font-locking?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I just tried
>> 
>> (while-no-input (let ((x 0)) (while t (cl-incf x) (when (> 3 (random 1000000)) (message "%d" x)))))
>> 
>> and it does not block.
>
> Do you understand why?  Specifically: which code in Emacs caused us to
> throw on input in this case?  And is that going to happen when Emacs
> calls a C function that hits the disk?

AFAIU, maybe_quit throws the necessary signal as needed.

For external C functions that interact with disk, they will indeed not
produce the signal. However, the signal will likely be thrown soon after
disk interaction ends. In my experience, helm command that opens a
number of files is successfully interrupted after populating a file
buffer but before initializing the major mode.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: michael_heerdegen@web.de, emacs-devel@gnu.org
> Date: Thu, 12 Jan 2023 14:44:11 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I just tried
> >> 
> >> (while-no-input (let ((x 0)) (while t (cl-incf x) (when (> 3 (random 1000000)) (message "%d" x)))))
> >> 
> >> and it does not block.
> >
> > Do you understand why?  Specifically: which code in Emacs caused us to
> > throw on input in this case?  And is that going to happen when Emacs
> > calls a C function that hits the disk?
> 
> AFAIU, maybe_quit throws the necessary signal as needed.
> 
> For external C functions that interact with disk, they will indeed not
> produce the signal. However, the signal will likely be thrown soon after
> disk interaction ends. In my experience, helm command that opens a
> number of files is successfully interrupted after populating a file
> buffer but before initializing the major mode.

My point is that when you call an expensive calculation inside
while-no-input, you are entirely at the mercy of the code run by the
body of while-no-input, wrt how soon after a keypress will
while-no-input be interrupted.  And since Michael was asking precisely
about such situations, I commented that it _might_ make Emacs less
responsive.

Re: Partly deferred font-locking?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> My point is that when you call an expensive calculation inside
> while-no-input, you are entirely at the mercy of the code run by the
> body of while-no-input, wrt how soon after a keypress will
> while-no-input be interrupted.  And since Michael was asking precisely
> about such situations, I commented that it _might_ make Emacs less
> responsive.

I'd say that the chances are slim. Primitives like `funcall_lambda',
`if', and `when' are all calling `maybe_quit'. So, pretty much any Lisp
code will trigger interruption.

The exceptions are subroutines and external library calls, but most of
the subroutines are also sparkled with maybe_quit.

I think that it is a reasonable assumption that Emacs will not block
inside `while-no-input' in the overwhelming majority of real-world
scenarios. Exceptions are possible, but should not be a blocker - few
exceptions are much better with the current stable inflow of font-lock
performance complaints that are coming from inefficient or buggy
font-lock code in third-party packages.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Partly deferred font-locking?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 12 Jan 2023 15:33:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > That would have to be done in Emacs, not in my code, right?
> >
> > Depends on whether the additional highlighting will be part of
> > jit-lock.el or not.  The display engine only cares whether the value
> > of 'fontified' is nil or not.
> 
> I can't image a solution that is not part of jit-lock.
> 
> > while-no-input is not a way to interrupt an on-going calculation,
> > because it requires Emacs to check whether any input arrived, and
> > Emacs only does that when it's idle.
> 
> Only when idle?  Why is
> 
>   (while-no-input (while t (+ 1 1)))
> 
> interruptable?

Yes, the Lisp interpreter also tests for input in many places.  But if
you call a C library function, that won't happen until it returns.

Re: Partly deferred font-locking?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> [...]

Thanks to you two for elaborating.

> Yes, the Lisp interpreter also tests for input in many places.  But if
> you call a C library function, that won't happen until it returns.

I did not expect that (or have already learned it).  It is not such a
big problem for what we discuss IME.  Unless maybe in cases where a disk needs
to spin up or so - but that blocks other things as font-lock (command
loop) as well.

Michael.