region-based face-remapping

region-based face-remapping

[JD Smith]

Has there been any discussion of implementing region-based face remapping, perhaps by adding 'face-remap as a property to overlays?  This would be very useful for modes in which local syntax highlighting depends on point, now broadly possible thanks to treesitter (think “brighten surrounding semantic unit” or similar).

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Mon, 1 Jan 2024 19:22:29 -0500
> 
> 
> Has there been any discussion of implementing region-based face remapping, perhaps by adding 'face-remap as a property to overlays?  This would be very useful for modes in which local syntax highlighting depends on point, now broadly possible thanks to treesitter (think “brighten surrounding semantic unit” or similar).

If you only want to change the appearance around point, you need a
much smaller revolution than position-dependent faces, which are
something much more general.  E.g., position-dependent faces need to
change all the face-related APIs to pass a position and a buffer
arguments, whereas if it's only "around point", you need nothing like
that because the position of point is always known in each buffer.

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]



> On Jan 2, 2024, at 8:00 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Mon, 1 Jan 2024 19:22:29 -0500
>> 
>> 
>> Has there been any discussion of implementing region-based face remapping, perhaps by adding 'face-remap as a property to overlays?  This would be very useful for modes in which local syntax highlighting depends on point, now broadly possible thanks to treesitter (think “brighten surrounding semantic unit” or similar).
> 
> If you only want to change the appearance around point, you need a
> much smaller revolution than position-dependent faces, which are
> something much more general.  E.g., position-dependent faces need to
> change all the face-related APIs to pass a position and a buffer
> arguments, whereas if it's only "around point", you need nothing like
> that because the position of point is always known in each buffer.


To make the entire face API position-dependent does seem complex. I presume face calculation first checks if there is a valid 'face-remapping-alist’ and does substitutions.  If 'face-remap were a valid (overlay/text) property, that would seem to obviate any deeper API changes: just add this property where remapping is desired.  This may relate to the text-property “planes” discussion from a couple years back.  Based on that discussion, perhaps targeting only ‘face is not enough. Is there another approach to “selectively enabling/substituting properties by region”, beyond just text-property-search-forward within the region (looking within ‘display properties too)?

To make this concrete, how would people approach the following:

On point motion, treesitter’s syntax tree is consulted to find an "enclosing semantic context region".
Within that region, pre-existing faces (or perhaps other properties) get updated to a “highlighted” variant.
Outdated/prior regions have the highlighted variants removed.

Since you don’t know in advance how large such regions might be, it seems problematic to search and replace all the various faces or other properties.   


[-- Attachment #2: Type: text/html, Size: 2786 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Tue, 2 Jan 2024 10:49:49 -0500
> Cc: emacs-devel@gnu.org
> 
>  On Jan 2, 2024, at 8:00 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  From: JD Smith <jdtsmith@gmail.com>
>  Date: Mon, 1 Jan 2024 19:22:29 -0500
> 
>  Has there been any discussion of implementing region-based face remapping, perhaps
>  by adding 'face-remap as a property to overlays?  This would be very useful for modes in
>  which local syntax highlighting depends on point, now broadly possible thanks to treesitter
>  (think “brighten surrounding semantic unit” or similar).
> 
>  If you only want to change the appearance around point, you need a
>  much smaller revolution than position-dependent faces, which are
>  something much more general.  E.g., position-dependent faces need to
>  change all the face-related APIs to pass a position and a buffer
>  arguments, whereas if it's only "around point", you need nothing like
>  that because the position of point is always known in each buffer.
> 
> To make the entire face API position-dependent does seem complex. I presume face calculation
> first checks if there is a valid 'face-remapping-alist’ and does substitutions.

Not exactly. You need to keep in mind that each buffer or string
position could have several sources of face information, see
"Displaying Faces" in the ELisp manual for the gory details.So first
Emacs needs to collect all those faces from all the relevant sources,
and then it merges them. Face remapping happens during the merge: each
face that is referenced by a symbol (a.k.a. "named face") is remapped
if needed.

> If 'face-remap were a
> valid (overlay/text) property, that would seem to obviate any deeper API changes: just add this
> property where remapping is desired.

I don't think I follow.

The workhorse we use now for obtaining face information is the
function face_at_buffer_position, which calls the various face-merging
routines.  Those merging routines and their subroutines consider
face-remapping-alist as part of face merging process outlined above.
I don't see how we can avoid passing the buffer position to them if
face-remapping depends on buffer position. Also, those routines will
now need to perform property or overlay search for the special
face-remapping property, using the position they are passed.

Am I missing something?

> This may relate to the text-property “planes” discussion from a
> couple years back.  Based on that discussion, perhaps targeting only ‘face is not enough. Is there
> another approach to “selectively enabling/substituting properties by region”, beyond just
> text-property-search-forward within the region (looking within ‘display properties too)?

I don't think I understand which discussion you have in mind and how
it is relevant to the issue at hand.

> To make this concrete, how would people approach the following:
> 
> 1 On point motion, treesitter’s syntax tree is consulted to find an "enclosing semantic context
>  region".
> 2 Within that region, pre-existing faces (or perhaps other properties) get updated to a “highlighted”
>  variant.
> 3 Outdated/prior regions have the highlighted variants removed.
> 
> Since you don’t know in advance how large such regions might be, it seems problematic to search
> and replace all the various faces or other properties.   

How do you envision to do items 2 and 3 above?  IOW, how do you
"update pre-existing faces"?  The naïve solution, of somehow changing
the face attributes on the fly, will be a performance killer, since
currently any change in any named face invalidates all the faces we
already know about and requires their "realization" anew -- because we
don't track the dependencies between faces, and thus don't know which
other faces need to be updated as result.  So this would mean that any
point movement could potentially cause recalculation of all the faces
on the selected frame.  And face realization is a relatively expensive
process, especially if there are a lot of them (we had reports about
users who routinely have hundreds or even thousands of faces on a
frame). Thus my question about items 2 and 3, which seem to do exactly
what we avoid doing: modify named faces.

Re: region-based face-remapping

[Dmitry Gutov]

On 03/01/2024 14:31, Eli Zaretskii wrote:
> The workhorse we use now for obtaining face information is the
> function face_at_buffer_position, which calls the various face-merging
> routines.  Those merging routines and their subroutines consider
> face-remapping-alist as part of face merging process outlined above.
> I don't see how we can avoid passing the buffer position to them if
> face-remapping depends on buffer position.

Perhaps face_at_buffer_position would first look up the overlay/text 
property 'face-remapping-alist, then merge its value with the variable 
face-remapping-alist (using plain 'append', for example), and pass on 
the resulting value to the face-merging routines?

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Wed, 3 Jan 2024 14:40:49 +0200
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 03/01/2024 14:31, Eli Zaretskii wrote:
> > The workhorse we use now for obtaining face information is the
> > function face_at_buffer_position, which calls the various face-merging
> > routines.  Those merging routines and their subroutines consider
> > face-remapping-alist as part of face merging process outlined above.
> > I don't see how we can avoid passing the buffer position to them if
> > face-remapping depends on buffer position.
> 
> Perhaps face_at_buffer_position would first look up the overlay/text 
> property 'face-remapping-alist, then merge its value with the variable 
> face-remapping-alist (using plain 'append', for example), and pass on 
> the resulting value to the face-merging routines?

Then any code running while face_at_buffer_position does its job will
see a tweaked value of face-remapping-alist, which is not necessarily
TRT, since it is not guaranteed that the buffer position being
examined by face_at_buffer_position is relevant to the rest of the
code.

Re: region-based face-remapping

[Dmitry Gutov]

On 03/01/2024 15:42, Eli Zaretskii wrote:
>> Date: Wed, 3 Jan 2024 14:40:49 +0200
>> Cc:emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 03/01/2024 14:31, Eli Zaretskii wrote:
>>> The workhorse we use now for obtaining face information is the
>>> function face_at_buffer_position, which calls the various face-merging
>>> routines.  Those merging routines and their subroutines consider
>>> face-remapping-alist as part of face merging process outlined above.
>>> I don't see how we can avoid passing the buffer position to them if
>>> face-remapping depends on buffer position.
>> Perhaps face_at_buffer_position would first look up the overlay/text
>> property 'face-remapping-alist, then merge its value with the variable
>> face-remapping-alist (using plain 'append', for example), and pass on
>> the resulting value to the face-merging routines?
> Then any code running while face_at_buffer_position does its job will
> see a tweaked value of face-remapping-alist, which is not necessarily
> TRT, since it is not guaranteed that the buffer position being
> examined by face_at_buffer_position is relevant to the rest of the
> code.

I'm not sure I understand your response.

IIUC the request is that one would be able to change the faces' 
appearances at any position in the buffer using a mechanism like 
additional text properties or overlay properties.

It seems to reason that any code computing the 'face' for a buffer 
position would need to account for the new property if it's set at that 
position.

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Thu, 4 Jan 2024 02:07:36 +0200
> Cc: jdtsmith@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 03/01/2024 15:42, Eli Zaretskii wrote:
> >> Date: Wed, 3 Jan 2024 14:40:49 +0200
> >> Cc:emacs-devel@gnu.org
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 03/01/2024 14:31, Eli Zaretskii wrote:
> >>> The workhorse we use now for obtaining face information is the
> >>> function face_at_buffer_position, which calls the various face-merging
> >>> routines.  Those merging routines and their subroutines consider
> >>> face-remapping-alist as part of face merging process outlined above.
> >>> I don't see how we can avoid passing the buffer position to them if
> >>> face-remapping depends on buffer position.
> >> Perhaps face_at_buffer_position would first look up the overlay/text
> >> property 'face-remapping-alist, then merge its value with the variable
> >> face-remapping-alist (using plain 'append', for example), and pass on
> >> the resulting value to the face-merging routines?
> > Then any code running while face_at_buffer_position does its job will
> > see a tweaked value of face-remapping-alist, which is not necessarily
> > TRT, since it is not guaranteed that the buffer position being
> > examined by face_at_buffer_position is relevant to the rest of the
> > code.
> 
> I'm not sure I understand your response.
> 
> IIUC the request is that one would be able to change the faces' 
> appearances at any position in the buffer using a mechanism like 
> additional text properties or overlay properties.
> 
> It seems to reason that any code computing the 'face' for a buffer 
> position would need to account for the new property if it's set at that 
> position.

AFAIU, you suggested to temporarily tweak the value of
face-remapping-alist when face_at_buffer_position is called for a
position where the new proposed property defines a "local" remapping.
Is that right?  If so, imagine the following scenario:

  . face_at_buffer_position is called and tweaks face-remapping-alist
  . one of the subroutines of face_at_buffer_position calls some Lisp
    hook
  . that Lisp hook calls code that calls face-font (or some other
    primitive which takes face-remapping-alist into account)

In this scenario, primitives called in this way will see the tweaked
value of face-remapping-alist, which is specific to the buffer
position passed to face_at_buffer_position, whereas the code which
calls those primitives doesn't expect to be affected by such
position-local face remapping (for example, because it is interested
in some face in some other position).

Did I clarify my concern?

Re: region-based face-remapping

[Dmitry Gutov]

On 04/01/2024 09:05, Eli Zaretskii wrote:
>    . one of the subroutines of face_at_buffer_position calls some Lisp
>      hook
>    . that Lisp hook calls code that calls face-font (or some other
>      primitive which takes face-remapping-alist into account)

Could you give an example of a Lisp hook which might be called from 
face_at_buffer_position's subroutines?

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Fri, 5 Jan 2024 05:49:24 +0200
> Cc: jdtsmith@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 04/01/2024 09:05, Eli Zaretskii wrote:
> >    . one of the subroutines of face_at_buffer_position calls some Lisp
> >      hook
> >    . that Lisp hook calls code that calls face-font (or some other
> >      primitive which takes face-remapping-alist into account)
> 
> Could you give an example of a Lisp hook which might be called from 
> face_at_buffer_position's subroutines?

Why is having a specific example important?

Are you saying that there can never be such an example?  With the
current tendency of moving stuff to Lisp and adding hooks to C code,
we clearly cannot convince ourselves such a hook will never happen in
the future, even if it doesn't exist in this particular moment in
time.  Assumptions that certain things "cannot happen" between points
A and B in display code already led to tricky bugs that needed tricky
solutions.  Thus, I don't want to design a new feature on such shaky
assumptions.

Re: region-based face-remapping

[Dmitry Gutov]

On 05/01/2024 10:50, Eli Zaretskii wrote:
>> Date: Fri, 5 Jan 2024 05:49:24 +0200
>> Cc:jdtsmith@gmail.com,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 04/01/2024 09:05, Eli Zaretskii wrote:
>>>     . one of the subroutines of face_at_buffer_position calls some Lisp
>>>       hook
>>>     . that Lisp hook calls code that calls face-font (or some other
>>>       primitive which takes face-remapping-alist into account)
>> Could you give an example of a Lisp hook which might be called from
>> face_at_buffer_position's subroutines?
> Why is having a specific example important?
> 
> Are you saying that there can never be such an example?

Yes, it would seem odd to me for face_at_buffer_position to call any hooks.

But if it did, I would consider whether any of the hooks being called 
would allow substituting the face with a different one (making it a 
different way to solve the present feature request). This sounds like a 
way to slow things down by an order of a magnitude, though, so it 
probably will not happen.

> With the
> current tendency of moving stuff to Lisp and adding hooks to C code,
> we clearly cannot convince ourselves such a hook will never happen in
> the future, even if it doesn't exist in this particular moment in
> time.

Even if it moved to Lisp, it won't necessarily add any hooks.

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Fri, 5 Jan 2024 16:18:28 +0200
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jdtsmith@gmail.com, emacs-devel@gnu.org
> 
> On 05/01/2024 10:50, Eli Zaretskii wrote:
> >> Date: Fri, 5 Jan 2024 05:49:24 +0200
> >> Cc:jdtsmith@gmail.com,emacs-devel@gnu.org
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 04/01/2024 09:05, Eli Zaretskii wrote:
> >>>     . one of the subroutines of face_at_buffer_position calls some Lisp
> >>>       hook
> >>>     . that Lisp hook calls code that calls face-font (or some other
> >>>       primitive which takes face-remapping-alist into account)
> >> Could you give an example of a Lisp hook which might be called from
> >> face_at_buffer_position's subroutines?
> > Why is having a specific example important?
> > 
> > Are you saying that there can never be such an example?
> 
> Yes, it would seem odd to me for face_at_buffer_position to call any hooks.

More strange hooks have been added.  For example, a face attribute
could have a function value, in which case face_at_buffer_position
will call into Lisp.

> But if it did, I would consider whether any of the hooks being called 
> would allow substituting the face with a different one (making it a 
> different way to solve the present feature request).

Any face merging looks at face-remapping-alist, so you cannot possibly
negate this.

> > With the
> > current tendency of moving stuff to Lisp and adding hooks to C code,
> > we clearly cannot convince ourselves such a hook will never happen in
> > the future, even if it doesn't exist in this particular moment in
> > time.
> 
> Even if it moved to Lisp, it won't necessarily add any hooks.

Once it's in Lisp, the probability of calling face-related functions
is much higher.

Re: region-based face-remapping

[Dmitry Gutov]

On 05/01/2024 16:34, Eli Zaretskii wrote:
>> Date: Fri, 5 Jan 2024 16:18:28 +0200
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> Cc: jdtsmith@gmail.com, emacs-devel@gnu.org
>>
>> On 05/01/2024 10:50, Eli Zaretskii wrote:
>>>> Date: Fri, 5 Jan 2024 05:49:24 +0200
>>>> Cc:jdtsmith@gmail.com,emacs-devel@gnu.org
>>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>>
>>>> On 04/01/2024 09:05, Eli Zaretskii wrote:
>>>>>      . one of the subroutines of face_at_buffer_position calls some Lisp
>>>>>        hook
>>>>>      . that Lisp hook calls code that calls face-font (or some other
>>>>>        primitive which takes face-remapping-alist into account)
>>>> Could you give an example of a Lisp hook which might be called from
>>>> face_at_buffer_position's subroutines?
>>> Why is having a specific example important?
>>>
>>> Are you saying that there can never be such an example?
>>
>> Yes, it would seem odd to me for face_at_buffer_position to call any hooks.
> 
> More strange hooks have been added.  For example, a face attribute
> could have a function value, in which case face_at_buffer_position
> will call into Lisp.

If we added such a feature someday, indeed realize_named_face and others 
would need to be changed, e.g. by passing the amended value of 
face-remapping-alist lexically to get_lface_attributes and all its 
callers (not necessarily the buffer position, though).

But until that happens, it might not be necessary.

>> But if it did, I would consider whether any of the hooks being called
>> would allow substituting the face with a different one (making it a
>> different way to solve the present feature request).
> 
> Any face merging looks at face-remapping-alist, so you cannot possibly
> negate this.

What I was saying is, if an attribute could be a Lisp function, it could 
be another way to implement what JD is asking for, one that doesn't 
introduce the 'face-remapping-alist' text/overlay property. But it would 
be a more complex, less declarative way of doing that. And again, I'm 
not sure how the overall performance would look with such an addition.

Anyway, the feature seems doable to me, and probably compatible with any 
addition similar to your example, given enough maintenance effort. I'm 
not saying we should necessarily have it, though.

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 6056 bytes --]



> On Jan 3, 2024, at 7:31 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> 
>> To make the entire face API position-dependent does seem complex. I presume face calculation
>> first checks if there is a valid 'face-remapping-alist’ and does substitutions.
> 
> Not exactly. You need to keep in mind that each buffer or string
> position could have several sources of face information, see
> "Displaying Faces" in the ELisp manual for the gory details.So first
> Emacs needs to collect all those faces from all the relevant sources,
> and then it merges them. Face remapping happens during the merge: each
> face that is referenced by a symbol (a.k.a. "named face") is remapped
> if needed.

I see.  But if there were a property at the position where Emacs is collecting and merging all the face information, say

	 ‘(face-remap ((some-face . another-face)))

would the merge process be able to easily use this style of remap-information, in addition to the buffer-local 'face-remapping-alist (with the local property taking precedence over the variable)?

>> If 'face-remap were a
>> valid (overlay/text) property, that would seem to obviate any deeper API changes: just add this
>> property where remapping is desired.
> 
> I don't think I follow.
> 
> The workhorse we use now for obtaining face information is the
> function face_at_buffer_position, which calls the various face-merging
> routines.  Those merging routines and their subroutines consider
> face-remapping-alist as part of face merging process outlined above.
> I don't see how we can avoid passing the buffer position to them if
> face-remapping depends on buffer position. Also, those routines will
> now need to perform property or overlay search for the special
> face-remapping property, using the position they are passed.
> 
> Am I missing something?

Perhaps we are talking about different things.  I was speaking from the point of the view of the ELISP user of the faces API.  From the ELISP face API, I don’t see a need to include any new position information for face definition and face property assignment.  I think now that you were speaking about the internal API.  Internally during the face merge process, I agree there would need to know the properties at a merging-the-face-info point under consideration, which could influence the merge.  

Maybe the issue is that face_at_buffer_position is called only at certain special positions, not on each position in the buffer?  If so, I can see how your concern about property/overlay search would apply (though what those positions would be I don’t know, since I can assign a different 'face property to arbitrary ranges as small as one character).

>> This may relate to the text-property “planes” discussion from a
>> couple years back.  Based on that discussion, perhaps targeting only ‘face is not enough. Is there
>> another approach to “selectively enabling/substituting properties by region”, beyond just
>> text-property-search-forward within the region (looking within ‘display properties too)?
> 
> I don't think I understand which discussion you have in mind and how
> it is relevant to the issue at hand.

I meant this discussion (and the related earlier one mentioned, by Stefan M):

https://lists.nongnu.org/archive/html/emacs-devel/2019-11/msg00599.html

My read of that discussion is it would allow external control of various “planes" of a text property: turn some on, others off, without searching through the entirety of the buffer finding and replacing those properties.  Similar to layers in a photo editor.  Whether that type of external control could be limited to a region I am unsure (though the MMM discussion makes me think that was envisioned).  The “layers” or “planes” concept certainly seems related.

>> To make this concrete, how would people approach the following:
>> 
>> 1 On point motion, treesitter’s syntax tree is consulted to find an "enclosing semantic context
>> region".
>> 2 Within that region, pre-existing faces (or perhaps other properties) get updated to a “highlighted”
>> variant.
>> 3 Outdated/prior regions have the highlighted variants removed.
>> 
>> Since you don’t know in advance how large such regions might be, it seems problematic to search
>> and replace all the various faces or other properties.   
> 
> How do you envision to do items 2 and 3 above?  IOW, how do you
> "update pre-existing faces"?  

In an ideal scenario, by applying a text or overlay property '(face-remap ((some-face . some-highlight-face))) to that region.

> The naïve solution, of somehow changing
> the face attributes on the fly, will be a performance killer, since
> currently any change in any named face invalidates all the faces we
> already know about and requires their "realization" anew -- because we
> don't track the dependencies between faces, and thus don't know which
> other faces need to be updated as result.  So this would mean that any
> point movement could potentially cause recalculation of all the faces
> on the selected frame.  

I do this very thing already in indent-bars[1], globally in the buffer, using face-remapping-alist.  The performance of that remapping is incredibly good in my experience.  So fast in fact that I’ve had to introduce a timer to artificially slow it down since it leads to rapid face flashing during smooth scrolling.

> And face realization is a relatively expensive
> process, especially if there are a lot of them (we had reports about
> users who routinely have hundreds or even thousands of faces on a
> frame). Thus my question about items 2 and 3, which seem to do exactly
> what we avoid doing: modify named faces.


Does “substitute a named face for another named or anonymous face” lead to similar concerns?  Or can those then all be precomputed without the constant face realization concern?

[1] https://github.com/jdtsmith/indent-bars


[-- Attachment #2.1: Type: text/html, Size: 35856 bytes --]

[-- Attachment #2.2: favicon.ico --]
[-- Type: image/vnd.microsoft.icon, Size: 1406 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Wed, 3 Jan 2024 18:15:04 -0500
> Cc: emacs-devel@gnu.org
> 
> > Not exactly. You need to keep in mind that each buffer or string
> > position could have several sources of face information, see
> > "Displaying Faces" in the ELisp manual for the gory details.So first
> > Emacs needs to collect all those faces from all the relevant sources,
> > and then it merges them. Face remapping happens during the merge: each
> > face that is referenced by a symbol (a.k.a. "named face") is remapped
> > if needed.
> 
> I see.  But if there were a property at the position where Emacs is collecting and merging all the face information, say
> 
> 	 ‘(face-remap ((some-face . another-face)))
> 
> would the merge process be able to easily use this style of remap-information, in addition to the buffer-local 'face-remapping-alist (with the local property taking precedence over the variable)?

Yes, but the buffer position will need to be known at the levels where
this remapping is handled, which is quite low-level code.

> > The workhorse we use now for obtaining face information is the
> > function face_at_buffer_position, which calls the various face-merging
> > routines.  Those merging routines and their subroutines consider
> > face-remapping-alist as part of face merging process outlined above.
> > I don't see how we can avoid passing the buffer position to them if
> > face-remapping depends on buffer position. Also, those routines will
> > now need to perform property or overlay search for the special
> > face-remapping property, using the position they are passed.
> > 
> > Am I missing something?
> 
> Perhaps we are talking about different things.  I was speaking from the point of the view of the ELISP user of the faces API.  From the ELISP face API, I don’t see a need to include any new position information for face definition and face property assignment.  I think now that you were speaking about the internal API.  Internally during the face merge process, I agree there would need to know the properties at a merging-the-face-info point under consideration, which could influence the merge.  

Of course, I was talking about the implementation and internal APIs!
You asked how about this idea, so I provided feedback about what it
would mean to implement such an idea.

> Maybe the issue is that face_at_buffer_position is called only at certain special positions, not on each position in the buffer?

It is called where the 'face' property changes.  The display engine
finds the next position where that happens, and arranges for calling
face_at_buffer_position when it gets to process that position.  The
results of face merging at that position are kept in a cache, to avoid
performing it more often than needed.

> If so, I can see how your concern about property/overlay search would apply (though what those positions would be I don’t know, since I can assign a different 'face property to arbitrary ranges as small as one character).

Try assigning different 'face' property to each character, and you
will see that redisplay will be slower due to the need to stop and
look up faces more often.

> >> This may relate to the text-property “planes” discussion from a
> >> couple years back.  Based on that discussion, perhaps targeting only ‘face is not enough. Is there
> >> another approach to “selectively enabling/substituting properties by region”, beyond just
> >> text-property-search-forward within the region (looking within ‘display properties too)?
> > 
> > I don't think I understand which discussion you have in mind and how
> > it is relevant to the issue at hand.
> 
> I meant this discussion (and the related earlier one mentioned, by Stefan M):
> 
> https://lists.nongnu.org/archive/html/emacs-devel/2019-11/msg00599.html

That's not different from face-remapping, and it is not
position-dependent, AFAIU.  So I don't think I see how it is relevant
to what you want to do.

> >> 1 On point motion, treesitter’s syntax tree is consulted to find an "enclosing semantic context
> >> region".
> >> 2 Within that region, pre-existing faces (or perhaps other properties) get updated to a “highlighted”
> >> variant.
> >> 3 Outdated/prior regions have the highlighted variants removed.
> >> 
> >> Since you don’t know in advance how large such regions might be, it seems problematic to search
> >> and replace all the various faces or other properties.   
> > 
> > How do you envision to do items 2 and 3 above?  IOW, how do you
> > "update pre-existing faces"?  
> 
> In an ideal scenario, by applying a text or overlay property '(face-remap ((some-face . some-highlight-face))) to that region.

Then we are back to the position information needed in low-level code
that merges faces.  Which is probably the simplest and most robust
solution; the only disadvantage is that we will need to "drag" the
buffer position through many existing APIs to a low level.  We'd
probably also need to extend face-font and the likes (more so if you
want the 'default' face to be mutable like this) with an optional
position argument.

> > The naïve solution, of somehow changing
> > the face attributes on the fly, will be a performance killer, since
> > currently any change in any named face invalidates all the faces we
> > already know about and requires their "realization" anew -- because we
> > don't track the dependencies between faces, and thus don't know which
> > other faces need to be updated as result.  So this would mean that any
> > point movement could potentially cause recalculation of all the faces
> > on the selected frame.  
> 
> I do this very thing already in indent-bars[1], globally in the buffer, using face-remapping-alist.  The performance of that remapping is incredibly good in my experience.  So fast in fact that I’ve had to introduce a timer to artificially slow it down since it leads to rapid face flashing during smooth scrolling.

How many faces do you have defined on that frame?  I'm guessing not
many, or maybe they are all almost identical (differ in only one or
two simple attributes).

> > And face realization is a relatively expensive
> > process, especially if there are a lot of them (we had reports about
> > users who routinely have hundreds or even thousands of faces on a
> > frame). Thus my question about items 2 and 3, which seem to do exactly
> > what we avoid doing: modify named faces.
> 
> Does “substitute a named face for another named or anonymous face” lead to similar concerns?  Or can those then all be precomputed without the constant face realization concern?

As long as the faces themselves don't change, no.  But that means we'd
need to store all those faces for all the regions that use such a
position-dependent remapping.

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]



> On Jan 4, 2024, at 1:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> It is called where the 'face' property changes.  The display engine
> finds the next position where that happens, and arranges for calling
> face_at_buffer_position when it gets to process that position.  The
> results of face merging at that position are kept in a cache, to avoid
> performing it more often than needed.

This makes sense, thanks.  In the ‘face-remap property scenario, the number of such ‘face property-change positions doesn’t actually change.  But at each such position, the face-merging code would have to check for an operative ‘face-remap property at point and use it together with `face-remapping-alist' to compute the final face merge. I would expect that performance would depend on whether the face-merge cache could be (largely) preserved, even as 'face-remap properties are added/removed/moved around (typically in repetitive fashion).

>> I do this very thing already in indent-bars[1], globally in the buffer, using face-remapping-alist.  The performance of that remapping is incredibly good in my experience.  So fast in fact that I’ve had to introduce a timer to artificially slow it down since it leads to rapid face flashing during smooth scrolling.
> 
> How many faces do you have defined on that frame?  I'm guessing not
> many, or maybe they are all almost identical (differ in only one or
> two simple attributes).

Right now it’s just a couple of faces (remove one face-remap, add another).  If I had access to a ‘face-remap property, I could imagine maybe 10 faces at most being affected as point changes.  Each face would change by one or two attributes like foreground, stipple, etc.  

I’m sure there could be lots of uses of this type of functionality, but “enhance the focus on one region of importance” seems like a very common one.  For example, you might imagine dim/gray version of all the font-lock faces outside the “treesitter region of interest” and bright ones inside.  Whenever that TS-prescribed region changes, an overlay with a ‘face-remap property gets moved.  

[-- Attachment #2: Type: text/html, Size: 5733 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Thu, 4 Jan 2024 19:51:46 -0500
> Cc: emacs-devel@gnu.org
> 
>  On Jan 4, 2024, at 1:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  It is called where the 'face' property changes.  The display engine
>  finds the next position where that happens, and arranges for calling
>  face_at_buffer_position when it gets to process that position.  The
>  results of face merging at that position are kept in a cache, to avoid
>  performing it more often than needed.
> 
> This makes sense, thanks.  In the ‘face-remap property scenario, the number of such ‘face
> property-change positions doesn’t actually change.

I don't think this is true. If the 'face-remap' property changes at
some buffer position, the realized faces need to change there,
although the 'face' property didn't change. So with the 'face-remap'
property, we'd need to arrange for face-merging also where the
'face-remap' property changes, not only where the 'face' property
changes.

> But at each such position, the face-merging
> code would have to check for an operative ‘face-remap property at
> point

(Please don't use "point" in this discussion to mean buffer position,
since the display engine doesn't move point while examining buffer
text and its properties.)

> and use it together with
> `face-remapping-alist' to compute the final face merge. I would expect that performance would
> depend on whether the face-merge cache could be (largely) preserved, even as 'face-remap
> properties are added/removed/moved around (typically in repetitive fashion).

The result of merging faces when 'face-remap' is in effect will be
different faces, even though the named faces didn't change.  Those
different faces will be realized and cached, and will have their own
internal face IDs, but will not be exposed to Lisp as symbols, via
APIs such as face-at-point.

>  I do this very thing already in indent-bars[1], globally in the buffer, using face-remapping-alist. 
>  The performance of that remapping is incredibly good in my experience.  So fast in fact that I’ve
>  had to introduce a timer to artificially slow it down since it leads to rapid face flashing during
>  smooth scrolling.
> 
>  How many faces do you have defined on that frame?  I'm guessing not
>  many, or maybe they are all almost identical (differ in only one or
>  two simple attributes).
> 
> Right now it’s just a couple of faces (remove one face-remap, add another).  If I had access to a
> ‘face-remap property, I could imagine maybe 10 faces at most being affected as point changes. 
> Each face would change by one or two attributes like foreground, stipple, etc.  

What matters is the total number of named faces defined for the frame,
not the number of faces affected by face-remap.

> I’m sure there could be lots of uses of this type of functionality, but “enhance the focus on one region
> of importance” seems like a very common one.  For example, you might imagine dim/gray version of
> all the font-lock faces outside the “treesitter region of interest” and bright ones inside.  Whenever that
> TS-prescribed region changes, an overlay with a ‘face-remap property gets moved.  

The question I suggest to ponder is whether simpler, less general ways
to implement these features are "good enough".  For example, changing
the appearance of a region of text can be handled by moving an overlay
there with a suitable face definition and high priority; changing the
appearance of text around point can be handled by a special variable
which the display engine will consider when working on text around
point; etc.  IOW, it is not always beneficial to generalize a feature
to the most general abstraction level, certainly not when talking
about Emacs, where a gazillion features related to the proposed new
one are already available and expected to work after the proposed
change.  Adding significant enhancements to Emacs that change previous
design assumptions can easily become a proverbial death by thousand
cuts.

Re: region-based face-remapping

[Dmitry Gutov]

On 05/01/2024 10:19, Eli Zaretskii wrote:
>> I’m sure there could be lots of uses of this type of functionality, but “enhance the focus on one region
>> of importance” seems like a very common one.  For example, you might imagine dim/gray version of
>> all the font-lock faces outside the “treesitter region of interest” and bright ones inside.  Whenever that
>> TS-prescribed region changes, an overlay with a ‘face-remap property gets moved.
> The question I suggest to ponder is whether simpler, less general ways
> to implement these features are "good enough".  For example, changing
> the appearance of a region of text can be handled by moving an overlay
> there with a suitable face definition and high priority; changing the
> appearance of text around point can be handled by a special variable
> which the display engine will consider when working on text around
> point;

An overlay with face and higher priority would override most faces in 
the region, wouldn't it? Unless it just sets the background, in which 
case we already have this capability.

JD, have you considered it? E.g. set the default face's background to 
some darker color and use an overlay which sets background to a lighter one.

But if that's not sufficient, we could use a hook similar to what you 
(Eli) mentioned in the other subthread. Except instead of having a 
functions for face properties, it could be a global function that would 
be called on a "realized" plist of face attributes and return the 
adjusted one, for the point. The upside is that the display code won't 
have to look for another text property (and where its intervals start 
and end).

The downside, though, is that the realized plist might no be cache-able 
(the function can return a different value based on any number of 
conditions, right?). So either we document the requirement in text and 
simply expect the programmer to honor them (e.g. the returned face 
attributes must be stable WRT to buffer contents and not depend on time, 
for example), or simply avoid such a design. Same goes for callable face 
attributes.

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]



> On Jan 5, 2024, at 11:35 AM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 05/01/2024 10:19, Eli Zaretskii wrote:
>>> I’m sure there could be lots of uses of this type of functionality, but “enhance the focus on one region
>>> of importance” seems like a very common one.  For example, you might imagine dim/gray version of
>>> all the font-lock faces outside the “treesitter region of interest” and bright ones inside.  Whenever that
>>> TS-prescribed region changes, an overlay with a ‘face-remap property gets moved.
>> The question I suggest to ponder is whether simpler, less general ways
>> to implement these features are "good enough".  For example, changing
>> the appearance of a region of text can be handled by moving an overlay
>> there with a suitable face definition and high priority; changing the
>> appearance of text around point can be handled by a special variable
>> which the display engine will consider when working on text around
>> point;
> 
> An overlay with face and higher priority would override most faces in the region, wouldn't it? Unless it just sets the background, in which case we already have this capability.
> 
> JD, have you considered it? E.g. set the default face's background to some darker color and use an overlay which sets background to a lighter one.

I’ve definitely considered “re-style all of one particular face attribute” options like this and none I’ve found are suitable in my case (though perhaps for the TS-current-context example I gave this could be a decent option).  The best analogy is the one I just offered in my reply to Eli: changing the class of an HTML element and having subsets of styled text within it automatically updated, thanks to the inheritance capabilities of CSS (not that that’s an explicit model for Emacs to emulate; just conveying the idea). 

> But if that's not sufficient, we could use a hook similar to what you (Eli) mentioned in the other subthread. Except instead of having a functions for face properties, it could be a global function that would be called on a "realized" plist of face attributes and return the adjusted one, for the point. The upside is that the display code won't have to look for another text property (and where its intervals start and end).

I see, so that function could look at some properties at point and decide whether to alter or “let stand” the computed faces.  I believe that would work and is obviously rather flexible, but it sounds even more general than the ‘face-remap property approach.

> The downside, though, is that the realized plist might no be cache-able (the function can return a different value based on any number of conditions, right?).

Right.   I don’t know how the caching is done, but I’d expect this would be harder.  Imagine for example every other invocation of this “final-face-remap-function” you propose changes from :foreground “blue” to :foreground “red” (ok not a very useful example, but illustrative).  Can the face merge code notice this repetition and keep the cache valid?  


[-- Attachment #2: Type: text/html, Size: 3796 bytes --]

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 6608 bytes --]



> On Jan 5, 2024, at 3:19 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Thu, 4 Jan 2024 19:51:46 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> On Jan 4, 2024, at 1:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> It is called where the 'face' property changes.  The display engine
>> finds the next position where that happens, and arranges for calling
>> face_at_buffer_position when it gets to process that position.  The
>> results of face merging at that position are kept in a cache, to avoid
>> performing it more often than needed.
>> 
>> This makes sense, thanks.  In the ‘face-remap property scenario, the number of such ‘face
>> property-change positions doesn’t actually change.
> 
> I don't think this is true. If the 'face-remap' property changes at
> some buffer position, the realized faces need to change there,
> although the 'face' property didn't change. So with the 'face-remap'
> property, we'd need to arrange for face-merging also where the
> 'face-remap' property changes, not only where the 'face' property
> changes.

I wondered about that.  Since I don’t know emacs internals, I am (blissfully?) unaware of how close at hand the information of (get-text-property (point) 'face-remap) is.  Since a typical buffer has maybe N=hundreds of ‘face changes, this puts an approximate upper limit on the number of sensible ‘face + ‘face-remap changes: 2N.  Much more likely would be just a handful of ‘face-remap changes.  So I doubt the overhead of “additional merging” would be significant.  Of course you have to find these special points, but already you must find ‘face and ‘font-lock-face (and others?).

>> and use it together with
>> `face-remapping-alist' to compute the final face merge. I would expect that performance would
>> depend on whether the face-merge cache could be (largely) preserved, even as 'face-remap
>> properties are added/removed/moved around (typically in repetitive fashion).
> 
> The result of merging faces when 'face-remap' is in effect will be
> different faces, even though the named faces didn't change.  Those
> different faces will be realized and cached, and will have their own
> internal face IDs, but will not be exposed to Lisp as symbols, via
> APIs such as face-at-point.

That’s good, since it sounds like even if a ‘face-remap property range is moved, but includes the same underlying face data, the cache for those internal merged faces should remain valid.

>> How many faces do you have defined on that frame?  I'm guessing not
>> many, or maybe they are all almost identical (differ in only one or
>> two simple attributes).
>> 
>> Right now it’s just a couple of faces (remove one face-remap, add another).  If I had access to a
>> ‘face-remap property, I could imagine maybe 10 faces at most being affected as point changes. 
>> Each face would change by one or two attributes like foreground, stipple, etc.  
> 
> What matters is the total number of named faces defined for the frame,
> not the number of faces affected by face-remap.

I define the faces on the fly based on max indentation depth, but in practice it’s of order 5 more than the normal contingent of a dozen or so font-lock-faces.  So not a huge number.

>> I’m sure there could be lots of uses of this type of functionality, but “enhance the focus on one region
>> of importance” seems like a very common one.  For example, you might imagine dim/gray version of
>> all the font-lock faces outside the “treesitter region of interest” and bright ones inside.  Whenever that
>> TS-prescribed region changes, an overlay with a ‘face-remap property gets moved.  
> 
> The question I suggest to ponder is whether simpler, less general ways
> to implement these features are "good enough".  For example, changing
> the appearance of a region of text can be handled by moving an overlay
> there with a suitable face definition and high priority;

Indeed I have pondered this for several months and found nothing suitable.  Do you have any specific approaches to consider, which would allow you to selectively affect the appearance of certain small ranges of text within a (much) larger region of text?  The problem is, the region is well known, but changing the appearance everywhere within it is not suitable to the problem at hand.  I could obviously scrub through the entire region on the fly replacing face properties as I go (including inside ‘display properties), but that’s painful and probably too slow for interactive use.

> changing the appearance of text around point can be handled by a special variable
> which the display engine will consider when working on text around
> point; etc.  

I’m afraid I don’t understand your “special variable to consider for text around point” idea.  Can you give an example of how this could work in practice?  How would such a variable be updated and associated with text, if not via text properties?

> IOW, it is not always beneficial to generalize a feature
> to the most general abstraction level, certainly not when talking
> about Emacs, where a gazillion features related to the proposed new
> one are already available and expected to work after the proposed
> change.  Adding significant enhancements to Emacs that change previous
> design assumptions can easily become a proverbial death by thousand
> cuts.

Of course, well understood.  It’s eminently sensible to guard against kitchen-sink-itis.  Lacking familiarity with the core face code, I wondered if there was a straightforward extension to the global face-remapping-alist which could be brought to bear on the problem.  But perhaps there are other more straightforward means.

If you have other simpler ideas in mind that would allow turning on and off a latent “plane” of varying face information within a region of text (information which, I should have mentioned, could easily be computed and pre-applied by font-lock), I’d be very glad to hear them.

In the world of CSS, you’d do this quite simply by (say) updating the class of the div which wraps your text of interest, and having special styling for, e.g., "alternate bold" and "alternate italic”, which is only activated when the “alternate” class is applied:

<div class=“normal”>
This is <i>some italic text</i> and <b>some bold</b> text.  This is unrelated text.
</div>

In CSS
	
.alternate b {...}
.alternate i {…}

(In JS later:)

 myDiv.classList.add('alternate');



[-- Attachment #2: Type: text/html, Size: 29538 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 6 Jan 2024 08:53:22 -0500
> Cc: emacs-devel@gnu.org
> 
>  This makes sense, thanks.  In the ‘face-remap property scenario, the number of such ‘face
>  property-change positions doesn’t actually change.
> 
>  I don't think this is true. If the 'face-remap' property changes at
>  some buffer position, the realized faces need to change there,
>  although the 'face' property didn't change. So with the 'face-remap'
>  property, we'd need to arrange for face-merging also where the
>  'face-remap' property changes, not only where the 'face' property
>  changes.
> 
> I wondered about that.  Since I don’t know emacs internals, I am (blissfully?) unaware of how close at
> hand the information of (get-text-property (point) 'face-remap) is.  Since a typical buffer has maybe
> N=hundreds of ‘face changes, this puts an approximate upper limit on the number of sensible ‘face +
> ‘face-remap changes: 2N.  Much more likely would be just a handful of ‘face-remap changes.  So I
> doubt the overhead of “additional merging” would be significant.  Of course you have to find these
> special points, but already you must find ‘face and ‘font-lock-face (and others?).

I only wrote the above because you said that "the number of such
'face' property changes will not change".  I'm saying that it _will_
change, and moreover, that the display engine will need to search for
this new property, in addition to the properties it searches for
already.  Whether this will add significantly to the processing
requirements, I don't know, because I don't have a clear idea what
application-level features you have in mind.  I do know that, like the
proverbial wisdom of "640KB should be enough for everyone", any
assumptions in Emacs that some feature will be used only "a little"
eventually becomes false.  We have ample examples of that:

  . the number of overlays in a buffer, which recently caused us to
    radically change how overlays are stored
  . the number of faces on a frame, which recently caused us to
    radically change how a frame's faces are stored

and there are other examples.

>  How many faces do you have defined on that frame?  I'm guessing not
>  many, or maybe they are all almost identical (differ in only one or
>  two simple attributes).
> 
>  Right now it’s just a couple of faces (remove one face-remap, add another).  If I had
>  access to a
>  ‘face-remap property, I could imagine maybe 10 faces at most being affected as point
>  changes. 
>  Each face would change by one or two attributes like foreground, stipple, etc.  
> 
>  What matters is the total number of named faces defined for the frame,
>  not the number of faces affected by face-remap.
> 
> I define the faces on the fly based on max indentation depth, but in practice it’s of order 5 more than
> the normal contingent of a dozen or so font-lock-faces.  So not a huge number.

Maybe not in your use patterns, but some users here reported many
hundreds of faces on each frame.  It is the reason why we changed
frame-face-alist to use a hash table instead.

With many faces defined for a frame, recomputing all of them is a
process that visibly slows down certain operations in Emacs.

>  The question I suggest to ponder is whether simpler, less general ways
>  to implement these features are "good enough".  For example, changing
>  the appearance of a region of text can be handled by moving an overlay
>  there with a suitable face definition and high priority; 
> 
> Indeed I have pondered this for several months and found nothing suitable.  Do you have any specific
> approaches to consider, which would allow you to selectively affect the appearance of certain small
> ranges of text within a (much) larger region of text?  The problem is, the region is well known, but
> changing the appearance everywhere within it is not suitable to the problem at hand.  I could
> obviously scrub through the entire region on the fly replacing face properties as I go (including inside
> ‘display properties), but that’s painful and probably too slow for interactive use.

I cannot be of more help here without understanding better what kinds
of application-level features you want to support.  Until now, you
only described that in very general terms.

>  changing the appearance of text around point can be handled by a special variable
>  which the display engine will consider when working on text around
>  point; etc.  
> 
> I’m afraid I don’t understand your “special variable to consider for text around point” idea.  Can you
> give an example of how this could work in practice?  How would such a variable be updated and
> associated with text, if not via text properties?

It could be a buffer-local variable, which defines the size of the
region around point where the faces should change their appearance,
and how to change the appearance.  The display engine then could take
that into consideration when processing buffer positions around point.

Whether this makes sense depends on the applications you have in mind.

> If you have other simpler ideas in mind that would allow turning on and off a latent “plane” of varying
> face information within a region of text (information which, I should have mentioned, could easily be
> computed and pre-applied by font-lock), I’d be very glad to hear them.
> 
> In the world of CSS, you’d do this quite simply by (say) updating the class of the div which wraps your
> text of interest, and having special styling for, e.g., "alternate bold" and "alternate italic”, which is only
> activated when the “alternate” class is applied:
> 
>  <div class=“normal”>
> 
>  This is <i>some italic text</i> and <b>some bold</b> text.  This is unrelated text.
> 
>  </div>
> 
>  In CSS
> 
>  .alternate b {...}
>  .alternate i {…}
> 
>  (In JS later:)
> 
>   myDiv.classList.add('alternate');

You are describing how to specify the effect, whereas I'm bothered by
the implementation which could support that.  Faces in Emacs are
specific to the entire frame, and we have face-remapping to change
that within limits buffer-locally.  We don't currently have any
infrastructure that allows the faces to change on a finer resolution,
except if the Lisp program modifies the 'face' text properties or
defines overlays with 'face' properties.

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 3482 bytes --]



> On Jan 6, 2024, at 9:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> I do know that, like the proverbial wisdom of "640KB should be enough for everyone", any
> assumptions in Emacs that some feature will be used only "a little"
> eventually becomes false.  We have ample examples

Point taken.  That said, I think my “at most a doubling” is a robust estimate, since it makes no sense to remap more face regions than there are in the buffer.  But never underestimate the intrepid elisp programmer I suppose...

> I cannot be of more help here without understanding better what kinds
> of application-level features you want to support.  Until now, you
> only described that in very general terms.

This is because I was discussing a general feature that would be useful for more than my own application.  

To make it more concrete, what I had in mind is an update to indent-bars which would changes the appearance of the set of bars in a “scope” region via treesitter queries in a post-command hook.  As point changes, the TS “enclosing scope” is calculated, and if it has changed, all the existing indent bars in that region would be updated with “alternate” styling (and formerly highlighted text would be returned to normal styling).  See [1] for some images to give you the idea of how the normal styling can look.  Important to note are that:

Bar styling can depend on indentation depth, so in most cases # faces = # indent levels.
‘display properties with ‘face propertized text are used to add bars to blank lines (or insufficiently long empty lines), and to properly handle tabs (when indent-tabs-mode=t).

> It could be a buffer-local variable, which defines the size of the
> region around point where the faces should change their appearance,
> and how to change the appearance.  The display engine then could take
> that into consideration when processing buffer positions around point.
> 
> Whether this makes sense depends on the applications you have in mind.

Since there are many small stretches of text (single character stretches) that would be impacted over a larger region, I’m afraid such a simple approach wouldn’t work.


>> In the world of CSS, you’d do this quite simply by (say) updating the class of the div which wraps your
>> text of interest, and having special styling for, e.g., "alternate bold" and "alternate italic”, which is only
>> activated when the “alternate” class is applied:
>> 
>> <div class=“normal”>
>> 
>> This is <i>some italic text</i> and <b>some bold</b> text.  This is unrelated text.
>> 
>> </div>
>> 
>> In CSS
>> 
>> .alternate b {...}
>> .alternate i {…}
>> 
>> (In JS later:)
>> 
>>  myDiv.classList.add('alternate');
> 
> You are describing how to specify the effect, whereas I'm bothered by
> the implementation which could support that.  Faces in Emacs are
> specific to the entire frame, and we have face-remapping to change
> that within limits buffer-locally.  We don't currently have any
> infrastructure that allows the faces to change on a finer resolution,
> except if the Lisp program modifies the 'face' text properties or
> defines overlays with 'face' properties.

I understand.  The question is whether it would be desirable, tractable, performant, and maintainable to add any such infrastructure.  Thanks for your thoughts and analysis.

[1] https://github.com/jdtsmith/indent-bars/blob/main/examples.md

[-- Attachment #2: Type: text/html, Size: 7200 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 6 Jan 2024 09:56:08 -0500
> Cc: emacs-devel@gnu.org
> 
>  I cannot be of more help here without understanding better what kinds
>  of application-level features you want to support.  Until now, you
>  only described that in very general terms.
> 
> This is because I was discussing a general feature that would be useful for more than my own
> application.  

And I'm trying to get a grip on the specific instances of that general
feature which perhaps could be much easier to implement and have much
lower costs.

> To make it more concrete, what I had in mind is an update to indent-bars which would changes the
> appearance of the set of bars in a “scope” region via treesitter queries in a post-command hook.  As
> point changes, the TS “enclosing scope” is calculated, and if it has changed, all the existing indent
> bars in that region would be updated with “alternate” styling (and formerly highlighted text would be
> returned to normal styling).  See [1] for some images to give you the idea of how the normal styling
> can look.  Important to note are that:

Why cannot this be done by modifying faces or overlays in the affected
region?

Btw, my advice is to use an idle timer, not post-command-hook, if that
is possible.  A timer-based implementation will not slow down Emacs
when the user moves point quickly or scrolls through a portion of the
buffer by leaning on an arrow key.  Also, the timer will always run
with point corresponding to what is on the screen, whereas
post-command-hook runs before point adjustment, so could use
inaccurate value of point.

>  It could be a buffer-local variable, which defines the size of the
>  region around point where the faces should change their appearance,
>  and how to change the appearance.  The display engine then could take
>  that into consideration when processing buffer positions around point.
> 
>  Whether this makes sense depends on the applications you have in mind.
> 
> Since there are many small stretches of text (single character stretches) that would be impacted over
> a larger region, I’m afraid such a simple approach wouldn’t work.

If all you need is change the faces, I think it will work.

> I understand.  The question is whether it would be desirable, tractable, performant, and maintainable
> to add any such infrastructure.

I don't know.  I do know it will not be simple.

Re: region-based face-remapping

[Dmitry Gutov]

On 06/01/2024 15:53, JD Smith wrote:
> In the world of CSS, you’d do this quite simply by (say) updating the 
> class of the div which wraps your text of interest, and having special 
> styling for, e.g., "alternate bold" and "alternate italic”, which is 
> only activated when the “alternate” class is applied:
> 
>     <div class=“normal”>
> 
>         This is <i>some italic text</i> and <b>some bold</b> text.  This
>         is unrelated text.
> 
>     </div>
> 
> 
>     In CSS
> 
>         .alternate b {...}
>         .alternate i {…}
> 
>     (In JS later:)
> 
>           myDiv.classList.add('alternate');

If we were to try to emulate CSS's nesting capability, perhaps we would 
add some new attributes which would alter the existing ones (using 
relative values). E.g. :background-relative and :foreground-relative, 
which could be used to lighten or darken the colors provided by the 
:background and :foreground attributes. The -relative ones could come 
from a separate face, but the question would be how to make sure to 
apply it - it would have to be the last rule in font-lock rules, to 
avoid forcing all other rules to use :override. Or, I don't know, if 
would need to be applied some other way at the end of font-lock. Main 
thing, it wouldn't work off an overlay.

Or a different spin, for better theme-ability: some declarative forms 
which allow to specify resulting attributes (colors, etc) for 
combinations of faces. I.e. if two faces are present together, the 
foreground and background should be this and that. The limitation is 
also like the above.

But to be able to set these faces through an overlay, or even a text 
property but without worrying about the application order vis-a-vis 
font-lock, it would probably need to be a new property. Not sure if we'd 
also need some new entities in its values, rather than regular faces.

Maybe the new property would specify one or several new values which 
would correspond to new conditions in the DISPLAY form of face 
definitions. But that would make defface forms incompatible with older 
emacsen, I guess, and it's not certain whether it would be easy enough 
to use for a third-party package author anyway.

Re: region-based face-remapping

[Stefan Monnier via Emacs development discussions.]

> To make this concrete, how would people approach the following:
>
> 1 On point motion, treesitter’s syntax tree is consulted to find an
>   "enclosing semantic context region".
> 2 Within that region, pre-existing faces (or perhaps other properties)
>   get updated to a “highlighted” variant.
> 3 Outdated/prior regions have the highlighted variants removed.
>
> Since you don’t know in advance how large such regions might be, it
> seems problematic to search and replace all the various faces or
> other properties.   

There' worse: you don't know in advance which faces can appear in the
region nor what those faces currently look like, so it's non-trivial to
find "highlighted variants".

Personally, I think the better way to the provide the above
functionality is by making faces more flexible.

A "face" specifies how to *modify* a "base appearance" w.r.t
various attributes.  In practice currently for most attributes the
modification is limited to one of two options:
- leave unchanged
- override

There are some exceptions, most importantly the size attribute which can
be a float to increase/reduce the base size.  I think such a third option
should be available for many other attributes, so we can have faces
which make the background color more red, or which increases/reduces
contrast, ...

[ I could also imagine a face which turns bolds into underlines (thus has
  an effect *across* attributes).  ]

With such a functionality, your request would be satisfied with a simple
overlay with a single `face` property.


        Stefan

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Mon, 15 Jan 2024 14:55:34 -0500
> From:  Stefan Monnier via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> There are some exceptions, most importantly the size attribute which can
> be a float to increase/reduce the base size.  I think such a third option
> should be available for many other attributes, so we can have faces
> which make the background color more red, or which increases/reduces
> contrast, ...

How do you express these "more red" or "more/less contrast" using just
the existing face attributes?  The :height example doesn't help
because it just defines special values for an existing attribute, so
the analogy would be to define a value for :foreground that would make
it "more red".  How to do that?  And the same with "more or less
contrast with the background color".

> With such a functionality, your request would be satisfied with a simple
> overlay with a single `face` property.

But only if the effect is to transform all the faces in the region in
the same way.  My understanding is that the request is not necessarily
limited to that.

Re: region-based face-remapping

[Eli Zaretskii]

> Date: Mon, 15 Jan 2024 22:19:28 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > With such a functionality, your request would be satisfied with a simple
> > overlay with a single `face` property.
> 
> But only if the effect is to transform all the faces in the region in
> the same way.  My understanding is that the request is not necessarily
> limited to that.

And another, IMO more serious issue: an overlay like that over a
region will change all the faces in the region, including the default
face, whereas the request was to change only a particular set of
faces, and I very much doubt that the default face was one of them.

Re: region-based face-remapping

[Stefan Monnier]

> How do you express these "more red" or "more/less contrast" using just
> the existing face attributes?  The :height example doesn't help
> because it just defines special values for an existing attribute, so
> the analogy would be to define a value for :foreground that would make
> it "more red".  How to do that?  And the same with "more or less
> contrast with the background color".

Allow a face to specify a function which takes the base face value and
returns the desired value of the attribute.  In order to change the
contrast, the "base face value" would need to include more than just the
base value of the specific attribute, but probably a whole vector giving
all the attributes.


        Stefan

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 7950 bytes --]



> On Jan 8, 2024, at 12:28 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> To make it more concrete, what I had in mind is an update to indent-bars which would changes the
>> appearance of the set of bars in a “scope” region via treesitter queries in a post-command hook.  As
>> point changes, the TS “enclosing scope” is calculated, and if it has changed, all the existing indent
>> bars in that region would be updated with “alternate” styling (and formerly highlighted text would be
>> returned to normal styling).  See [1] for some images to give you the idea of how the normal styling
>> can look.  Important to note are that:
> 
> Why cannot this be done by modifying faces or overlays in the affected
> region?

It can, but the region where such face modification is needed could be arbitrarily large (including the entire buffer), which makes this more like a font-lock-problem in a PCH or via a timer (with the bonus that it could occur either on modification or just point change).  So at worst it would be like re-font-locking the entire buffer (many thousands of locations, most just 1 character wide) on every key press (or rapidly with a timer).  See below for an attempted “better” approach.

> Btw, my advice is to use an idle timer, not post-command-hook, if that
> is possible.  A timer-based implementation will not slow down Emacs
> when the user moves point quickly or scrolls through a portion of the
> buffer by leaning on an arrow key.  Also, the timer will always run
> with point corresponding to what is on the screen, whereas
> post-command-hook runs before point adjustment, so could use
> inaccurate value of point

For sure.  I already by default use a delay timer via a PCH to avoid updating the current-depth (single bar) highlight too quickly (though not an idle timer: instead I schedule the update say 75ms in the future, and push it forward another 75ms when new commands come in before the timer fires — what I call a “kick the can” timer [1, for aside on timers]).  As I briefly mentioned before, face-remapping-alist on my system is actually performant enough to do this as fast as users can fire commands (e.g. ~10ms for smooth scrolling).  That speed and my subsequent inference about what the display engine might be able to do in this space is what motivated my notion of a ‘face-remap property.

What I’m struggling with is how to do something “like font lock” — i.e. refontify some potentially substantial fraction of all the faces in a buffer, not (just) on modifications in an after-change-hook, but also on point-dependent “region of interest” changes, with a priority given to the displayed region in the window.  IMO, tree-sitter will make this kind of idea more common.

I’ve thought of simply performing the highlight update operation on the intersection between the treesitter region-of-interest and the window-start/end region, maybe with 20% space padding on either side of the window.  And then, when new regions are indicated, highlight the (new-old) difference, and unhighlight the (old-new) difference (leaving their union unchanged).  Something like this (in pseudo-code):

(defvar-local old-roi nil)
(let* ((ts-roi (some-treesitter-command)) ; a (start . end) "region of interest"
       (ws (window-start)) (we (window-end))
       (space (round (/ (- we ws) 5)))
       regs)
  ;; Add 20% padding to window region
  (setq ws (max (point-min) (- ws space))
	we (min (point-max) (+ we space)))
  (setq ts-roi (region-intersection ts-roi (cons ws we))) ;find intersection range, possibly nil
  (if (null ts-roi)
      (do-unhighlight old-roi)
    (if old-roi
	(progn
	  ;; newly highlighted
	  (setq regs (region-differences ts-roi old-roi))
	  (while regs
	    (do-highlight (car regs))
	    (setq regs (cdr regs)))
	  ;; newly unhighlighted
	  (setq regs (region-differences old-roi ts-roi))
	  (while regs
	    (do-unhighlight (car regs))
	    (setq regs (cdr regs))))
      (do-highlight ts-roi)))
  (setq old-roi ts-roi))

A bit fiddly, but not terrible.  Would be better to see if point has changed, and if not, use a cached ts-roi.

For the highlighting/unhighlighting operation, I think I also mentioned that the faces of interest can and will live in display properties, so you’d need to check all ‘display strings within the (un-)highlit region too, and rebuild those with the updated faces.

But suppose the change of region was precipitated by the removal or addition of text in the buffer, not just point movement?  Now your old region (old-roi, above) is outdated, likely wrong, and possibly had holes put in it by the edits.

So instead of saving old-roi in a variable, reach for a marker text property, call it 'indent-bars-ts-highlighted (rear-nonsticky of course).  Then, when the region of interest has changed, find (via `next-single-property-changes’) all changes to this marker property across the entire buffer (hopefully fast enough?).   Intersect this (possibly disjoint set of) old region(s) against the new (window-region + ts-roi) intersection, and (de-)highlight as before.

OK, getting there, but what about invisible text?  What if the window includes a huge range within a large buffer, most of it hidden, and your ts-roi is large (like the whole buffer)?  There’s no point doing the (un-)highlighting operation on invisible text.  But you can’t just skip over invisible text, it better have its marker property removed, and the act of hiding/unhiding anything is now cause for recalculating everything.

You can see how tangled it can get, compared to the simplicity (for the elisp programmer) of setting a ‘face-remap overlay and moving it around (i.e. similar to updating the class of a div in HTML), and letting the display engine sort it out.

So I think it’s possible, but it’s also painful, and I’d hazard to guess most package authors wouldn’t go to such lengths.

>> It could be a buffer-local variable, which defines the size of the
>> region around point where the faces should change their appearance,
>> and how to change the appearance.  The display engine then could take
>> that into consideration when processing buffer positions around point.
>> 
>> Whether this makes sense depends on the applications you have in mind.
>> 
>> Since there are many small stretches of text (single character stretches) that would be impacted over
>> a larger region, I’m afraid such a simple approach wouldn’t work.
> 
> If all you need is change the faces, I think it will work.

Maybe here you mean something like “within the window region, updating as that changes”, similar to what I outlined above?

>> I understand.  The question is whether it would be desirable, tractable, performant, and maintainable
>> to add any such infrastructure.
> 
> I don't know.  I do know it will not be simple.

OK, fair enough.  Possibly it’s worth pressing on this to see what I can do with current capabilities then.  Please let me know if I’m missing any obvious strategies that would simplify the mess above.

Thanks for your thoughts.

[1]  Short aside on timers: do you think an idle timer that repeatedly runs every 75ms of idle time and asks “did point change?” then, if so, “did TS region of interest change?” would be preferable to a post command hook that kicks a timer to do the same?  I already use `timer-set-time' to avoid rapidly reallocating a timer.  I’d guess these two approaches are ~equivalent performance-wise, but PCH’s can be buffer-local and idle-timers can’t so they are always running.

Aside within aside: it would be great if `timer-activate' included an optional no-error argument so you don’t have to check if it is on `timer-list’ twice.  I.e. if a timer is already on timer-list and `timer-activate’ (with no-error) is called on it, do nothing.

[-- Attachment #2: Type: text/html, Size: 13556 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Mon, 8 Jan 2024 16:49:21 -0500
> Cc: emacs-devel@gnu.org
> 
>  Why cannot this be done by modifying faces or overlays in the affected
>  region?
> 
> It can, but the region where such face modification is needed could be arbitrarily large (including the
> entire buffer), which makes this more like a font-lock-problem in a PCH or via a timer (with the bonus
> that it could occur either on modification or just point change).  So at worst it would be like
> re-font-locking the entire buffer (many thousands of locations, most just 1 character wide) on every
> key press (or rapidly with a timer).

Not sure changing faces will be significantly slower than what you had
in mind.  Maybe simpler (for a Lisp program), but not necessarily
slower.  After all, that's what JIT font-lock does all the time, and
we accept the performance it provides.

> What I’m struggling with is how to do something “like font lock” — i.e. refontify some potentially
> substantial fraction of all the faces in a buffer, not (just) on modifications in an after-change-hook, but
> also on point-dependent “region of interest” changes, with a priority given to the displayed region in
> the window.  IMO, tree-sitter will make this kind of idea more common.

Did you try to use jit-lock-register machinery?

Maybe Stefan (CC'ed) will have better ideas.

> For the highlighting/unhighlighting operation, I think I also mentioned that the faces of interest can and
> will live in display properties, so you’d need to check all ‘display strings within the (un-)highlit region
> too, and rebuild those with the updated faces.

I hope the number of faces you use in those display strings is small,
in which case the feature should still work reasonably well, since you
presumably are familiar with those faces and are managing them in your
code.

> But suppose the change of region was precipitated by the removal or addition of text in the buffer,
> not just point movement?  Now your old region (old-roi, above) is outdated, likely wrong, and possibly
> had holes put in it by the edits.

That's exactly what JIT font-lock solves, right?  Editing a buffer
changes the text, and thus the faces that need to be put on the text
change also.  For example, something that was a comment can become
non-comment, or vice versa.  Isn't what you describe similar (read:
identical), except that the faces themselves are different, i.e. not
font-lock faces?

> OK, getting there, but what about invisible text?  What if the window includes a huge range within a
> large buffer, most of it hidden, and your ts-roi is large (like the whole buffer)?  There’s no point doing
> the (un-)highlighting operation on invisible text.  But you can’t just skip over invisible text, it better have
> its marker property removed, and the act of hiding/unhiding anything is now cause for recalculating
> everything.

Again, JIT font-lock and the display engine already solve this
problem.  So you shouldn't need to reinvent the wheel.

> You can see how tangled it can get, compared to the simplicity (for the elisp programmer) of setting a
> ‘face-remap overlay and moving it around (i.e. similar to updating the class of a div in HTML), and
> letting the display engine sort it out.

Yes, your original idea is much simpler for a Lisp program.  But my
point is that its price in complexity of implementation is
significantly higher, so much so that I doubt we can have it,
certainly soon.  By contrast, if jit-lock-register satisfies your
needs, you should have almost all the tricky machinery in-place
already, and what remains for you to implement is relatively little.

>  It could be a buffer-local variable, which defines the size of the
>  region around point where the faces should change their appearance,
>  and how to change the appearance.  The display engine then could take
>  that into consideration when processing buffer positions around point.
> 
>  Whether this makes sense depends on the applications you have in mind.
> 
>  Since there are many small stretches of text (single character stretches) that would be
>  impacted over
>  a larger region, I’m afraid such a simple approach wouldn’t work.
> 
>  If all you need is change the faces, I think it will work.
> 
> Maybe here you mean something like “within the window region, updating as that changes”, similar to
> what I outlined above?

This was related to my idea of asking the display engine to handle
faces specially in the vicinity of point.  Since the display engine
examines each change in the face property, it could apply special
handling for faces of buffer positions around point.  For a simple
example, suppose that the buffer-local variable I mention above has
the value

   (BEFORE AFTER :background "red")

where BEFORE and AFTER are numbers of positions before and after point
to give the specified background color to all the faces in that
region.  Then the display engine could automatically merge each face
in the region with an anonymous face '(:background "red")', and use
that for the characters in the region.

> [1]  Short aside on timers: do you think an idle timer that repeatedly runs every 75ms of idle time and
> asks “did point change?” then, if so, “did TS region of interest change?” would be preferable to a
> post command hook that kicks a timer to do the same?

I don't think it matters much: a post-command-hook that just calls
run-with-idle-timer is not expensive.

> I already use `timer-set-time' to avoid rapidly
> reallocating a timer.  I’d guess these two approaches are ~equivalent performance-wise, but PCH’s
> can be buffer-local and idle-timers can’t so they are always running.

If you need a timer to be buffer-local, then indeed starting it from a
post-command-hook is more convenient.  Alternatively, the timer
function could test for the buffer and do nothing if it isn't the
buffer where the feature is activated.

> Aside within aside: it would be great if `timer-activate' included an optional no-error argument so you
> don’t have to check if it is on `timer-list’ twice.  I.e. if a timer is already on timer-list and
> `timer-activate’ (with no-error) is called on it, do nothing.

This is usually done by checking whether a timer variable is non-nil,
so it isn't hard to achieve.

Re: region-based face-remapping

[Stefan Monnier]

>> What I’m struggling with is how to do something “like font lock” —
>> i.e. refontify some potentially substantial fraction of all the faces
>> in a buffer, not (just) on modifications in an after-change-hook, but
>> also on point-dependent “region of interest” changes, with a priority
>> given to the displayed region in the window.  IMO, tree-sitter will
>> make this kind of idea more common.

[ Sorry, I missed the beginning of the thread and haven't had time to go
  and read the previous messages.  ]

IIUC you're discussing features where the appearance of parts of the
buffer depends on the position of point.  The main design issue with it
is what to do when the buffer is displayed in several windows (so there
are several points).  Depending on this, the implementation strategy may
need to be very different.

>> Aside within aside: it would be great if `timer-activate' included an
>> optional no-error argument so you don’t have to check if it is on
>> `timer-list’ twice.  I.e. if a timer is already on timer-list and
>> `timer-activate’ (with no-error) is called on it, do nothing.

The `timer.el` API is geared towards creating/destroying timers.
The other functions (like `timer-activate` and friends) seem to have
been thought mostly for internal use.

In order to reuse timer objects the API needs a few changes.
One of them could be to expose a `timer-active-p`, indeed.
[ Another would be to merge `timer-activate-when-idle` and
  `timer-activate` so the caller doesn't need to care which
  one to call.  ]


        Stefan

Re: region-based face-remapping

[JD Smith]

> On Jan 9, 2024, at 9:15 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> What I’m struggling with is how to do something “like font lock” —
>>> i.e. refontify some potentially substantial fraction of all the faces
>>> in a buffer, not (just) on modifications in an after-change-hook, but
>>> also on point-dependent “region of interest” changes, with a priority
>>> given to the displayed region in the window.  IMO, tree-sitter will
>>> make this kind of idea more common.
> 
> [ Sorry, I missed the beginning of the thread and haven't had time to go
>  and read the previous messages.  ]

No worries, thanks for joining.

> IIUC you're discussing features where the appearance of parts of the
> buffer depends on the position of point.  The main design issue with it
> is what to do when the buffer is displayed in several windows (so there
> are several points).  Depending on this, the implementation strategy may
> need to be very different.

That’s a very good point that I had not considered.  In my case, the selected widow would take precedence, and other windows just get what they get.

>>> Aside within aside: it would be great if `timer-activate' included an
>>> optional no-error argument so you don’t have to check if it is on
>>> `timer-list’ twice.  I.e. if a timer is already on timer-list and
>>> `timer-activate’ (with no-error) is called on it, do nothing.
> 
> The `timer.el` API is geared towards creating/destroying timers.
> The other functions (like `timer-activate` and friends) seem to have
> been thought mostly for internal use.
> 
> In order to reuse timer objects the API needs a few changes.
> One of them could be to expose a `timer-active-p`, indeed.
> [ Another would be to merge `timer-activate-when-idle` and
>  `timer-activate` so the caller doesn't need to care which
>  one to call.  ]

I had once heard that constantly allocating and destroying timers (say every 50ms) is memory inefficient, but that reusing the same timer overcomes this.  Hence timer-activate instead of constantly run-at-time’ing.  It’s possible that’s apocryphal (haven’t tested).

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Tue, 9 Jan 2024 15:20:53 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> > IIUC you're discussing features where the appearance of parts of the
> > buffer depends on the position of point.  The main design issue with it
> > is what to do when the buffer is displayed in several windows (so there
> > are several points).  Depending on this, the implementation strategy may
> > need to be very different.
> 
> That’s a very good point that I had not considered.  In my case, the selected widow would take precedence, and other windows just get what they get.

If you use overlays, they can be defined specific to a window.

Re: region-based face-remapping

[JD Smith]

> On Jan 9, 2024, at 9:15 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> What I’m struggling with is how to do something “like font lock” —
>>> i.e. refontify some potentially substantial fraction of all the faces
>>> in a buffer, not (just) on modifications in an after-change-hook, but
>>> also on point-dependent “region of interest” changes, with a priority
>>> given to the displayed region in the window.  IMO, tree-sitter will
>>> make this kind of idea more common.
> 
> [ Sorry, I missed the beginning of the thread and haven't had time to go
>  and read the previous messages.  ]

No worries, thanks for joining.

> IIUC you're discussing features where the appearance of parts of the
> buffer depends on the position of point.  The main design issue with it
> is what to do when the buffer is displayed in several windows (so there
> are several points).  Depending on this, the implementation strategy may
> need to be very different.

That’s a very good point that I had not considered.  In my case, the selected widow would take precedence, and other windows just get what they get.

>>> Aside within aside: it would be great if `timer-activate' included an
>>> optional no-error argument so you don’t have to check if it is on
>>> `timer-list’ twice.  I.e. if a timer is already on timer-list and
>>> `timer-activate’ (with no-error) is called on it, do nothing.
> 
> The `timer.el` API is geared towards creating/destroying timers.
> The other functions (like `timer-activate` and friends) seem to have
> been thought mostly for internal use.
> 
> In order to reuse timer objects the API needs a few changes.
> One of them could be to expose a `timer-active-p`, indeed.
> [ Another would be to merge `timer-activate-when-idle` and
>  `timer-activate` so the caller doesn't need to care which
>  one to call.  ]

I had once heard that constantly allocating and destroying timers (say every 50ms) is memory inefficient, but that reusing the same timer overcomes this.  Hence timer-activate instead of constantly run-at-time’ing.  It’s possible that’s apocryphal (haven’t tested).

Re: region-based face-remapping

[Stefan Monnier via Emacs development discussions.]

>> IIUC you're discussing features where the appearance of parts of the
>> buffer depends on the position of point.  The main design issue with it
>> is what to do when the buffer is displayed in several windows (so there
>> are several points).  Depending on this, the implementation strategy may
>> need to be very different.
>
> That’s a very good point that I had not considered.  In my case, the
> selected widow would take precedence, and other windows just get what
> they get.

While rendering window A, the selected window is always window A.
But yeah, I guess you could use jit-lock as in:

      ... call (jit-lock-register #'my-region-hl-fontify)
      ... and also make sure `my-region-hl` is placed last on
          `jit-lock-functions` (jit-lock currently doesn't offer
          a clean way to do that, sorry).
      ... and use `post-command-hook` or `pre-redisplay-functions`
      ... to call `my-region-hl-update-point`.

    (defvar-local my-region-hl-pos nil)

    (defun my-region-hl-update-point ()
      ;; compute the highlighted region according to point.
      (let ((beg ...)
            (end ...))
        (unless (and my-region-hl-pos
                     (= beg (car my-region-hl-pos))
                     (= end (cdr my-region-hl-pos)))
          (put-text-property (min beg (car my-region-hl-pos))
                             (max beg (car my-region-hl-pos))
                             'fontified nil))
          (put-text-property (min end (cdr my-region-hl-pos))
                             (max end (cdr my-region-hl-pos))
                             'fontified nil)
          (setq my-region-hl-pos
                (cons (copy-marker beg) (copy-marker end)))))

    (defun my-region-hl-fontify (beg end)
      (when my-region-hl-pos
        (setq beg (max beg (car my-region-hl-pos)))
        (setq end (min end (cdr my-region-hl-pos)))
        (when (> end beg)
          ... modify faces between `beg` and `end`.
          )))
          
        
-- Stefan

Re: region-based face-remapping

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 6216 bytes --]



> On Jan 9, 2024, at 8:03 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Mon, 8 Jan 2024 16:49:21 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> Why cannot this be done by modifying faces or overlays in the affected
>> region?
>> 
>> It can, but the region where such face modification is needed could be arbitrarily large (including the
>> entire buffer), which makes this more like a font-lock-problem in a PCH or via a timer (with the bonus
>> that it could occur either on modification or just point change).  So at worst it would be like
>> re-font-locking the entire buffer (many thousands of locations, most just 1 character wide) on every
>> key press (or rapidly with a timer).
> 
> Not sure changing faces will be significantly slower than what you had
> in mind.  Maybe simpler (for a Lisp program), but not necessarily
> slower.  After all, that's what JIT font-lock does all the time, and
> we accept the performance it provides.

As long as the changes happen at the right places (ala font-lock) and not all over the buffer,  you may be right.  And it’s reasonable to consider this since font-lock is already the one applying those faces in the first place.  The trick is how to actually implement it. 

>> What I’m struggling with is how to do something “like font lock” — i.e. refontify some potentially
>> substantial fraction of all the faces in a buffer, not (just) on modifications in an after-change-hook, but
>> also on point-dependent “region of interest” changes, with a priority given to the displayed region in
>> the window.  IMO, tree-sitter will make this kind of idea more common.
> 
> Did you try to use jit-lock-register machinery?
> 
> Maybe Stefan (CC'ed) will have better ideas.

I haven’t.  In my (limited) understanding and experience, all the font-lock apparatus is highly tuned to after-change modifications.  Here we need to (in addition) mark regions (old and new) as outdated, even when the buffer hasn’t changed at all, just based on the movement of the (selected window’s) point.  I can know how to expand modified regions with `font-lock-extend-region-functions’, but I don’t know how to inform font-lock that “even though you didn’t see changes, trust me, these regions are in need of updating”.  I would do this in a PCH, rate-limited by a timer.

Maybe it’s as simple as calling `font-lock-flush’ on those regions?  If so, can you call font-lock-flush multiple times in a row on overlapping regions without creating issues?  Does font-lock-flush do the right (JIT) thing, even when you mark a very large region?

And (this is the big one) how do you handle two chef’s both with their spoons in the font-locking pot at the same time?

Normal font-lock responding to changes in the buffer to refontify regions.
PCH+timer marks regions for refontification when the buffer-modified-tick or treesitter region-of-interest changes (the former because you can no longer trust that equal regions are in fact equivalent).

Seem like there’d be a bunch of unnecessarily repetitive refontification in such a scheme.  And marking large regions with `font-lock-flush’ every 75ms as you type also seems problematic.

For disclosure, I’ve never used jit-lock-register, and personally find the font-lock documentation light on details for implementing non-standard fontification scenarios such as this.

>> For the highlighting/unhighlighting operation, I think I also mentioned that the faces of interest can and
>> will live in display properties, so you’d need to check all ‘display strings within the (un-)highlit region
>> too, and rebuild those with the updated faces.
> 
> I hope the number of faces you use in those display strings is small,
> in which case the feature should still work reasonably well, since you
> presumably are familiar with those faces and are managing them in your
> code.

Yes, here again it's just a handful of faces that may appear inside ‘display strings.  But there may be many such strings (e.g. one for every empty/tab-indented line).  In the pure font-lock approach we are discussing, these display strings would just be regenerated as text is altered. 

>> But suppose the change of region was precipitated by the removal or addition of text in the buffer,
>> not just point movement?  Now your old region (old-roi, above) is outdated, likely wrong, and possibly
>> had holes put in it by the edits.
> 
> That's exactly what JIT font-lock solves, right?  Editing a buffer
> changes the text, and thus the faces that need to be put on the text
> change also.  For example, something that was a comment can become
> non-comment, or vice versa.  Isn't what you describe similar (read:
> identical), except that the faces themselves are different, i.e. not
> font-lock faces?

I guess so. The difficulty seems perhaps the other way around: updating font-lock JIT for both buffer modifications and (potentially) point movement (simultaneously, without stepping on each other!).

>> It could be a buffer-local variable, which defines the size of the
>> region around point where the faces should change their appearance

> This was related to my idea of asking the display engine to handle
> faces specially in the vicinity of point.  Since the display engine
> examines each change in the face property, it could apply special
> handling for faces of buffer positions around point.  For a simple
> example, suppose that the buffer-local variable I mention above has
> the value
> 
>   (BEFORE AFTER :background "red")
> 
> where BEFORE and AFTER are numbers of positions before and after point
> to give the specified background color to all the faces in that
> region.  Then the display engine could automatically merge each face
> in the region with an anonymous face '(:background "red")', and use
> that for the characters in the region.

This could indeed be useful, but in my situation, it’s a bunch of smaller text regions between BEFORE and AFTER that need altering individually. Hence the interest in a top-level “alternative class” type specifier, ala CSS. 

[-- Attachment #2: Type: text/html, Size: 7461 bytes --]

Re: region-based face-remapping

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Tue, 9 Jan 2024 16:31:30 -0500
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
>  Did you try to use jit-lock-register machinery?
> 
>  Maybe Stefan (CC'ed) will have better ideas.
> 
> I haven’t.  In my (limited) understanding and experience, all the font-lock apparatus is highly tuned to
> after-change modifications.  Here we need to (in addition) mark regions (old and new) as outdated,
> even when the buffer hasn’t changed at all, just based on the movement of the (selected window’s)
> point.  I can know how to expand modified regions with `font-lock-extend-region-functions’, but I don’t
> know how to inform font-lock that “even though you didn’t see changes, trust me, these regions are in
> need of updating”.  I would do this in a PCH, rate-limited by a timer.

It is not true that jit-lock is based on editing changes.  It uses a
special text property to mark regions already fontified, so resetting
that property will cause the corresponding region to be re-fontified
on the next redisplay cycle.

> Maybe it’s as simple as calling `font-lock-flush’ on those regions?  If so, can you call font-lock-flush
> multiple times in a row on overlapping regions without creating issues?  Does font-lock-flush do the
> right (JIT) thing, even when you mark a very large region?

Yes, AFAIU.  But it might be overkill for your case.

> And (this is the big one) how do you handle two chef’s both with their spoons in the font-locking pot
> at the same time?
> 
> 1 Normal font-lock responding to changes in the buffer to refontify regions.
> 2 PCH+timer marks regions for refontification when the buffer-modified-tick or treesitter
>  region-of-interest changes (the former because you can no longer trust that equal regions are in
>  fact equivalent).

Don't worry about that, redisplay and jit-lock know how to deal with
that.

> Seem like there’d be a bunch of unnecessarily repetitive refontification in such a scheme.  And
> marking large regions with `font-lock-flush’ every 75ms as you type also seems problematic.

That could be, but I'm quite sure this can be optimized if it works
for your use case.

> For disclosure, I’ve never used jit-lock-register, and personally find the font-lock documentation light
> on details for implementing non-standard fontification scenarios such as this.

That's why I asked Stefan to join this discussion.

>  That's exactly what JIT font-lock solves, right?  Editing a buffer
>  changes the text, and thus the faces that need to be put on the text
>  change also.  For example, something that was a comment can become
>  non-comment, or vice versa.  Isn't what you describe similar (read:
>  identical), except that the faces themselves are different, i.e. not
>  font-lock faces?
> 
> I guess so. The difficulty seems perhaps the other way around: updating font-lock JIT for both buffer
> modifications and (potentially) point movement (simultaneously, without stepping on each other!).

Again, don't worry about that.  Once redisplay cycle starts, changes
are basically frozen, so whatever was done in the buffer gets
fontified.  If there's a followup change, it will be fontified again.