bug#75291: Redisplay not updating fringe when face filter changes

[Eli Zaretskii <eliz@gnu.org>]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: 75291@debbugs.gnu.org,  mina86@mina86.com
> Date: Thu, 02 Jan 2025 14:50:54 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, redrawing everything will work, but will also cause flicker, and
> > is generally expensive, thus undesirable.
> 
> FWIW, it doesn't seem to cause flicker in practice.  I see flicker only
> when walking through messages in mu4e --- we do redisplay and draw only
> the background, and I haven't figured out why yet.  But in general, on a
> modern window system, turning a given redisplay into a full redisplay
> shouldn't cause flicker, even if it's inefficient.

I think it depends on whether you use double-buffering (some people
don't or cannot) and whether you have the mouse pointer over an Emacs
frame.  Also, depending on the GUI toolkit, the decorations might
flicker.

> I came across overlay_arrows_changed_p --- isn't this function trying to
> deal with exactly the case of something in the fringe changing outside
> the changed text region?

So you want to add to display_line code that sets each glyph_row's
redraw_fringe_bitmaps_p flag when the fringe face changes?  That could
probably work, provided that we disable redisplay optimizations which
might avoid calling display_line (you will see that we already disable
such optimizations when overlay_arrows_changed_p returns non-zero).
We might actually need to disable more of the optimizations, because
the overlay-arrow thing doesn't contradict the optimizations that
scroll the pixels, something that reaction to changes in the fringe
face cannot tolerate.

> > Regarding your idea about Lisp function that would invalidate GUI
> > parts: it is not very easy, since a Lisp program cannot easily know
> > where on the screen a given region of buffer positions will be.
> > There is posn-at-point, of course, but (a) it is quite expensive, and
> > (b) when Lisp runs, display could be outdated, so what posn-at-point
> > returns could be inaccurate.
> 
> I was imagining a lisp function that would make the next redisplay of a
> window do what you suggest in the next paragraph.

What does Lisp know about the fringe face that the display engine
doesn't?

> I'm not sure we'd even need an explicit Lisp function though.
> Face filters with :window are defined to compare window parameter values
> with eq, so couldn't we keep track of all the :filtered face
> specifications we encounter during face resolution and have
> set-window-parameter check whether the parameter it's setting is on the
> list of possible face filters and, if so, force next redisplay to
> evaluate faces? set-window-parameter wouldn't even have to do a deep
> comparison, because it's just eq.

First, the fringe face might not be window-specific (by default, it
isn't).  So I'm not sure how a window parameter will help.
face-remapping-alist is per-buffer, not per-window.

Next, the main problem with faces is face inheritance and face merging
(the latter is not relevant to fringe, I think).  Given that some
attribute of some face changes, how do you know whether such a change
causes the fringe face to change?  We'd probably need to realize it
anew, and then compare to the cached one?  And we'd need to do that
for each window, because of face-remapping-alist?

> > The way to "notice the change in the face" is not a simple problem to
> > solve, btw.  We currently don't know which faces change when some face
> > is modified.  So we have a frame-global flag that is set when any face
> > changes its attributes.  If we use that flag for detecting potential
> > changes in the fringe face, we'd start redrawing fringes
> > unnecessarily.
> 
> How many unnecessary face recalculations would we do? ISTM we could make
> the invalidation pretty precise as long as we're just looking at
> window parameters.

See above.

I think the proof of the proverbial pudding is in eating: maybe doing
the above will produce reasonable performance.