Drawing UI elements behind text

Drawing UI elements behind text

[Cecilio Pardo]

Hello,

I am working on a mechanism to draw things on the background, behind
the text of a buffer.

This would allow to draw UI elements (such as indent lines or a fill
column indicator) and eye candy like background images or colored
windows.

This two first features (indent lines and fill indicator) are controlled
by declaring a list of vertical segments linked to buffer
positions.

Is there any other feature that could be implemented like this? Not just
by segments, but by any graphics drawn behind the text, vectors or
bitmap.

Thanks.

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 8:39 AM, Cecilio Pardo wrote:
> Is there any other feature that could be implemented like this? Not just
> by segments, but by any graphics drawn behind the text, vectors or
> bitmap.

It's not "behind" text, but with some work on setting per-line fill 
widths, you could perhaps use this in EWW to display images alongside 
text (like with the "float: right" CSS rule).

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 10:09:04 -0800
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 8:39 AM, Cecilio Pardo wrote:
> > Is there any other feature that could be implemented like this? Not just
> > by segments, but by any graphics drawn behind the text, vectors or
> > bitmap.
> 
> It's not "behind" text, but with some work on setting per-line fill 
> widths, you could perhaps use this in EWW to display images alongside 
> text (like with the "float: right" CSS rule).

Given the right Lisp API (which is not easy to come up with, AFAIR),
this can be done today, because Emacs already knows how to show only a
part of an image on a screen line.  So we need a way to tell Emacs
"split this image's display between these N lines".

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 10:44 AM, Eli Zaretskii wrote:
> Given the right Lisp API (which is not easy to come up with, AFAIR),
> this can be done today, because Emacs already knows how to show only a
> part of an image on a screen line.  So we need a way to tell Emacs
> "split this image's display between these N lines".

I've thought about this before. Doing it via sliced images scares me a 
bit since it would require extreme care to make sure everything lines up 
correctly. For example, what if the text has mixed font sizes resulting 
in different heights for some lines? Or what about applying text scaling?

I suppose if we deemed this feature important, we could probably address 
all of those, but I'm not sure it's worth the effort. In fact, I'm not 
sure it would be worth using this new feature to help EWW out here 
either (though that's not to say the feature isn't worthwhile 
elsewhere). I just though I'd mention it as another case that's worth at 
least a cursory look.

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 11:06:13 -0800
> Cc: cpardo@imayhem.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 10:44 AM, Eli Zaretskii wrote:
> > Given the right Lisp API (which is not easy to come up with, AFAIR),
> > this can be done today, because Emacs already knows how to show only a
> > part of an image on a screen line.  So we need a way to tell Emacs
> > "split this image's display between these N lines".
> 
> I've thought about this before. Doing it via sliced images scares me a 
> bit since it would require extreme care to make sure everything lines up 
> correctly. For example, what if the text has mixed font sizes resulting 
> in different heights for some lines?

Once the display engine knows at what X coordinate the image slice
should start, it can stop laying out other display elements and switch
to the slice when that X coordinate is reached.

> Or what about applying text scaling?

How is this different from what we do today when text-scaling happens
in a buffer where images are displayed together with text?

> I suppose if we deemed this feature important, we could probably address 
> all of those, but I'm not sure it's worth the effort.

What alternative do we have?  Showing the images alongside text
without letting the images scroll with the text (vertically and
horizontally) will not fly.  How do you scroll the images with the
text except by displaying them as part of the "normal" display layout?

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 11:22 AM, Eli Zaretskii wrote:
> Once the display engine knows at what X coordinate the image slice
> should start, it can stop laying out other display elements and switch
> to the slice when that X coordinate is reached.

I was thinking more about the Y coordinates and having gaps between 
consecutive slices if we made a mistake. That's probably fixable, but it 
seems like it would require a lot of care to avoid bugs.

>> Or what about applying text scaling?
> 
> How is this different from what we do today when text-scaling happens
> in a buffer where images are displayed together with text?

I'm just worried about this due to seeing some similar bugs come up in 
the visual-wrap-prefix-mode code.

>> I suppose if we deemed this feature important, we could probably address
>> all of those, but I'm not sure it's worth the effort.
> 
> What alternative do we have?  Showing the images alongside text
> without letting the images scroll with the text (vertically and
> horizontally) will not fly.  How do you scroll the images with the
> text except by displaying them as part of the "normal" display layout?

Personally, I'm ok with saying we just don't support laying out images 
like this in EWW. For it to have a practical effect, we'd probably need 
to have more-complete CSS support anyway. That's not easy either (though 
I have some ideas about how we could do it if we thought it would be 
worthwhile).

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 11:29:27 -0800
> Cc: cpardo@imayhem.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 11:22 AM, Eli Zaretskii wrote:
> > Once the display engine knows at what X coordinate the image slice
> > should start, it can stop laying out other display elements and switch
> > to the slice when that X coordinate is reached.
> 
> I was thinking more about the Y coordinates and having gaps between 
> consecutive slices if we made a mistake. That's probably fixable, but it 
> seems like it would require a lot of care to avoid bugs.

Yes, it requires care.

> >> Or what about applying text scaling?
> > 
> > How is this different from what we do today when text-scaling happens
> > in a buffer where images are displayed together with text?
> 
> I'm just worried about this due to seeing some similar bugs come up in 
> the visual-wrap-prefix-mode code.

Bugs will always be with us, no way around that.

> >> I suppose if we deemed this feature important, we could probably address
> >> all of those, but I'm not sure it's worth the effort.
> > 
> > What alternative do we have?  Showing the images alongside text
> > without letting the images scroll with the text (vertically and
> > horizontally) will not fly.  How do you scroll the images with the
> > text except by displaying them as part of the "normal" display layout?
> 
> Personally, I'm ok with saying we just don't support laying out images 
> like this in EWW. For it to have a practical effect, we'd probably need 
> to have more-complete CSS support anyway. That's not easy either (though 
> I have some ideas about how we could do it if we thought it would be 
> worthwhile).

This entire sub-thread is about how to provide such a feature;
leaving it unsupported is easy.  If we do want to provide it, I think
going the image-slice way is a better alternative.