Drawing UI elements behind text

Drawing UI elements behind text

[Cecilio Pardo]

Hello,

I am working on a mechanism to draw things on the background, behind
the text of a buffer.

This would allow to draw UI elements (such as indent lines or a fill
column indicator) and eye candy like background images or colored
windows.

This two first features (indent lines and fill indicator) are controlled
by declaring a list of vertical segments linked to buffer
positions.

Is there any other feature that could be implemented like this? Not just
by segments, but by any graphics drawn behind the text, vectors or
bitmap.

Thanks.

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 8:39 AM, Cecilio Pardo wrote:
> Is there any other feature that could be implemented like this? Not just
> by segments, but by any graphics drawn behind the text, vectors or
> bitmap.

It's not "behind" text, but with some work on setting per-line fill 
widths, you could perhaps use this in EWW to display images alongside 
text (like with the "float: right" CSS rule).

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 10:09:04 -0800
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 8:39 AM, Cecilio Pardo wrote:
> > Is there any other feature that could be implemented like this? Not just
> > by segments, but by any graphics drawn behind the text, vectors or
> > bitmap.
> 
> It's not "behind" text, but with some work on setting per-line fill 
> widths, you could perhaps use this in EWW to display images alongside 
> text (like with the "float: right" CSS rule).

Given the right Lisp API (which is not easy to come up with, AFAIR),
this can be done today, because Emacs already knows how to show only a
part of an image on a screen line.  So we need a way to tell Emacs
"split this image's display between these N lines".

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 10:44 AM, Eli Zaretskii wrote:
> Given the right Lisp API (which is not easy to come up with, AFAIR),
> this can be done today, because Emacs already knows how to show only a
> part of an image on a screen line.  So we need a way to tell Emacs
> "split this image's display between these N lines".

I've thought about this before. Doing it via sliced images scares me a 
bit since it would require extreme care to make sure everything lines up 
correctly. For example, what if the text has mixed font sizes resulting 
in different heights for some lines? Or what about applying text scaling?

I suppose if we deemed this feature important, we could probably address 
all of those, but I'm not sure it's worth the effort. In fact, I'm not 
sure it would be worth using this new feature to help EWW out here 
either (though that's not to say the feature isn't worthwhile 
elsewhere). I just though I'd mention it as another case that's worth at 
least a cursory look.

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 11:06:13 -0800
> Cc: cpardo@imayhem.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 10:44 AM, Eli Zaretskii wrote:
> > Given the right Lisp API (which is not easy to come up with, AFAIR),
> > this can be done today, because Emacs already knows how to show only a
> > part of an image on a screen line.  So we need a way to tell Emacs
> > "split this image's display between these N lines".
> 
> I've thought about this before. Doing it via sliced images scares me a 
> bit since it would require extreme care to make sure everything lines up 
> correctly. For example, what if the text has mixed font sizes resulting 
> in different heights for some lines?

Once the display engine knows at what X coordinate the image slice
should start, it can stop laying out other display elements and switch
to the slice when that X coordinate is reached.

> Or what about applying text scaling?

How is this different from what we do today when text-scaling happens
in a buffer where images are displayed together with text?

> I suppose if we deemed this feature important, we could probably address 
> all of those, but I'm not sure it's worth the effort.

What alternative do we have?  Showing the images alongside text
without letting the images scroll with the text (vertically and
horizontally) will not fly.  How do you scroll the images with the
text except by displaying them as part of the "normal" display layout?

Re: Drawing UI elements behind text

[Jim Porter]

On 11/10/2024 11:22 AM, Eli Zaretskii wrote:
> Once the display engine knows at what X coordinate the image slice
> should start, it can stop laying out other display elements and switch
> to the slice when that X coordinate is reached.

I was thinking more about the Y coordinates and having gaps between 
consecutive slices if we made a mistake. That's probably fixable, but it 
seems like it would require a lot of care to avoid bugs.

>> Or what about applying text scaling?
> 
> How is this different from what we do today when text-scaling happens
> in a buffer where images are displayed together with text?

I'm just worried about this due to seeing some similar bugs come up in 
the visual-wrap-prefix-mode code.

>> I suppose if we deemed this feature important, we could probably address
>> all of those, but I'm not sure it's worth the effort.
> 
> What alternative do we have?  Showing the images alongside text
> without letting the images scroll with the text (vertically and
> horizontally) will not fly.  How do you scroll the images with the
> text except by displaying them as part of the "normal" display layout?

Personally, I'm ok with saying we just don't support laying out images 
like this in EWW. For it to have a practical effect, we'd probably need 
to have more-complete CSS support anyway. That's not easy either (though 
I have some ideas about how we could do it if we thought it would be 
worthwhile).

Re: Drawing UI elements behind text

[Eli Zaretskii]

> Date: Sun, 10 Nov 2024 11:29:27 -0800
> Cc: cpardo@imayhem.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 11/10/2024 11:22 AM, Eli Zaretskii wrote:
> > Once the display engine knows at what X coordinate the image slice
> > should start, it can stop laying out other display elements and switch
> > to the slice when that X coordinate is reached.
> 
> I was thinking more about the Y coordinates and having gaps between 
> consecutive slices if we made a mistake. That's probably fixable, but it 
> seems like it would require a lot of care to avoid bugs.

Yes, it requires care.

> >> Or what about applying text scaling?
> > 
> > How is this different from what we do today when text-scaling happens
> > in a buffer where images are displayed together with text?
> 
> I'm just worried about this due to seeing some similar bugs come up in 
> the visual-wrap-prefix-mode code.

Bugs will always be with us, no way around that.

> >> I suppose if we deemed this feature important, we could probably address
> >> all of those, but I'm not sure it's worth the effort.
> > 
> > What alternative do we have?  Showing the images alongside text
> > without letting the images scroll with the text (vertically and
> > horizontally) will not fly.  How do you scroll the images with the
> > text except by displaying them as part of the "normal" display layout?
> 
> Personally, I'm ok with saying we just don't support laying out images 
> like this in EWW. For it to have a practical effect, we'd probably need 
> to have more-complete CSS support anyway. That's not easy either (though 
> I have some ideas about how we could do it if we thought it would be 
> worthwhile).

This entire sub-thread is about how to provide such a feature;
leaving it unsupported is easy.  If we do want to provide it, I think
going the image-slice way is a better alternative.

Re: Drawing UI elements behind text

[JD Smith]

> On Nov 10, 2024, at 11:39 AM, Cecilio Pardo <cpardo@imayhem.com> wrote:
> 
> Hello,
> 
> I am working on a mechanism to draw things on the background, behind
> the text of a buffer.
> 
> This would allow to draw UI elements (such as indent lines or a fill
> column indicator) and eye candy like background images or colored
> windows.

I support this idea.  For indent-bars, I use display :stipple as a "separate band" of graphical information that can lie atop text in a buffer, but remain independent of it.  Stipples are flexible and very performant, but have a few issues:

- There is inconsistent stipple support across emacs versions.
- Stipple patterns begin at the frame origin, so up-to-date window-specific remapping is required to prevent unwanted pattern aliasing.  
- Aligning stipples vertically requires fixed space content.

More commonly, modes also make use of :box styling, e.g. for producing boxed "labels", as in org-modern.

An API for drawing simple styled lines, rounded corners, boxes etc. — either in absolute window coordinates, or associated with, but "behind" text in the buffer — would represent a real step forward. 

Re: Drawing UI elements behind text

[Cecilio Pardo]

On 25/11/2024 0:37, JD Smith wrote:
> I support this idea.  For indent-bars, I use display :stipple as a "separate band" of graphical information that can lie atop text in a buffer, but remain independent of it.  Stipples are flexible and very performant, but have a few issues:
> 
> - There is inconsistent stipple support across emacs versions.
> - Stipple patterns begin at the frame origin, so up-to-date window-specific remapping is required to prevent unwanted pattern aliasing.
> - Aligning stipples vertically requires fixed space content.

Stipples also disable the scrolling of buffer pixels, affecting 
performance of redisplay.

> 
> More commonly, modes also make use of :box styling, e.g. for producing boxed "labels", as in org-modern.
> 
> An API for drawing simple styled lines, rounded corners, boxes etc. — either in absolute window coordinates, or associated with, but "behind" text in the buffer — would represent a real step forward.

I haven't found a way to draw behind the text that is not too 
complicated or affects redisplay too much. So now the idea is drawing 
over the text. For the applications I have in mind this works equally well.

The starting drawing elements are going to be vertical and horizontal 
segments, with color (maybe with alpha), width and different patterns. 
They can be placed on pixel positions or character positions when using 
fixed size fonts, using floating point numbers so you can put, for 
example, indent lines in the middle of characters.

They are associated with a buffer, and have a 'source' marker so the 
facility can be shared by several programs.

Any ideas for more elements or applications are welcome.

--
Cecilio Pardo.

Re: Drawing UI elements behind text

[JD Smith]

--

> On Nov 26, 2024, at 6:19 PM, Cecilio Pardo <cpardo@imayhem.com> wrote:
> 
> On 25/11/2024 0:37, JD Smith wrote:
>> I support this idea.  For indent-bars, I use display :stipple as a "separate band" of graphical information that can lie atop text in a buffer, but remain independent of it.  Stipples are flexible and very performant, but have a few issues:
>> - There is inconsistent stipple support across emacs versions.
>> - Stipple patterns begin at the frame origin, so up-to-date window-specific remapping is required to prevent unwanted pattern aliasing.
>> - Aligning stipples vertically requires fixed space content.
> 
> Stipples also disable the scrolling of buffer pixels, affecting performance of redisplay.

On all ports?  I didn’t know this, though haven’t had any related issues. 

>> More commonly, modes also make use of :box styling, e.g. for producing boxed "labels", as in org-modern.
>> An API for drawing simple styled lines, rounded corners, boxes etc. — either in absolute window coordinates, or associated with, but "behind" text in the buffer — would represent a real step forward.
> 
> I haven't found a way to draw behind the text that is not too complicated or affects redisplay too much. So now the idea is drawing over the text. For the applications I have in mind this works equally well.

For mine as well that would work, but features like this are added so rarely it makes sense to make it robust for various uses, if possible.  

> The starting drawing elements are going to be vertical and horizontal segments, with color (maybe with alpha), width and different patterns.

Nice.  Is there a patch to try?  Bit patterns or similar are the most flexible.  

> They can be placed on pixel positions or character positions when using fixed size fonts, using floating point numbers so you can put, for example, indent lines in the middle of characters.

Would they be associated with buffer text and move with it (aka :box), or fixed to the buffer window?  What API did you have in mind?