prettify-symbols-mode, derived modes, and compose-region

prettify-symbols-mode, derived modes, and compose-region

[D]

Hello,

I would like to ask what the general consensus is regarding the way
prettify-symbols-mode (and similar modes) work, given my current use
case.  For context, I am the developer and maintainer of
org-superstar-mode[1], a purely cosmetic minor mode for Org.  It
serves as a "spiritual successor" to the popular org-bullets[2] (of
which I maintain the form used for MELPA[3]).  Both of these minor
modes (ab)use static character composition to achieve their visual
effect, instead of utilizing the display text property.  Recently, the
following comment has been brought to my attention on this subject
recently[4] (by a user named eli-zaretskii):

> I'm sorry, but I don't share your enthusiasm about
> prettify-symbols-mode. It piggy-backs a feature, called "static
> character composition", which was never designed for this job, and
> which is nowadays rarely if ever used for the job it was designed
> for. So it has bit-rotted, and in some not-too-far future we will
> probably want to obsolete and remove the static character composition
> from Emacs, at which point prettify-symbols-mode will be a huge drag.

I empathize with the sentiment, which consequently brings me here to
ask: Should I consider relying on composition "future-proof" or is
this use discouraged?  Because if it is, I may run into an issue.  In
the case of org-bullets, I have no qualms to just rid the code of the
composition hack, as it (to my knowledge) serves no purpose there that
can't trivially be fulfilled by the display property.  Meanwhile,
Superstar has a more "justified" reason to favor composition over
display: The decorations it adds to a buffer are not necessarily
monospaced in a given user's font set.  This can lead to alignment
problems for entries that should be the same level of indentation.
Superstar solves this problem by letting the user superimpose the
decoration character with whitespace.  This way, a user can have an
asterisk display as a <too-thin-char> while keeping alignment by using
the string "\s<too-thin-char>", which is passed to compose-region and
produces the desired result.  Conversely, too wide chars can be
"normalized" by using U2003 (EM SPACE) and similar characters to widen
all affected "bullets" equally.  If static composition were to be
deprecated, I would not be able to provide this feature anymore, and
it's the primary reason I don't think I could avoid the composition
hack in Superstar (given it directly benefits from actual
composition).

Kind regards,
D.

Footnotes:
[1]  https://github.com/integral-dw/org-superstar-mode
[2]  https://github.com/sabof/org-bullets
[3]  https://github.com/integral-dw/org-bullets
[4]
https://www.reddit.com/r/emacs/comments/brt0sk/prettifysymbolsmode_is_so_cool/eomb5av?utm_source=share&utm_medium=web2x&context=3

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> From: D <d.williams@posteo.net>
> Date: Thu, 4 Mar 2021 22:33:01 +0100
> 
> > I'm sorry, but I don't share your enthusiasm about
> > prettify-symbols-mode. It piggy-backs a feature, called "static
> > character composition", which was never designed for this job, and
> > which is nowadays rarely if ever used for the job it was designed
> > for. So it has bit-rotted, and in some not-too-far future we will
> > probably want to obsolete and remove the static character composition
> > from Emacs, at which point prettify-symbols-mode will be a huge drag.
> 
> I empathize with the sentiment, which consequently brings me here to
> ask: Should I consider relying on composition "future-proof" or is
> this use discouraged?

Yes, I'd very much like to discourage the use of static compositions.
Especially in text-related modes such as Org, where human-readable
text is the main content (as opposed to programming-language modes).
That's because static compositions don't support bidirectional editing
and display, and so some of the scripts and languages we support
elsewhere in Emacs are off-limits when static compositions are
involved.  So prettify-symbols-mode is semi-okay, as it is usually
turned on in PL modes, where everything is left-to-right.  But using
the same techniques in Org is from my POV much worse, and I'd be
happier if that could be avoided.

> Because if it is, I may run into an issue.  In
> the case of org-bullets, I have no qualms to just rid the code of the
> composition hack, as it (to my knowledge) serves no purpose there that
> can't trivially be fulfilled by the display property.  Meanwhile,
> Superstar has a more "justified" reason to favor composition over
> display: The decorations it adds to a buffer are not necessarily
> monospaced in a given user's font set.  This can lead to alignment
> problems for entries that should be the same level of indentation.
> Superstar solves this problem by letting the user superimpose the
> decoration character with whitespace.  This way, a user can have an
> asterisk display as a <too-thin-char> while keeping alignment by using
> the string "\s<too-thin-char>", which is passed to compose-region and
> produces the desired result.  Conversely, too wide chars can be
> "normalized" by using U2003 (EM SPACE) and similar characters to widen
> all affected "bullets" equally.  If static composition were to be
> deprecated, I would not be able to provide this feature anymore, and
> it's the primary reason I don't think I could avoid the composition
> hack in Superstar (given it directly benefits from actual
> composition).

I'm not sure I understand the problem, but can't you use :align-to in
display properties to solve the alignment issues?

Thanks.

Re: prettify-symbols-mode, derived modes, and compose-region

[D]

> I'm not sure I understand the problem, but can't you use :align-to in
> display properties to solve the alignment issues?

To provide a concrete example, consider the following (closed) issue:
https://github.com/integral-dw/org-superstar-mode/issues/9

The data structure org-superstar-todo-bullet-alist associates a todo
keyword with a character to be displayed.  The image shows the alignment
problem.  The compose-region solution to this is simply to compose each
character with a wide enough space, so the user simply has to use
"\N{EM SPACE}<character>" instead of ?<character> in the alist.
compose-region will then just superimpose the two and you get a "padded"
character.  It is very much possible that :align-to could fix this as
well, judging from the documentation it does sound like it can reproduce
this feature easily.  To see how well it works out I would likely have
to implement a display-based version first, though.

On a side note: Are there any quirks I should be aware of when
transitioning to the display property? I doubt there will be any
difference in terms of performance (contrast overlays, which my mode
can't afford use), but is there anything else I should be aware of?  I'm
asking in case there have been efforts like mine in the past which I am
not aware of.

Re: prettify-symbols-mode, derived modes, and compose-region

[Stefan Monnier]

> problem.  The compose-region solution to this is simply to compose each
> character with a wide enough space, so the user simply has to use
> "\N{EM SPACE}<character>" instead of ?<character> in the alist.
> compose-region will then just superimpose the two and you get a "padded"
> character.  It is very much possible that :align-to could fix this as
> well, judging from the documentation it does sound like it can reproduce
> this feature easily.

You should definitely try, but I suspect it's not going to work
well: :align-to wants to get an "absolute" position of the position
where the flexible space ends whereas in your case you only know how
wide "char + space" should be, but you don't know at which absolute
column "char" will happen to be displayed, so you can't know to which
column to :align-to.


        Stefan

Re: prettify-symbols-mode, derived modes, and compose-region

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> You should definitely try, but I suspect it's not going to work
> well: [...]

Another thing to be aware of is that the :align-to doesn't adjust when
you change the font size with `text-scale-adjust'.

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 4 Mar 2021 19:20:22 -0800
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > You should definitely try, but I suspect it's not going to work
> > well: [...]
> 
> Another thing to be aware of is that the :align-to doesn't adjust when
> you change the font size with `text-scale-adjust'.

I think it should adjust if you specify the value in columns, not in
pixels.  You can also specify the width of the whitespace stretch in
'width' units, which stands for the canonical width of the current
face (assuming that the current face is itself affected by
text-scale-adjust).

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 04 Mar 2021 21:30:33 -0500
> 
> You should definitely try, but I suspect it's not going to work
> well: :align-to wants to get an "absolute" position of the position
> where the flexible space ends whereas in your case you only know how
> wide "char + space" should be, but you don't know at which absolute
> column "char" will happen to be displayed, so you can't know to which
> column to :align-to.

If the width of the additional space is known, there's also the :width
display property.

Re: prettify-symbols-mode, derived modes, and compose-region

[Stefan Monnier]

>> You should definitely try, but I suspect it's not going to work
>> well: :align-to wants to get an "absolute" position of the position
>> where the flexible space ends whereas in your case you only know how
>> wide "char + space" should be, but you don't know at which absolute
>> column "char" will happen to be displayed, so you can't know to which
>> column to :align-to.
>
> If the width of the additional space is known, there's also the :width
> display property.

Indeed, there's `:width` but in the case under discussion the desired
with of the space is something like `1 - width-of-previous-char`, IOW
something ELisp can't know when it puts the text property.

I think it would take something like a `width` property that is
non-replacing (i.e. can be applied to non-spaces and doesn't hide the
text to which it's applied).


        Stefan

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: d.williams@posteo.net,  emacs-devel@gnu.org
> Date: Fri, 05 Mar 2021 10:14:50 -0500
> 
> > If the width of the additional space is known, there's also the :width
> > display property.
> 
> Indeed, there's `:width` but in the case under discussion the desired
> with of the space is something like `1 - width-of-previous-char`, IOW
> something ELisp can't know when it puts the text property.

Yes, it can: shr.el uses these capabilities.

> I think it would take something like a `width` property that is
> non-replacing (i.e. can be applied to non-spaces and doesn't hide the
> text to which it's applied).

Yes, that's what I had in mind in my other message.

Re: prettify-symbols-mode, derived modes, and compose-region

[Stefan Monnier]

>> > If the width of the additional space is known, there's also the :width
>> > display property.
>> 
>> Indeed, there's `:width` but in the case under discussion the desired
>> with of the space is something like `1 - width-of-previous-char`, IOW
>> something ELisp can't know when it puts the text property.
>
> Yes, it can: shr.el uses these capabilities.

IIRC you can query Emacs about the size the character would have if it
were to be displayed right now in the currently selected window.
But you don't know that it's the same size as the character will have
when it will actually be displayed (and that char could have
simultaneously two different sizes in two different windows, of course).

I guess we could extend the "pixel specification for spaces" so you can
use "the width of this char" or "the width of that string" in your
computation, but it sounds pretty ugly ;-)


        Stefan

Re: prettify-symbols-mode, derived modes, and compose-region

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> IIRC you can query Emacs about the size the character would have if it
> were to be displayed right now in the currently selected window.
> But you don't know that it's the same size as the character will have
> when it will actually be displayed (and that char could have
> simultaneously two different sizes in two different windows, of course).

That's true, but for shr it doesn't (in general) make much difference.
That is, if you display a rendered HTML document on two different
displays, it'll look awful anyway, until you re-render tables and the
like with the new width of the glyphs.  That is, a layout with

| foo bar | more foo |
| zot     |          |

rarely looks on two different displays -- even if the :width/:align-to
elements were to magically adjust themselves.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: D <d.williams@posteo.net>
> Date: Fri, 5 Mar 2021 01:22:30 +0100
> 
> The data structure org-superstar-todo-bullet-alist associates a todo
> keyword with a character to be displayed.  The image shows the alignment
> problem.  The compose-region solution to this is simply to compose each
> character with a wide enough space, so the user simply has to use
> "\N{EM SPACE}<character>" instead of ?<character> in the alist.
> compose-region will then just superimpose the two and you get a "padded"
> character.

But how do you know that no character associated with a todo keyword
could ever be wider than the EM SPACE in the same font?  That's
entirely up to the font designer.  And that's even before you consider
complex text shaping which could combine several codepoints into a
single wide grapheme cluster.

IOW, I think this solution is fragile.

Anyway, I think it should be very easy to add a new display property
type that would override the pixel width of a font glyph with a fixed
value calculated in some way by a Lisp program.

> On a side note: Are there any quirks I should be aware of when
> transitioning to the display property? I doubt there will be any
> difference in terms of performance (contrast overlays, which my mode
> can't afford use), but is there anything else I should be aware of?  I'm
> asking in case there have been efforts like mine in the past which I am
> not aware of.

This question is too general, and the only answer I can give to such a
question is YES.  I suggest to try that and bring up any issues you
have, then we could talk specifically about those issues.

Re: prettify-symbols-mode, derived modes, and compose-region

[Clément Pit-Claudel]

On 3/5/21 2:08 AM, Eli Zaretskii wrote:
> Anyway, I think it should be very easy to add a new display property
> type that would override the pixel width of a font glyph with a fixed
> value calculated in some way by a Lisp program.

Some previous discussion about this: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38485

Re: prettify-symbols-mode, derived modes, and compose-region

[D]

> But how do you know that no character associated with a todo keyword
> could ever be wider than the EM SPACE in the same font?  That's
> entirely up to the font designer.
> IOW, I think this solution is fragile.

In theory, yes. In practice this has (to my knowledge) not cropped up.
My approach is, however, a hack at the end of the day.  I mean, the
reason I acknowledge is that why I'm here :)

> And that's even before you consider
> complex text shaping which could combine several codepoints into a
> single wide grapheme cluster.

I don't think that using a character capable of that would be of any
use as a decorative element.  I'd say decorating a title or
representing an item bullet is largely the domain of dingbats,
geometric shapes, and to a lesser extent pictograms ("Emoji").

> Anyway, I think it should be very easy to add a new display property
> type that would override the pixel width of a font glyph with a fixed
> value calculated in some way by a Lisp program.

That would of course be the perfect solution.  But, doesn't this have
to exist in one way or another already?  For example, the checkbox
widget in custom interfaces to my knowledge visually replaces a simple
"[X]" with a bitmap.  When I examine a checkbox with C-u C-x = it
seems to be the case that display does the legwork and somehow obtains
the image size from the xpm file (I suppose).

(On an unrelated side note, I adore the custom interface.  Especially
the way you can specify complex composite types and tweak the
interface to be more expressive, and how you can safety check values
using :set)

> I suggest to try that and bring up any issues you
> have, then we could talk specifically about those issues.

Fair.

>> IIRC you can query Emacs about the size the character would have if it
>> were to be displayed right now in the currently selected window.
>> But you don't know that it's the same size as the character will have
>> when it will actually be displayed (and that char could have
>> simultaneously two different sizes in two different windows, of course).
>
> That's true, but for shr it doesn't (in general) make much difference.
> That is, if you display a rendered HTML document on two different
> displays, it'll look awful anyway, until you re-render tables and the
> like with the new width of the glyphs.  That is, a layout with
>
> | foo bar | more foo |
> | zot     |          |
>
> rarely looks on two different displays -- even if the :width/:align-to
> elements were to magically adjust themselves.

Same with superstar, essentially.  For example, every decorative
bullet can have a "fallback character" for terminal displays, and this
behaves correctly from the perspective of the frame you opened the
buffer with.  I include a simple redisplay command for emacsclient
users for this reason.  So I suppose I operate under the same
limitations as shr?

In that case, I probably should look into that character size
function.  Could you point me in the right general direction?  I'm not
quite sure how to find it.

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> From: D <d.williams@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Fri, 5 Mar 2021 22:24:44 +0100
> 
> > And that's even before you consider
> > complex text shaping which could combine several codepoints into a
> > single wide grapheme cluster.
> 
> I don't think that using a character capable of that would be of any
> use as a decorative element.

I was talking about the need to align the "decorative elements" with
text that includes such grapheme clusters.

> > Anyway, I think it should be very easy to add a new display property
> > type that would override the pixel width of a font glyph with a fixed
> > value calculated in some way by a Lisp program.
> 
> That would of course be the perfect solution.  But, doesn't this have
> to exist in one way or another already?  For example, the checkbox
> widget in custom interfaces to my knowledge visually replaces a simple
> "[X]" with a bitmap.  When I examine a checkbox with C-u C-x = it
> seems to be the case that display does the legwork and somehow obtains
> the image size from the xpm file (I suppose).

In your original use case you didn't have images, you had characters.
Or did I misunderstand?

> Same with superstar, essentially.  For example, every decorative
> bullet can have a "fallback character" for terminal displays, and this
> behaves correctly from the perspective of the frame you opened the
> buffer with.  I include a simple redisplay command for emacsclient
> users for this reason.  So I suppose I operate under the same
> limitations as shr?

Yes, I think so.  And on text terminals this issue is moot anyway,
because every character takes a single column (if you forget about
those rare ones that take 2).

> In that case, I probably should look into that character size
> function.  Could you point me in the right general direction?  I'm not
> quite sure how to find it.

shr-fill-line, I guess.

You can also try using font-get-glyphs, it might be more suitable for
your needs.  I think shr.el doesn't use it because that turned out to
be slower than the current approach, but your use case is different.

Re: prettify-symbols-mode, derived modes, and compose-region

[D]

> In your original use case you didn't have images, you had characters.
> Or did I misunderstand?

That's correct, my point was whether there was any common underlying
logic one could make use of.

> Yes, I think so.  And on text terminals this issue is moot anyway,
> because every character takes a single column (if you forget about
> those rare ones that take 2).

Exactly, it's the straightforward case.

> shr-fill-line, I guess.
> 
> You can also try using font-get-glyphs, it might be more suitable for
> your needs.  I think shr.el doesn't use it because that turned out to
> be slower than the current approach, but your use case is different.

I'll look into both, thank you!

Re: prettify-symbols-mode, derived modes, and compose-region

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, larsi@gnus.org, emacs-devel@gnu.org
> From: D <d.williams@posteo.net>
> Date: Sat, 6 Mar 2021 11:43:45 +0100
> 
> > In your original use case you didn't have images, you had characters.
> > Or did I misunderstand?
> 
> That's correct, my point was whether there was any common underlying
> logic one could make use of.

Calculating the size of images is different from that of character
glyphs.  So the general logic may be similar, but the implementation
is very different.