Supporting stylistic sets

Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

Hi,
I was trying to add stylistic sets in Emacs.
I was able to successfully able to switch to a different set of a Bengali
font by hardcoding it  to the hbfont.c file, but that is obviously not a
solution.

Should I try adding a new font property name "stylistic-set" which would
accept a list of values for a font. If so could I get some pointers on how
to add a new font property to emacs.

Thanks.

[-- Attachment #2: Type: text/html, Size: 517 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh
>  <lumarzeli30@gmail.com>
> Date: Fri, 23 Sep 2022 18:24:48 +0530
> 
> I was trying to add stylistic sets in Emacs.

Thanks.  We have a TODO item for that, please see it.

> I was able to successfully able to switch to a different set of a Bengali font by hardcoding it  to the hbfont.c
> file, but that is obviously not a solution.
> 
> Should I try adding a new font property name "stylistic-set" which would accept a list of values for a font. If so
> could I get some pointers on how to add a new font property to emacs.

How did you envision this font property to be used in Emacs?

The etc/TODO item for that suggests to implement this as a text
property instead.  That is, of course, not carved in stone, but I
think we should in any case decide on how this will be used from Lisp
before we talk about how to expose it to Lisp.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

>
> > I was trying to add stylistic sets in Emacs.
>
> Thanks.  We have a TODO item for that, please see it.
>

Yes I had read it and also the mailing list discussion linked there.

How did you envision this font property to be used in Emacs?
>
> The etc/TODO item for that suggests to implement this as a text
> property instead.  That is, of course, not carved in stone, but I
> think we should in any case decide on how this will be used from Lisp
> before we talk about how to expose it to Lisp.
>

I imagined that it would be something like this:


  (set-fontset-font t 'bengali
   (font-spec :family "Tiro Bangla"
      :stylistic-set '("ss03")))


  (set-face-attribute 'default nil
     :font "Fira Code"
     :weight 'regular
     :height 170
     :stylistic-set '("cv01" "ss05" "ss03"))

Just like how weight, foundry, size etc are set and they seem to be font
properties.

On Fri, Sep 23, 2022 at 9:07 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh
> >  <lumarzeli30@gmail.com>
> > Date: Fri, 23 Sep 2022 18:24:48 +0530
> >
> > I was trying to add stylistic sets in Emacs.
>
> Thanks.  We have a TODO item for that, please see it.
>
> > I was able to successfully able to switch to a different set of a
> Bengali font by hardcoding it  to the hbfont.c
> > file, but that is obviously not a solution.
> >
> > Should I try adding a new font property name "stylistic-set" which would
> accept a list of values for a font. If so
> > could I get some pointers on how to add a new font property to emacs.
>
> How did you envision this font property to be used in Emacs?
>
> The etc/TODO item for that suggests to implement this as a text
> property instead.  That is, of course, not carved in stone, but I
> think we should in any case decide on how this will be used from Lisp
> before we talk about how to expose it to Lisp.
>

[-- Attachment #2: Type: text/html, Size: 3025 bytes --]

Re: Supporting stylistic sets

[Nicolas Ouellet-payeur]

> How did you envision this font property to be used in Emacs?
>
> The etc/TODO item for that suggests to implement this as a text
> property instead.  That is, of course, not carved in stone, but I
> think we should in any case decide on how this will be used from Lisp
> before we talk about how to expose it to Lisp.

Personally, I think a font property is easier to configure. I can use
customize or set-face-attribute to set how I want a font to look.
Ideally, I would like to do something like this:

  (set-face-attribute
   'default nil
   :stylistic-sets '("cv01" "cv02" "cv03"))

This uses my chosen stylistic sets everywhere with a single line of
Lisp. If I don't like having it *everywhere* (e.g. minibuffer,
mode-line), I can still set it separately for each face.

With a text property, I'd have to hook into everything that displays
text, somehow, and add the text property there.

> Thanks.  We have a TODO item for that, please see it.

Also, while we're on this topic: I'm working on a patch to pass *all*
strings/buffer contents to hbfont_shape() during redisplay, and making
all text a composition. That way we could have stylistic sets for Latin
scripts as well (and most other scripts), not just "composed" scripts
like Bengali and Arabic. It'd also achieve "Support ligatures out of the
box" from etc/TODO, by giving HarfBuzz the means to shape text properly.

This would be nice for fonts like Fira Code [1], which has both
ligatures and stylistic sets.

My patch is very experimental (i.e. hacky) at this point though, and it
crashes all the time. I don't want to derail this discussion, so I'll
share my work in another thread once it stops crashing so much.

[1] https://github.com/tonsky/FiraCode

On Fri, Sep 23, 2022 at 1:29 PM समीर सिंह Sameer Singh
<lumarzeli30@gmail.com> wrote:
>>
>> > I was trying to add stylistic sets in Emacs.
>>
>> Thanks.  We have a TODO item for that, please see it.
>
>
> Yes I had read it and also the mailing list discussion linked there.
>
>> How did you envision this font property to be used in Emacs?
>>
>> The etc/TODO item for that suggests to implement this as a text
>> property instead.  That is, of course, not carved in stone, but I
>> think we should in any case decide on how this will be used from Lisp
>> before we talk about how to expose it to Lisp.
>
>
> I imagined that it would be something like this:
>
>
>   (set-fontset-font t 'bengali
>    (font-spec :family "Tiro Bangla"
>       :stylistic-set '("ss03")))
>
>
>   (set-face-attribute 'default nil
>      :font "Fira Code"
>      :weight 'regular
>      :height 170
>      :stylistic-set '("cv01" "ss05" "ss03"))
>
> Just like how weight, foundry, size etc are set and they seem to be font properties.
>
> On Fri, Sep 23, 2022 at 9:07 PM Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> > From: समीर सिंह Sameer Singh
>> >  <lumarzeli30@gmail.com>
>> > Date: Fri, 23 Sep 2022 18:24:48 +0530
>> >
>> > I was trying to add stylistic sets in Emacs.
>>
>> Thanks.  We have a TODO item for that, please see it.
>>
>> > I was able to successfully able to switch to a different set of a Bengali font by hardcoding it  to the hbfont.c
>> > file, but that is obviously not a solution.
>> >
>> > Should I try adding a new font property name "stylistic-set" which would accept a list of values for a font. If so
>> > could I get some pointers on how to add a new font property to emacs.
>>
>> How did you envision this font property to be used in Emacs?
>>
>> The etc/TODO item for that suggests to implement this as a text
>> property instead.  That is, of course, not carved in stone, but I
>> think we should in any case decide on how this will be used from Lisp
>> before we talk about how to expose it to Lisp.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Fri, 23 Sep 2022 22:57:41 +0530
> Cc: emacs-devel@gnu.org
> 
>  How did you envision this font property to be used in Emacs?
> 
>  The etc/TODO item for that suggests to implement this as a text
>  property instead.  That is, of course, not carved in stone, but I
>  think we should in any case decide on how this will be used from Lisp
>  before we talk about how to expose it to Lisp.
> 
> I imagined that it would be something like this:
> 
>   (set-fontset-font t 'bengali
>    (font-spec :family "Tiro Bangla"
>       :stylistic-set '("ss03")))
> 
>   (set-face-attribute 'default nil
>      :font "Fira Code"
>      :weight 'regular
>      :height 170
>      :stylistic-set '("cv01" "ss05" "ss03"))
> 
> Just like how weight, foundry, size etc are set and they seem to be font properties.

That means all text that uses the font will use that stylistic-set.
Is that good enough?  What if you want to have the same font with and
without a specific stylistic-set in the same document?.

And if the font is not the default face's font, it means you need to
setup a special face for using it anyway.

So I think a font attribute is not flexible enough for Emacs.  But I'm
interested in what others think.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Nicolas Ouellet-payeur <nicolaso@google.com>
> Date: Fri, 23 Sep 2022 13:31:30 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > How did you envision this font property to be used in Emacs?
> >
> > The etc/TODO item for that suggests to implement this as a text
> > property instead.  That is, of course, not carved in stone, but I
> > think we should in any case decide on how this will be used from Lisp
> > before we talk about how to expose it to Lisp.
> 
> Personally, I think a font property is easier to configure. I can use
> customize or set-face-attribute to set how I want a font to look.
> Ideally, I would like to do something like this:
> 
>   (set-face-attribute
>    'default nil
>    :stylistic-sets '("cv01" "cv02" "cv03"))

This is not a font property, this is a face property.  It will be in
effect for every font used for the default face, including fonts used
for non-ASCII characters, like CJK and Emoji.  Are you sure this is
what you want?

I'm not against adding a face property, I just think it isn't enough,
even if that's what people will want.

Also, AFAIU this feature is meant for special styling of select text
segments, not for the entire buffer.

> This uses my chosen stylistic sets everywhere with a single line of
> Lisp. If I don't like having it *everywhere* (e.g. minibuffer,
> mode-line), I can still set it separately for each face.

That'd be very tedious, since Emacs uses so many different faces.

> With a text property, I'd have to hook into everything that displays
> text, somehow, and add the text property there.

Yes.

Maybe we should talk about higher-level use cases: when and why would
one want to use stylistic-sets in Emacs?

> Also, while we're on this topic: I'm working on a patch to pass *all*
> strings/buffer contents to hbfont_shape() during redisplay, and making
> all text a composition. That way we could have stylistic sets for Latin
> scripts as well (and most other scripts), not just "composed" scripts
> like Bengali and Arabic. It'd also achieve "Support ligatures out of the
> box" from etc/TODO, by giving HarfBuzz the means to shape text properly.

I think you will find out that this makes Emacs redisplay unbearably
slow.  IMO, it is impractical to shape everything via a shaping engine
without completely redesigning how we handle character compositions in
the display engine, because what we have now was not designed to be
used for all the text we display.  And the main difficulty with such a
redesign is to do it in a way that still allows easy customization of
composition rules from Lisp.

> This would be nice for fonts like Fira Code [1], which has both
> ligatures and stylistic sets.

You can have ligatures today without passing everything through the
shaping engine.  The number of character combinations that ligate is
not very large, so this is a far cry from shaping everything.  The
only known problem with using ligatures in current Emacs codebase is
that some of the character sequences you'd like to ligate happen in
places where you don't want that, such as on the mode line.  So we
need some way of preventing ligation in certain contexts.

Re: Supporting stylistic sets

[Visuwesh]

[வெள்ளி செப்டம்பர் 23, 2022] Eli Zaretskii wrote:

>> I imagined that it would be something like this:
>> 
>>   (set-fontset-font t 'bengali
>>    (font-spec :family "Tiro Bangla"
>>       :stylistic-set '("ss03")))
>> 
>>   (set-face-attribute 'default nil
>>      :font "Fira Code"
>>      :weight 'regular
>>      :height 170
>>      :stylistic-set '("cv01" "ss05" "ss03"))
>> 
>> Just like how weight, foundry, size etc are set and they seem to be font properties.
>
> That means all text that uses the font will use that stylistic-set.
> Is that good enough?  What if you want to have the same font with and
> without a specific stylistic-set in the same document?.

Unless there are performance implications, I don't see why using an
anonymous face won't work i.e., 

    (propertize "text" '(face (:stylistic-set ("cv01" ..)))

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: समीर सिंह Sameer Singh
>  <lumarzeli30@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 23 Sep 2022 23:56:58 +0530
> 
> [வெள்ளி செப்டம்பர் 23, 2022] Eli Zaretskii wrote:
> 
> >> I imagined that it would be something like this:
> >> 
> >>   (set-fontset-font t 'bengali
> >>    (font-spec :family "Tiro Bangla"
> >>       :stylistic-set '("ss03")))
> >> 
> >>   (set-face-attribute 'default nil
> >>      :font "Fira Code"
> >>      :weight 'regular
> >>      :height 170
> >>      :stylistic-set '("cv01" "ss05" "ss03"))
> >> 
> >> Just like how weight, foundry, size etc are set and they seem to be font properties.
> >
> > That means all text that uses the font will use that stylistic-set.
> > Is that good enough?  What if you want to have the same font with and
> > without a specific stylistic-set in the same document?.
> 
> Unless there are performance implications, I don't see why using an
> anonymous face won't work i.e., 
> 
>     (propertize "text" '(face (:stylistic-set ("cv01" ..)))

How is this different or better than having a special text property?

Re: Supporting stylistic sets

[Nicolas Ouellet-payeur]

> From: Nicolas Ouellet-payeur <nicolaso@google.com>
> Date: Fri, 23 Sep 2022 13:31:30 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org

> This is not a font property, this is a face property.  It will be in
> effect for every font used for the default face, including fonts used
> for non-ASCII characters, like CJK and Emoji.  Are you sure this is
> what you want?

Thanks for clearing up the confusion RE: font vs. face properites.

> I'm not against adding a face property, I just think it isn't enough,
> even if that's what people will want.
>
> Also, AFAIU this feature is meant for special styling of select text
> segments, not for the entire buffer.

Ah, that's a good point. I hadn't considered what happens to different
scripts if, say, the 'ss03' variant means a different thing in each
font.

There's no reason it *can't* be for the entire buffer though.
See below.

> > This uses my chosen stylistic sets everywhere with a single line of
> > Lisp. If I don't like having it *everywhere* (e.g. minibuffer,
> > mode-line), I can still set it separately for each face.
>
> That'd be very tedious, since Emacs uses so many different faces.

Well yes, but what's the alternative? And in the specific use-case of

> > With a text property, I'd have to hook into everything that displays
> > text, somehow, and add the text property there.
>
> Yes.
>
> Maybe we should talk about higher-level use cases: when and why would
> one want to use stylistic-sets in Emacs?

In my case specific: because the 'cv31' variant of my font changes the
shape of parentheses, and I like that shape better. I would rather set
it everywhere, even the minibuffer, mode-line, header-line.

This is all just ASCII though. And ligatures, which might be weirder.
IIUC, Sameer's suggestion with set-fontset-font would address this
use-case.

  (set-fontset-font t 'ascii
   (font-spec
    :family "Fira Code"
    :stylistic-set '("cv31")))

> > Also, while we're on this topic: I'm working on a patch to pass *all*
> > strings/buffer contents to hbfont_shape() during redisplay, and making
> > all text a composition. That way we could have stylistic sets for Latin
> > scripts as well (and most other scripts), not just "composed" scripts
> > like Bengali and Arabic. It'd also achieve "Support ligatures out of the
> > box" from etc/TODO, by giving HarfBuzz the means to shape text properly.
>
> I think you will find out that this makes Emacs redisplay unbearably
> slow.
>
> IMO, it is impractical to shape everything via a shaping engine
> without completely redesigning how we handle character compositions in
> the display engine, because what we have now was not designed to be
> used for all the text we display.

That's one of my big worries, as well. That's why I'm anxious to share
a proof-of-concept ASAP, and see if it's worth pursuing.

Initially it *was* super janky, because I did the naïve thing and passed
strings through hbfont_shape() without caching the result. Each
redisplay would pass the strings to HarfBuzz again, and create a bunch
of new Lisp vectors... Then the GC would go crazy, and cause a *lot* of
jank.

I then tried to put everything into `gstring_hash_table' in composite.c.
That made things better, but the hashtable would keep growing in size
all the time. Basically leaking memory on purpose...

My latest attempt is to tweak `region_cache' so it can store arbitrary
Lisp_Objects, rather than just 1s or 0s for known-ness of regions. Then
I stick the lgstrings into that cache. Different frames have different
face properties, so there's one region_cache per (frame,buffer) pair.
This works, but right now it's incomplete and crashes all the time like
I said.

It's not *noticeably* jankier so far, and I even used the
`gstring_hash_table' version as my main editor for a while. But maybe my
computer is just fast enough, or my use-cases are too simple. It's
*definitely* way too slow for very long lines. For example, a JSON file
that's a single line and multiple megabytes. We'd need to fall back to
the existing code in cases like that.

I haven't tried running it through `gprof' yet, and this is all
subjective. So I'm still worried about performance.

> You can have ligatures today without passing everything through the
> shaping engine.

Yes, but you need to configure those manually. Here, the advantage of
passing them to HarfBuzz is that it can read those ligatures from the
font file without any extra configuration. i.e., "Support ligatures out
of the box" as described in etc/TODO. That's not going to change the
world, but if we can make it work it'll be neat.

> And the main difficulty with such a redesign is to do it in a way that
> still allows easy customization of composition rules from Lisp.

A quick test shows that `prettify-symbols-mode' still works, but I'm
sure there's subtle bugs hiding underneath.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Nicolas Ouellet-payeur <nicolaso@google.com>
> Date: Fri, 23 Sep 2022 15:20:54 -0400
> Cc: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>, 
> 	emacs-devel@gnu.org
> 
> In my case specific: because the 'cv31' variant of my font changes the
> shape of parentheses, and I like that shape better. I would rather set
> it everywhere, even the minibuffer, mode-line, header-line.

You could do something similar to what jit-lock does: place a text
property or an overlay on each parenthesis that is going to be
displayed.

> > IMO, it is impractical to shape everything via a shaping engine
> > without completely redesigning how we handle character compositions in
> > the display engine, because what we have now was not designed to be
> > used for all the text we display.
> 
> That's one of my big worries, as well. That's why I'm anxious to share
> a proof-of-concept ASAP, and see if it's worth pursuing.
> 
> Initially it *was* super janky, because I did the naïve thing and passed
> strings through hbfont_shape() without caching the result. Each
> redisplay would pass the strings to HarfBuzz again, and create a bunch
> of new Lisp vectors... Then the GC would go crazy, and cause a *lot* of
> jank.
> 
> I then tried to put everything into `gstring_hash_table' in composite.c.
> That made things better, but the hashtable would keep growing in size
> all the time. Basically leaking memory on purpose...
> 
> My latest attempt is to tweak `region_cache' so it can store arbitrary
> Lisp_Objects, rather than just 1s or 0s for known-ness of regions. Then
> I stick the lgstrings into that cache. Different frames have different
> face properties, so there's one region_cache per (frame,buffer) pair.
> This works, but right now it's incomplete and crashes all the time like
> I said.
> 
> It's not *noticeably* jankier so far, and I even used the
> `gstring_hash_table' version as my main editor for a while. But maybe my
> computer is just fast enough, or my use-cases are too simple. It's
> *definitely* way too slow for very long lines. For example, a JSON file
> that's a single line and multiple megabytes. We'd need to fall back to
> the existing code in cases like that.

To measure the slowdown, run the benchmark of scrolling through a
large file, like xdisp.c, one line at a time, and compare with the
current code.

I think one other source of slowdown is that the glyph production is
interrupted too frequently.  This is something unrelated to caching
the compositions.

> > You can have ligatures today without passing everything through the
> > shaping engine.
> 
> Yes, but you need to configure those manually. Here, the advantage of
> passing them to HarfBuzz is that it can read those ligatures from the
> font file without any extra configuration. i.e., "Support ligatures out
> of the box" as described in etc/TODO. That's not going to change the
> world, but if we can make it work it'll be neat.

But what do you do if the user wants to disable ligation, or control
which ligatures should and shouldn't happen?  The fact that a font
that supports ligatures is used in some buffer (or even the default
face) doesn't yet mean the user wants these ligatures everywhere.  As
a simple example, consider stuff like := or **-, which is sometimes
shown on the mode line -- we don't want ligation there, do we?  If you
leave it to the font to decide when to ligate, Emacs users lose
control of that.  That might be OK in the lies of VSCode, but not in
Emacs.

> > And the main difficulty with such a redesign is to do it in a way that
> > still allows easy customization of composition rules from Lisp.
> 
> A quick test shows that `prettify-symbols-mode' still works, but I'm
> sure there's subtle bugs hiding underneath.

prettify-symbols-mode uses static compositions, not automatic
compositions, so it's little wonder prettify-symbols-mode isn't
affected: it uses completely different code paths.

I was talking about the way we control automatic compositions through
composition-function-table.  This gives us all the freedom and
convenience to control compositions, but it comes with a price: we
need to call into Lisp (which then calls back into C) to produce
glyphs, and we need to match the text against regexps to know what
sequences can and cannot be composed.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

>
> That means all text that uses the font will use that stylistic-set.
> Is that good enough?  What if you want to have the same font with and
> without a specific stylistic-set in the same document?.
>

Isn't the point of stylistic-set is that if you don't like a particular
glyph of a font you can interchange it for another, therefore I don't
see why someone would have the same font with and without the stylistic-set
in the same document.
For example: Tiro Bangla had set the stylistic set because the users could
not agree how the হ্+ন and হ্+ণ conjuncts should look[1].
It is the case for FiraCode too, simple variations of glyphs which the user
can switch out according to their preference.

And if the font is not the default face's font, it means you need to
> setup a special face for using it anyway.
>

You mean for scripts like Bangla, Devanagari etc?
For them defining their font-spec wouldn't work?

[1]: https://github.com/TiroTypeworks/Indigo/issues/30

On Fri, Sep 23, 2022 at 11:25 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Fri, 23 Sep 2022 22:57:41 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  How did you envision this font property to be used in Emacs?
> >
> >  The etc/TODO item for that suggests to implement this as a text
> >  property instead.  That is, of course, not carved in stone, but I
> >  think we should in any case decide on how this will be used from Lisp
> >  before we talk about how to expose it to Lisp.
> >
> > I imagined that it would be something like this:
> >
> >   (set-fontset-font t 'bengali
> >    (font-spec :family "Tiro Bangla"
> >       :stylistic-set '("ss03")))
> >
> >   (set-face-attribute 'default nil
> >      :font "Fira Code"
> >      :weight 'regular
> >      :height 170
> >      :stylistic-set '("cv01" "ss05" "ss03"))
> >
> > Just like how weight, foundry, size etc are set and they seem to be font
> properties.
>
> That means all text that uses the font will use that stylistic-set.
> Is that good enough?  What if you want to have the same font with and
> without a specific stylistic-set in the same document?.
>
> And if the font is not the default face's font, it means you need to
> setup a special face for using it anyway.
>
> So I think a font attribute is not flexible enough for Emacs.  But I'm
> interested in what others think.
>

[-- Attachment #2: Type: text/html, Size: 3592 bytes --]

Re: Supporting stylistic sets

[Visuwesh]

[வெள்ளி செப்டம்பர் 23, 2022] Eli Zaretskii wrote:

>> Unless there are performance implications, I don't see why using an
>> anonymous face won't work i.e., 
>> 
>>     (propertize "text" '(face (:stylistic-set ("cv01" ..)))
>
> How is this different or better than having a special text property?

It has the added advantage of being able to easily alter existing code
to use stylistic sets.  E.g., if I want outline headings to use the
small caps set, then I can change the relevant faces which is much
simpler than writing new code to put the special text property.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: lumarzeli30@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 24 Sep 2022 10:29:12 +0530
> 
> [வெள்ளி செப்டம்பர் 23, 2022] Eli Zaretskii wrote:
> 
> >> Unless there are performance implications, I don't see why using an
> >> anonymous face won't work i.e., 
> >> 
> >>     (propertize "text" '(face (:stylistic-set ("cv01" ..)))
> >
> > How is this different or better than having a special text property?
> 
> It has the added advantage of being able to easily alter existing code
> to use stylistic sets.  E.g., if I want outline headings to use the
> small caps set, then I can change the relevant faces which is much
> simpler than writing new code to put the special text property.

I'm not talking about changing the existing faces, I'm talking about
the above proposal to use propertize.  That is exactly equivalent to
putting a named text property on the text, isn't it?  IOW, the text
property you use to request stylistic variants doesn't have to be
'face', it can be some other property specifically designed for this
purpose.

Using faces for this introduces complications and performance aspects
that we had better avoided, unless we really must have this as part of
a face.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 10:13:47 +0530
> Cc: emacs-devel@gnu.org
> 
>  That means all text that uses the font will use that stylistic-set.
>  Is that good enough?  What if you want to have the same font with and
>  without a specific stylistic-set in the same document?.
> 
> Isn't the point of stylistic-set is that if you don't like a particular glyph of a font you can interchange it for
> another, therefore I don't
> see why someone would have the same font with and without the stylistic-set in the same document.

You are thinking about one particular use case.  There might be
others.

> For example: Tiro Bangla had set the stylistic set because the users could not agree how the হ্+ন and হ্+ণ
> conjuncts should look[1].
> It is the case for FiraCode too, simple variations of glyphs which the user can switch out according to their
> preference.

If we want to support the use case of "just change the appearance of a
particular character", then a much simpler implementation would be to
use the existing character-composition machinery.  Specifically:

  . add a suitable entry to composition-function-table, with a regexp
    that matches the characters you want to affect
  . provide a new variable that can be used to specify which, if any,
    stylistic feature to use for each character (this could be a
    char-table, for example)
  . modify hbfont_shape to consult the above variable and apply the
    requested features to the affected character(s)

Btw, is it always the case that, when using a particular font, users
will want to apply the same stylistic-set feature to all of the
characters from the font that have alternative glyphs?  If not, then
font-level feature sets are not going to solve that, and we will need
to be able to specify this for each character individually.

>  And if the font is not the default face's font, it means you need to
>  setup a special face for using it anyway.
> 
> You mean for scripts like Bangla, Devanagari etc?
> For them defining their font-spec wouldn't work?

I meant that the way to use a special font in Emacs, for ASCII
characters, is to define a special face that uses that font.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]

>
> You are thinking about one particular use case.  There might be
> others.


What might be the other use cases?

Btw, is it always the case that, when using a particular font, users
> will want to apply the same stylistic-set feature to all of the
> characters from the font that have alternative glyphs?  If not, then
> font-level feature sets are not going to solve that, and we will need
> to be able to specify this for each character individually.
>

There are different stylistic-set features for different glyphs.
For example in FiraCode to change "a" there is cv01, for "g" cv02 etc.
or am I misunderstanding the question? (most likely)

On Sat, Sep 24, 2022 at 12:25 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 10:13:47 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  That means all text that uses the font will use that stylistic-set.
> >  Is that good enough?  What if you want to have the same font with and
> >  without a specific stylistic-set in the same document?.
> >
> > Isn't the point of stylistic-set is that if you don't like a particular
> glyph of a font you can interchange it for
> > another, therefore I don't
> > see why someone would have the same font with and without the
> stylistic-set in the same document.
>
> You are thinking about one particular use case.  There might be
> others.
>
> > For example: Tiro Bangla had set the stylistic set because the users
> could not agree how the হ্+ন and হ্+ণ
> > conjuncts should look[1].
> > It is the case for FiraCode too, simple variations of glyphs which the
> user can switch out according to their
> > preference.
>
> If we want to support the use case of "just change the appearance of a
> particular character", then a much simpler implementation would be to
> use the existing character-composition machinery.  Specifically:
>
>   . add a suitable entry to composition-function-table, with a regexp
>     that matches the characters you want to affect
>   . provide a new variable that can be used to specify which, if any,
>     stylistic feature to use for each character (this could be a
>     char-table, for example)
>   . modify hbfont_shape to consult the above variable and apply the
>     requested features to the affected character(s)
>
> Btw, is it always the case that, when using a particular font, users
> will want to apply the same stylistic-set feature to all of the
> characters from the font that have alternative glyphs?  If not, then
> font-level feature sets are not going to solve that, and we will need
> to be able to specify this for each character individually.
>
> >  And if the font is not the default face's font, it means you need to
> >  setup a special face for using it anyway.
> >
> > You mean for scripts like Bangla, Devanagari etc?
> > For them defining their font-spec wouldn't work?
>
> I meant that the way to use a special font in Emacs, for ASCII
> characters, is to define a special face that uses that font.
>

[-- Attachment #2: Type: text/html, Size: 4120 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 13:30:00 +0530
> Cc: emacs-devel@gnu.org
> 
>  You are thinking about one particular use case.  There might be
>  others.
> 
> What might be the other use cases? 

I don't really know.  That was a comment asking for responses from
anyone who could know.

>  Btw, is it always the case that, when using a particular font, users
>  will want to apply the same stylistic-set feature to all of the
>  characters from the font that have alternative glyphs?  If not, then
>  font-level feature sets are not going to solve that, and we will need
>  to be able to specify this for each character individually.
> 
> There are different stylistic-set features for different glyphs.
> For example in FiraCode to change "a" there is cv01, for "g" cv02 etc.

Then font-level specification of this would not be as simple as was
originally suggested, I think: the characters were missing from that
suggestion.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

>
> Then font-level specification of this would not be as simple as was
> originally suggested, I think: the characters were missing from that
> suggestion.


We don't have to specify each character for which we want the
stylistic-set, we just simply would have to pass the tags
and hb_shape_full would take care of the rest.

On Sat, Sep 24, 2022 at 1:57 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 13:30:00 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  You are thinking about one particular use case.  There might be
> >  others.
> >
> > What might be the other use cases?
>
> I don't really know.  That was a comment asking for responses from
> anyone who could know.
>
> >  Btw, is it always the case that, when using a particular font, users
> >  will want to apply the same stylistic-set feature to all of the
> >  characters from the font that have alternative glyphs?  If not, then
> >  font-level feature sets are not going to solve that, and we will need
> >  to be able to specify this for each character individually.
> >
> > There are different stylistic-set features for different glyphs.
> > For example in FiraCode to change "a" there is cv01, for "g" cv02 etc.
>
> Then font-level specification of this would not be as simple as was
> originally suggested, I think: the characters were missing from that
> suggestion.
>

[-- Attachment #2: Type: text/html, Size: 2161 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 14:04:42 +0530
> Cc: emacs-devel@gnu.org
> 
>  Then font-level specification of this would not be as simple as was
>  originally suggested, I think: the characters were missing from that
>  suggestion.
> 
> We don't have to specify each character for which we want the stylistic-set, we just simply would have to
> pass the tags
> and hb_shape_full would take care of the rest. 

I don't understand: how does this handle the case where a certain
character has more than 2 alternative glyphs? which alternative will
be selected?

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

>
> I don't understand: how does this handle the case where a certain
> character has more than 2 alternative glyphs? which alternative will
> be selected?


Different glyphs would have different tags.
For example in FiraCode "0" has four alternatives and each one has a
different tag:
zero, cv11, cv12, cv13

On Sat, Sep 24, 2022 at 2:09 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 14:04:42 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  Then font-level specification of this would not be as simple as was
> >  originally suggested, I think: the characters were missing from that
> >  suggestion.
> >
> > We don't have to specify each character for which we want the
> stylistic-set, we just simply would have to
> > pass the tags
> > and hb_shape_full would take care of the rest.
>
> I don't understand: how does this handle the case where a certain
> character has more than 2 alternative glyphs? which alternative will
> be selected?
>

[-- Attachment #2: Type: text/html, Size: 1714 bytes --]

Re: Supporting stylistic sets

[Visuwesh]

[சனி செப்டம்பர் 24, 2022] Eli Zaretskii wrote:

> I'm not talking about changing the existing faces, I'm talking about
> the above proposal to use propertize.  That is exactly equivalent to
> putting a named text property on the text, isn't it?  IOW, the text
> property you use to request stylistic variants doesn't have to be
> 'face', it can be some other property specifically designed for this
> purpose.

Yes, it isn't different but my point was that introducing stylistic sets
as a face attribute is more desirable than just having a text property.
Using a face attribute means we do not need to rewrite/add code to use
stylistic sets in certain cases, it is a matter of customisation.  IOW,
I am not saying that we should only have a face attribute.

> Using faces for this introduces complications and performance aspects
> that we had better avoided, unless we really must have this as part of
> a face.

See above.  Hopefully, I made my point clear.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Oops, it seems like I misunderstood the question.
For glyphs with more than one tag it seems like the one with the lower
value is prioritised when all of the tags are passed. (atleast in
LibreOffice)
I would have to look into it more.

On Sat, Sep 24, 2022 at 2:09 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 14:04:42 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  Then font-level specification of this would not be as simple as was
> >  originally suggested, I think: the characters were missing from that
> >  suggestion.
> >
> > We don't have to specify each character for which we want the
> stylistic-set, we just simply would have to
> > pass the tags
> > and hb_shape_full would take care of the rest.
>
> I don't understand: how does this handle the case where a certain
> character has more than 2 alternative glyphs? which alternative will
> be selected?
>

[-- Attachment #2: Type: text/html, Size: 1477 bytes --]

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

It seems to be the case in hb-view too.
Here is the priority order for "0" in FiraCode
hb-view --features='cv11,zero' FiraCode-Regular.ttf '0000' -O png >
test.png gives
zero
hb-view --features='cv11,cv12' FiraCode-Regular.ttf '0000' -O png >
test.png gives
cv11
hb-view --features='cv12,cv13' FiraCode-Regular.ttf '0000' -O png >
test.png gives
cv12

It was also in the case of the Iosevka font file.

[-- Attachment #2: Type: text/html, Size: 1503 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 17:39:55 +0530
> Cc: emacs-devel@gnu.org
> 
> It seems to be the case in hb-view too.
> Here is the priority order for "0" in FiraCode
> hb-view --features='cv11,zero' FiraCode-Regular.ttf '0000' -O png > test.png gives zero
> hb-view --features='cv11,cv12' FiraCode-Regular.ttf '0000' -O png > test.png gives cv11
> hb-view --features='cv12,cv13' FiraCode-Regular.ttf '0000' -O png > test.png gives cv12
> 
> It was also in the case of the Iosevka font file.

Sorry, I don't understand: what is "it" in this case?

And how is the priority order relevant to our discussion?  I'm
probably missing something here.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

>
> I don't understand: how does this handle the case where a certain
> character has more than 2 alternative glyphs? which alternative will
> be selected?


I was answering this question, I thought you were asking what would happen
if we pass multiple tags for the same glyph.
If that was not what you meant then look at my previous answer:

Different glyphs would have different tags.
For example in FiraCode "0" has four alternatives and each one has a
different tag:
zero, cv11, cv12, cv13

On Sat, Sep 24, 2022 at 6:05 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 17:39:55 +0530
> > Cc: emacs-devel@gnu.org
> >
> > It seems to be the case in hb-view too.
> > Here is the priority order for "0" in FiraCode
> > hb-view --features='cv11,zero' FiraCode-Regular.ttf '0000' -O png >
> test.png gives zero
> > hb-view --features='cv11,cv12' FiraCode-Regular.ttf '0000' -O png >
> test.png gives cv11
> > hb-view --features='cv12,cv13' FiraCode-Regular.ttf '0000' -O png >
> test.png gives cv12
> >
> > It was also in the case of the Iosevka font file.
>
> Sorry, I don't understand: what is "it" in this case?
>
> And how is the priority order relevant to our discussion?  I'm
> probably missing something here.
>

[-- Attachment #2: Type: text/html, Size: 2092 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 18:15:23 +0530
> Cc: emacs-devel@gnu.org
> 
>  I don't understand: how does this handle the case where a certain
>  character has more than 2 alternative glyphs? which alternative will
>  be selected?
> 
> I was answering this question, I thought you were asking what would happen if we pass multiple tags for the
> same glyph.

You are suggesting that we let the font and the shaping engine decide
which alternative to use?  That sounds un-Emacsy to me.  For the use
case where only specific characters should be affected, I think we
should have a facility whereby users could specify, for each character
that has alternatives, which one to use.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1.1: Type: text/plain, Size: 1382 bytes --]

>
> You are suggesting that we let the font and the shaping engine decide
> which alternative to use?  That sounds un-Emacsy to me.  For the use
> case where only specific characters should be affected, I think we
> should have a facility whereby users could specify, for each character
> that has alternatives, which one to use.


No, I am not suggesting that every variation has its own tag and the user
can choose which tag they would like or not.
For example look at the FiraCode variations attached

On Sat, Sep 24, 2022 at 6:31 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 18:15:23 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  I don't understand: how does this handle the case where a certain
> >  character has more than 2 alternative glyphs? which alternative will
> >  be selected?
> >
> > I was answering this question, I thought you were asking what would
> happen if we pass multiple tags for the
> > same glyph.
>
> You are suggesting that we let the font and the shaping engine decide
> which alternative to use?  That sounds un-Emacsy to me.  For the use
> case where only specific characters should be affected, I think we
> should have a facility whereby users could specify, for each character
> that has alternatives, which one to use.
>

[-- Attachment #1.2: Type: text/html, Size: 2071 bytes --]

[-- Attachment #2: character_variants.png --]
[-- Type: image/png, Size: 40797 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 18:40:55 +0530
> Cc: emacs-devel@gnu.org
> 
>  You are suggesting that we let the font and the shaping engine decide
>  which alternative to use?  That sounds un-Emacsy to me.  For the use
>  case where only specific characters should be affected, I think we
>  should have a facility whereby users could specify, for each character
>  that has alternatives, which one to use.
> 
> No, I am not suggesting that every variation has its own tag and the user can choose which tag they would
> like or not.
> For example look at the FiraCode variations attached

So what do I do if I want variant 4 for 'i' and variant 2 for 'l'?

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Simply pass both of their tags to hb_shape_full, which I think would be
done something like this:

hb_feature_t features[] = {{HB_TAG("c","v","0","6"), 1, start, end},
{HB_TAG("c","v","0","8"), 1, start, end}};
hb_shape_full(font, buffer, features, 2, NULL);

On Sat, Sep 24, 2022 at 7:48 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 18:40:55 +0530
> > Cc: emacs-devel@gnu.org
> >
> >  You are suggesting that we let the font and the shaping engine decide
> >  which alternative to use?  That sounds un-Emacsy to me.  For the use
> >  case where only specific characters should be affected, I think we
> >  should have a facility whereby users could specify, for each character
> >  that has alternatives, which one to use.
> >
> > No, I am not suggesting that every variation has its own tag and the
> user can choose which tag they would
> > like or not.
> > For example look at the FiraCode variations attached
>
> So what do I do if I want variant 4 for 'i' and variant 2 for 'l'?
>

[-- Attachment #2: Type: text/html, Size: 1711 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 19:56:47 +0530
> Cc: emacs-devel@gnu.org
> 
> Simply pass both of their tags to hb_shape_full, which I think would be done something like this:
> 
> hb_feature_t features[] = {{HB_TAG("c","v","0","6"), 1, start, end}, {HB_TAG("c","v","0","8"), 1, start, end}};
> hb_shape_full(font, buffer, features, 2, NULL);

With different values of START and END, I presume?

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

no, the same would work

On Sat, Sep 24, 2022 at 8:31 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 19:56:47 +0530
> > Cc: emacs-devel@gnu.org
> >
> > Simply pass both of their tags to hb_shape_full, which I think would be
> done something like this:
> >
> > hb_feature_t features[] = {{HB_TAG("c","v","0","6"), 1, start, end},
> {HB_TAG("c","v","0","8"), 1, start, end}};
> > hb_shape_full(font, buffer, features, 2, NULL);
>
> With different values of START and END, I presume?
>

[-- Attachment #2: Type: text/html, Size: 1105 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 20:32:26 +0530
> Cc: emacs-devel@gnu.org
> 
> no, the same would work

Then I don't understand how will HarfBuzz know to apply the first tag
to 'i', but the second one to 'l'.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1.1: Type: text/plain, Size: 419 bytes --]

On Sat, Sep 24, 2022 at 8:52 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 20:32:26 +0530
> > Cc: emacs-devel@gnu.org
> >
> > no, the same would work
>
> Then I don't understand how will HarfBuzz know to apply the first tag
> to 'i', but the second one to 'l'.
>

The font file has a GSUB table for this purpose

[-- Attachment #1.2: Type: text/html, Size: 884 bytes --]

[-- Attachment #2: gsub.png --]
[-- Type: image/png, Size: 178749 bytes --]

[-- Attachment #3: cv06.png --]
[-- Type: image/png, Size: 45984 bytes --]

[-- Attachment #4: cv08.png --]
[-- Type: image/png, Size: 41032 bytes --]

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> Date: Sat, 24 Sep 2022 21:20:03 +0530
> Cc: emacs-devel@gnu.org
> 
> On Sat, Sep 24, 2022 at 8:52 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
>  > Date: Sat, 24 Sep 2022 20:32:26 +0530
>  > Cc: emacs-devel@gnu.org
>  > 
>  > no, the same would work
> 
>  Then I don't understand how will HarfBuzz know to apply the first tag
>  to 'i', but the second one to 'l'.
> 
> The font file has a GSUB table for this purpose 

I'm sorry, I still don't understand.  (I'm not an expert on fonts, so
I'd appreciate a slightly more detailed explanations.)  Are you saying
that each tag can be used by a single character in a font?  IOW, no
feature tag can ever be shared by two or more characters in a font?

The HarfBuzz docs says:

  The hb_feature_t is the structure that holds information about
  requested feature application. The feature will be applied with the
  given value to all glyphs which are in clusters between start
  (inclusive) and end (exclusive).

This seems to say that we need to set START and END of each feature to
buffer positions of the characters to which we want the feature to be
applied.  That is why I said that START and END should be set
separately for each feature, according to the positions of the
corresponding characters in the buffer submitted to hb_shape_full.

Re: Supporting stylistic sets

[Yuri Khan]

On Sat, 24 Sept 2022 at 23:15, Eli Zaretskii <eliz@gnu.org> wrote:

> I'm sorry, I still don't understand.  (I'm not an expert on fonts, so
> I'd appreciate a slightly more detailed explanations.)  Are you saying
> that each tag can be used by a single character in a font?  IOW, no
> feature tag can ever be shared by two or more characters in a font?

As I understand it, the font has instructions like “if stylistic set X
is in effect, use glyphs Y1, Y2, …, Yn for characters Z1, Z2, …, Zn”.
Depending on the font designer’s whim, there could be separate
stylistic sets for each character, sets for multiple characters
similar in shape, and/or sets for a group of characters where the user
is unlikely to want to control each character separately.

For example, Fira Code has a stylistic set ‘onum’ that applies to all
digits. By enabling this, the user signals that they want old-style,
aka minuscule, numerals, where certain digits are seated lower
relative to the baseline, and/or have a smaller glyph height. (On
Windows, see the Georgia font for an example of minuscule digits.) It
is a choice that is normally made for all digits as a set, not
individually “I want odd digits to be set as minuscule but even digits
as default”.

For another example, the stylistic set ‘cv31’ discussed above applies
to both opening and closing parentheses, as it is unlikely that a user
would want different styles for those.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 24 Sep 2022 23:42:33 +0700
> Cc: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>, 
> 	emacs-devel@gnu.org
> 
> On Sat, 24 Sept 2022 at 23:15, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I'm sorry, I still don't understand.  (I'm not an expert on fonts, so
> > I'd appreciate a slightly more detailed explanations.)  Are you saying
> > that each tag can be used by a single character in a font?  IOW, no
> > feature tag can ever be shared by two or more characters in a font?
> 
> As I understand it, the font has instructions like “if stylistic set X
> is in effect, use glyphs Y1, Y2, …, Yn for characters Z1, Z2, …, Zn”.
> Depending on the font designer’s whim, there could be separate
> stylistic sets for each character, sets for multiple characters
> similar in shape, and/or sets for a group of characters where the user
> is unlikely to want to control each character separately.

"Unlikely" doesn't fly with Emacs users.  Someone will want to do just
that.  AFAIR, MS Office allows the user to select variants separately
for each character.

And if we want to support only the "likely" use cases, then the idea
of a special text property seems more attractive, since that means
entire segments of text will have the same collection of stylistic
sets.

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On Sat, Sep 24, 2022 at 9:44 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> > Date: Sat, 24 Sep 2022 21:20:03 +0530
> > Cc: emacs-devel@gnu.org
> >
> > On Sat, Sep 24, 2022 at 8:52 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >  > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
> >  > Date: Sat, 24 Sep 2022 20:32:26 +0530
> >  > Cc: emacs-devel@gnu.org
> >  >
> >  > no, the same would work
> >
> >  Then I don't understand how will HarfBuzz know to apply the first tag
> >  to 'i', but the second one to 'l'.
> >
> > The font file has a GSUB table for this purpose
>
> I'm sorry, I still don't understand.  (I'm not an expert on fonts, so
> I'd appreciate a slightly more detailed explanations.)  Are you saying
> that each tag can be used by a single character in a font?  IOW, no
> feature tag can ever be shared by two or more characters in a font?
>

IIUC, then yes.


> The HarfBuzz docs says:
>
>   The hb_feature_t is the structure that holds information about
>   requested feature application. The feature will be applied with the
>   given value to all glyphs which are in clusters between start
>   (inclusive) and end (exclusive).
>
> This seems to say that we need to set START and END of each feature to
> buffer positions of the characters to which we want the feature to be
> applied.  That is why I said that START and END should be set
> separately for each feature, according to the positions of the
> corresponding characters in the buffer submitted to hb_shape_full.
>

I don't think that is the case, I have attached a slightly modified
harfbuzz-tutorial[1],
where I have defined four features for a,g,i and l respectively with the
same start and end variables.
It works as expected.

You can provide it with the path to a firacode font file and the text which
it has to display and it will produce a png.

[1] https://github.com/harfbuzz/harfbuzz-tutorial

[-- Attachment #2: Type: text/html, Size: 3090 bytes --]

Re: Supporting stylistic sets

[समीर सिंह Sameer Singh]

[-- Attachment #1.1: Type: text/plain, Size: 2229 bytes --]

Forgot to attach, here it is.

On Sat, Sep 24, 2022 at 10:39 PM समीर सिंह Sameer Singh <
lumarzeli30@gmail.com> wrote:

> On Sat, Sep 24, 2022 at 9:44 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
>> > Date: Sat, 24 Sep 2022 21:20:03 +0530
>> > Cc: emacs-devel@gnu.org
>> >
>> > On Sat, Sep 24, 2022 at 8:52 PM Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> >  > From: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>
>> >  > Date: Sat, 24 Sep 2022 20:32:26 +0530
>> >  > Cc: emacs-devel@gnu.org
>> >  >
>> >  > no, the same would work
>> >
>> >  Then I don't understand how will HarfBuzz know to apply the first tag
>> >  to 'i', but the second one to 'l'.
>> >
>> > The font file has a GSUB table for this purpose
>>
>> I'm sorry, I still don't understand.  (I'm not an expert on fonts, so
>> I'd appreciate a slightly more detailed explanations.)  Are you saying
>> that each tag can be used by a single character in a font?  IOW, no
>> feature tag can ever be shared by two or more characters in a font?
>>
>
> IIUC, then yes.
>
>
>> The HarfBuzz docs says:
>>
>>   The hb_feature_t is the structure that holds information about
>>   requested feature application. The feature will be applied with the
>>   given value to all glyphs which are in clusters between start
>>   (inclusive) and end (exclusive).
>>
>> This seems to say that we need to set START and END of each feature to
>> buffer positions of the characters to which we want the feature to be
>> applied.  That is why I said that START and END should be set
>> separately for each feature, according to the positions of the
>> corresponding characters in the buffer submitted to hb_shape_full.
>>
>
> I don't think that is the case, I have attached a slightly modified
> harfbuzz-tutorial[1],
> where I have defined four features for a,g,i and l respectively with the
> same start and end variables.
> It works as expected.
>
> You can provide it with the path to a firacode font file and the text
> which it has to display and it will produce a png.
>
> [1] https://github.com/harfbuzz/harfbuzz-tutorial
>

[-- Attachment #1.2: Type: text/html, Size: 3542 bytes --]

[-- Attachment #2: harfbuzz-tutorial.tar.gz --]
[-- Type: application/gzip, Size: 24389 bytes --]

Re: Supporting stylistic sets

[Clément Pit-Claudel]

On 9/24/22 09:14, Eli Zaretskii wrote:
> Are you saying
> that each tag can be used by a single character in a font?  IOW, no
> feature tag can ever be shared by two or more characters in a font?

Sometimes, but not always:

- Sometimes each stylistic set tag is used to toggle a different display for a single character.  See e.g. https://ctan.org/tex-archive/fonts/garamond-math .  This works like a GUI that lets you pick variants individually for each character.

- Sometimes a stylistic set is used to change the appearance of multiple characters at once.  See e.g. https://github.com/newbee1905/Iosevka-personal-config/blob/master/doc/stylistic-sets.md  . This is more akin to customizing the feel of the whole face.

I haven't thought deeply about how to best make these accessible in Emacs, but in most cases I think the preference for one stylistic set is "global".  For example, I use the Iosevka font, and I use the stylistic set that imitates the "Ubuntu mono" font everywhere.  So, for this use case, a font-level parameter would work well.

There is one use case for stylistic sets where a global, per-font setting doesn't cut it, which is sometimes called "ligation sets" (see e.g. https://typeof.net/Iosevka/).  The idea is that some font use stylistic sets to allow you to chose different sets of ligatures, and the appropriate set of ligatures depends on the context.  For example, Iosevka has one "ligation set" per programming language: the expectation is that code in F# would be rendered with a different stylistic set than code written in Coq; same for OCaml, Haskell, etc.  In an Org buffer with multiple code fragments, I would want to use a different stylistic set for each fragment.  I'm not sure how to do this in Emacs.

Clément.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Nicolas Ouellet-payeur <nicolaso@google.com>
> Date: Mon, 26 Sep 2022 12:35:34 -0400
> Cc: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>, 
> 	emacs-devel@gnu.org
> 
> > To measure the slowdown, run the benchmark of scrolling through a
> > large file, like xdisp.c, one line at a time, and compare with the
> > current code.
> 
> It's not as bad as I feared, but it's still pretty bad. Scrolling 2000
> lines through xdisp.c is ~2.3x slower in the patched version. The jank
> is noticeable though, especially because scrolling *should* be limited
> by the keyboard repeat rate.

I think you tried a relatively benign situation.  To see how this
could _really_ be slow, make a composition rule that will catch any
sequence of any non-whitespace characters.

This slowness is the main disadvantage of our current method of
handling character compositions.  (It also has numerous advantages.)
If we want to progressively increase the use of text-shaping engines
when rendering text, we will need to redesign this part, because the
way we do this now cannot be extended in those directions without
causing significant slowdown.

> > But what do you do if the user wants to disable ligation, or control
> > which ligatures should and shouldn't happen?
> 
> To disable ligation, one can turn off font features selectively. For
> instance disabling 'rlig' in Fira Code disables the code ligatures. We
> could disable it by default on the mode-line, for instance. Or just a
> section of the default mode-line.

That would disable ligation in complex script shaping, where it is a
must.  For example, displaying the Arabic script requires ligatures.
So I think disabling ligation this way is unworkable; we must find
some other ways of doing that.

> - Ignore the shaping engine. Read the GSUB table from the font file
>   ourselves, and populate `composition-function-table' with that.
>   Supporting *everything* is really hard, but basic use-cases might work
>   fine out-of-the-box.

That's a non-starter, IMO.  We don't want to develop our own
font-related and text-shaping-related capabilities, as it's not our
expertise and we will never be able to have enough knowledge and
development resources to do it.  We must rely on external shaping
engines such as HarfBuzz.

> - Give up on OOTB ligature support (status quo).

The status quo is NOT that we don't support ligatures.  The support is
available; you can have it at the price of a few minutes' work by
installing the composition rules in composition-function-table.  What
is missing is a convenient interface and UI for users to control where
and which ligatures will be available.  See the related TODO item.

Re: Supporting stylistic sets

[Nicolas Ouellet-payeur]

> That would disable ligation in complex script shaping, where it is a
> must.

That's why the default mode-line would have a text property on the parts
that are likely to ligate.

e.g. if my mode-line reads like:

-:--- foo.c   All   L1    (C Abbrev)

Then we'd apply this text property on the "-:---" part.

> This slowness is the main disadvantage of our current method of
> handling character compositions.  (It also has numerous advantages.)
> If we want to progressively increase the use of text-shaping engines
> when rendering text, we will need to redesign this part

What do you mean by "this part" specifically? The part where we call
into Lisp for any character in `composition-function-table'? Or the fact
that we do a regex match each time?

> The status quo is NOT that we don't support ligatures.  The support is
> available

Of course. By OOTB (out-of-the-box) I meant reading the GSUB table so we
can apply ligatures *without* requiring manual configuration. i.e.
making the shaping engine do the heavy lifting.

Are we OK with the status quo then, where ligatures have to be manually
configured? In that case, I could send a patch to remove that item from
etc/TODO.

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Nicolas Ouellet-payeur <nicolaso@google.com>
> Date: Mon, 26 Sep 2022 13:46:40 -0400
> Cc: समीर सिंह Sameer Singh <lumarzeli30@gmail.com>, 
> 	emacs-devel@gnu.org
> 
> > That would disable ligation in complex script shaping, where it is a
> > must.
> 
> That's why the default mode-line would have a text property on the parts
> that are likely to ligate.
> 
> e.g. if my mode-line reads like:
> 
> -:--- foo.c   All   L1    (C Abbrev)
> 
> Then we'd apply this text property on the "-:---" part.

First, this is the first time that you mention a text property in this
discussion.  We don't have such a text property in Emacs, so it would
have to be added, character-composition code will need to be amended
to be aware of the property.

And second, it is not good enough to have the property on the "-:---"
part, because that's where the input-method mnemonics are displayed,
and those frequently do need ligations, if they use characters from
scripts which require that.

> > This slowness is the main disadvantage of our current method of
> > handling character compositions.  (It also has numerous advantages.)
> > If we want to progressively increase the use of text-shaping engines
> > when rendering text, we will need to redesign this part
> 
> What do you mean by "this part" specifically? The part where we call
> into Lisp for any character in `composition-function-table'? Or the fact
> that we do a regex match each time?

Mainly the former.

> > The status quo is NOT that we don't support ligatures.  The support is
> > available
> 
> Of course. By OOTB (out-of-the-box) I meant reading the GSUB table so we
> can apply ligatures *without* requiring manual configuration. i.e.
> making the shaping engine do the heavy lifting.

The shaping engine will do the heavy lifting anyway, but we must give
users control on what is and what isn't ligated, so just asking the
shaping engine to ligate everything the font supports is also out of
the question.

> Are we OK with the status quo then, where ligatures have to be manually
> configured? In that case, I could send a patch to remove that item from
> etc/TODO.

The item in TODO explains in detail what it means by "OOTB".  What is
described there is missing from the current Emacs, and I hope it will
be added.  But the way to add it must allow user control on what
ligatures will be produced and in what places.  It also should allow
this control via a UI that is much easier to use than the current raw
support which requires manually writing entries for
composition-function-table.  It's all in the TODO item's text, and I
don't understand how that could lead you to the conclusion that we are
"OK with the status quo".  All I said that it's incorrect to say that
we don't have any support for ligatures.

Re: Supporting stylistic sets

[Yuri Khan]

On Tue, 27 Sept 2022 at 01:55, Eli Zaretskii <eliz@gnu.org> wrote:

> And second, it is not good enough to have the property on the "-:---"
> part, because that's where the input-method mnemonics are displayed,
> and those frequently do need ligations, if they use characters from
> scripts which require that.

Maybe just put a zwnj between every two directly adjacent fields to
prevent ligation?

Re: Supporting stylistic sets

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Tue, 27 Sep 2022 02:06:35 +0700
> Cc: Nicolas Ouellet-payeur <nicolaso@google.com>, lumarzeli30@gmail.com, emacs-devel@gnu.org
> 
> On Tue, 27 Sept 2022 at 01:55, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > And second, it is not good enough to have the property on the "-:---"
> > part, because that's where the input-method mnemonics are displayed,
> > and those frequently do need ligations, if they use characters from
> > scripts which require that.
> 
> Maybe just put a zwnj between every two directly adjacent fields to
> prevent ligation?

It's possible.  But then (1) we'd need to always have the ZWNJ there;
(2) some mode-line constructs are produced by formatting text with
tricky format specs, and some could come from the user or some Lisp
program that are utterly unaware of these subtleties; and (3) Emacs by
default displays ZWNJ as a thin space.  So this is not the best
solution, IMO.

I tend to think this is just one aspect of a more general problem,
which is: how do users/Lisp programs control which characters will or
will not ligate in what situations?  For example, take ffi: you might
not want the characters f f i to ligate in a variable name, or in some
major modes as opposed to others, etc.  When we figure out how to
specify when to ligate, we need that specification support the mode
line and other places (like maybe be tab bar/tab line).