Unicode 13 Emoji ranges composed with wrong font on NS port

Unicode 13 Emoji ranges composed with wrong font on NS port

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

First of all, thanks for the new emoji support, it's great!

However, despite what the NEWS file claims, I was visiting Unicode 13's 
emoji-style.txt downloaded from here 
<https://www.unicode.org/emoji/charts-13.0/emoji-style.txt> and noticed 
that certain composed character in the modifier and zwj blocks do not 
compose using Apple Color Emoji on the NS port running on macOS 11, but 
somehow went to Symbola, even after setting (set-fontset-font t 'emoji 
'("Apple Color Emoji" . "iso10646-1") nil 'prepend). Specifically, the 
characters are #x270c, #x261d, #x270d, and #x26f9.

I also noticed that Emoji Presentation isn't handled correctly according 
to spec yet, tho much closer than most browers except Firefox.

Are these omissions intentional? Sorry I couldn't follow the long emoji 
thread.

-- 
Jimmy Wong

[-- Attachment #2: Type: text/html, Size: 1145 bytes --]

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> Date: Sun, 10 Oct 2021 09:27:17 +0100
> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> 
> However, despite what the NEWS file claims, I was visiting Unicode 13's emoji-style.txt downloaded from
> here and noticed that certain composed character in the modifier and zwj blocks do not compose using
> Apple Color Emoji on the NS port running on macOS 11, but somehow went to Symbola, even after setting
> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend). Specifically, the characters are
> #x270c, #x261d, #x270d, and #x26f9.
> 
> I also noticed that Emoji Presentation isn't handled correctly according to spec yet, tho much closer than
> most browers except Firefox.
> 
> Are these omissions intentional? Sorry I couldn't follow the long emoji thread.

No, it should work, provided that your fonts don't get in the way.
The idea is that the VS-16 character in the sequence is supposed to be
supported by your Emoji font, and that causes the entire sequence to
be handled as an Emoji sequence.  It is not clear to me whether the
characters you mention belong to sequences that end in VS-16; if not,
then we don't want to display them as Emoji, because they are just
normal symbol characters.

The set-fontset-font customization you tried won't help because #x270C
and others do not belong to the Emoji script.

I don't know how font selection works on macOS, so I cannot help you
more, sorry.  Perhaps Robert could.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Sun, 10 Oct 2021 13:52:01 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> Date: Sun, 10 Oct 2021 09:27:17 +0100
    >> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
    >> 
    >> However, despite what the NEWS file claims, I was visiting Unicode 13's emoji-style.txt downloaded from
    >> here and noticed that certain composed character in the modifier and zwj blocks do not compose using
    >> Apple Color Emoji on the NS port running on macOS 11, but somehow went to Symbola, even after setting
    >> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend). Specifically, the characters are
    >> #x270c, #x261d, #x270d, and #x26f9.
    >>

As Eli mentions below, theyʼre not emoji (in Emacs, at least)

    >> I also noticed that Emoji Presentation isn't handled correctly according to spec yet, tho much closer than
    >> most browers except Firefox.
    >>

You'll have to be more specific. Details matter when dealing with
Unicode :-)

    >> Are these omissions intentional? Sorry I couldn't follow the long emoji thread.

    Eli> No, it should work, provided that your fonts don't get in the way.
    Eli> The idea is that the VS-16 character in the sequence is supposed to be
    Eli> supported by your Emoji font, and that causes the entire sequence to
    Eli> be handled as an Emoji sequence.  It is not clear to me whether the
    Eli> characters you mention belong to sequences that end in VS-16; if not,
    Eli> then we don't want to display them as Emoji, because they are just
    Eli> normal symbol characters.

They're all Emoji_Presentation=No codepoints, so they should only be
displayed with emoji font if followed by VS-16

    Eli> The set-fontset-font customization you tried won't help because #x270C
    Eli> and others do not belong to the Emoji script.

Right

    Eli> I don't know how font selection works on macOS, so I cannot help you
    Eli> more, sorry.  Perhaps Robert could.

I donʼt think font selection is that different on macOS. Certainly
set-fontset-font works the same.

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Sun, 10 Oct 2021 17:49:49 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
>     Eli> I don't know how font selection works on macOS, so I cannot help you
>     Eli> more, sorry.  Perhaps Robert could.
> 
> I donʼt think font selection is that different on macOS. Certainly
> set-fontset-font works the same.

I mean font selection when the fontset is not specific enough (as in
"dictates the font for a specific range of codepoints").  That is, how
candidate system fonts are found and filtered.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Jimmy Yuen Ho Wong]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On 10/10/2021 4:49 PM, Robert Pluim wrote:
>>>>>> On Sun, 10 Oct 2021 13:52:01 +0300, Eli Zaretskii <eliz@gnu.org> said:
>      >> Date: Sun, 10 Oct 2021 09:27:17 +0100
>      >> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
>      >>
>      >> However, despite what the NEWS file claims, I was visiting Unicode 13's emoji-style.txt downloaded from
>      >> here and noticed that certain composed character in the modifier and zwj blocks do not compose using
>      >> Apple Color Emoji on the NS port running on macOS 11, but somehow went to Symbola, even after setting
>      >> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend). Specifically, the characters are
>      >> #x270c, #x261d, #x270d, and #x26f9.
>      >>
>
> As Eli mentions below, theyʼre not emoji (in Emacs, at least)
x270c is Victory Hand, x261d is Index Pointing Up, x270d is Writing 
Hand, and x26f9 is Person Bouncing Ball. They are found in Unicode 14's, 
along with Unicode 13's emoji-data.txt. All of which have been a part of 
Emoji 1.0 from 2015.
>
>      >> I also noticed that Emoji Presentation isn't handled correctly according to spec yet, tho much closer than
>      >> most browers except Firefox.
>      >>
>
> You'll have to be more specific. Details matter when dealing with
> Unicode :-)

As the comments in emoji-style.txt states, emojis with 
Emoji_Presentation=False when followed by VS15 (text+ts), should be 
displayed as text, but currently most are displayed as emojis, whereas 
most of those followed by VS16 (text+es) are currently displayed text. 
See the attached screenshot.


[-- Attachment #2: Screen Shot 2021-10-11 at 10.35.56 pm.png --]
[-- Type: image/png, Size: 568667 bytes --]

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Mon, 11 Oct 2021 22:39:19 +0100, Jimmy Yuen Ho Wong <wyuenho@gmail.com> said:

    Jimmy> On 10/10/2021 4:49 PM, Robert Pluim wrote:
    >>>>>>> On Sun, 10 Oct 2021 13:52:01 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> >> Date: Sun, 10 Oct 2021 09:27:17 +0100
    >> >> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
    >> >>
    >> >> However, despite what the NEWS file claims, I was visiting Unicode 13's emoji-style.txt downloaded from
    >> >> here and noticed that certain composed character in the modifier and zwj blocks do not compose using
    >> >> Apple Color Emoji on the NS port running on macOS 11, but somehow went to Symbola, even after setting
    >> >> (set-fontset-font t 'emoji '("Apple Color Emoji"
    >> >> . "iso10646-1") nil 'prepend). Specifically, the characters
    >> >> are
    >> >> #x270c, #x261d, #x270d, and #x26f9.
    >> >>
    >> 
    >> As Eli mentions below, theyʼre not emoji (in Emacs, at least)
    Jimmy> x270c is Victory Hand, x261d is Index Pointing Up, x270d is Writing
    Jimmy> Hand, and x26f9 is Person Bouncing Ball. They are found in Unicode
    Jimmy> 14's, along with Unicode 13's emoji-data.txt. All of which have been a
    Jimmy> part of Emoji 1.0 from 2015.
    >>

You missed the "in Emacs" qualifier in what I wrote.

    >> >> I also noticed that Emoji Presentation isn't handled correctly according to spec yet, tho much closer than
    >> >> most browers except Firefox.
    >> >>
    >> 
    >> You'll have to be more specific. Details matter when dealing with
    >> Unicode :-)

    Jimmy> As the comments in emoji-style.txt states, emojis with
    Jimmy> Emoji_Presentation=False when followed by VS15 (text+ts), should be
    Jimmy> displayed as text, but currently most are displayed as
    Jimmy> emojis, whereas

They are? Thatʼs not what Iʼm seeing in your screenshot (nor
locally). And Emacs does nothing with VS15 for now.

    Jimmy> most of those followed by VS16 (text+es) are currently displayed
    Jimmy> text. See the attached screenshot.

On Gnu/Linux with Noto Color Emoji they all look very colourful. For
some reason on macOS the code thatʼs supposed to handle VS16 is not
working correctly.

Hmm, the bit in font_range that does the lookup based off
Vscript_representative_chars is working correctly, which means Iʼll
need to look into the macOS font/display code. Given that
glyphless-char-display appears not to be honored on macOS, maybe
thereʼs some code missing. Alan?

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> Date: Mon, 11 Oct 2021 22:39:19 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> 
> >      >> #x270c, #x261d, #x270d, and #x26f9.
> >      >>
> >
> > As Eli mentions below, theyʼre not emoji (in Emacs, at least)
> x270c is Victory Hand, x261d is Index Pointing Up, x270d is Writing 
> Hand, and x26f9 is Person Bouncing Ball. They are found in Unicode 14's, 
> along with Unicode 13's emoji-data.txt. All of which have been a part of 
> Emoji 1.0 from 2015.

But characters like U+0023 hash sign and U+00A9 copyright sign are
also in emoji-data.txt with the same "Emoji=Yes" property, and yet
that doesn't mean we should treat them as Emoji, right?

These belong to symbols and punctuation, outside of the Emoticons
block, so we decided to treat them as Emoji only when they are
followed by VS-16, which means the text explicitly requests to show
them in their Emoji representation.

> > You'll have to be more specific. Details matter when dealing with
> > Unicode :-)
> 
> As the comments in emoji-style.txt states, emojis with 
> Emoji_Presentation=False when followed by VS15 (text+ts), should be 
> displayed as text, but currently most are displayed as emojis, whereas 
> most of those followed by VS16 (text+es) are currently displayed text. 
> See the attached screenshot.

Please be even more specific: which parts of that screenshot are in
your opinion incorrect, and why?

In general, sequences that end in VS-16 should be displayed as Emoji
if they are designated as such in emoji-sequences.txt and
emoji-zwj-sequences.txt.  See admin/unidata/emoji-zwj.awk.  The rest
we decided not to display as Empji for various good reasons.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,
>     Alan Third <alan@idiocy.org>
> Date: Tue, 12 Oct 2021 15:12:20 +0200
> 
> Emacs does nothing with VS15 for now.

If you force Emacs to compose sequences that end with VS-15 (by
tweaking composition-function-table), do any good Emoji fonts produce
a different representation?  If they do, perhaps we should do
something about VS-15 as well.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

    Robert> On Gnu/Linux with Noto Color Emoji they all look very colourful. For
    Robert> some reason on macOS the code thatʼs supposed to handle VS16 is not
    Robert> working correctly.

    Robert> Hmm, the bit in font_range that does the lookup based off
    Robert> Vscript_representative_chars is working correctly, which means Iʼll
    Robert> need to look into the macOS font/display code. Given that
    Robert> glyphless-char-display appears not to be honored on macOS, maybe
    Robert> thereʼs some code missing. Alan?

This turns out not to be due to macOS differences, but because of this
code in lisp/composite.el:

(let ((elt '([".." 1 compose-gstring-for-variation-glyph])))
  (set-char-table-range composition-function-table '(#xFE00 . #xFE0F) elt)
  (set-char-table-range composition-function-table '(#xE0100 . #xE01EF) elt))

If I change that to use `compose-gstring-for-graphic' instead, then
the emoji+VS-16 display works correctly on macOS and GNU/Linux.

Eli, something like the following for emacs-28?

diff --git a/lisp/composite.el b/lisp/composite.el
index 859253ec7e..983398f469 100644
--- a/lisp/composite.el
+++ b/lisp/composite.el
@@ -835,8 +835,11 @@ compose-gstring-for-variation-glyph
 	      (throw 'tag gstring)))))))
 
 (let ((elt '([".." 1 compose-gstring-for-variation-glyph])))
-  (set-char-table-range composition-function-table '(#xFE00 . #xFE0F) elt)
+  (set-char-table-range composition-function-table '(#xFE00 . #xFE0E) elt)
   (set-char-table-range composition-function-table '(#xE0100 . #xE01EF) elt))
+;; We don't want variation selectors to be used to look up glyphs for FE0F
+(set-char-table-range composition-function-table #xFE0F
+                      '([".." 1 compose-gstring-for-graphic]))
 
 (defun auto-compose-chars (func from to font-object string direction)
   "Compose the characters at FROM by FUNC.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Jimmy Yuen Ho Wong]

On 12/10/2021 2:12 PM, Robert Pluim wrote:
> You missed the "in Emacs" qualifier in what I wrote.
Well, lol, why are these characters in the script script anyway? What 
even is script script? That's not a Microsoft script tag.
> They are? Thatʼs not what Iʼm seeing in your screenshot (nor
> locally). And Emacs does nothing with VS15 for now.
Sorry there was a type, I mean, text+ts should be all monochrome, but 
some are currently colorful. text+es should all be colorful.

-- 
Jimmy Wong

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Jimmy Yuen Ho Wong <wyuenho@gmail.com>,  Alan Third <alan@idiocy.org>,
>   emacs-devel@gnu.org
> Date: Tue, 12 Oct 2021 19:13:59 +0200
> 
> This turns out not to be due to macOS differences, but because of this
> code in lisp/composite.el:
> 
> (let ((elt '([".." 1 compose-gstring-for-variation-glyph])))
>   (set-char-table-range composition-function-table '(#xFE00 . #xFE0F) elt)
>   (set-char-table-range composition-function-table '(#xE0100 . #xE01EF) elt))
> 
> If I change that to use `compose-gstring-for-graphic' instead, then
> the emoji+VS-16 display works correctly on macOS and GNU/Linux.
> 
> Eli, something like the following for emacs-28?

That's because VS-16 is already handled by other entries, including
its own?  Or how will VS-16 be handled if you remove it from that
range?

And if the reason is that VS-16 is not about glyph variations, then
why not exempt VS-15 as well?  Or why we need that special entry for
VS-16 at all?

> +;; We don't want variation selectors to be used to look up glyphs for FE0F

The comment should explain why.  Right now, the comment just says what
the code does, but gives no reasons.

Thanks.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> Date: Tue, 12 Oct 2021 18:28:36 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>  Alan Third <alan@idiocy.org>
> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> 
> On 12/10/2021 2:12 PM, Robert Pluim wrote:
> > You missed the "in Emacs" qualifier in what I wrote.
> Well, lol, why are these characters in the script script anyway? What 
> even is script script? That's not a Microsoft script tag.

'symbol', not 'script':

  (aref char-script-table #x270c) => symbol

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> Date: Tue, 12 Oct 2021 18:40:07 +0100
> Cc: rpluim@gmail.com, emacs-devel@gnu.org
> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> 
> What I'm talking about specifically is when these characters are used as 
> a modifier base, the skin color or sex symbol that comes after them 
> should take effect and turn them into emoji presentation, but they are 
> currently not. See the attached screenshots.

Is this about the sequences that end with VS-16?  Or are there others?
(You again show a very large image without any indication what parts
of it are problematic, and what sequence of codepoints was there in
the file.)

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Alan Third]

On Tue, Oct 12, 2021 at 03:12:20PM +0200, Robert Pluim wrote:
> Hmm, the bit in font_range that does the lookup based off
> Vscript_representative_chars is working correctly, which means Iʼll
> need to look into the macOS font/display code. Given that
> glyphless-char-display appears not to be honored on macOS, maybe
> thereʼs some code missing. Alan?

I'm afraid I don't have even the faintest understanding of what you're
talking about. If you think there's something wrong with the font
handling in macfont.m you may be better asking Yamamoto Mitsuharu.

If it's not the font backend then it's quite possible there's code
missing, but I'm afraid I don't know what that might be.
-- 
Alan Third

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Tue, 12 Oct 2021 21:27:46 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Jimmy Yuen Ho Wong <wyuenho@gmail.com>,  Alan Third <alan@idiocy.org>,
    >> emacs-devel@gnu.org
    >> Date: Tue, 12 Oct 2021 19:13:59 +0200
    >> 
    >> This turns out not to be due to macOS differences, but because of this
    >> code in lisp/composite.el:
    >> 
    >> (let ((elt '([".." 1 compose-gstring-for-variation-glyph])))
    >> (set-char-table-range composition-function-table '(#xFE00 . #xFE0F) elt)
    >> (set-char-table-range composition-function-table '(#xE0100 . #xE01EF) elt))
    >> 
    >> If I change that to use `compose-gstring-for-graphic' instead, then
    >> the emoji+VS-16 display works correctly on macOS and GNU/Linux.
    >> 
    >> Eli, something like the following for emacs-28?

    Eli> That's because VS-16 is already handled by other entries, including
    Eli> its own?  Or how will VS-16 be handled if you remove it from that
    Eli> range?

Digging deeper, we could actually remove the VS-16 entry completely,
since itʼs then handled by composite.el:737 :

    (when unicode-category-table
      (let ((elt `([,(purecopy "\\c.\\c^+") 1 compose-gstring-for-graphic]
                   [nil 0 compose-gstring-for-graphic])))

although I think Iʼd prefer to be explicit about it.

    Eli> And if the reason is that VS-16 is not about glyph variations, then
    Eli> why not exempt VS-15 as well?  Or why we need that special entry for
    Eli> VS-16 at all?

I donʼt know the history. I also donʼt know how common fonts which
contain variant glyphs are. Iʼll have to run some experiments.

    >> +;; We don't want variation selectors to be used to look up glyphs for FE0F

    Eli> The comment should explain why.  Right now, the comment just says what
    Eli> the code does, but gives no reasons.

Iʼll fix that.

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Tue, 12 Oct 2021 22:00:39 +0100, Alan Third <alan@idiocy.org> said:

    Alan> On Tue, Oct 12, 2021 at 03:12:20PM +0200, Robert Pluim wrote:
    >> Hmm, the bit in font_range that does the lookup based off
    >> Vscript_representative_chars is working correctly, which means Iʼll
    >> need to look into the macOS font/display code. Given that
    >> glyphless-char-display appears not to be honored on macOS, maybe
    >> thereʼs some code missing. Alan?

    Alan> I'm afraid I don't have even the faintest understanding of what you're
    Alan> talking about. If you think there's something wrong with the font
    Alan> handling in macfont.m you may be better asking Yamamoto Mitsuharu.

False alarm, macfont.m is fine.

    Alan> If it's not the font backend then it's quite possible there's code
    Alan> missing, but I'm afraid I don't know what that might be.

glyphless-char-display/glyphless-char-display-control gives you a way
to display e.g. U+FE0F as a thin-space, or a box, etc. It appears that
this doesnʼt work on macOS (but thatʼs totally separate from this issue).

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Tue, 12 Oct 2021 21:53:22 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> Date: Tue, 12 Oct 2021 18:40:07 +0100
    >> Cc: rpluim@gmail.com, emacs-devel@gnu.org
    >> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
    >> 
    >> What I'm talking about specifically is when these characters are used as 
    >> a modifier base, the skin color or sex symbol that comes after them 
    >> should take effect and turn them into emoji presentation, but they are 
    >> currently not. See the attached screenshots.

    Eli> Is this about the sequences that end with VS-16?  Or are there others?
    Eli> (You again show a very large image without any indication what parts
    Eli> of it are problematic, and what sequence of codepoints was there in
    Eli> the file.)

Based on what I see here, itʼs the usual suspects: eg U+261D. Itʼs not
part of the emoji script in Emacs. The font_range change should cause

    U+261D U+FE0F

to switch to emoji presentation, but it doesnʼt since thereʼs a
composition-function-table entry for U+261D, so the backwards looking
entry for U+FE0F never comes into play.

Iʼm not sure how to fix this.

- We canʼt add U+261D to the emoji script.
- We could add a hack in font_range for U+261D, U+26F9,
U+270C..U+270D, and U+2764 to use the emoji font to check for
composition the same way as we do now when the composition trigger is
an emoji
- We could fix the backwards looking issue with
composition-function-table (I have no idea how complex that is)

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: alan@idiocy.org,  wyuenho@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 13 Oct 2021 11:35:22 +0200
> 
>     Eli> That's because VS-16 is already handled by other entries, including
>     Eli> its own?  Or how will VS-16 be handled if you remove it from that
>     Eli> range?
> 
> Digging deeper, we could actually remove the VS-16 entry completely,
> since itʼs then handled by composite.el:737 :
> 
>     (when unicode-category-table
>       (let ((elt `([,(purecopy "\\c.\\c^+") 1 compose-gstring-for-graphic]
>                    [nil 0 compose-gstring-for-graphic])))
> 
> although I think Iʼd prefer to be explicit about it.

Why do you prefer that?

>     Eli> And if the reason is that VS-16 is not about glyph variations, then
>     Eli> why not exempt VS-15 as well?  Or why we need that special entry for
>     Eli> VS-16 at all?
> 
> I donʼt know the history. I also donʼt know how common fonts which
> contain variant glyphs are. Iʼll have to run some experiments.

But VS-15 is not about variant glyphs, it's about requesting the "text
representation" of certain characters, including Emoji.  So it sounds
like the rule for compose-gstring-for-variation-glyph is not really
relevant for it, as well as for VS-16.  Right?

>     >> +;; We don't want variation selectors to be used to look up glyphs for FE0F
> 
>     Eli> The comment should explain why.  Right now, the comment just says what
>     Eli> the code does, but gives no reasons.
> 
> Iʼll fix that.

Thanks.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Jimmy Yuen Ho Wong <wyuenho@gmail.com>,  emacs-devel@gnu.org
> Date: Wed, 13 Oct 2021 12:22:26 +0200
> 
> Based on what I see here, itʼs the usual suspects: eg U+261D. Itʼs not
> part of the emoji script in Emacs. The font_range change should cause
> 
>     U+261D U+FE0F
> 
> to switch to emoji presentation, but it doesnʼt since thereʼs a
> composition-function-table entry for U+261D, so the backwards looking
> entry for U+FE0F never comes into play.
> 
> Iʼm not sure how to fix this.
> 
> - We canʼt add U+261D to the emoji script.

These composition-function-table entries for these characters
come from emoji-zwj.el, so we could simply add the above sequences to
that, by suitable change in emoji-zwj.awk, right?  And then the
font_ranges trick should cause us select an Emoji font, right?

But is it worth the trouble?  IOW, do these characters have pretty
appearance when they are followed by VS-16?

I wouldn't add the characters themselves to the 'emoji' script, as
that would have much wider consequences, not necessarily good ones.
For example, I don't think that these characters, when they aren't
followed by VS-16 or something Emoji-like, should be displayed using
the Emoji fonts.

> - We could fix the backwards looking issue with
> composition-function-table (I have no idea how complex that is)

I don't think it's complicated, and I already asked Kenichi Hand about
that, hopefully he will either agree with my conclusions or explain
where I'm wrong.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Wed, 13 Oct 2021 16:12:42 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Jimmy Yuen Ho Wong <wyuenho@gmail.com>,  emacs-devel@gnu.org
    >> Date: Wed, 13 Oct 2021 12:22:26 +0200
    >> 
    >> Based on what I see here, itʼs the usual suspects: eg U+261D. Itʼs not
    >> part of the emoji script in Emacs. The font_range change should cause
    >> 
    >> U+261D U+FE0F
    >> 
    >> to switch to emoji presentation, but it doesnʼt since thereʼs a
    >> composition-function-table entry for U+261D, so the backwards looking
    >> entry for U+FE0F never comes into play.
    >> 
    >> Iʼm not sure how to fix this.
    >> 
    >> - We canʼt add U+261D to the emoji script.

    Eli> These composition-function-table entries for these characters
    Eli> come from emoji-zwj.el, so we could simply add the above sequences to
    Eli> that, by suitable change in emoji-zwj.awk, right?  And then the
    Eli> font_ranges trick should cause us select an Emoji font, right?

The triggering character would not be an emoji,so weʼd need to change
font_range. See below

    Eli> But is it worth the trouble?  IOW, do these characters have pretty
    Eli> appearance when they are followed by VS-16?

They do. The difference is quite pronounced.

    Eli> I wouldn't add the characters themselves to the 'emoji' script, as
    Eli> that would have much wider consequences, not necessarily good ones.
    Eli> For example, I don't think that these characters, when they aren't
    Eli> followed by VS-16 or something Emoji-like, should be displayed using
    Eli> the Emoji fonts.

So if I change emoji-zwj.awk to emit codepoint+U+FE0F rules for these
problematic codepoints which have Emoji_presentation = No, and change
font_range to try looking up the emoji font for them, all the
'codepoint+U+FE0F' from Jimmy's test file display as Emoji, and the
'codepoint+U+FE0E' display as text.

It would mean emitting rules in emoji-zwj.awk for, and checking in
font_range for, the following codepoints.

U+261D, U+26F9, U+270C, U+270D, U+2764, U+1F3CB, U+1F3CC, U+1F3F3,
U+1F3F4, U+1F441, U+1F574, U+1F575, U+1F590

As an added bonus, those U+1xxxx codepoints would no longer be part of the
'emoji' script.

    >> - We could fix the backwards looking issue with
    >> composition-function-table (I have no idea how complex that is)

    Eli> I don't think it's complicated, and I already asked Kenichi Hand about
    Eli> that, hopefully he will either agree with my conclusions or explain
    Eli> where I'm wrong.

OK. Let's cross that bridge when we get to it.

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Wed, 13 Oct 2021 15:58:19 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: alan@idiocy.org,  wyuenho@gmail.com,  emacs-devel@gnu.org
    >> Date: Wed, 13 Oct 2021 11:35:22 +0200
    >> 
    Eli> That's because VS-16 is already handled by other entries, including
    Eli> its own?  Or how will VS-16 be handled if you remove it from that
    Eli> range?
    >> 
    >> Digging deeper, we could actually remove the VS-16 entry completely,
    >> since itʼs then handled by composite.el:737 :
    >> 
    >> (when unicode-category-table
    >> (let ((elt `([,(purecopy "\\c.\\c^+") 1 compose-gstring-for-graphic]
    >> [nil 0 compose-gstring-for-graphic])))
    >> 
    >> although I think Iʼd prefer to be explicit about it.

    Eli> Why do you prefer that?

I was thinking it would be better to give more information to later
maintainers, but the explicit entry would need to be the same as
what's already there, so best to leave it alone.

    Eli> And if the reason is that VS-16 is not about glyph variations, then
    Eli> why not exempt VS-15 as well?  Or why we need that special entry for
    Eli> VS-16 at all?
    >> 
    >> I donʼt know the history. I also donʼt know how common fonts which
    >> contain variant glyphs are. Iʼll have to run some experiments.

    Eli> But VS-15 is not about variant glyphs, it's about requesting the "text
    Eli> representation" of certain characters, including Emoji.  So it sounds
    Eli> like the rule for compose-gstring-for-variation-glyph is not really
    Eli> relevant for it, as well as for VS-16.  Right?

The VS-15 rule actually needs to continue to use
compose-gstring-for-variation-glyph, because if I change it to use
compose-gstring-for-graphic, the font_range change kicks in, and Emacs
displays the Emoji presentation for codepoints followed by VS-15.

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: wyuenho@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 13 Oct 2021 16:52:52 +0200
> 
> So if I change emoji-zwj.awk to emit codepoint+U+FE0F rules for these
> problematic codepoints which have Emoji_presentation = No, and change
> font_range to try looking up the emoji font for them, all the
> 'codepoint+U+FE0F' from Jimmy's test file display as Emoji, and the
> 'codepoint+U+FE0E' display as text.
> 
> It would mean emitting rules in emoji-zwj.awk for, and checking in
> font_range for, the following codepoints.
> 
> U+261D, U+26F9, U+270C, U+270D, U+2764, U+1F3CB, U+1F3CC, U+1F3F3,
> U+1F3F4, U+1F441, U+1F574, U+1F575, U+1F590
> 
> As an added bonus, those U+1xxxx codepoints would no longer be part of the
> 'emoji' script.

SGTM, thanks.

I guess we should have those special codepoints in some list exposed
to Lisp?

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: alan@idiocy.org,  wyuenho@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 13 Oct 2021 16:59:15 +0200
> 
>     Eli> But VS-15 is not about variant glyphs, it's about requesting the "text
>     Eli> representation" of certain characters, including Emoji.  So it sounds
>     Eli> like the rule for compose-gstring-for-variation-glyph is not really
>     Eli> relevant for it, as well as for VS-16.  Right?
> 
> The VS-15 rule actually needs to continue to use
> compose-gstring-for-variation-glyph, because if I change it to use
> compose-gstring-for-graphic, the font_range change kicks in, and Emacs
> displays the Emoji presentation for codepoints followed by VS-15.

OK.  We should carefully document all this complexity in some
comments, I think.

Thanks.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Wed, 13 Oct 2021 18:54:33 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: wyuenho@gmail.com,  emacs-devel@gnu.org
    >> Date: Wed, 13 Oct 2021 16:52:52 +0200
    >> 
    >> So if I change emoji-zwj.awk to emit codepoint+U+FE0F rules for these
    >> problematic codepoints which have Emoji_presentation = No, and change
    >> font_range to try looking up the emoji font for them, all the
    >> 'codepoint+U+FE0F' from Jimmy's test file display as Emoji, and the
    >> 'codepoint+U+FE0E' display as text.
    >> 
    >> It would mean emitting rules in emoji-zwj.awk for, and checking in
    >> font_range for, the following codepoints.
    >> 
    >> U+261D, U+26F9, U+270C, U+270D, U+2764, U+1F3CB, U+1F3CC, U+1F3F3,
    >> U+1F3F4, U+1F441, U+1F574, U+1F575, U+1F590
    >> 
    >> As an added bonus, those U+1xxxx codepoints would no longer be part of the
    >> 'emoji' script.

    Eli> SGTM, thanks.

    Eli> I guess we should have those special codepoints in some list exposed
    Eli> to Lisp?

That would avoid having to update two locations if the list ever
changes, since we could use the list to do the checking in
font_range.. Would you prefer 'list of codepoints' or 'list of 1-char
strings containing codepoints'? The latter has a friendlier
appearance, but would be slightly less efficient due to the need to
convert between codepoints and 1-char strings.

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Wed, 13 Oct 2021 18:56:04 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: alan@idiocy.org,  wyuenho@gmail.com,  emacs-devel@gnu.org
    >> Date: Wed, 13 Oct 2021 16:59:15 +0200
    >> 
    Eli> But VS-15 is not about variant glyphs, it's about requesting the "text
    Eli> representation" of certain characters, including Emoji.  So it sounds
    Eli> like the rule for compose-gstring-for-variation-glyph is not really
    Eli> relevant for it, as well as for VS-16.  Right?
    >> 
    >> The VS-15 rule actually needs to continue to use
    >> compose-gstring-for-variation-glyph, because if I change it to use
    >> compose-gstring-for-graphic, the font_range change kicks in, and Emacs
    >> displays the Emoji presentation for codepoints followed by VS-15.

    Eli> OK.  We should carefully document all this complexity in some
    Eli> comments, I think.

I suspect getting the comments right will take at least as long as
getting the code right :-)

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: wyuenho@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 14 Oct 2021 12:27:58 +0200
> 
>     Eli> I guess we should have those special codepoints in some list exposed
>     Eli> to Lisp?
> 
> That would avoid having to update two locations if the list ever
> changes, since we could use the list to do the checking in
> font_range.. Would you prefer 'list of codepoints' or 'list of 1-char
> strings containing codepoints'? The latter has a friendlier
> appearance, but would be slightly less efficient due to the need to
> convert between codepoints and 1-char strings.

I think a list of codepoints is better, since usually that's how this
stuff gets communicated.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Thu, 14 Oct 2021 12:28:42 +0200
> Cc: alan@idiocy.org, wyuenho@gmail.com, emacs-devel@gnu.org
> 
>     Eli> OK.  We should carefully document all this complexity in some
>     Eli> comments, I think.
> 
> I suspect getting the comments right will take at least as long as
> getting the code right :-)

Feel free to leave that to me.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Thu, 14 Oct 2021 14:31:51 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: wyuenho@gmail.com,  emacs-devel@gnu.org
    >> Date: Thu, 14 Oct 2021 12:27:58 +0200
    >> 
    Eli> I guess we should have those special codepoints in some list exposed
    Eli> to Lisp?
    >> 
    >> That would avoid having to update two locations if the list ever
    >> changes, since we could use the list to do the checking in
    >> font_range.. Would you prefer 'list of codepoints' or 'list of 1-char
    >> strings containing codepoints'? The latter has a friendlier
    >> appearance, but would be slightly less efficient due to the need to
    >> convert between codepoints and 1-char strings.

    Eli> I think a list of codepoints is better, since usually that's how this
    Eli> stuff gets communicated.

OK. emacs-28 or master? (Iʼd argue this is a bugfix to a new feature in
emacs-28)

Robert
-- 

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: wyuenho@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 18 Oct 2021 12:09:26 +0200
> 
>     Eli> I guess we should have those special codepoints in some list exposed
>     Eli> to Lisp?
>     >> 
>     >> That would avoid having to update two locations if the list ever
>     >> changes, since we could use the list to do the checking in
>     >> font_range.. Would you prefer 'list of codepoints' or 'list of 1-char
>     >> strings containing codepoints'? The latter has a friendlier
>     >> appearance, but would be slightly less efficient due to the need to
>     >> convert between codepoints and 1-char strings.
> 
>     Eli> I think a list of codepoints is better, since usually that's how this
>     Eli> stuff gets communicated.
> 
> OK. emacs-28 or master? (Iʼd argue this is a bugfix to a new feature in
> emacs-28)

Yes, emacs-28, please.

Thanks.

Re: Unicode 13 Emoji ranges composed with wrong font on NS port

[Robert Pluim]

>>>>> On Mon, 18 Oct 2021 15:58:43 +0300, Eli Zaretskii <eliz@gnu.org> said:
 
    >> OK. emacs-28 or master? (Iʼd argue this is a bugfix to a new feature in
    >> emacs-28)

    Eli> Yes, emacs-28, please.

Pushed to emacs-28 as 9bd2f59db6

thanks

Robert
--