Entering emojis

Entering emojis

[Lars Ingebrigtsen]

Emacs can display such nice glyphs as 🏴‍☠️, 🧝🏾‍♂️ and  👨🏾‍🦰 now, but has anybody
made a package to compose them?  Or rather -- to navigate the Emoji
Space Of Possibilities?

I don't really have any idea of what that would look like in an Emacs
context, though.  Perhaps something with transient.el?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 25 Oct 2021 20:32:24 +0200
> 
> Emacs can display such nice glyphs as 🏴‍☠️, 🧝🏾‍♂️ and  👨🏾‍🦰 now, but has anybody
> made a package to compose them?  Or rather -- to navigate the Emoji
> Space Of Possibilities?
> 
> I don't really have any idea of what that would look like in an Emacs
> context, though.  Perhaps something with transient.el?

Rather an input method, I'd say.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't really have any idea of what that would look like in an Emacs
>> context, though.  Perhaps something with transient.el?
>
> Rather an input method, I'd say.

I think you want to see how things look?  With an input method you'd
just end up with the result, but won't see the possibilities?

Say you want to insert a construction worker, then these are the
options:

construction worker (👷🏻)
construction worker (👷🏼)
construction worker (👷🏽)
construction worker (👷🏾)
construction worker (👷🏿)
man construction worker  (👷‍♂️)
man construction worker: (👷🏿‍♂️)
man construction worker: (👷🏻‍♂️)
man construction worker: (👷🏽‍♂️)
man construction worker: (👷🏾‍♂️)
man construction worker: (👷🏼‍♂️)
woman construction worker  (👷‍♀️)
woman construction worker: (👷🏿‍♀️)
woman construction worker: (👷🏻‍♀️)
woman construction worker: (👷🏽‍♀️)
woman construction worker: (👷🏾‍♀️)
woman construction worker: (👷🏼‍♀️)

We could do an input method that's thing/gender/hue (when it's about a
person; other things have other possibilities), and after typing that
triplet, you end up with, for instance, 👷🏿‍♀️.  But I think that after
deciding on "thing", you probably want to see the 17 options you have,
and when you're narrowing down by gender or hue, you see the options
you're left with.

Can you do that with an input method?  I've only used fancy
input methods when trying to debug something, so perhaps this is
something you can do with input methods, but I assumed not.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 25 Oct 2021 21:21:06 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I don't really have any idea of what that would look like in an Emacs
> >> context, though.  Perhaps something with transient.el?
> >
> > Rather an input method, I'd say.
> 
> I think you want to see how things look?  With an input method you'd
> just end up with the result, but won't see the possibilities?

That's not true, an input method can show the possible candidates in
the echo area.  Try some CJK input method, and you will see that.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That's not true, an input method can show the possible candidates in
> the echo area.  Try some CJK input method, and you will see that.

Hm, yes...  I tried chinese-zozy, and that does work, but it feels like
it requires some training to get used to.  Probably very efficient once
you can the system, but for something that you don't use all the time?

Do you know an example input method that has a lot of variants?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 25 Oct 2021 21:36:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That's not true, an input method can show the possible candidates in
> > the echo area.  Try some CJK input method, and you will see that.
> 
> Hm, yes...  I tried chinese-zozy, and that does work, but it feels like
> it requires some training to get used to.  Probably very efficient once
> you can the system, but for something that you don't use all the time?

Well, all you need is to type the ordinal number, how hard is that?
And it isn't like you are going to type too many Emoji in a row.

> Do you know an example input method that has a lot of variants?

How many would qualify as "a lot"?  Is 12 enough?

Alternatively, we could have a special command, insert-emoji, which
would just complete your input using the table of all the known Emoji
sequences.  Then you type "constr TAB TAB" and get the list you've
shown in the earlier message.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Well, all you need is to type the ordinal number, how hard is that?
> And it isn't like you are going to type too many Emoji in a row.

You have to know how to get started.

>> Do you know an example input method that has a lot of variants?
>
> How many would qualify as "a lot"?  Is 12 enough?

Sure.  It looks like 18 variants is the max in emoji-land.

> Alternatively, we could have a special command, insert-emoji, which
> would just complete your input using the table of all the known Emoji
> sequences.  Then you type "constr TAB TAB" and get the list you've
> shown in the earlier message.

I think emojis are more in the area of "I know what I want when I've
seen it".

I've now actually had a look at what my Android phone does here, and it
seems quite nice, and is probably what people are used to.

It pops up a long field of emojis to choose from, grouped by theme:
First you get all the smileys, then all the people, then "animals and
nature", and then "food and beverage", etc.

When you long-hold the people, you then get to see the (up to) 18
variations per person.

I didn't know 😹 existed before I saw it, and I don't think I would have
navigated to it via a textual interface.

Perhaps somebody who uses emojis a lot has some ideas for how this
should be in Emacs.  😎

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Gregory Heytings]

>
> I've now actually had a look at what my Android phone does here, and it 
> seems quite nice, and is probably what people are used to.
>

Actually this is not the best interface, it was okay five years ago or so, 
when the number of individual emojis was still reasonable, but that method 
doesn't scale well, now that there are more and more emojis.  I think the 
ideal interface to enter emojis would be to have:

1. a "recently used" list,

2. a search field in which you type something to describe the emoji you 
want, like "pig", which displays a list with the "pig", "pig nose" and 
"pig face" emojis,

3. and on the elements of these two lists, a way to select one of its 
variants.

Re: Entering emojis

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> I think the ideal interface to enter emojis would be to have:
>
> 1. a "recently used" list,
>
> 2. a search field in which you type something to describe the emoji
> you want, like "pig", which displays a list with the "pig", "pig nose"
> and "pig face" emojis,

But how do you discover which emojis exist?  It didn't occur to me that
💰 existed before I saw it, but now that I know it exists, I'll find a
way to work it into a conversation.  😪

I think exploration has to be a feature of the interface, as well as a
fast way to get to the ones you know you want.  So, sure, "recently
used" would be useful, but we need something more than textual search in
addition.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Howard Melman]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Gregory Heytings <gregory@heytings.org> writes:
>
>> I think the ideal interface to enter emojis would be to have:
>>
>> 1. a "recently used" list,
>>
>> 2. a search field in which you type something to describe the emoji
>> you want, like "pig", which displays a list with the "pig", "pig nose"
>> and "pig face" emojis,
>
> But how do you discover which emojis exist?  It didn't occur to me that
> 💰 existed before I saw it, but now that I know it exists, I'll find a
> way to work it into a conversation.  😪
>
> I think exploration has to be a feature of the interface, as well as a
> fast way to get to the ones you know you want.  So, sure, "recently
> used" would be useful, but we need something more than textual search in
> addition.

IOS is similar to what you described, and the first block is
"recents" and there's a search field.  So it covers all of
the above.

insert-char can be used, but I agree it would be nice if
there was a command that limited the completion table to
just emoji.  I do like the idea of a transient for selecting variants.

-- 

Howard

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 25 Oct 2021 22:49:16 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> But how do you discover which emojis exist?  It didn't occur to me that
> 💰 existed before I saw it, but now that I know it exists, I'll find a
> way to work it into a conversation.  😪

We could have a separate list-emojis-display command for those who
want to explore.  IOW, that doesn't have to be part of the way we
provide to input Emoji for those who already know what they want.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We could have a separate list-emojis-display command for those who
> want to explore.  IOW, that doesn't have to be part of the way we
> provide to input Emoji for those who already know what they want.

Yup.

Anyway, I couldn't help myself, so I implemented an input command based
on transient.el (pushed to a branch for those who are interested), but
transient.el doesn't really give us much here (creating these interfaces
on the fly), and it doesn't know how to line up stuff that's not
monospaced, so I think I'll rewrite the display bit tomorrow without
transient.el.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Anyway, I couldn't help myself, so I implemented an input command based
> on transient.el (pushed to a branch for those who are interested),

So I checked out the branch, but what do I type to test it?

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> So I checked out the branch, but what do I type to test it?

I forgot to add a top-level command.  Now pushed -- `M-x emoji-insert'.

It bugs out on certain sequences of commands due to some transient.el
formatting issues.  I'm not sure whether to rewrite it not to use
transient or to fix transient so that it works with variable width
glyphs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas, I've been experimenting with using transient.el to implement a
new UI for entering emojis.  This works fine except for one major
detail -- it seems like transient doesn't really like lining things up
into columns by their displayed pixel width?

I haven't looked at transient.el internals in details, but extending a
columnar display to variable-pitch circumstances usually isn't a lot of
work: You basically just use window-text-pixel-size to get the width of
the texts, and align with an :align-to display spec.

Have you looked at extending transient this way?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I haven't looked at transient.el internals in details, but extending a
> columnar display to variable-pitch circumstances usually isn't a lot of
> work: You basically just use window-text-pixel-size to get the width of
> the texts, and align with an :align-to display spec.
>
> Have you looked at extending transient this way?

I've now done so -- by adding a new slot to the transient-prefix class.
I don't know whether that's the most likely place to stash the info, but
this seems to work.  Perhaps stashing it in transient-columns would make
more sense...  Or at least propagating it there.

What do you think?  (It should be fully backwards compatible -- the
output when not supplying this keyword should be identical to the old
output.)

diff --git a/lisp/transient.el b/lisp/transient.el
index c33a4c722a..13dfbf53ec 100644
--- a/lisp/transient.el
+++ b/lisp/transient.el
@@ -590,7 +590,8 @@ transient-prefix
    (transient-suffix     :initarg :transient-suffix     :initform nil)
    (transient-non-suffix :initarg :transient-non-suffix :initform nil)
    (incompatible         :initarg :incompatible         :initform nil)
-   (suffix-description   :initarg :suffix-description))
+   (suffix-description   :initarg :suffix-description)
+   (variable-pitch       :initarg :variable-pitch       :initform nil))
   "Transient prefix command.
 
 Each transient prefix command consists of a command, which is
@@ -2941,6 +2942,18 @@ transient--insert-group
       (unless (string-match-p ".\n\\'" str)
         (insert ?\n)))))
 
+(defun transient--pixel-width (string)
+  (with-temp-buffer
+    (insert string)
+    (if (not (get-buffer-window (current-buffer)))
+        (save-window-excursion
+          ;; Avoid errors if the selected window is a dedicated one,
+          ;; and they just want to insert a document into it.
+          (set-window-dedicated-p nil nil)
+	  (set-window-buffer nil (current-buffer))
+	  (car (window-text-pixel-size nil (line-beginning-position) (point))))
+      (car (window-text-pixel-size nil (line-beginning-position) (point))))))
+
 (cl-defmethod transient--insert-group ((group transient-columns))
   (let* ((columns
           (mapcar
@@ -2951,11 +2964,18 @@ transient--insert-group
                  (push desc rows))
                rows))
            (oref group suffixes)))
+         (vp (oref transient--prefix variable-pitch))
          (rs (apply #'max (mapcar #'length columns)))
          (cs (length columns))
-         (cw (mapcar (lambda (col) (apply #'max (mapcar #'length col)))
+         (cw (mapcar (lambda (col)
+                       (apply #'max (mapcar (if vp
+                                                #'transient--pixel-width
+                                              #'length)
+                                            col)))
                      columns))
-         (cc (transient--seq-reductions-from (apply-partially #'+ 3) cw 0)))
+         (cc (transient--seq-reductions-from
+              (apply-partially #'+ (if vp 30 3))
+              cw 0)))
     (if transient-force-single-column
         (dotimes (c cs)
           (dotimes (r rs)
@@ -2966,11 +2986,21 @@ transient--insert-group
             (insert ?\n)))
       (dotimes (r rs)
         (dotimes (c cs)
-          (insert (make-string (- (nth c cc) (current-column)) ?\s))
-          (when-let ((cell (nth r (nth c columns))))
-            (insert cell))
-          (when (= c (1- cs))
-            (insert ?\n)))))))
+          (if vp
+              (progn
+                (when-let ((cell (nth r (nth c columns))))
+                  (insert cell))
+                (if (= c (1- cs))
+                    (insert ?\n)
+                  (insert
+                   (propertize
+	            " " 'display
+                    `(space :align-to (,(nth (1+ c) cc)))))))
+            (insert (make-string (- (nth c cc) (current-column)) ?\s))
+            (when-let ((cell (nth r (nth c columns))))
+              (insert cell))
+            (when (= c (1- cs))
+              (insert ?\n))))))))
 
 (cl-defmethod transient--insert-group ((group transient-subgroups))
   (let* ((subgroups (oref group suffixes))


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> +(defun transient--pixel-width (string)

(That was an old, buggy version of that function; new version of it on
the branch.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It bugs out on certain sequences of commands due to some transient.el
> formatting issues.  I'm not sure whether to rewrite it not to use
> transient or to fix transient so that it works with variable width
> glyphs.

I tested it, and I like it.  Transient seems like a good choice for
something this.  Another idea is to have a grid where you can pick one
using arrow keys and RET, which might feel more familiar to some users
(I'm not sure if transient supports this).  Or perhaps a combination
of the two?

(Unrelatedly, almost no smileys display on my macOS machine, but I
believe this is a known issue.)

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> I tested it, and I like it.  Transient seems like a good choice for
> something this.

Thanks; yes, it feels quite efficient to choose stuff via the transient
interface.  (And I've hacked it up with a proof-of-concept variable
pitch alignment on that branch.)

You can now pick among all the normal emojis, but I've yet to add the
hue/gender variations and stuff.  There's also the question of the
VARIATION SELECTOR-16 stuff and how that relates to all this, which I'm
still quite not sure...

> Another idea is to have a grid where you can pick one
> using arrow keys and RET, which might feel more familiar to some users
> (I'm not sure if transient supports this).  Or perhaps a combination
> of the two?

Yes, a huge grid (divided into sections) is probably also a good thing,
but that would be a separate command, like `list-emojis' or something.

I also thing a separate command for searching emojis by name would be
good (and then drop into the transient interface to choose the
hue/gender variations), probably.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

So I'm trying to figure out how this all maps up.

In the labels file, we have (for instance) 👮‍♂️ (a male police officer).
I can find that glyph in emoji-zwj-sequences:

1F46E 200D 2642 FE0F                          ; man police officer                                             # E4.0   [1] (👮‍♂️)
1F46E 1F3FB 200D 2640 FE0F                    ; woman police officer: light skin tone                          # E4.0   [1] (👮🏻‍♀️)
1F46E 1F3FB 200D 2642 FE0F                    ; man police officer: light skin tone                            # E4.0   [1] (👮🏻‍♂️)
1F46E 1F3FC 200D 2640 FE0F                    ; woman police officer: medium-light skin tone                   # E4.0   [1] (👮🏼‍♀️)
1F46E 1F3FC 200D 2642 FE0F                    ; man police officer: medium-light skin tone                     # E4.0   [1] (👮🏼‍♂️)
1F46E 1F3FD 200D 2640 FE0F                    ; woman police officer: medium skin tone                         # E4.0   [1] (👮🏽‍♀️)

etc.  But there's no mapping from that glyph to these other ones except
by ... being in the vicinity...  and the "woman" forms aren't variants.
Hm...

Aha!  common/annotationsDerived/en.xml has

		<annotation cp="👮🏻‍♂" type="tts">man police officer: light skin tone</annotation>
		<annotation cp="👮🏼‍♂" type="tts">man police officer: medium-light skin tone</annotation>
		<annotation cp="👮🏽‍♂" type="tts">man police officer: medium skin tone</annotation>

So I can find "man police officer" in the sequences file, and then get
the derivations from that XML file?  Geez.  Well, that sounds doable,
and I hope that those names for the glyphs are the same in both files.
:-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> etc.  But there's no mapping from that glyph to these other ones except
> by ... being in the vicinity...  and the "woman" forms aren't variants.
> Hm...

Dur!  It's really simple -- the standalone-ish glyphs are the ones
without a colon in the name, and then the varieties are with a colon:

1F46E 200D 2640 FE0F             ; woman police officer                                           # E4.0   [1] (👮‍♀️)
1F46E 200D 2642 FE0F             ; man police officer                                             # E4.0   [1] (👮‍♂️)
1F46E 1F3FB 200D 2640 FE0F       ; woman police officer: light skin tone                          # E4.0   [1] (👮🏻‍♀️)
1F46E 1F3FB 200D 2642 FE0F       ; man police officer: light skin tone                            # E4.0   [1] (👮🏻‍♂️)

So that should be easy enough without adding any new data files.  I
wonder how one gets from "police officer" to "man police officer"/"woman
police officer", but perhaps these names are also totally regular and
just adds "man " and "woman " to the start?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>   Emacs developers <emacs-devel@gnu.org>
> Date: Tue, 26 Oct 2021 18:42:00 +0200
> 
> I wonder how one gets from "police officer" to "man police
> officer"/"woman police officer", but perhaps these names are also
> totally regular and just adds "man " and "woman " to the start?

What exactly is the problem?  The ♀ or ♂ determine the gender, but I'm
sure you already know that.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I wonder how one gets from "police officer" to "man police
>> officer"/"woman police officer", but perhaps these names are also
>> totally regular and just adds "man " and "woman " to the start?
>
> What exactly is the problem?  The ♀ or ♂ determine the gender, but I'm
> sure you already know that.

I'm wondering whether anybody has any insight into how these files are
meant to be mapped to/from glyphs to variants.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>   Emacs developers <emacs-devel@gnu.org>
> Date: Tue, 26 Oct 2021 18:36:25 +0200
> 
> So I'm trying to figure out how this all maps up.
> 
> In the labels file, we have (for instance) 👮‍♂️ (a male police officer).
> I can find that glyph in emoji-zwj-sequences:
> 
> 1F46E 200D 2642 FE0F                          ; man police officer                                             # E4.0   [1] (👮‍♂️)
> 1F46E 1F3FB 200D 2640 FE0F                    ; woman police officer: light skin tone                          # E4.0   [1] (👮🏻‍♀️)
> 1F46E 1F3FB 200D 2642 FE0F                    ; man police officer: light skin tone                            # E4.0   [1] (👮🏻‍♂️)
> 1F46E 1F3FC 200D 2640 FE0F                    ; woman police officer: medium-light skin tone                   # E4.0   [1] (👮🏼‍♀️)
> 1F46E 1F3FC 200D 2642 FE0F                    ; man police officer: medium-light skin tone                     # E4.0   [1] (👮🏼‍♂️)
> 1F46E 1F3FD 200D 2640 FE0F                    ; woman police officer: medium skin tone                         # E4.0   [1] (👮🏽‍♀️)
> 
> etc.  But there's no mapping from that glyph to these other ones except
> by ... being in the vicinity...  and the "woman" forms aren't variants.
> Hm...
> 
> Aha!  common/annotationsDerived/en.xml has
> 
> 		<annotation cp="👮🏻‍♂" type="tts">man police officer: light skin tone</annotation>
> 		<annotation cp="👮🏼‍♂" type="tts">man police officer: medium-light skin tone</annotation>
> 		<annotation cp="👮🏽‍♂" type="tts">man police officer: medium skin tone</annotation>
> 
> So I can find "man police officer" in the sequences file, and then get
> the derivations from that XML file?  Geez.  Well, that sounds doable,
> and I hope that those names for the glyphs are the same in both files.
> :-/

I don't think I understand the problem.  The first 2 codepoints are in
admin/unidata/emoji-sequences.txt, and the gender thingy is what
determines if its "man" or "woman".  VS-16 is a no-op, and I'm not
even sure you should produce it in these sequences.  It is only needed
when the original character is not an emoji.

Re: Entering emojis

[Robert Pluim]

>>>>> On Tue, 26 Oct 2021 19:49:56 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
    >> Emacs developers <emacs-devel@gnu.org>
    >> Date: Tue, 26 Oct 2021 18:36:25 +0200
    >> 
    >> So I'm trying to figure out how this all maps up.
    >> 
    >> In the labels file, we have (for instance) 👮‍♂️ (a male police officer).
    >> I can find that glyph in emoji-zwj-sequences:
    >> 
    >> 1F46E 200D 2642 FE0F                          ; man police officer                                             # E4.0   [1] (👮‍♂️)
    >> 1F46E 1F3FB 200D 2640 FE0F                    ; woman police officer: light skin tone                          # E4.0   [1] (👮🏻‍♀️)
    >> 1F46E 1F3FB 200D 2642 FE0F                    ; man police officer: light skin tone                            # E4.0   [1] (👮🏻‍♂️)
    >> 1F46E 1F3FC 200D 2640 FE0F                    ; woman police officer: medium-light skin tone                   # E4.0   [1] (👮🏼‍♀️)
    >> 1F46E 1F3FC 200D 2642 FE0F                    ; man police officer: medium-light skin tone                     # E4.0   [1] (👮🏼‍♂️)
    >> 1F46E 1F3FD 200D 2640 FE0F                    ; woman police officer: medium skin tone                         # E4.0   [1] (👮🏽‍♀️)
    >> 
    >> etc.  But there's no mapping from that glyph to these other ones except
    >> by ... being in the vicinity...  and the "woman" forms aren't variants.
    >> Hm...
    >> 
    >> Aha!  common/annotationsDerived/en.xml has
    >> 
    >> <annotation cp="👮🏻‍♂" type="tts">man police officer: light skin tone</annotation>
    >> <annotation cp="👮🏼‍♂" type="tts">man police officer: medium-light skin tone</annotation>
    >> <annotation cp="👮🏽‍♂" type="tts">man police officer: medium skin tone</annotation>
    >> 
    >> So I can find "man police officer" in the sequences file, and then get
    >> the derivations from that XML file?  Geez.  Well, that sounds doable,
    >> and I hope that those names for the glyphs are the same in both files.
    >> :-/

    Eli> I don't think I understand the problem.  The first 2 codepoints are in
    Eli> admin/unidata/emoji-sequences.txt, and the gender thingy is what
    Eli> determines if its "man" or "woman".  VS-16 is a no-op, and I'm not
    Eli> even sure you should produce it in these sequences.  It is only needed
    Eli> when the original character is not an emoji.

Itʼs not a no-op: it modifies U+2640 or U+2642

Iʼm not sure I understand the issue either: the base codepoint is
U+1F46E, and emoji-zwj-sequences tells you what the sequences
are. What else is needed?

Robert
-- 

Re: Entering emojis

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Gmane-Reply-To-List: yes
> Date: Tue, 26 Oct 2021 19:09:03 +0200
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org,
>  gregory@heytings.org, stefankangas@gmail.com
> 
> >>>>> On Tue, 26 Oct 2021 19:49:56 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> I don't think I understand the problem.  The first 2 codepoints are in
>     Eli> admin/unidata/emoji-sequences.txt, and the gender thingy is what
>     Eli> determines if its "man" or "woman".  VS-16 is a no-op, and I'm not
>     Eli> even sure you should produce it in these sequences.  It is only needed
>     Eli> when the original character is not an emoji.
> 
> Itʼs not a no-op: it modifies U+2640 or U+2642

OK, so a different kind of "no-op": always follows those two, if they
are present.

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Itʼs not a no-op: it modifies U+2640 or U+2642
>
> Iʼm not sure I understand the issue either: the base codepoint is
> U+1F46E, and emoji-zwj-sequences tells you what the sequences
> are. What else is needed?

There's no VS-16 mixed up with these glyphs -- that a separate issue
that I haven't even tried to understand yet.  :-)

But back to the people.  For instance:

1F3CC 1F3FB 200D 2640 FE0F                  ; RGI_Emoji_ZWJ_Sequence  ; woman golfing: light skin tone                                 # E4.0   [1] (🏌🏻‍♀️)

(Oh, heh, that one even had a VS-16...)

How am I supposed to go from GOLFER to that glyph?  From POLICE OFFICER
it's no problem getting to "woman police officer: light skin tone",
because those use the same name in the UCS file and in the zwj file, but
GOLFING isn't the same as GOLFER.

Are the names just informational, and I'm supposed to look this up based
on 🏌GOLFER being #x1F3CC?  So the first codepoint is what matters for
determining the variants?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  gregory@heytings.org,
>   stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 26 Oct 2021 19:33:35 +0200
> 
> How am I supposed to go from GOLFER to that glyph?  From POLICE OFFICER
> it's no problem getting to "woman police officer: light skin tone",
> because those use the same name in the UCS file and in the zwj file, but
> GOLFING isn't the same as GOLFER.

How many such cases are there?  Can't you have a small database of
such "translations"?

> So the first codepoint is what matters for determining the variants?

Yes, AFAIK.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> How am I supposed to go from GOLFER to that glyph?  From POLICE OFFICER
>> it's no problem getting to "woman police officer: light skin tone",
>> because those use the same name in the UCS file and in the zwj file, but
>> GOLFING isn't the same as GOLFER.
>
> How many such cases are there?  Can't you have a small database of
> such "translations"?

I'd prefer things to work automatically -- then there'll be no need to
maintain this stuff as Unicode adds new things every year.  But...
that's perhaps a forlorn hope.  We'll see; things seem to be working
pretty well now with:

>> So the first codepoint is what matters for determining the variants?
>
> Yes, AFAIK.

I'm now doing (some) mapping based on that instead, and that does indeed
fix the problem with golfing.  But it seems like it might give some
false positives (i.e., it thinks that some things that shouldn't be
derived are derived), so more tweaking might be needed in the algorithm.
I think it's "basically working", but I need to rewrite that algo
anyway, because it's a bit of a mess with all the tweaking back and
forth, and I may just have confused myself...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> There's no VS-16 mixed up with these glyphs -- that a separate issue
> that I haven't even tried to understand yet.  :-)

OK, here's my VS-16 question.

WARNING SIGN looks like ⚠.  With a VS-16 it looks like ⚠️.  How do I know
that I'm supposed to add a VS-16 to that character to get the emoji?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]


>
> OK, here's my VS-16 question.
>
> WARNING SIGN looks like ⚠.  With a VS-16 it looks like ⚠️.  How do I 
> know that I'm supposed to add a VS-16 to that character to get the 
> emoji?
>

Each emoji has an Emoji_Presentation property.

Those with Emoji_Presentation = yes should by default have an emoji 
presentation (the second warning sign above).

Those with Emoji_Presentation = no should by default have a text 
presentation (the first warning sign above).

The variation selectors are used as a toggle: VS-15 is used to request a 
text presentation for an emoji which has an Emoji_Presentation = yes, 
VS-15 is used to request an emoji presentation for an emoji which has an 
Emoji_Presentation = no.

Re: Entering emojis

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> The variation selectors are used as a toggle: VS-15 is used to request
> a text presentation for an emoji which has an Emoji_Presentation =
> yes, VS-15 is used to request an emoji presentation for an emoji which
> has an Emoji_Presentation = no.

Ah, thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Tue, 26 Oct 2021 19:43:37 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Lars Ingebrigtsen <larsi@gnus.org> writes:
    >> There's no VS-16 mixed up with these glyphs -- that a separate issue
    >> that I haven't even tried to understand yet.  :-)

    Lars> OK, here's my VS-16 question.

    Lars> WARNING SIGN looks like ⚠.  With a VS-16 it looks like ⚠️.  How do I know
    Lars> that I'm supposed to add a VS-16 to that character to get the emoji?

If itʼs not in the 'emoji script you need the VS-16. Itʼs an emoji,
but it has Emoji_Presentation = No. Donʼt ask.

(aref char-script-table #x26a0) => symbol

Robert
-- 

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> If itʼs not in the 'emoji script you need the VS-16. Itʼs an emoji,
> but it has Emoji_Presentation = No. Donʼt ask.

:-)

> (aref char-script-table #x26a0) => symbol

Thanks; with that I'm getting all the correct emojis inserted in that
area of the woods, like ☢️.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Tue, 26 Oct 2021 19:33:35 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> Itʼs not a no-op: it modifies U+2640 or U+2642
    >> 
    >> Iʼm not sure I understand the issue either: the base codepoint is
    >> U+1F46E, and emoji-zwj-sequences tells you what the sequences
    >> are. What else is needed?

    Lars> There's no VS-16 mixed up with these glyphs -- that a separate issue
    Lars> that I haven't even tried to understand yet.  :-)

If you want them to display correctly in Emacs youʼre going to have to
use the sequences in emoji-zwj-sequences.txt and emoji-sequences.txt

    Lars> But back to the people.  For instance:

    Lars> 1F3CC 1F3FB 200D 2640 FE0F ; RGI_Emoji_ZWJ_Sequence ; woman golfing:
    Lars> light skin tone # E4.0 [1] (🏌🏻‍♀️)

    Lars> (Oh, heh, that one even had a VS-16...)

Yes, because of the U+2640

    Lars> How am I supposed to go from GOLFER to that glyph?  From POLICE OFFICER
    Lars> it's no problem getting to "woman police officer: light skin tone",
    Lars> because those use the same name in the UCS file and in the zwj file, but
    Lars> GOLFING isn't the same as GOLFER.

    Lars> Are the names just informational, and I'm supposed to look this up based
    Lars> on 🏌GOLFER being #x1F3CC?  So the first codepoint is what matters for
    Lars> determining the variants?

The field in emoji-zwj-sequences.txt is called 'descriptions', and I
think theyʼre not intended to be normative in any way, so probably
best to use the base codepoint.

Robert
-- 

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> If you want them to display correctly in Emacs youʼre going to have to
> use the sequences in emoji-zwj-sequences.txt and emoji-sequences.txt

I'm using the labels file, because it had better taxonomy...

> The field in emoji-zwj-sequences.txt is called 'descriptions', and I
> think theyʼre not intended to be normative in any way, so probably
> best to use the base codepoint.

Right...  So in emoji-sequences, we have

1F46E..1F4AC  ; Basic_Emoji                  ; police officer                                                 # E0.6  [63] (👮..💬)

and then

1F46E 1F3FB   ; RGI_Emoji_Modifier_Sequence  ; police officer: light skin tone                                # E1.0   [1] (👮🏻)
1F46E 1F3FC   ; RGI_Emoji_Modifier_Sequence  ; police officer: medium-light skin tone                         # E1.0   [1] (👮🏼)
1F46E 1F3FD   ; RGI_Emoji_Modifier_Sequence  ; police officer: medium skin tone                               # E1.0   [1] (👮🏽)

and then in the zwj file we have

1F46E 200D 2640 FE0F                        ; RGI_Emoji_ZWJ_Sequence  ; woman police officer                                           # E4.0   [1] (👮‍♀️)
1F46E 200D 2642 FE0F                        ; RGI_Emoji_ZWJ_Sequence  ; man police officer                                             # E4.0   [1] (👮‍♂️)
1F46E 1F3FB 200D 2640 FE0F                  ; RGI_Emoji_ZWJ_Sequence  ; woman police officer: light skin tone                          # E4.0   [1] (👮🏻‍♀️)
1F46E 1F3FB 200D 2642 FE0F                  ; RGI_Emoji_ZWJ_Sequence  ; man police officer: light skin tone                            # E4.0   [1] (👮🏻‍♂️)

So that all matches up, and I can go from 1F46E and get all the
variants.  Seems very promising; I'll give it a go.

But...  I can't go from "woman police officer" to "woman police officer:
light skin tone" by looking at the first code point.  Er...  what's the
rule here, then?

1F46E plus 2640 as the next-to-last code point?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Tue, 26 Oct 2021 19:58:28 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:


    Lars> Right...  So in emoji-sequences, we have

    Lars> 1F46E..1F4AC  ; Basic_Emoji                  ; police officer                                                 # E0.6  [63] (👮..💬)

    Lars> and then

    Lars> 1F46E 1F3FB   ; RGI_Emoji_Modifier_Sequence  ; police officer: light skin tone                                # E1.0   [1] (👮🏻)
    Lars> 1F46E 1F3FC   ; RGI_Emoji_Modifier_Sequence  ; police officer: medium-light skin tone                         # E1.0   [1] (👮🏼)
    Lars> 1F46E 1F3FD   ; RGI_Emoji_Modifier_Sequence  ; police officer: medium skin tone                               # E1.0   [1] (👮🏽)

    Lars> and then in the zwj file we have

    Lars> 1F46E 200D 2640 FE0F ; RGI_Emoji_ZWJ_Sequence ; woman police officer #
    Lars> E4.0 [1] (👮‍♀️)
    Lars> 1F46E 200D 2642 FE0F ; RGI_Emoji_ZWJ_Sequence ; man police officer #
    Lars> E4.0 [1] (👮‍♂️)
    Lars> 1F46E 1F3FB 200D 2640 FE0F ; RGI_Emoji_ZWJ_Sequence ; woman police
    Lars> officer: light skin tone # E4.0 [1] (👮🏻‍♀️)
    Lars> 1F46E 1F3FB 200D 2642 FE0F ; RGI_Emoji_ZWJ_Sequence ; man police
    Lars> officer: light skin tone # E4.0 [1] (👮🏻‍♂️)

    Lars> So that all matches up, and I can go from 1F46E and get all the
    Lars> variants.  Seems very promising; I'll give it a go.

    Lars> But...  I can't go from "woman police officer" to "woman police officer:
    Lars> light skin tone" by looking at the first code point.  Er...  what's the
    Lars> rule here, then?

    Lars> 1F46E plus 2640 as the next-to-last code point?

You could look at the second code point, which would be one of the
Emoji_Component ones (theyʼre in emoji-data.txt).

Robert
-- 

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

    Lars> 1F46E plus 2640 as the next-to-last code point?

> You could look at the second code point, which would be one of the
> Emoji_Component ones (theyʼre in emoji-data.txt).

Uhm, second...

1F46E 200D 2640 FE0F is woman police officer
1F46E 1F3FB 200D 2640 FE0F is woman police officer: light skin tone

Well, that's the next-to-last code point?  2640?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Tue, 26 Oct 2021 20:14:06 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    Lars> 1F46E plus 2640 as the next-to-last code point?

    >> You could look at the second code point, which would be one of the
    >> Emoji_Component ones (theyʼre in emoji-data.txt).

    Lars> Uhm, second...

    Lars> 1F46E 200D 2640 FE0F is woman police officer
    Lars> 1F46E 1F3FB 200D 2640 FE0F is woman police officer: light skin tone

    Lars> Well, that's the next-to-last code point?  2640?

1F3FB, in the second sequence, no? So, look for base + Emoji_Component

Robert
-- 

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Lars> 1F46E 200D 2640 FE0F is woman police officer
>     Lars> 1F46E 1F3FB 200D 2640 FE0F is woman police officer: light skin tone
>
>     Lars> Well, that's the next-to-last code point?  2640?
>
> 1F3FB, in the second sequence, no? So, look for base + Emoji_Component

1F3FB is the hue, isn't it?  So it's also there for the male police
officers (of that hue), but other women police officers don't have
1F3FB.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

💂🏾‍♂️

gives us

             to input: type "C-x 8 RET 1f482" or "C-x 8 RET GUARDSMAN"
          buffer code: #xF0 #x9F #x92 #x82
            file code: #xF0 #x9F #x92 #x82 (encoded by coding system utf-8-emacs)
              display: composed to form "💂🏾‍♂️" (see below)

Composed with the following character(s) "🏾‍♂️" using this font:
  ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 4 128130 2700 24 0 24 18 5 nil]
with these character(s):
  🏾 (#x1f3fe) EMOJI MODIFIER FITZPATRICK TYPE-5
  ‍ (#x200d) ZERO WIDTH JOINER
  ♂ (#x2642) MALE SIGN
  ️ (#xfe0f) VARIATION SELECTOR-16

This should be amended -- the "to input" part isn't correct, for
instance.  Perhaps it should be changed completely -- it's confusing
having the base character up there, and then the composed bits displayed
differently.

And the name is "man guard: medium-light skin tone", which we should
probably output somewhere.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 26 Oct 2021 21:18:03 +0200
> 
> 💂🏾‍♂️
> 
> gives us
> 
>              to input: type "C-x 8 RET 1f482" or "C-x 8 RET GUARDSMAN"
>           buffer code: #xF0 #x9F #x92 #x82
>             file code: #xF0 #x9F #x92 #x82 (encoded by coding system utf-8-emacs)
>               display: composed to form "💂🏾‍♂️" (see below)
> 
> Composed with the following character(s) "🏾‍♂️" using this font:
>   ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 4 128130 2700 24 0 24 18 5 nil]
> with these character(s):
>   🏾 (#x1f3fe) EMOJI MODIFIER FITZPATRICK TYPE-5
>   ‍ (#x200d) ZERO WIDTH JOINER
>   ♂ (#x2642) MALE SIGN
>   ️ (#xfe0f) VARIATION SELECTOR-16
> 
> This should be amended -- the "to input" part isn't correct, for
> instance.

Yes, it is correct: it reports on the character at a certain buffer
position (you elided that part).

If you want a command that could show how to input a sequence of
characters that were composed, it should be a different command, and
the way to type such a sequence cannot be automatically generated,
because how would Emacs know that there's a particular command that
would produce such a sequence?

> And the name is "man guard: medium-light skin tone", which we should
> probably output somewhere.

That's not a character, while this command describes a character.

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> 💂🏾‍♂️

[...]

> Yes, it is correct: it reports on the character at a certain buffer
> position (you elided that part).

No, the glyph in question is the one at the start of the email: 💂🏾‍♂️.

> If you want a command that could show how to input a sequence of
> characters that were composed, it should be a different command, and
> the way to type such a sequence cannot be automatically generated,
> because how would Emacs know that there's a particular command that
> would produce such a sequence?

It's just a sequence of Unicode code points, surely?  (And the help
buffer lists them, but not in the format needed to enter them.)

>> And the name is "man guard: medium-light skin tone", which we should
>> probably output somewhere.
>
> That's not a character, while this command describes a character.

Well...  it's a glyph, and the command describes the glyph perfectly
(i.e., all the elements that are part of it), but it doesn't output the
resulting name for the glyph.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[James Cloos]

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

LI> 💂🏾‍♂️.

I just recompiled master yesterday.

The sequence you mention still shows as five separate glyphs.

(using font:
ftcrhb:-unknown-Symbola-normal-normal-semicondensed-*-22-*-*-*-*-0-iso10646-1
(#x2207) for the first two, DejaVu Serif for the Male and JuliaMono for
the variation selector and the zero width joiner.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> The sequence you mention still shows as five separate glyphs.
>
> (using font:
> ftcrhb:-unknown-Symbola-normal-normal-semicondensed-*-22-*-*-*-*-0-iso10646-1
> (#x2207) for the first two, DejaVu Serif for the Male and JuliaMono for
> the variation selector and the zero width joiner.)

I'm guessing this means that you don't have an emoji font installed?  I
use Noto Color Emoji.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Lars Ingebrigtsen wrote:

> I'm guessing this means that you don't have an emoji font installed?  I
> use Noto Color Emoji.

Doesn't work either.

$ fc-match 'Noto Color Emoji' 
NotoColorEmoji.ttf: "Noto Color Emoji" "Regular"

             position: 1 of 5 (0%), column: 0
            character: 💂 (displayed as 💂) (codepoint 128130, #o372202, #x1f482)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x1F482
               script: emoji
               syntax: w 	which means: word
             category: .:Base
             to input: type "C-x 8 RET 1f482" or "C-x 8 RET GUARDSMAN"
          buffer code: #xF0 #x9F #x92 #x82
            file code: #xF0 #x9F #x92 #x82 (encoded by coding system utf-8-unix)
              display: composed to form "💂🏾‍♂️" (see below)

Composed with the following character(s) "🏾‍♂️" using this font:
  ftcrhb:-Free-Symbola-medium-normal-semicondensed-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 4 128130 8392 12 0 12 10 3 nil]
  [0 4 127998 8260 13 0 13 10 3 nil]
  [0 4 8205 3 3 0 0 0 0 [0 0 0]]
  [0 4 9794 3549 11 0 12 10 1 nil]
  [0 4 65039 3 3 0 0 0 0 [0 0 0]]
with these character(s):
  🏾 (#x1f3fe) EMOJI MODIFIER FITZPATRICK TYPE-5
  ‍ (#x200d) ZERO WIDTH JOINER
  ♂ (#x2642) MALE SIGN
  ️ (#xfe0f) VARIATION SELECTOR-16

Character code properties: customize what to show
  name: GUARDSMAN
  general-category: So (Symbol, Other)
  decomposition: (128130) ('💂')

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Composed with the following character(s) "🏾‍♂️" using this font:
>   ftcrhb:-Free-Symbola-medium-normal-semicondensed-*-13-*-*-*-*-0-iso10646-1
> by these glyphs:

Odd.  I get

  ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1

in emacs -Q (on Debian/bookworm with the current trunk).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 17:07:13 +0200, Andreas Schwab <schwab@linux-m68k.org> said:

    Andreas> On Okt 27 2021, Lars Ingebrigtsen wrote:
    >> I'm guessing this means that you don't have an emoji font installed?  I
    >> use Noto Color Emoji.

    Andreas> Doesn't work either.

    Andreas> $ fc-match 'Noto Color Emoji' 
    Andreas> NotoColorEmoji.ttf: "Noto Color Emoji" "Regular"

    Andreas>              position: 1 of 5 (0%), column: 0
    Andreas>             character: 💂 (displayed as 💂) (codepoint 128130, #o372202, #x1f482)
    Andreas>               charset: unicode (Unicode (ISO10646))
    Andreas> code point in charset: 0x1F482
    Andreas>                script: emoji
    Andreas>                syntax: w 	which means: word
    Andreas>              category: .:Base
    Andreas>              to input: type "C-x 8 RET 1f482" or "C-x 8 RET GUARDSMAN"
    Andreas>           buffer code: #xF0 #x9F #x92 #x82
    Andreas>             file code: #xF0 #x9F #x92 #x82 (encoded by coding system utf-8-unix)
    Andreas>               display: composed to form "💂🏾‍♂️" (see below)

And what font is used if you have just U+1F482?

Robert
-- 

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Robert Pluim wrote:

> And what font is used if you have just U+1F482?

The same.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: James Cloos <cloos@jhcloos.com>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 17:07:13 +0200
> 
> On Okt 27 2021, Lars Ingebrigtsen wrote:
> 
> > I'm guessing this means that you don't have an emoji font installed?  I
> > use Noto Color Emoji.
> 
> Doesn't work either.
> 
> $ fc-match 'Noto Color Emoji' 
> NotoColorEmoji.ttf: "Noto Color Emoji" "Regular"
> 
>              position: 1 of 5 (0%), column: 0
>             character: 💂 (displayed as 💂) (codepoint 128130, #o372202, #x1f482)
>               charset: unicode (Unicode (ISO10646))
> code point in charset: 0x1F482
>                script: emoji
>                syntax: w 	which means: word
>              category: .:Base
>              to input: type "C-x 8 RET 1f482" or "C-x 8 RET GUARDSMAN"
>           buffer code: #xF0 #x9F #x92 #x82
>             file code: #xF0 #x9F #x92 #x82 (encoded by coding system utf-8-unix)
>               display: composed to form "💂🏾‍♂️" (see below)
> 
> Composed with the following character(s) "🏾‍♂️" using this font:
>   ftcrhb:-Free-Symbola-medium-normal-semicondensed-*-13-*-*-*-*-0-iso10646-1
> by these glyphs:

Very strange.  We have the default fontset explicitly request Noto
Color Emoji for these characters, so why isn't it working for you?  is
this Emacs 28 and "emacs -Q"?  If so, perhaps Noto Color Emoji is of
an old version that doesn't support some of the characters we require
for it to be considered a match for emoji?

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Eli Zaretskii wrote:

> If so, perhaps Noto Color Emoji is of an old version that doesn't
> support some of the characters we require for it to be considered a
> match for emoji?

noto-coloremoji-fonts-20191119-1.9.noarch

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> noto-coloremoji-fonts-20191119-1.9.noarch

Debian/bookworm has:

fonts-noto-color-emoji/testing,now 2.028-1 all [installed,automatic]
  color emoji font from Google

So it sounds like 1.9 is too old to be detected?  Should we adjust the
detection logic?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Lars Ingebrigtsen wrote:

> So it sounds like 1.9 is too old to be detected?

1.9 is the release, not the version.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  cloos@jhcloos.com,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 18:23:07 +0200
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> > noto-coloremoji-fonts-20191119-1.9.noarch
> 
> Debian/bookworm has:
> 
> fonts-noto-color-emoji/testing,now 2.028-1 all [installed,automatic]
>   color emoji font from Google
> 
> So it sounds like 1.9 is too old to be detected?  Should we adjust the
> detection logic?

We need to understand how to adjust it.  Does 1.9 lack support for
one of the characters we require for Emoji in
script-representative-chars?  If so, the solution is to upgrade to a
newer version of the font, because we have these 2 characters there
for a reason.

But I'm not yet sure this is the root cause, we need to be sure.

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Eli Zaretskii wrote:

> We need to understand how to adjust it.  Does 1.9 lack support for
> one of the characters we require for Emoji in
> script-representative-chars?

How do I check?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  cloos@jhcloos.com,
>   emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 18:43:43 +0200
> 
> On Okt 27 2021, Eli Zaretskii wrote:
> 
> > We need to understand how to adjust it.  Does 1.9 lack support for
> > one of the characters we require for Emoji in
> > script-representative-chars?
> 
> How do I check?

The easiest way is to start Emacs like this:

  emacs -Q -fn "Noto Color Emoji"

then oinsert the two characters we have for Emoji in
script-representative-chars, and see if they display as glyphs or as
"tofu".

(It could be that Emacs will refuse to start with Noto Color Emoji as
the default font, for some reason.  Then I will suggest a slightly
more complex method.)

Re: describe-char on emoji sequences

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 20:02:41 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Andreas Schwab <schwab@linux-m68k.org>
    >> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  cloos@jhcloos.com,
    >> emacs-devel@gnu.org
    >> Date: Wed, 27 Oct 2021 18:43:43 +0200
    >> 
    >> On Okt 27 2021, Eli Zaretskii wrote:
    >> 
    >> > We need to understand how to adjust it.  Does 1.9 lack support for
    >> > one of the characters we require for Emoji in
    >> > script-representative-chars?
    >> 
    >> How do I check?

    Eli> The easiest way is to start Emacs like this:

    Eli>   emacs -Q -fn "Noto Color Emoji"

    Eli> then oinsert the two characters we have for Emoji in
    Eli> script-representative-chars, and see if they display as glyphs or as
    Eli> "tofu".

    Eli> (It could be that Emacs will refuse to start with Noto Color Emoji as
    Eli> the default font, for some reason.  Then I will suggest a slightly
    Eli> more complex method.)

I downloaded noto-coloremoji-fonts-20191119-1.9.noarch.rpm and
extracted the .ttf from it. Emacs-28 here has no problem with U+1f482,
U+1f300, and U+1f600

Does

FC_DEBUG=1 emacs -Q|grep :file

give any clues as to where things may be going wrong? Or maybe
hb-view:

hb-view  --output-file=foo.svg --font-size=14 \
/usr/share/fonts/truetype/noto/NotoColorEmoji.ttf  -u 1f482

should produce a nice looking svg.

Robert
-- 

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Robert Pluim wrote:

>     Eli> The easiest way is to start Emacs like this:
>
>     Eli>   emacs -Q -fn "Noto Color Emoji"

=> Font ‘Noto Color Emoji’ is not defined

> FC_DEBUG=1 emacs -Q|grep :file

Nothing.

> hb-view  --output-file=foo.svg --font-size=14 \
> /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf  -u 1f482

Works for me.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[Gregory Heytings]

>> FC_DEBUG=1 emacs -Q|grep :file
>
> Nothing.
>

It should be

FC_DEBUG=1 emacs -Q|grep file:

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: larsi@gnus.org,  cloos@jhcloos.com,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 18:12:54 +0200
> 
> On Okt 27 2021, Eli Zaretskii wrote:
> 
> > If so, perhaps Noto Color Emoji is of an old version that doesn't
> > support some of the characters we require for it to be considered a
> > match for emoji?
> 
> noto-coloremoji-fonts-20191119-1.9.noarch

That's the latest, AFAICT.

Can you show the details of your build, the stuff that
report-emacs-bug collects?  Then Lars and Robert could compare with
what they get, and maybe that would give us a hint?

Another idea is to compare font-log from your build with that from a
build that does display the Emoji with Noto Color Emoji.

Re: describe-char on emoji sequences

[Andreas Schwab]

On Okt 27 2021, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: larsi@gnus.org,  cloos@jhcloos.com,  emacs-devel@gnu.org
>> Date: Wed, 27 Oct 2021 18:12:54 +0200
>> 
>> On Okt 27 2021, Eli Zaretskii wrote:
>> 
>> > If so, perhaps Noto Color Emoji is of an old version that doesn't
>> > support some of the characters we require for it to be considered a
>> > match for emoji?
>> 
>> noto-coloremoji-fonts-20191119-1.9.noarch
>
> That's the latest, AFAICT.

openSUSE Tunbleweed has 20200916, FWIW.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: describe-char on emoji sequences

[James Cloos]

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

LI> I'm guessing this means that you don't have an emoji font installed?  I
LI> use Noto Color Emoji.

:; fc-list|grep -i emoji
/usr/local/share/fonts/DOZE10/seguiemj.ttf: Segoe UI Emoji:style=Regular,Normal,obyčejné,Standard,Κανονικά,Normaali,Normál,Normale,Standaard,Normalny,Обычный,Normálne,Navadno,Arrunta
/home/cloos/.fonts/GeckoEmoji.ttf: Gecko Emoji:style=Regular
/home/cloos/.fonts/AndroidEmoji.ttf: Android Emoji:style=Regular
/usr/share/fonts/noto-emoji/NotoEmoji-Regular.ttf: Noto Emoji:style=Regular

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Copyright: Copyright 2021 James Cloos
> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
> Date: Wed, 27 Oct 2021 13:08:30 -0400
> 
> >>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> LI> I'm guessing this means that you don't have an emoji font installed?  I
> LI> use Noto Color Emoji.
> 
> :; fc-list|grep -i emoji
> /usr/local/share/fonts/DOZE10/seguiemj.ttf: Segoe UI Emoji:style=Regular,Normal,obyčejné,Standard,Κανονικά,Normaali,Normál,Normale,Standaard,Normalny,Обычный,Normálne,Navadno,Arrunta
> /home/cloos/.fonts/GeckoEmoji.ttf: Gecko Emoji:style=Regular
> /home/cloos/.fonts/AndroidEmoji.ttf: Android Emoji:style=Regular
> /usr/share/fonts/noto-emoji/NotoEmoji-Regular.ttf: Noto Emoji:style=Regular

AFAIU, this means you don't have Noto Color Emoji, so Emacs picks up
the first matching font it finds, and that is Symbola, which doesn't
support Emoji sequences.

The solution is either to install Noto Color Emoji, or to customize
your fontset to specify one of the above fonts you do have, assuming
some of them do support Emoji sequences.

Re: describe-char on emoji sequences

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 20:13:14 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: James Cloos <cloos@jhcloos.com>
    >> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
    >> Copyright: Copyright 2021 James Cloos
    >> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
    >> Date: Wed, 27 Oct 2021 13:08:30 -0400
    >> 
    >> >>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:
    >> 
    LI> I'm guessing this means that you don't have an emoji font installed?  I
    LI> use Noto Color Emoji.
    >> 
    >> :; fc-list|grep -i emoji
    >> /usr/local/share/fonts/DOZE10/seguiemj.ttf: Segoe UI
    >> Emoji:style=Regular,Normal,obyčejné,Standard,Κανονικά,Normaali,Normál,Normale,Standaard,Normalny,Обычный,Normálne,Navadno,Arrunta
    >> /home/cloos/.fonts/GeckoEmoji.ttf: Gecko Emoji:style=Regular
    >> /home/cloos/.fonts/AndroidEmoji.ttf: Android Emoji:style=Regular
    >> /usr/share/fonts/noto-emoji/NotoEmoji-Regular.ttf: Noto Emoji:style=Regular

    Eli> AFAIU, this means you don't have Noto Color Emoji, so Emacs picks up
    Eli> the first matching font it finds, and that is Symbola, which doesn't
    Eli> support Emoji sequences.

    Eli> The solution is either to install Noto Color Emoji, or to customize
    Eli> your fontset to specify one of the above fonts you do have, assuming
    Eli> some of them do support Emoji sequences.

If using one or the above fonts works, and itʼs a free font, then we
could add it to the default fontset for emoji (Iʼd hazard a guess that
anything under the 'DOZE10' directory doesnʼt fit that description)

Robert
-- 

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: James Cloos <cloos@jhcloos.com>,  larsi@gnus.org,  emacs-devel@gnu.org
> Gmane-Reply-To-List: yes
> Date: Wed, 27 Oct 2021 19:36:49 +0200
> 
> >>>>> On Wed, 27 Oct 2021 20:13:14 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> :; fc-list|grep -i emoji
>     >> /usr/local/share/fonts/DOZE10/seguiemj.ttf: Segoe UI
>     >> Emoji:style=Regular,Normal,obyčejné,Standard,Κανονικά,Normaali,Normál,Normale,Standaard,Normalny,Обычный,Normálne,Navadno,Arrunta
>     >> /home/cloos/.fonts/GeckoEmoji.ttf: Gecko Emoji:style=Regular
>     >> /home/cloos/.fonts/AndroidEmoji.ttf: Android Emoji:style=Regular
>     >> /usr/share/fonts/noto-emoji/NotoEmoji-Regular.ttf: Noto Emoji:style=Regular
> 
>     Eli> AFAIU, this means you don't have Noto Color Emoji, so Emacs picks up
>     Eli> the first matching font it finds, and that is Symbola, which doesn't
>     Eli> support Emoji sequences.
> 
>     Eli> The solution is either to install Noto Color Emoji, or to customize
>     Eli> your fontset to specify one of the above fonts you do have, assuming
>     Eli> some of them do support Emoji sequences.
> 
> If using one or the above fonts works, and itʼs a free font, then we
> could add it to the default fontset for emoji (Iʼd hazard a guess that
> anything under the 'DOZE10' directory doesnʼt fit that description)

None of these fonts is free, AFAICT.

Re: describe-char on emoji sequences

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> The solution is either to install Noto Color Emoji, or to customize
EZ> your fontset to specify one of the above fonts you do have, assuming
EZ> some of them do support Emoji sequences.

Emacs really should accept Noto Emoji and not only coloured stuff.

Reading main&news would be vastly worse with non-monochrome glyphs.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: describe-char on emoji sequences

[Robert Pluim]

>>>>> On Thu, 28 Oct 2021 03:02:43 -0400, James Cloos <cloos@jhcloos.com> said:

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
    EZ> The solution is either to install Noto Color Emoji, or to customize
    EZ> your fontset to specify one of the above fonts you do have, assuming
    EZ> some of them do support Emoji sequences.

    James> Emacs really should accept Noto Emoji and not only coloured stuff.

Noto Emoji is not monochrome. It also doesnʼt have as complete
coverage as Noto Color Emoji, but itʼs not too bad. I guess we could
add it to the default fontset as a fallback, but nothing is stopping
you from doing it locally.

Robert
-- 

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 09:46:16 +0200
> 
> >>>>> On Thu, 28 Oct 2021 03:02:43 -0400, James Cloos <cloos@jhcloos.com> said:
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>     EZ> The solution is either to install Noto Color Emoji, or to customize
>     EZ> your fontset to specify one of the above fonts you do have, assuming
>     EZ> some of them do support Emoji sequences.
> 
>     James> Emacs really should accept Noto Emoji and not only coloured stuff.
> 
> Noto Emoji is not monochrome. It also doesnʼt have as complete
> coverage as Noto Color Emoji, but itʼs not too bad. I guess we could
> add it to the default fontset as a fallback, but nothing is stopping
> you from doing it locally.

The version of Noto Emoji I have here doesn't seem to support Emoji
sequences.  Which version of the font does?

If this font doesn't support sequences, I don't think we should have
it in our default fontset.

Re: describe-char on emoji sequences

[Robert Pluim]

>>>>> On Thu, 28 Oct 2021 12:34:05 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Eli Zaretskii <eliz@gnu.org>,  larsi@gnus.org,  emacs-devel@gnu.org
    >> Date: Thu, 28 Oct 2021 09:46:16 +0200
    >> 
    >> >>>>> On Thu, 28 Oct 2021 03:02:43 -0400, James Cloos <cloos@jhcloos.com> said:
    >> 
    >> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
    EZ> The solution is either to install Noto Color Emoji, or to customize
    EZ> your fontset to specify one of the above fonts you do have, assuming
    EZ> some of them do support Emoji sequences.
    >> 
    James> Emacs really should accept Noto Emoji and not only coloured stuff.
    >> 
    >> Noto Emoji is not monochrome. It also doesnʼt have as complete
    >> coverage as Noto Color Emoji, but itʼs not too bad. I guess we could
    >> add it to the default fontset as a fallback, but nothing is stopping
    >> you from doing it locally.

    Eli> The version of Noto Emoji I have here doesn't seem to support Emoji
    Eli> sequences.  Which version of the font does?

Youʼre right, it doesnʼt. Iʼd only checked the basic emoji support
(and even there it has missing codepoints).

    Eli> If this font doesn't support sequences, I don't think we should have
    Eli> it in our default fontset.

Makes sense.

Robert
-- 

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Copyright: Copyright 2021 James Cloos
> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
> Date: Thu, 28 Oct 2021 03:02:43 -0400
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> The solution is either to install Noto Color Emoji, or to customize
> EZ> your fontset to specify one of the above fonts you do have, assuming
> EZ> some of them do support Emoji sequences.
> 
> Emacs really should accept Noto Emoji and not only coloured stuff.

What do you mean by "should accept"?  Emacs tests the fonts for
support of the script it needs to display, and generally uses the
first one that passes the tests.  If you want Emacs to use a
particular font for a particular script, you should customize your
fontset, as I suggested above.

And btw, in my testing Noto Emoji's support for Emoji sequences is
inadequate: it seems like it doesn't have the information necessary
for displaying sequences as a single glyph, even without the color.
So I'm not sure you really do want Emacs to use Noto Emoji.

> Reading main&news would be vastly worse with non-monochrome glyphs.

Most people think otherwise.  But again, you can customize the fontset
to have what you want, just make sure you have a font that supports
monochrome display of Emoji sequences.

Re: describe-char on emoji sequences

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> Emacs really should accept Noto Emoji and not only coloured stuff.

EZ> What do you mean by "should accept"?

i made apparently false presumption that noto emoji had, w/o the colour,
whast noto color emoji had.

oh well...
>> Reading main&news would be vastly worse with non-monochrome glyphs.

EZ> Most people think otherwise.  But again, you can customize the fontset
EZ> to have what you want, just make sure you have a font that supports
EZ> monochrome display of Emoji sequences.

goes w/o saying the *I* can; i was thinking of newer users.

(you know; those who've started using gnu emacs w/in the last third cent. :)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 15:24:07 -0400
> 
> >> Reading main&news would be vastly worse with non-monochrome glyphs.
> 
> EZ> Most people think otherwise.  But again, you can customize the fontset
> EZ> to have what you want, just make sure you have a font that supports
> EZ> monochrome display of Emoji sequences.
> 
> goes w/o saying the *I* can; i was thinking of newer users.
> 
> (you know; those who've started using gnu emacs w/in the last third cent. :)

Won't those want color Emoji?

Re: describe-char on emoji sequences

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm guessing this means that you don't have an emoji font installed?  I
  > use Noto Color Emoji.

Is that font libre?

Is there a libre font for emojis?
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > I'm guessing this means that you don't have an emoji font installed?  I
>   > use Noto Color Emoji.
>
> Is that font libre?

Yes.  It uses Apache License 2.0.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 09:35:14 -0400
> 
> >>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> LI> 💂🏾‍♂️.
> 
> I just recompiled master yesterday.
> 
> The sequence you mention still shows as five separate glyphs.
> 
> (using font:
> ftcrhb:-unknown-Symbola-normal-normal-semicondensed-*-22-*-*-*-*-0-iso10646-1
> (#x2207) for the first two, DejaVu Serif for the Male and JuliaMono for
> the variation selector and the zero width joiner.)

Symbola doesn't support Emoji sequences.  Install a better font.

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 26 Oct 2021 21:42:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 💂🏾‍♂️
> 
> [...]
> 
> > Yes, it is correct: it reports on the character at a certain buffer
> > position (you elided that part).
> 
> No, the glyph in question is the one at the start of the email: 💂🏾‍♂️.

The command is describe-char, and it reports on the character at
point:

  (describe-char POS &optional BUFFER)

  Describe position POS (interactively, point) and the char after POS.

> > If you want a command that could show how to input a sequence of
> > characters that were composed, it should be a different command, and
> > the way to type such a sequence cannot be automatically generated,
> > because how would Emacs know that there's a particular command that
> > would produce such a sequence?
> 
> It's just a sequence of Unicode code points, surely?  (And the help
> buffer lists them, but not in the format needed to enter them.)

How can Emacs know that there is a special command that can be used to
insert this entire sequence of codepoints in one go?

> >> And the name is "man guard: medium-light skin tone", which we should
> >> probably output somewhere.
> >
> > That's not a character, while this command describes a character.
> 
> Well...  it's a glyph, and the command describes the glyph perfectly
> (i.e., all the elements that are part of it), but it doesn't output the
> resulting name for the glyph.

Because it doesn't necessarily have a name.  This is a general-purpose
command, it is capable of describing any result of any character
composition, including those which yield more than one glyph and
glyphs that have no name.  (Technically, the correct terminology is
"grapheme cluster", not "glyph".)

We could, of course, program describe-char to give special treatment
to glyphs produced from the Emoji sequences, but that has to be coded
explicitly and specially for Emoji, because I don't see how you can do
that for an arbitrary composition.

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It's just a sequence of Unicode code points, surely?  (And the help
>> buffer lists them, but not in the format needed to enter them.)
>
> How can Emacs know that there is a special command that can be used to
> insert this entire sequence of codepoints in one go?

There isn't (well, there is now with emoji-insert), but...

Take ⚠️:

             to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"
          buffer code: #xE2 #x9A #xA0
            file code: #xE2 #x9A #xA0 (encoded by coding system utf-8-emacs)
              display: composed to form "⚠️" (see below)

Composed with the following character(s) "️" using this font:
  ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 9888 112 24 0 24 18 5 nil]
with these character(s):
  ️ (#xfe0f) VARIATION SELECTOR-16

If you insert

C-x 8 RET WARNING SIGN and then C-x 8 RET VARIATION SELECTOR-16 you
indeed get:

⚠️

So 

             to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"

could just be expanded to include all the code points in the
decomposition.  Of course, with some of these sequences (with five code
points), doing this is completely impractical, so perhaps it's not worth
doing. 

> Because it doesn't necessarily have a name.  This is a general-purpose
> command, it is capable of describing any result of any character
> composition, including those which yield more than one glyph and
> glyphs that have no name.  (Technically, the correct terminology is
> "grapheme cluster", not "glyph".)

I had a feeling that "glyph" wasn't correct.  :-)

> We could, of course, program describe-char to give special treatment
> to glyphs produced from the Emoji sequences, but that has to be coded
> explicitly and specially for Emoji, because I don't see how you can do
> that for an arbitrary composition.

Yes, that's what I was thinking -- if we had a table that goes from
grapheme cluster to name (and those would only be filled in for emojis),
then we could output that name.  (emoji.el creates such a table, but I
don't think we'd want to load that from this command, so the table
should perhaps be created in a more central location.)

The point of this is that it's not always clear what an emoji is trying
to express.  For instance, if somebody writes you a message about 👨🏻‍✈️,
it'd be nice if Emacs could tell you that it's a "man pilot".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 16:25:49 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> Take ⚠️:
> 
>              to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"
>           buffer code: #xE2 #x9A #xA0
>             file code: #xE2 #x9A #xA0 (encoded by coding system utf-8-emacs)
>               display: composed to form "⚠️" (see below)
> 
> Composed with the following character(s) "️" using this font:
>   ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 9888 112 24 0 24 18 5 nil]
> with these character(s):
>   ️ (#xfe0f) VARIATION SELECTOR-16
> 
> If you insert
> 
> C-x 8 RET WARNING SIGN and then C-x 8 RET VARIATION SELECTOR-16 you
> indeed get:
> 
> ⚠️
> 
> So 
> 
>              to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"
> 
> could just be expanded to include all the code points in the
> decomposition.  Of course, with some of these sequences (with five code
> points), doing this is completely impractical, so perhaps it's not worth
> doing. 

We could add such a feature, but note that the existing display all
but gives it to you already: it shows the codepoints of the other
characters in parentheses.

> The point of this is that it's not always clear what an emoji is trying
> to express.  For instance, if somebody writes you a message about 👨🏻‍✈️,
> it'd be nice if Emacs could tell you that it's a "man pilot".

Here, you are talking about a different feature: a kind of "parser" of
Emoji sequences that would convert a sequence of characters into its
Emoji description.  I don't think it belongs to describe-char, but it
could be a useful feature on its own.

Re: describe-char on emoji sequences

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 3102 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:


If we could do that, that would be awesome.

I know I'm a minority use case, but that type of reverse translation
from a sequence of glyphs to meaningful description would potentially
help blind and low-vision users of Emacs better comprehend things on
microblogging sites like twitter where emojis get used heavily.
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> It's just a sequence of Unicode code points, surely?  (And the help
>>> buffer lists them, but not in the format needed to enter them.)
>>
>> How can Emacs know that there is a special command that can be used to
>> insert this entire sequence of codepoints in one go?
>
> There isn't (well, there is now with emoji-insert), but...
>
> Take 7²1„1„1:
>
>              to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"
>           buffer code: #xE2 #x9A #xA0
>             file code: #xE2 #x9A #xA0 (encoded by coding system utf-8-emacs)
>               display: composed to form "7²1„1„1" (see below)
>
> Composed with the following character(s) "„1„1" using this font:
>   ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-19-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 9888 112 24 0 24 18 5 nil]
> with these character(s):
>   „1„1 (#xfe0f) VARIATION SELECTOR-16
>
> If you insert
>
> C-x 8 RET WARNING SIGN and then C-x 8 RET VARIATION SELECTOR-16 you
> indeed get:
>
> 7²1„1„1
>
> So 
>
>              to input: type "C-x 8 RET 26a0" or "C-x 8 RET WARNING SIGN"
>
> could just be expanded to include all the code points in the
> decomposition.  Of course, with some of these sequences (with five code
> points), doing this is completely impractical, so perhaps it's not worth
> doing. 
>
>> Because it doesn't necessarily have a name.  This is a general-purpose
>> command, it is capable of describing any result of any character
>> composition, including those which yield more than one glyph and
>> glyphs that have no name.  (Technically, the correct terminology is
>> "grapheme cluster", not "glyph".)
>
> I had a feeling that "glyph" wasn't correct.  :-)
>
>> We could, of course, program describe-char to give special treatment
>> to glyphs produced from the Emoji sequences, but that has to be coded
>> explicitly and specially for Emoji, because I don't see how you can do
>> that for an arbitrary composition.
>
> Yes, that's what I was thinking -- if we had a table that goes from
> grapheme cluster to name (and those would only be filled in for emojis),
> then we could output that name.  (emoji.el creates such a table, but I
> don't think we'd want to load that from this command, so the table
> should perhaps be created in a more central location.)
>
> The point of this is that it's not always clear what an emoji is trying
> to express.  For instance, if somebody writes you a message about ”9Ó8”9È96¤87¼5„1„1,
> it'd be nice if Emacs could tell you that it's a "man pilot".

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

"T.V Raman" <raman@google.com> writes:

> If we could do that, that would be awesome.
>
> I know I'm a minority use case, but that type of reverse translation
> from a sequence of glyphs to meaningful description would potentially
> help blind and low-vision users of Emacs better comprehend things on
> microblogging sites like twitter where emojis get used heavily.

Yup -- it's not just generally helpful, but an accessibility issue.

I've factored out these bits, so I think I'll make `describe-char'
output this data for these emoji grapheme clusters (in Emacs 29).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: describe-char on emoji sequences

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've factored out these bits, so I think I'll make `describe-char'
> output this data for these emoji grapheme clusters (in Emacs 29).

I've now implemented this on the branch:

              display: composed to form "👩🏼‍🤝‍👨🏿" (see below)
     composition name: woman and man holding hands: medium-light skin tone, dark skin tone

It only loads the emoji data if 1) we're looking at an emoji glyph and
2) it's a composition.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, a huge grid (divided into sections) is probably also a good thing,
> but that would be a separate command, like `list-emojis' or something.

If we set aside the difficulties of implementing it, wouldn't it be
natural to just have one unified interface, that supports both using
keys like a-z and arrow keys+RET to pick the emoji you want?

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Yes, a huge grid (divided into sections) is probably also a good thing,
>> but that would be a separate command, like `list-emojis' or something.
>
> If we set aside the difficulties of implementing it, wouldn't it be
> natural to just have one unified interface, that supports both using
> keys like a-z and arrow keys+RET to pick the emoji you want?

The huge grid wouldn't really be navigable by keyboard input, I think?
I mean, there's thousands of the darn things, and my keyboard doesn't
have that many keys.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[William Xu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Yes, a huge grid (divided into sections) is probably also a good thing,
>>> but that would be a separate command, like `list-emojis' or something.
>>
>> If we set aside the difficulties of implementing it, wouldn't it be
>> natural to just have one unified interface, that supports both using
>> keys like a-z and arrow keys+RET to pick the emoji you want?
>
> The huge grid wouldn't really be navigable by keyboard input, I think?
> I mean, there's thousands of the darn things, and my keyboard doesn't
> have that many keys.

If the names of emojis are also displayed, one may just search by its
name.. (or use somthing like avy-goto-char) Sounds more convenient than
having to go deep several levels to pick up an emoji.

-- 
William

Re: Entering emojis

[Stefan Monnier]

Stefan Kangas [2021-10-26 17:46:05] wrote:
> (Unrelatedly, almost no smileys display on my macOS machine, but I
> believe this is a known issue.)

Of course, you need to be on GNU/Linux to find happiness,


        Stefan

Broken

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Stefan Kangas [2021-10-26 17:46:05] wrote:
>> (Unrelatedly, almost no smileys display on my macOS machine, but I
>> believe this is a known issue.)
>
> Of course, you need to be on GNU/Linux to find happiness,

Of course!  I have a GNU/Linux machine at home to cozy up next to.  I'm
not equally lucky at work, however.  ;-)

Re: Entering emojis

[Stefan Kangas]

Stefan Kangas <stefankangas@gmail.com> writes:

> (Unrelatedly, almost no smileys display on my macOS machine, but I
> believe this is a known issue.)

Why exactly are emojis broken on macOS?  I'm surprised to learn that I
have to say:

    (set-fontset-font t 'emoji
                      '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

Is that correct?  Why can't we just do that automatically?

I think that most users will just assume that emojis are broken and move
on.  Maybe some will soldier on and find the fix.  I very nearly didn't.

The NEWS entry on this is also not very useful, as it gives:

    (set-fontset-font t 'emoji
                      '("My New Emoji Font" . "iso10646-1") nil 'prepend)

But there is of course no such font "My New Emoji Font".

---

Also, and I'm sorry in advance, but can we please change the text to
something understandable?

     ** New character script 'emoji' has been created.
     Various blocks of codepoints have been split out of the 'symbol'
     script into their own 'emoji' script to allow easier specification of
     their treatment.  Which codepoints are treated as emoji is derived
     from the Unicode specifications.

Uh, what?  I have no idea what practical difference any of the words in
the above would make.  Blocks of codepoints?  What is a 'symbol' script?
Am I a horrible programmer and human being if I don't know what any of
this means?

Maybe I have a general concept of some of these things, but not enough
to know how this affects me as a user.  Did emojis already work, but
they are now just better?  Are emojis an entirely new feature in Emacs
28?  Do they work even a little bit in Emacs 27?  Do they work, but are
half-broken?  Do they work perfectly?

The entry after start talking about Zero Width Joiners and like,
seriously people, can we just *really* dumb this down to something like:

    *** Emojis now display nicely under X Windows and macOS.

Or like:

    *** Emacs can display 50 new Emojis and also with skin tones.

Or whatever the above is supposed to mean.  You can then of course go
into whatever low-level technical details you want, but please at least
explain first in casual terms what this even is.

Unless of course this is only of interest to actual Unicode experts, in
which case ignore me and carry on.

Re: Entering emojis

[Howard Melman]

Stefan Kangas <stefankangas@gmail.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
> I think that most users will just assume that emojis are broken and move
> on.  Maybe some will soldier on and find the fix.  I very nearly didn't.

FWIW I use the mac port (aka carbon port) of Emacs 27.2 and
emojis "just work" reasonably well.

> Also, and I'm sorry in advance, but can we please change the text to
> something understandable?
>
>      ** New character script 'emoji' has been created.
>      Various blocks of codepoints have been split out of the 'symbol'
>      script into their own 'emoji' script to allow easier specification of
>      their treatment.  Which codepoints are treated as emoji is derived
>      from the Unicode specifications.
>
> Uh, what?  I have no idea what practical difference any of the words in
> the above would make.  Blocks of codepoints?  What is a 'symbol' script?
> Am I a horrible programmer and human being if I don't know what any of
> this means?

I've been using emacs since the 1980s, on a mac since 2005,
and I don't really know what any of that means either.

> Unless of course this is only of interest to actual Unicode experts, in
> which case ignore me and carry on.

-- 

Howard

Re: Entering emojis

[Mattias Engdegård]

26 okt. 2021 kl. 23.12 skrev Stefan Kangas <stefankangas@gmail.com>:

> I think that most users will just assume that emojis are broken and move
> on.

Some of us think that they clearly are, and fundamentally so.

It would be useful to have a 'literate-mode' where they would be replaced with the empty string, a space, U+FFFD, a suitable hex escape, or something else. A compromise would be to show them as monochrome glyphs that respect the typographic metrics of the surrounding text.

Re: Entering emojis

[Andreas Schwab]

On Okt 27 2021, Mattias Engdegård wrote:

> 26 okt. 2021 kl. 23.12 skrev Stefan Kangas <stefankangas@gmail.com>:
>
>> I think that most users will just assume that emojis are broken and move
>> on.
>
> Some of us think that they clearly are, and fundamentally so.

Emojis are one of those solutions that are still in search for a problem.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 738 bytes --]

Andreas Schwab <schwab@linux-m68k.org> writes:


While at it I hear the Egyptians used heiroglyphics -- could we add
those too:-)

A picture is worth a 1000 words, no one knows which 1000.
Emojis and heiroglyphics both have the same problem:-)

> On Okt 27 2021, Mattias Engdeg0Š2rd wrote:
>
>> 26 okt. 2021 kl. 23.12 skrev Stefan Kangas <stefankangas@gmail.com>:
>>
>>> I think that most users will just assume that emojis are broken and move
>>> on.
>>
>> Some of us think that they clearly are, and fundamentally so.
>
> Emojis are one of those solutions that are still in search for a problem.
>
> Andreas.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Gregory Heytings]

>
> While at it I hear the Egyptians used heiroglyphics -- could we add 
> those too:-)
>

There's no need to add them, they are there already: C-x 8 RET hie TAB, 
and you'll get a list of the 1695 hieroglyphs defined by Unicode.  With an 
appropriate font, they'll be displayed correctly.

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 15:14:57 +0000
> From: Gregory Heytings <gregory@heytings.org>
> Cc: Mattias Engdegård <mattiase@acm.org>,
>  Andreas Schwab <schwab@linux-m68k.org>, Stefan Kangas <stefankangas@gmail.com>,
>  emacs-devel@gnu.org
> 
> There's no need to add them, they are there already: C-x 8 RET hie TAB, 
> and you'll get a list of the 1695 hieroglyphs defined by Unicode.  With an 
> appropriate font, they'll be displayed correctly.

It's not enough to know their names and codepoints.  Egyptian
hieroglyphs have unique shaping requirements, which needed appropriate
character composition rules to be written.  We have those as well,
though.  It's the fonts to support this shaping that are missing.
Which is to say our composition rules were not really tested well, so
they could include bugs.

Re: Entering emojis

[Eli Zaretskii]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 696 bytes --]

> From: "T.V Raman" <raman@google.com>
> Cc: Mattias Engdeg0Š2rd <mattiase@acm.org>,  Stefan Kangas
>  <stefankangas@gmail.com>,  Emacs developers <emacs-devel@gnu.org>
> Date: Wed, 27 Oct 2021 07:14:26 -0700
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> 
> While at it I hear the Egyptians used heiroglyphics -- could we add
> those too:-)
> 
> A picture is worth a 1000 words, no one knows which 1000.
> Emojis and heiroglyphics both have the same problem:-)

Jokes aside, we already support Egyptian hieroglyphs, see HELLO.  But
there are no fonts out there yet which can display them well enough,
so for now this is a capability that waits for the typography world to
catch up.

Re: Entering emojis

[Gregory Heytings]

>
> Jokes aside, we already support Egyptian hieroglyphs, see HELLO.  But 
> there are no fonts out there yet which can display them well enough, so 
> for now this is a capability that waits for the typography world to 
> catch up.
>

I know next to nothing about hieroglyphs, but on Debian the 
fonts-noto-core contains the Noto Sans Egyptian Hieroglyphs font, which 
has a glyph for each egyptian hieroglyph.

Debian also has a fonts-ancient-scripts package, which contains the 
Aegyptus font, which also has a glyph for each egyptian hieroglyph.  A 
more recent version of that font (but apparently with a more restrictive 
licence) can be seen here: 
https://dn-works.com/wp-content/uploads/2020/UFAS-Docs/Aegyptus.pdf and 
downloaded here: 
https://dn-works.com/wp-content/uploads/2020/UFAS-Fonts/Aegyptus.zip .

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 17:50:43 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: "T.V Raman" <raman@google.com>, mattiase@acm.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> 
> >
> > Jokes aside, we already support Egyptian hieroglyphs, see HELLO.  But 
> > there are no fonts out there yet which can display them well enough, so 
> > for now this is a capability that waits for the typography world to 
> > catch up.
> >
> 
> I know next to nothing about hieroglyphs, but on Debian the 
> fonts-noto-core contains the Noto Sans Egyptian Hieroglyphs font, which 
> has a glyph for each egyptian hieroglyph.
> 
> Debian also has a fonts-ancient-scripts package, which contains the 
> Aegyptus font, which also has a glyph for each egyptian hieroglyph.  A 
> more recent version of that font (but apparently with a more restrictive 
> licence) can be seen here: 
> https://dn-works.com/wp-content/uploads/2020/UFAS-Docs/Aegyptus.pdf and 
> downloaded here: 
> https://dn-works.com/wp-content/uploads/2020/UFAS-Fonts/Aegyptus.zip .

I know about these fonts.  I tried to explain in a followup message
that there's more about this script than just a glyph for each
codepoint: the layout of the glyphs in readable text is unusual, and
requires horizontal and vertical offsets under control of special
formatting characters specific to this script.  AFAIK, no existing
font that is distributed, let alone distributed freely, supports that
yet.

Re: Entering emojis

[Gregory Heytings]

>> Debian also has a fonts-ancient-scripts package, which contains the 
>> Aegyptus font, which also has a glyph for each egyptian hieroglyph.  A 
>> more recent version of that font (but apparently with a more 
>> restrictive licence) can be seen here: 
>> https://dn-works.com/wp-content/uploads/2020/UFAS-Docs/Aegyptus.pdf and 
>> downloaded here: 
>> https://dn-works.com/wp-content/uploads/2020/UFAS-Fonts/Aegyptus.zip .
>
> I know about these fonts.  I tried to explain in a followup message that 
> there's more about this script than just a glyph for each codepoint: the 
> layout of the glyphs in readable text is unusual, and requires 
> horizontal and vertical offsets under control of special formatting 
> characters specific to this script.  AFAIK, no existing font that is 
> distributed, let alone distributed freely, supports that yet.
>

Unless I misunderstand what you mean, page 4 and 5 of the above PDF 
describes such special formattings in which multiple glyphs are combined, 
and they are supported by the Aegyptus font.

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 18:38:46 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: raman@google.com, mattiase@acm.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> Unless I misunderstand what you mean, page 4 and 5 of the above PDF 
> describes such special formattings in which multiple glyphs are combined, 
> and they are supported by the Aegyptus font.

I don't think they are (at least they weren't last time I tried), but
please show the display of HELLO with that font.  If they fixed the
font, it's good news.

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]


>> Unless I misunderstand what you mean, page 4 and 5 of the above PDF 
>> describes such special formattings in which multiple glyphs are 
>> combined, and they are supported by the Aegyptus font.
>
> I don't think they are (at least they weren't last time I tried), but 
> please show the display of HELLO with that font.  If they fixed the 
> font, it's good news.
>

Here they are.  One with Noto Sans, one with the free Aegyptus font, and 
one with the latest Aegyptus font.  As far as I can see, the X001 and Q003 
hieroglyphs are (correctly?) combined, which is (I guess) what you wanted 
to see.

[-- Attachment #2: Type: image/png, Size: 5458 bytes --]

[-- Attachment #3: Type: image/png, Size: 4798 bytes --]

[-- Attachment #4: Type: image/png, Size: 5336 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 18:56:54 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: raman@google.com, mattiase@acm.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> > I don't think they are (at least they weren't last time I tried), but 
> > please show the display of HELLO with that font.  If they fixed the 
> > font, it's good news.
> >
> 
> Here they are.  One with Noto Sans, one with the free Aegyptus font, and 
> one with the latest Aegyptus font.  As far as I can see, the X001 and Q003 
> hieroglyphs are (correctly?) combined, which is (I guess) what you wanted 
> to see.

Thanks, but that's not how this is supposed to look.  It should look
like here:

  https://en.wikipedia.org/wiki/Ancient_Egypt#Historical_development

The layout in the vertical direction is completely incorrect in all of
the images you show.  Basically, it looks like the vertical layout of
hieroglyphs above other hieroglyphs is either unsupported or doesn't
work.

Maybe I made mistakes in the composition machinery, and that's why it
doesn't display correctly.  But it sounds unlikely, because I see in
your images glyphs that shouldn't be there (they are control
characters which should disappear from display).  Or maybe HarfBuzz
doesn't yet support this script well enough.

Re: Entering emojis

[Gregory Heytings]

>> Here they are.  One with Noto Sans, one with the free Aegyptus font, 
>> and one with the latest Aegyptus font.  As far as I can see, the X001 
>> and Q003 hieroglyphs are (correctly?) combined, which is (I guess) what 
>> you wanted to see.
>
> Thanks, but that's not how this is supposed to look.  It should look 
> like here:
>
>  https://en.wikipedia.org/wiki/Ancient_Egypt#Historical_development
>
> The layout in the vertical direction is completely incorrect in all of 
> the images you show.  Basically, it looks like the vertical layout of 
> hieroglyphs above other hieroglyphs is either unsupported or doesn't 
> work.
>

It should, again if you look at the PDF I linked earlier, you'll see many 
examples of up to four glyphs combined together, vertically and 
horizontally.

>
> Maybe I made mistakes in the composition machinery, and that's why it 
> doesn't display correctly.  But it sounds unlikely, because I see in 
> your images glyphs that shouldn't be there (they are control characters 
> which should disappear from display).  Or maybe HarfBuzz doesn't yet 
> support this script well enough.
>

All this is possible, indeed, and I'd even say probable.

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 19:20:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: raman@google.com, mattiase@acm.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> >  https://en.wikipedia.org/wiki/Ancient_Egypt#Historical_development
> >
> > The layout in the vertical direction is completely incorrect in all of 
> > the images you show.  Basically, it looks like the vertical layout of 
> > hieroglyphs above other hieroglyphs is either unsupported or doesn't 
> > work.
> 
> It should, again if you look at the PDF I linked earlier, you'll see many 
> examples of up to four glyphs combined together, vertically and 
> horizontally.

We don't know what software they used for layout, do we?

> > Maybe I made mistakes in the composition machinery, and that's why it 
> > doesn't display correctly.  But it sounds unlikely, because I see in 
> > your images glyphs that shouldn't be there (they are control characters 
> > which should disappear from display).  Or maybe HarfBuzz doesn't yet 
> > support this script well enough.
> >
> 
> All this is possible, indeed, and I'd even say probable.

What does hb-view show for that text with those fonts?  If it shows
correct display, the problem is somewhere in Emacs.

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]


>> It should, again if you look at the PDF I linked earlier, you'll see 
>> many examples of up to four glyphs combined together, vertically and 
>> horizontally.
>
> We don't know what software they used for layout, do we?
>

We do: the metadata of the PDF file indicate that this file has been 
produced by LibreOffice 5.4.

But why does it matter?  The author of that font produced that PDF to 
demonstrate what the font does.  He did not place the various elements of 
each hieroglyphs by hand.  The PDF even demonstrates that with various 
(ligature) options it's possible to automatically change the vertical and 
horizontal placement of elements in a combined hieroglyph.

>
> What does hb-view show for that text with those fonts?  If it shows 
> correct display, the problem is somewhere in Emacs.
>

See attached.  The two vertical joiners are not rendered correctly, 
perhaps they are misplaced/misused in the input?  But the rendering seems 
better than that of Emacs.

[-- Attachment #2: Type: image/png, Size: 19542 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 06:50:57 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, emacs-devel@gnu.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, raman@google.com
> 
> > We don't know what software they used for layout, do we?
> 
> We do: the metadata of the PDF file indicate that this file has been 
> produced by LibreOffice 5.4.
> 
> But why does it matter?  The author of that font produced that PDF to 
> demonstrate what the font does.

Complex text layout needs two players: the font and the shaping
engine.  Each one must do its part of the job correctly; the font
alone is not enough.  The shaping engine is part of the editor that
produces the display, that's why I asked which one was that.

> > What does hb-view show for that text with those fonts?  If it shows 
> > correct display, the problem is somewhere in Emacs.
> 
> See attached.  The two vertical joiners are not rendered correctly, 
> perhaps they are misplaced/misused in the input?  But the rendering seems 
> better than that of Emacs.

Better in what sense?  The vertical arrangement is still wrong; it
looks like HarfBuzz understood incorrectly which glyph should be above
which, and the formatting controls were not removed from display.  Of
course, all of that could be due to my own mistakes, in how I decided
to type the codepoints for this display and/or in the
character-composition support for Egyptian hieroglyphs that I added.
Feel free to find what part(s) I did wrongly.

How does LibreOffice display the same text?

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]


>
> Complex text layout needs two players: the font and the shaping engine. 
> Each one must do its part of the job correctly; the font alone is not 
> enough.  The shaping engine is part of the editor that produces the 
> display, that's why I asked which one was that.
>

Yes, I understand this.

>
> How does LibreOffice display the same text?
>

Correctly.

I attach three files:

- a minimal LibreOffice text document, which displays "Egyptian language" 
as expected,

- the output of hb-view on the "Egyptian language" string, with the 
Aegyptus font,

- a patch for etc/HELLO, which replaces the "Egyptian language" string 
with the correct one (I don't know how the text "Egyptian language" should 
look like).

[-- Attachment #2: Type: application/vnd.oasis.opendocument.text, Size: 1188 bytes --]

[-- Attachment #3: Type: image/png, Size: 20511 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: Type: text/x-diff; name=Correct-Egyptian-language-in-etc-HELLO.patch, Size: 872 bytes --]

From b68596af102cd86016efc866e0f0f664294732a2 Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Thu, 28 Oct 2021 09:24:42 +0000
Subject: [PATCH] Correct Egyptian language in etc/HELLO

---
 etc/HELLO | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/etc/HELLO b/etc/HELLO
index 577c2828de..b83c323cf3 100644
--- a/etc/HELLO
+++ b/etc/HELLO
@@ -38,7 +38,7 @@ Czech (čeština)	Dobrý den
 Danish (dansk)	Hej / Goddag / Halløj
 Dutch (Nederlands)	Hallo / Dag
 Efik  /ˈɛfɪk/	Mɔkɔm
-Egyptian Hieroglyphs (𓂋𓐰𓏤𓈖𓆎𓅓𓏏𓐰𓊖)	𓅓𓊵𓐰𓐷𓏏𓊪𓐸, 𓇍𓇋𓂻𓍘𓇋
+Egyptian Hieroglyphs (𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖)	𓅓𓊵𓐰𓐷𓏏𓊪𓐸, 𓇍𓇋𓂻𓍘𓇋
 Emacs	emacs --no-splash -f view-hello-file
 Emoji	👋
 English /ˈɪŋɡlɪʃ/	Hello
-- 
2.33.0

Re: Entering emojis

[Gregory Heytings]

>
> (I don't know how the text "Egyptian language" should look like).
>

Sorry: how the text *after* "Egyptian language".

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 09:37:33 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, raman@google.com, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> > How does LibreOffice display the same text?
> >
> 
> Correctly.
> 
> I attach three files:
> 
> - a minimal LibreOffice text document, which displays "Egyptian language" 
> as expected,
> 
> - the output of hb-view on the "Egyptian language" string, with the 
> Aegyptus font,
> 
> - a patch for etc/HELLO, which replaces the "Egyptian language" string 
> with the correct one

Hmm... it seems you removed U+13430 VERTICAL JOINER?  Why is that?  Is
it only because removing them yields the correct display, or there's
some deeper, non-phenomenological reason?

> (I don't know how the text "Egyptian language" should look like).

Like shown here:

  http://www.mummyfriends.com/medu/phrases/fun

Thanks.

Re: Entering emojis

[Gregory Heytings]

>
> Hmm... it seems you removed U+13430 VERTICAL JOINER?  Why is that?  Is 
> it only because removing them yields the correct display, or there's 
> some deeper, non-phenomenological reason?
>

Indeed, it's because removing them yields the correct display with both 
LibreOffice and hb-view.  Which I guess means that it's the correct 
string.  But I'm not an aegyptologist...

>> (I don't know how the text "Egyptian language" should look like).
>
> Like shown here:
>
>  http://www.mummyfriends.com/medu/phrases/fun
>

"em hotep" and "ii-ti"?

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 249 bytes --]


And here's a patch which corrects both the "Egyptian language" string, and 
the "em-hotep" one.  Together with the output of hb-view on the "em-hotep" 
and "ii-ti" strings with the Aegyptus font (the recent one, not the one 
distributed by Debian).

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-diff; name=Correct-Egyptian-language-in-etc-HELLO.patch, Size: 860 bytes --]

From ad58dc93db4d06492edc75a0f69772d3dbcc0fdb Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Thu, 28 Oct 2021 10:21:44 +0000
Subject: [PATCH] Correct Egyptian language in etc/HELLO

---
 etc/HELLO | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/etc/HELLO b/etc/HELLO
index 577c2828de..8bd489fb40 100644
--- a/etc/HELLO
+++ b/etc/HELLO
@@ -38,7 +38,7 @@ Czech (čeština)	Dobrý den
 Danish (dansk)	Hej / Goddag / Halløj
 Dutch (Nederlands)	Hallo / Dag
 Efik  /ˈɛfɪk/	Mɔkɔm
-Egyptian Hieroglyphs (𓂋𓐰𓏤𓈖𓆎𓅓𓏏𓐰𓊖)	𓅓𓊵𓐰𓐷𓏏𓊪𓐸, 𓇍𓇋𓂻𓍘𓇋
+Egyptian Hieroglyphs (𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖)	𓅓𓊵𓏏𓊪, 𓇍𓇋𓂻𓍘𓇋
 Emacs	emacs --no-splash -f view-hello-file
 Emoji	👋
 English /ˈɪŋɡlɪʃ/	Hello
-- 
2.33.0


[-- Attachment #3: Type: image/png, Size: 11421 bytes --]

[-- Attachment #4: Type: image/png, Size: 11546 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 10:05:31 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, raman@google.com, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> > Hmm... it seems you removed U+13430 VERTICAL JOINER?  Why is that?  Is 
> > it only because removing them yields the correct display, or there's 
> > some deeper, non-phenomenological reason?
> 
> Indeed, it's because removing them yields the correct display with both 
> LibreOffice and hb-view.  Which I guess means that it's the correct 
> string.  But I'm not an aegyptologist...

There are examples of hieroglyph formatting in section 11.4 of the
Unicode Standard: they show the sequence of codepoints and the
expected display.  Do these work correctly with that font?  I mean
figures 11-2, 11-3, and 11-4, as well as tables 11-2 and 11-3.  Here,
we are free from the need to be aegyptologists, since Someone™ did the
work for us.

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]


>
> There are examples of hieroglyph formatting in section 11.4 of the 
> Unicode Standard: they show the sequence of codepoints and the expected 
> display.  Do these work correctly with that font?  I mean figures 11-2, 
> 11-3, and 11-4, as well as tables 11-2 and 11-3.  Here, we are free from 
> the need to be aegyptologists, since Someone™ did the work for us.
>

This is such a specialized area that I trust the work of an aegyptologist 
way more than that of the Unicode consortium.  The Aegyptus font contains 
~10000 glyphs and ligatures, more precisely, ~7500 glyphs and ~2500 
ligatures.

The method chosen in that font is to use ligatures instead of explicit 
formatting characters, with an "escape" (zero-width non-joiner U+200C) to 
display adjacent characters that would have been subject to a ligature 
separately (one after the other).  Both LibreOffice and Harfbuzz 
understand this, and render Egyptian texts correctly.

I just looked at section 11.4 of the Unicode Standard.  The first 
character sequence in figure 11-2 is not a valid quadrat.  The second one 
are two characters that would normally be placed one above each other, so 
to obtain what is displayed in the Unicode Standard it's necessary to 
separate them with a zero-width non-joiner.

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 11:20:37 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, raman@google.com, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> > There are examples of hieroglyph formatting in section 11.4 of the 
> > Unicode Standard: they show the sequence of codepoints and the expected 
> > display.  Do these work correctly with that font?  I mean figures 11-2, 
> > 11-3, and 11-4, as well as tables 11-2 and 11-3.  Here, we are free from 
> > the need to be aegyptologists, since Someone™ did the work for us.
> 
> This is such a specialized area that I trust the work of an aegyptologist 
> way more than that of the Unicode consortium.  The Aegyptus font contains 
> ~10000 glyphs and ligatures, more precisely, ~7500 glyphs and ~2500 
> ligatures.

That font was released more than a year ago, and the Unicode
formatting characters for the hieroglyphs are relatively new, as you
see from the standard's description.  So it could be the fonts didn't
yet catch up.

In any case, I'd be very surprised to learn that the Unicode Standard
is so wrong.  They have specialists aboard as well.

> The method chosen in that font is to use ligatures instead of explicit 
> formatting characters, with an "escape" (zero-width non-joiner U+200C) to 
> display adjacent characters that would have been subject to a ligature 
> separately (one after the other).  Both LibreOffice and Harfbuzz 
> understand this, and render Egyptian texts correctly.

Like I said: the new way of formatting this script is not yet
supported widely enough.  There was a discussion on the HarfBuzz forum
about a year ago, from which I understood that these controls are not
yet supported by fonts.

> I just looked at section 11.4 of the Unicode Standard.  The first 
> character sequence in figure 11-2 is not a valid quadrat.

Why not?  And it isn't supposed to be a quadrat, AFAIU, anyway.

> The second one are two characters that would normally be placed one
> above each other, so to obtain what is displayed in the Unicode
> Standard it's necessary to separate them with a zero-width
> non-joiner.

AFAIU, U+13431 is the joiner to be used in that case.

And you didn't answer my question: does LibreOffice with the Aegyptus
font display those sequences correctly?

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]


>
> Like I said: the new way of formatting this script is not yet supported 
> widely enough.
>

Okay, so IIUC, Unicode has decided to do something that goes against the 
existing practice.  Given the limited available manpower in that narrow 
subfield, I'm not quite sure it was the best thing to do.  Using a font 
with predefined ligatures is much easier to enter text.

>> I just looked at section 11.4 of the Unicode Standard.  The first 
>> character sequence in figure 11-2 is not a valid quadrat.
>
> Why not?
>

I should have said: it's not a valid quadrat from the point of view of the 
Aegyptus font, which defines ~2500 valid quadrats.  Which probably means 
that this combination A1 above O1 (combining a man and a house) does not 
happen in practice (or happens rarely enough that it isn't yet included in 
the quadrats known by that font).

>
> And it isn't supposed to be a quadrat, AFAIU, anyway.
>

It is, a quadrat is a combination of two to four glyphs.

>> The second one are two characters that would normally be placed one 
>> above each other, so to obtain what is displayed in the Unicode 
>> Standard it's necessary to separate them with a zero-width non-joiner.
>
> AFAIU, U+13431 is the joiner to be used in that case.
>

But it's not a joiner, it's a non-joiner.  The logic is the opposite of 
what Unicode decided to do: known quadrats are automatically recognized 
and combined appropriately when their individual characters appear one 
after the other in a string.  It's only when you want to avoid this that 
you have to add a non-joiner.

>
> And you didn't answer my question: does LibreOffice with the Aegyptus 
> font display those sequences correctly?
>

I'm not sure what you mean by "those sequences".  The sequences in the 
patch are rendred correctly.  The sequences in figure 11-2 of section 11.4 
of the Unicode standard are not rendered correctly.  See the attached two 
pictures.

[-- Attachment #2: Type: image/png, Size: 13406 bytes --]

[-- Attachment #3: Type: image/png, Size: 2062 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 13:55:21 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, raman@google.com, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> > Like I said: the new way of formatting this script is not yet supported 
> > widely enough.
> 
> Okay, so IIUC, Unicode has decided to do something that goes against the 
> existing practice.

Looks like that, yes.

> Given the limited available manpower in that narrow subfield, I'm
> not quite sure it was the best thing to do.  Using a font with
> predefined ligatures is much easier to enter text.

I think the issue is not whether the font delivers ligatures or not,
the issue is whether the font recognizes the sequences which should
produce either ligatures or series of glyphs with offsets, when the
formatting controls are in the sequence.  It sounds like the existing
fonts don't recognize such sequences for what they are supposed to
produce.

> > AFAIU, U+13431 is the joiner to be used in that case.
> 
> But it's not a joiner, it's a non-joiner.  The logic is the opposite of 
> what Unicode decided to do: known quadrats are automatically recognized 
> and combined appropriately when their individual characters appear one 
> after the other in a string.  It's only when you want to avoid this that 
> you have to add a non-joiner.

That's not what the Unicode Standard says.

> > And you didn't answer my question: does LibreOffice with the Aegyptus 
> > font display those sequences correctly?
> >
> 
> I'm not sure what you mean by "those sequences".  The sequences in the 
> patch are rendred correctly.  The sequences in figure 11-2 of section 11.4 
> of the Unicode standard are not rendered correctly.  See the attached two 
> pictures.

OK, thanks.

So I think we should indeed install you patch, and add a comment that
if and when the fonts learn to support these formatting controls, we
should go back to the version with the controls.  Because etc/HELLO is
supposed to demonstrate our advanced text shaping and display
capabilities, not just that we can find suitable fonts for scripts.

WDYT?

Re: Entering emojis

[Gregory Heytings]

>> Given the limited available manpower in that narrow subfield, I'm not 
>> quite sure it was the best thing to do.  Using a font with predefined 
>> ligatures is much easier to enter text.
>
> I think the issue is not whether the font delivers ligatures or not, the 
> issue is whether the font recognizes the sequences which should produce 
> either ligatures or series of glyphs with offsets, when the formatting 
> controls are in the sequence.  It sounds like the existing fonts don't 
> recognize such sequences for what they are supposed to produce.
>

This is not how ligatures work.  Ligatures automatically translate a 
sequence of characters into an appropriate glyph, which may or may not be 
a combination of other glyphs.  For example, the Computer Modern font 
translates the two-character sequence "fi" into a character which looks 
better than "f" followed by "i".  If for some reason you don't want that 
ligature to take place, you write "f{}i" (in TeX), and you get "f" 
followed by "i".

Aegyptus does the same, for about ~2500 sequences of two to four 
hieroglyphs, known as quadrats.

>> But it's not a joiner, it's a non-joiner.  The logic is the opposite of 
>> what Unicode decided to do: known quadrats are automatically recognized 
>> and combined appropriately when their individual characters appear one 
>> after the other in a string.  It's only when you want to avoid this 
>> that you have to add a non-joiner.
>
> That's not what the Unicode Standard says.
>

I don't know what you mean by this.  As I said, the ligatures logic (used 
by Aegyptus) is the opposite of what Unicode decided to do.  For some 
reason, Unicode decided to not use ligatures, and to use explicit 
positioning instructions instead.  The "non-joiner" is the "{}" in "f{}i" 
above, it's an instruction of the writer which means "do not use a 
ligature here".

The logic chosen by Unicode amounts to insert a special character between 
each "f" and "i" when you want the "fi" ligature, instead of inserting a 
special character between "f" and "i" when you don't want the "fi" 
ligature.

With the ligature logic, to enter "em-hotep", which is composed of four 
characters, you just enter these four characters: G17, R4, X1, Q3.

With the Unicode logic, to enter "em-hotep", you enter seven characters: 
G17, R4, vertical joiner, begin segment, X1, Q3, end segment.

>
> So I think we should indeed install you patch, and add a comment that if 
> and when the fonts learn to support these formatting controls, we should 
> go back to the version with the controls.  Because etc/HELLO is supposed 
> to demonstrate our advanced text shaping and display capabilities, not 
> just that we can find suitable fonts for scripts.
>
> WDYT?
>

I don't know.  The problem is that the sequence of egyptian characters in 
etc/HELLO that are displayed correctly by hb-view and LibreOffice (and 
that are included in my patch) is for some reason not displayed correctly 
by Emacs, even with the recent Aegyptus font installed.  Are ligatures 
disabled for some reason in Emacs?

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 17:06:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, raman@google.com, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, emacs-devel@gnu.org
> 
> >> Given the limited available manpower in that narrow subfield, I'm not 
> >> quite sure it was the best thing to do.  Using a font with predefined 
> >> ligatures is much easier to enter text.
> >
> > I think the issue is not whether the font delivers ligatures or not, the 
> > issue is whether the font recognizes the sequences which should produce 
> > either ligatures or series of glyphs with offsets, when the formatting 
> > controls are in the sequence.  It sounds like the existing fonts don't 
> > recognize such sequences for what they are supposed to produce.
> 
> This is not how ligatures work.  Ligatures automatically translate a 
> sequence of characters into an appropriate glyph, which may or may not be 
> a combination of other glyphs.  For example, the Computer Modern font 
> translates the two-character sequence "fi" into a character which looks 
> better than "f" followed by "i".  If for some reason you don't want that 
> ligature to take place, you write "f{}i" (in TeX), and you get "f" 
> followed by "i".

Yes, I know.  But ligatures are not the only way of handling this.
When a font produces a ligature, i.e. a precomposed glyph that should
be displayed instead of several characters, it produces a single font
glyph.  The other way is to produce several font glyphs, each one with
offsets relative to the base-line.  Emacs supports both ways.
However, for any of the two to work, both the shaping engine and the
font should recognize the sequence, and the font should produce one or
more glyphs with the offsets for that sequence.

> >> But it's not a joiner, it's a non-joiner.  The logic is the opposite of 
> >> what Unicode decided to do: known quadrats are automatically recognized 
> >> and combined appropriately when their individual characters appear one 
> >> after the other in a string.  It's only when you want to avoid this 
> >> that you have to add a non-joiner.
> >
> > That's not what the Unicode Standard says.
> 
> I don't know what you mean by this.

I mean what the Unicode Standard says: it says that two hieroglyphs
should be displayed "normally", i.e. as separate characters at the
same vertical position, unless there's the vertical joiner between
them, in which case one should be above the other.

> With the ligature logic, to enter "em-hotep", which is composed of four 
> characters, you just enter these four characters: G17, R4, X1, Q3.

Which then means that you cannot write anything where these 4
characters are displayed in an arrangement different from that
particular one that's encoded in the font, unless you use non-joiners.

> I don't know.  The problem is that the sequence of egyptian characters in 
> etc/HELLO that are displayed correctly by hb-view and LibreOffice (and 
> that are included in my patch) is for some reason not displayed correctly 
> by Emacs, even with the recent Aegyptus font installed.  Are ligatures 
> disabled for some reason in Emacs?

No, they aren't.  In Emacs, ligatures are just part of character
composition, they aren't a separate feature.

However, the composition rules I defined went with Unicode, and need
to be fixed to support what the Aegyptus font does.  Does the patch
below help?

diff --git a/lisp/language/misc-lang.el b/lisp/language/misc-lang.el
index a2ca678..141349a 100644
--- a/lisp/language/misc-lang.el
+++ b/lisp/language/misc-lang.el
@@ -192,7 +192,12 @@ egyptian-shape-grouping
    composition-function-table
    #x13437
    (list (vector "\U00013437[\U00013000-\U0001343F]+"
-                 0 #'egyptian-shape-grouping))))
+                 0 #'egyptian-shape-grouping)))
+  (set-char-table-range
+   composition-function-table
+   '(#x13000 . #x1342E)
+   (list (vector "[\U00013000-\U0001342E]+"
+                 0 #'font-shape-gstring))))
 
 (provide 'misc-lang)
 

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 3085 bytes --]


>
> Yes, I know.  But ligatures are not the only way of handling this. When 
> a font produces a ligature, i.e. a precomposed glyph that should be 
> displayed instead of several characters, it produces a single font 
> glyph.  The other way is to produce several font glyphs, each one with 
> offsets relative to the base-line.  Emacs supports both ways. However, 
> for any of the two to work, both the shaping engine and the font should 
> recognize the sequence, and the font should produce one or more glyphs 
> with the offsets for that sequence.
>

In this case, ISTM that the problem is not the font, but the shaping 
engine.  If Harfbuzz does not know how handle the joiners and segment 
delimiters, it should behave as they did not exist, and use the font 
ligatures (if the font does have ligatures), instead of displaying the 
fictitious glyph at that codepoint (at the codepoint of the joiner or 
delimiter).  If it knows how to handle joiners and delimiters, it should 
ignore the font ligatures and create the final glyph with the individual 
glyphs in the font.  IIUC, that way a single font can be used with and 
without ligatures and with and without joiners and segment delimiters.

>
> I mean what the Unicode Standard says: it says that two hieroglyphs 
> should be displayed "normally", i.e. as separate characters at the same 
> vertical position, unless there's the vertical joiner between them, in 
> which case one should be above the other.
>

I understand.  But what the standard says doesn't make sense I think, 
because it means that it requires one to ignore font ligatures and instead 
requires one to use joiners and segment delimiters.  Again I know next to 
nothing about hieroglyphs (although I did learn a few things about them 
today), but I'd guess that if the Aegyptus font uses ligatures, it's 
because the number of combinations between hieroglyphs to produce a 
quadrat is, in practice, limited.  Just like in ancient greek you cannot 
put a rough breathing above an arbitrary letter for example.

>
> Which then means that you cannot write anything where these 4 characters 
> are displayed in an arrangement different from that particular one 
> that's encoded in the font, unless you use non-joiners.
>

That's not entirely correct.  Aegyptus provides an alternative arrangement 
for most quadrats, sometimes it even provides two alternative 
arrangements.  But indeed you have to use non-joiners if you want to 
ignore the automatic arrangement provided by the font.  Which is I guess 
much easier to use in practice than joiners and segment delimiters.  It 
would be a pain to type, in a language with diacritics, the letter itself, 
a joiner to indicate where the diacritic should be placed, and the 
diacritic itself, if in the vast majority of cases the diacritic is always 
placed in the same way.

>
> However, the composition rules I defined went with Unicode, and need to 
> be fixed to support what the Aegyptus font does.  Does the patch below 
> help?
>

It does!  Thanks.

Here's a combined patch, with the comment you asked.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-diff; name=Make-hieroglyphs-display-correctly-in-Emacs-for-the-.patch, Size: 2362 bytes --]

From c16dd1bb2b45eb4c41d6efc0ca825e3e6da4fb78 Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Thu, 28 Oct 2021 20:58:02 +0000
Subject: [PATCH] Make hieroglyphs display correctly in Emacs for the moment.

* etc/HELLO: Remove hieroglyph format controls.

* lisp/language/misc-lang.el: Use font-shape-gstring instead of
egyptian-shape-grouping until hieroglyph format controls are supported
by fonts and shaping engines.
---
 etc/HELLO                  |  2 +-
 lisp/language/misc-lang.el | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/etc/HELLO b/etc/HELLO
index 577c2828de..8bd489fb40 100644
--- a/etc/HELLO
+++ b/etc/HELLO
@@ -38,7 +38,7 @@ Czech (čeština)	Dobrý den
 Danish (dansk)	Hej / Goddag / Halløj
 Dutch (Nederlands)	Hallo / Dag
 Efik  /ˈɛfɪk/	Mɔkɔm
-Egyptian Hieroglyphs (𓂋𓐰𓏤𓈖𓆎𓅓𓏏𓐰𓊖)	𓅓𓊵𓐰𓐷𓏏𓊪𓐸, 𓇍𓇋𓂻𓍘𓇋
+Egyptian Hieroglyphs (𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖)	𓅓𓊵𓏏𓊪, 𓇍𓇋𓂻𓍘𓇋
 Emacs	emacs --no-splash -f view-hello-file
 Emoji	👋
 English /ˈɪŋɡlɪʃ/	Hello
diff --git a/lisp/language/misc-lang.el b/lisp/language/misc-lang.el
index a2ca678b2b..de4f092dc1 100644
--- a/lisp/language/misc-lang.el
+++ b/lisp/language/misc-lang.el
@@ -192,7 +192,20 @@ egyptian-shape-grouping
    composition-function-table
    #x13437
    (list (vector "\U00013437[\U00013000-\U0001343F]+"
-                 0 #'egyptian-shape-grouping))))
+                 0 #'egyptian-shape-grouping)))
+  ;; As of late 2021, Egyptian Hieroglyph Format Controls are not yet
+  ;; supported in existing fonts and shaping engines, but some fonts
+  ;; do provide ligatures with which texts in Egyptian Hieroglyphs are
+  ;; correctly displayed.  If and when these format controls are
+  ;; supported, the five lines below (which cancel the effect of the
+  ;; above lines) can be removed, and the entry in etc/HELLO can be
+  ;; restored to:
+  ;; Egyptian Hieroglyphs (𓂋𓐰𓏤𓈖𓆎𓅓𓏏𓐰𓊖)	𓅓𓊵𓐰𓐷𓏏𓊪𓐸, 𓇍𓇋𓂻𓍘𓇋
+  (set-char-table-range
+   composition-function-table
+   '(#x13000 . #x1342E)
+   (list (vector "[\U00013000-\U0001342E]+"
+                 0 #'font-shape-gstring))))
 
 (provide 'misc-lang)
 
-- 
2.33.0

Re: Entering emojis

[Eli Zaretskii]

> Date: Thu, 28 Oct 2021 21:10:41 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, emacs-devel@gnu.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, raman@google.com
> 
> > Yes, I know.  But ligatures are not the only way of handling this. When 
> > a font produces a ligature, i.e. a precomposed glyph that should be 
> > displayed instead of several characters, it produces a single font 
> > glyph.  The other way is to produce several font glyphs, each one with 
> > offsets relative to the base-line.  Emacs supports both ways. However, 
> > for any of the two to work, both the shaping engine and the font should 
> > recognize the sequence, and the font should produce one or more glyphs 
> > with the offsets for that sequence.
> 
> In this case, ISTM that the problem is not the font, but the shaping 
> engine.  If Harfbuzz does not know how handle the joiners and segment 
> delimiters, it should behave as they did not exist, and use the font 
> ligatures (if the font does have ligatures)

AFAICT, this is what happens here.

> instead of displaying the fictitious glyph at that codepoint (at the
> codepoint of the joiner or delimiter).

I don't think I understand what fictitious glyph you allude to here.
The joiners were displayed as thin spaces by the Emacs
glyphless-char-display feature, because HarfBuzz+font didn't compose
the sequence, and returned the separate font glyphs for each codepoint
in the sequence.  IOW, the composition failed, and therefore Emacs
displayed each of these characters as it's supposed to do.

> If it knows how to handle joiners and delimiters, it should ignore
> the font ligatures and create the final glyph with the individual
> glyphs in the font.

This doesn't happen in general, only in some very specific cases,
where the HarfBuzz developers decide to implement a fallback strategy
of this kind.  In the general case, HarfBuzz (and any other shaping
engine) recognizes the sequence, and looks into the font tables for
how to display that sequence; it then returns one or more font glyphs
from the font information about that sequence.  It only "ignores" the
font glyphs when some reasonable fallback is required, which is
generally avoided, because it's assumed that the font designers know
better.

> > I mean what the Unicode Standard says: it says that two hieroglyphs 
> > should be displayed "normally", i.e. as separate characters at the same 
> > vertical position, unless there's the vertical joiner between them, in 
> > which case one should be above the other.
> >
> 
> I understand.  But what the standard says doesn't make sense I think, 
> because it means that it requires one to ignore font ligatures and instead 
> requires one to use joiners and segment delimiters.

No, the joiners are supposed to tell the shaping engine and the font
that we want the ligatures and not separate font glyphs.

> > However, the composition rules I defined went with Unicode, and need to 
> > be fixed to support what the Aegyptus font does.  Does the patch below 
> > help?
> >
> 
> It does!  Thanks.
> 
> Here's a combined patch, with the comment you asked.

Thanks, installed on the release branch.

Re: Entering emojis

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 3562 bytes --]


>> In this case, ISTM that the problem is not the font, but the shaping 
>> engine.  If Harfbuzz does not know how handle the joiners and segment 
>> delimiters, it should behave as they did not exist, and use the font 
>> ligatures (if the font does have ligatures)
>
> AFAICT, this is what happens here.
>

No, because Harfbuzz displays the fictitious joiner and deelimiter glyphs, 
and does not try to use the ligatures that the font provides.

>> instead of displaying the fictitious glyph at that codepoint (at the 
>> codepoint of the joiner or delimiter).
>
> I don't think I understand what fictitious glyph you allude to here. The 
> joiners were displayed as thin spaces by the Emacs 
> glyphless-char-display feature, because HarfBuzz+font didn't compose the 
> sequence, and returned the separate font glyphs for each codepoint in 
> the sequence.  IOW, the composition failed, and therefore Emacs 
> displayed each of these characters as it's supposed to do.
>

A picture is worth a thousand words.  I attach four files:

In 1.png you see what Harfbuzz displays with the previous HELLO entry. 
The three glyphs with a thick rectangle above and a crossed rectangle 
below are a fictitious glyph in the Aegyptus font for the codepoint 
hieroglyph vertical joiner, and the opening and closing parentheses are 
fictitious glyphs in the Aegyptus font for the codepoints hieroglyph begin 
and end segment.

In 2.png you see what I would expect Harfbuzz to do with the previous 
HELLO entry, if it knows that it cannot handle the joiner and segment 
delimiters or if it detects that the font does not provide enough 
information to handle them appropriately, and if the font has no 
ligatures: displaying the hieroglyphs one after the other.  That's what I 
would expect to see with the Noto Hieroglyph font for example.

In 3.png you see what I would expect Harfbuzz to do with the previous 
HELLO entry, if it knows that it cannot handle the joiner and segment 
delimiters or if it detects that the font does not provide enough 
information to handle them appropriately, and if the font does have 
ligatures: displaying the hieroglyphs with the ligatures provided by the 
font.  That's what I would expect to see with the Aegyptus font.

In 4.png you see what I would expect Harfbuzz to do with the previous 
HELLO entry, if it knows that it can handle the joiner and segment 
delimiters and if it detects that the font does provide enough information 
to handle them appropriately.

>>> I mean what the Unicode Standard says: it says that two hieroglyphs 
>>> should be displayed "normally", i.e. as separate characters at the 
>>> same vertical position, unless there's the vertical joiner between 
>>> them, in which case one should be above the other.
>>
>> I understand.  But what the standard says doesn't make sense I think, 
>> because it means that it requires one to ignore font ligatures and 
>> instead requires one to use joiners and segment delimiters.
>
> No, the joiners are supposed to tell the shaping engine and the font 
> that we want the ligatures and not separate font glyphs.
>

Unless I misunderstand something, a text without joiners and delimiters 
would thus be displayed as 2.png, even if the underlying font provides 
ligatures with which it could be displayed as 3.png.  And ZWNJ would be 
ignored.  Which makes, as I said, the task of those who want to edit 
egyptian texts much harder, and unnecessarily so.

>> Here's a combined patch, with the comment you asked.
>
> Thanks, installed on the release branch.
>

Thanks!

[-- Attachment #2: Type: image/png, Size: 38372 bytes --]

[-- Attachment #3: Type: image/png, Size: 25177 bytes --]

[-- Attachment #4: Type: image/png, Size: 30444 bytes --]

[-- Attachment #5: Type: image/png, Size: 25734 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> Date: Fri, 29 Oct 2021 10:32:03 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, emacs-devel@gnu.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, raman@google.com
> 
> >> In this case, ISTM that the problem is not the font, but the shaping 
> >> engine.  If Harfbuzz does not know how handle the joiners and segment 
> >> delimiters, it should behave as they did not exist, and use the font 
> >> ligatures (if the font does have ligatures)
> >
> > AFAICT, this is what happens here.
> 
> No, because Harfbuzz displays the fictitious joiner and deelimiter glyphs, 
> and does not try to use the ligatures that the font provides.

Because the font and/or HarfBuzz don't support the formatting
controls.

If you have evidence that the font does support the formatting
controls, but HarfBuzz doesn't, please show it.  I think that's not
the case, because the same or similar display problems happen with
LibreOffice when using the format controls.

When a sequence of codepoints is not recognized as a composable one,
what we get as result is either a separate glyph for each codepoint,
or maybe composed glyphs for some sub-sequence.  That is normal and
expected when a sequence is not recognized.

> >> instead of displaying the fictitious glyph at that codepoint (at the 
> >> codepoint of the joiner or delimiter).
> >
> > I don't think I understand what fictitious glyph you allude to here. The 
> > joiners were displayed as thin spaces by the Emacs 
> > glyphless-char-display feature, because HarfBuzz+font didn't compose the 
> > sequence, and returned the separate font glyphs for each codepoint in 
> > the sequence.  IOW, the composition failed, and therefore Emacs 
> > displayed each of these characters as it's supposed to do.
> >
> 
> A picture is worth a thousand words.  I attach four files:
> 
> In 1.png you see what Harfbuzz displays with the previous HELLO entry. 
> The three glyphs with a thick rectangle above and a crossed rectangle 
> below are a fictitious glyph in the Aegyptus font for the codepoint 
> hieroglyph vertical joiner, and the opening and closing parentheses are 
> fictitious glyphs in the Aegyptus font for the codepoints hieroglyph begin 
> and end segment.

Why do you call them "fictitious"?  If those are the glyphs returned
by the font, then that's what the font designers want us to display

> In 2.png you see what I would expect Harfbuzz to do with the previous 
> HELLO entry, if it knows that it cannot handle the joiner and segment 
> delimiters or if it detects that the font does not provide enough 
> information to handle them appropriately, and if the font has no 
> ligatures: displaying the hieroglyphs one after the other.  That's what I 
> would expect to see with the Noto Hieroglyph font for example.
> 
> In 3.png you see what I would expect Harfbuzz to do with the previous 
> HELLO entry, if it knows that it cannot handle the joiner and segment 
> delimiters or if it detects that the font does not provide enough 
> information to handle them appropriately, and if the font does have 
> ligatures: displaying the hieroglyphs with the ligatures provided by the 
> font.  That's what I would expect to see with the Aegyptus font.
> 
> In 4.png you see what I would expect Harfbuzz to do with the previous 
> HELLO entry, if it knows that it can handle the joiner and segment 
> delimiters and if it detects that the font does provide enough information 
> to handle them appropriately.

Your expectations are incorrect.  The job of producing the correct
glyphs for a sequence involves both the font and the shaping engine.
The shaping engine in general should not produce any glyphs that the
font didn't return, and AFAIU has no means to "detect that the font
doesn't provide enough information" or "doesn't have ligatures".  The
shaping engine is supposed to trust the font that it knows what it's
doing.  The role of the shaping engine is to collect information about
the context of the character sequence (language, script,
directionality, etc.), and communicate that information to the font so
that the font could select the appropriate glyphs.

> > No, the joiners are supposed to tell the shaping engine and the font 
> > that we want the ligatures and not separate font glyphs.
> 
> Unless I misunderstand something, a text without joiners and delimiters 
> would thus be displayed as 2.png, even if the underlying font provides 
> ligatures with which it could be displayed as 3.png.  And ZWNJ would be 
> ignored.  Which makes, as I said, the task of those who want to edit 
> egyptian texts much harder, and unnecessarily so.

It is irrelevant what you and me think about whether this makes
sense.  We try to follow the Unicode Standard where we don't have a
reason to deviate from it.  If and when the fonts will implement what
Unicode says, we should also do what Unicode says, regardless of our
private opinions about the convenience of writing this script.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, I know.  But ligatures are not the only way of handling this.
> When a font produces a ligature, i.e. a precomposed glyph that should
> be displayed instead of several characters, it produces a single font
> glyph.  The other way is to produce several font glyphs, each one with
> offsets relative to the base-line.  Emacs supports both ways.

Oh, I didn't know that Emacs had support for ligatures now -- last time
I looked at it, it didn't seem to work.  Or am I misreading?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Gregory Heytings <gregory@heytings.org>,  mattiase@acm.org,
>   emacs-devel@gnu.org,  schwab@linux-m68k.org,  stefankangas@gmail.com,
>   raman@google.com
> Date: Thu, 28 Oct 2021 23:40:44 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, I know.  But ligatures are not the only way of handling this.
> > When a font produces a ligature, i.e. a precomposed glyph that should
> > be displayed instead of several characters, it produces a single font
> > glyph.  The other way is to produce several font glyphs, each one with
> > offsets relative to the base-line.  Emacs supports both ways.
> 
> Oh, I didn't know that Emacs had support for ligatures now -- last time
> I looked at it, it didn't seem to work.  Or am I misreading?

We have the infrastructure that supports ligatures in builds with
HarfBuzz since Emacs 27.  There's a TODO item which describes what is
left to do before we can claim a decent support for ligatures, not
just the potential for it.  Right now, to have ligatures, you need to
manually define character-composition rules for them, something that
is too low-level to advertise as a feature ready to be used.  And the
definitions of those rules should be specific to the font you are
using, because different fonts support different ligatures.  (Some
3rd-party packages use this infrastructure, but I have yet to see a
package that does it in a satisfactory manner, per the description in
TODO.)

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Right now, to have ligatures, you need to manually define
> character-composition rules for them, something that is too low-level
> to advertise as a feature ready to be used.

Ah, I see.

> And the definitions of those rules should be specific to the font you
> are using, because different fonts support different ligatures.

Huh.  Well, I know nothing about this, but...  don't the fonts come with
this metadata?  It seems very odd if every program has to have separate
support for each font.

> (Some 3rd-party packages use this infrastructure, but I have yet to
> see a package that does it in a satisfactory manner, per the
> description in TODO.)

It'd be cool if Emacs got ligature support that works out of the box,
because then people could use

  https://github.com/tonsky/FiraCode

😋

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 29 Oct 2021 14:51:30 +0200
> 
> > And the definitions of those rules should be specific to the font you
> > are using, because different fonts support different ligatures.
> 
> Huh.  Well, I know nothing about this, but...  don't the fonts come with
> this metadata?

Maybe they do, I don't know.  If someone does know, please tell how to
get that meta-data.

> It'd be cool if Emacs got ligature support that works out of the box,
> because then people could use
> 
>   https://github.com/tonsky/FiraCode

Yes; thus the TODO item about this.  Patches welcome.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> > And the definitions of those rules should be specific to the font you
>> > are using, because different fonts support different ligatures.
>> 
>> Huh.  Well, I know nothing about this, but...  don't the fonts come with
>> this metadata?
>
> Maybe they do, I don't know.  If someone does know, please tell how to
> get that meta-data.

I don't know anything about this, but I'm just poking around.

So, I've got a font called Jolie that has a very noticeable ligature --
it turns the string "#03" (1-3 are ligatures, the rest aren't) into a
swoosh.

I tried the hb-shape harfbuzz debug command:

larsi@elva:~/src$ hb-shape -V ~/.fonts/Jolie\ Romantique.ttf "#03"
trace: start table GSUB	buffer: [numbersign=0|zero=1|three=2]
trace: start lookup 0	buffer: [numbersign=0|zero=1|three=2]
trace: end lookup 0	buffer: [end3.swsh=0]
trace: end table GSUB	buffer: [end3.swsh=0]
trace: start table GPOS	buffer: [end3.swsh=0+362]
trace: start lookup 0	buffer: [end3.swsh=0+362]
trace: end lookup 0	buffer: [end3.swsh=0+362]
trace: end table GPOS	buffer: [end3.swsh=0+362]
[end3.swsh=0+362]

And indeed, it finds the swoosh in the GSUB/GPOS ligature tables in the
font file.  If I try a combination that doesn't have a swoosh:

larsi@elva:~/src$ hb-shape -V ~/.fonts/Jolie\ Romantique.ttf "#05"
trace: start table GSUB	buffer: [numbersign=0|zero=1|five=2]
trace: start lookup 0	buffer: [numbersign=0|zero=1|five=2]
trace: end lookup 0	buffer: [numbersign=0|zero=1|five=2]
trace: end table GSUB	buffer: [numbersign=0|zero=1|five=2]
trace: start table GPOS	buffer: [numbersign=0+407|zero=1+289|five=2+313]
trace: start lookup 0	buffer: [numbersign=0+407|zero=1+289|five=2+313]
trace: end lookup 0	buffer: [numbersign=0+407|zero=1+289|five=2+313]
trace: end table GPOS	buffer: [numbersign=0+407|zero=1+289|five=2+313]
[numbersign=0+407|zero=1+289|five=2+313]

It returns three characters instead of a cluster, if I interpret this
correctly.

So apparently by traversing the GSUB/GPOS tables (whatever they are),
this data can be found, and then we can feed it to font-shape-gstring
and get ligatures?

I've had a peek at the hb-shape source code, but it's unfortunately
written in the C++ style "everything happens somewhere else", so it's a
bit difficult to read.

But apparently we already parse these tables in hbfont_otf_capability?
So...  we need...  to parse them more to get all the ligature data out
of them and then...  put it in a ... char table range?  How all this
connects is very vague to me.  😀

Does this have anything to do with anything:

https://harfbuzz.github.io/harfbuzz-hb-ot-layout.html#hb-ot-layout-get-ligature-carets

?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support (was: Entering emojis)

[Eli Zaretskii]

[I've changed the Subject, because the original thread is unrelated,
and it's too long already.]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 05 Nov 2021 06:04:11 +0100
> 
> So apparently by traversing the GSUB/GPOS tables (whatever they are),
> this data can be found, and then we can feed it to font-shape-gstring
> and get ligatures?

You are describing what the text shaper does when it looks for
possible ligatures.  Why should we reproduce this in Emacs?  It's the
job of the shaping engine to understand how to query a font for some
information.  Delegating that job to the shaping engine is TRT, so
that we don't need to make changes in Emacs when some new variant of
font appears in the wild.

And I'm not sure I understand the idea behind this.  Are you trying to
add a feature to Emacs whereby it queries the font up from about the
ligatures it supports?  What would we do with such an information?
And for which font(s) would we request it?

We have font-shape-gstring.  If the font being used doesn't have a
ligature for the character sequence we pass to it, that function
returns nil, and we then display those characters "normally".  Isn't
that enough?

> But apparently we already parse these tables in hbfont_otf_capability?

Only for script and language information, because some scripts require
fonts that support special features of those scripts, and so our
default fontset requires the presence of those features in the font;
those features are queried when we determine whether some font can be
used for a character from those scripts.

> So...  we need...  to parse them more to get all the ligature data out
> of them and then...  put it in a ... char table range?  How all this
> connects is very vague to me.  😀

As I say above, I'm not sure I understand the goal of this, given what
we have already.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We have font-shape-gstring.  If the font being used doesn't have a
> ligature for the character sequence we pass to it, that function
> returns nil, and we then display those characters "normally".  Isn't
> that enough?

That's what I thought the design here was originally -- but that's not
what's happening in general in Emacs.  If I instrument
font-shape-gstring and start "emacs -Q", that function is never called.
It's not until I insert something that we've set up something in...
composition-function-table?...  that the function is called and harfbuzz
is consulted.  (You can confirm by setting a breakpoint on
hbfont_shape.)

I thought we'd just send all the text through hb_shape_full, and it
would handle all this stuff.  But we don't -- we only send a very small
subset of the strings through that function, as far as I can tell.

>> So...  we need...  to parse them more to get all the ligature data out
>> of them and then...  put it in a ... char table range?  How all this
>> connects is very vague to me.  😀
>
> As I say above, I'm not sure I understand the goal of this, given what
> we have already.

The goal is to send all text that represents ligatures (in the current
font) through font-shape-gstring.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 05 Nov 2021 14:42:39 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We have font-shape-gstring.  If the font being used doesn't have a
> > ligature for the character sequence we pass to it, that function
> > returns nil, and we then display those characters "normally".  Isn't
> > that enough?
> 
> That's what I thought the design here was originally -- but that's not
> what's happening in general in Emacs.  If I instrument
> font-shape-gstring and start "emacs -Q", that function is never called.
> It's not until I insert something that we've set up something in...
> composition-function-table?...  that the function is called and harfbuzz
> is consulted.  (You can confirm by setting a breakpoint on
> hbfont_shape.)

Yes, because characters whose slot in composition-function-table is
nil aren't handed to the shaping engine; we display them directly.

> I thought we'd just send all the text through hb_shape_full, and it
> would handle all this stuff.  But we don't -- we only send a very small
> subset of the strings through that function, as far as I can tell.

Right, and by design.  Character composition is somewhat slow in
Emacs: the display code calls into Lisp, and Lisp then turns around
and calls into C.  Doing that for every piece of text would slow down
redisplay, so we don't do that unless composition-function-table tells
us we should.

So for having ligatures you need to set up composition-function-table
for the characters that could/should ligate.  This article is a good
starting point:

  https://en.wikipedia.org/wiki/Ligature_(writing)

But please read the TODO item: there are pitfalls here that we need to
solve before this could be considered a reasonable user-level
feature.  Setting up composition-function-table to produce ligatures
for, say, "fi" and "ff" is easy for testing, but if you do that in
production, every word with these characters, everywhere on display in
Emacs, will display as that ligature, and that is hardly TRT.  We need
to make this more fine-tunable.  TODO has the details.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So for having ligatures you need to set up composition-function-table
> for the characters that could/should ligate.

Yes.  And to do that, we can parse the tables in the font, apparently.
It's not obviously clear how, but it looks like harfbuzz has the
functions needed.

> But please read the TODO item: there are pitfalls here that we need to
> solve before this could be considered a reasonable user-level
> feature.  Setting up composition-function-table to produce ligatures
> for, say, "fi" and "ff" is easy for testing, but if you do that in
> production, every word with these characters, everywhere on display in
> Emacs, will display as that ligature, and that is hardly TRT.  We need
> to make this more fine-tunable.  TODO has the details.

I guess it needs some more customisation to be a per-font thing?

But first we need to parse the Gsub/Gpos tables so that we can get
started at all.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 05 Nov 2021 15:38:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So for having ligatures you need to set up composition-function-table
> > for the characters that could/should ligate.
> 
> Yes.  And to do that, we can parse the tables in the font, apparently.
> It's not obviously clear how, but it looks like harfbuzz has the
> functions needed.

Why do we need to depend on the font?  As I said, if the font doesn't
have a ligature, font-shape-gstring will return nil, and we will then
display the characters as individual glyphs.  Isn't that what is
expected?

> > But please read the TODO item: there are pitfalls here that we need to
> > solve before this could be considered a reasonable user-level
> > feature.  Setting up composition-function-table to produce ligatures
> > for, say, "fi" and "ff" is easy for testing, but if you do that in
> > production, every word with these characters, everywhere on display in
> > Emacs, will display as that ligature, and that is hardly TRT.  We need
> > to make this more fine-tunable.  TODO has the details.
> 
> I guess it needs some more customisation to be a per-font thing?

I'm not sure it's per font.  I think it should be per buffer/mode, and
perhaps also something special for the mode line.

> But first we need to parse the Gsub/Gpos tables so that we can get
> started at all.

I don't think we should or need to do that.  See above.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do we need to depend on the font?  As I said, if the font doesn't
> have a ligature, font-shape-gstring will return nil, and we will then
> display the characters as individual glyphs.  Isn't that what is
> expected?

Different fonts have different ligatures (or "ligatures").  Since we
don't want to send all strings through font-shape-gstring (because
that'll be slow), we'll have to be selective as usual, I think?

For instance, my test font has these ligatures:

          <LigatureSet glyph="numbersign">
            <Ligature components="zero,one" glyph="end1.swsh"/>
            <Ligature components="zero,two" glyph="end2.swsh"/>
            <Ligature components="zero,three" glyph="end3.swsh"/>
          </LigatureSet>

I.e. "#01" is translated into a very different glyph, and we need to
know that so that we can send "#01" to harfbuzz to get the glyph out.
(This is how all the "neat" programming fonts that translate => into fun
glyphs work.)

This data is in the GPOS table, and I just found this fun utility:

ttx -t GPOS ~/.fonts/Jolie\ Romantique.ttf

I'll have a peek at the source code to see how much work it is to
extract this data.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'll have a peek at the source code to see how much work it is to
> extract this data.

Darn; that's a Python library and doesn't use any of the HarfBuzz
functions to parse the fonts, so it's less useful as a way to get
pointers...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Lars Ingebrigtsen]

This has some interesting information about how the data is stored:

https://simoncozens.github.io/fonts-and-layout/features.html#how-features-are-stored

Reading the harfbuzz documentation is slightly frustrating -- it's not
that it's badly documented, because it's not.  It just doesn't have that
much information about how the functions work together...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 05 Nov 2021 16:40:19 +0100
> 
> This has some interesting information about how the data is stored:
> 
> https://simoncozens.github.io/fonts-and-layout/features.html#how-features-are-stored
> 
> Reading the harfbuzz documentation is slightly frustrating -- it's not
> that it's badly documented, because it's not.  It just doesn't have that
> much information about how the functions work together...

We can ask the HarfBuzz developers questions, they are usually quite
helpful.  Their Github has Issues, just ask there.

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 05 Nov 2021 16:20:19 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why do we need to depend on the font?  As I said, if the font doesn't
> > have a ligature, font-shape-gstring will return nil, and we will then
> > display the characters as individual glyphs.  Isn't that what is
> > expected?
> 
> Different fonts have different ligatures (or "ligatures").  Since we
> don't want to send all strings through font-shape-gstring (because
> that'll be slow), we'll have to be selective as usual, I think?

How many ligatures are there in the best of all fonts?  If the number
of the ligatures is not too large, then passing them to the shaper may
not be such a terrible idea.

So I think we should first consider which ligatures we'd like to
support, and whether the answer differs depending on the major mode.
For example, ligatures like ==> etc. are relevant to prog-mode modes,
but probably not to text modes.  By contrast, "ff" and "fi" are
relevant only to text mode, I think.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> How many ligatures are there in the best of all fonts?  If the number
> of the ligatures is not too large, then passing them to the shaper may
> not be such a terrible idea.

I don't really know -- my test font has 27 ligatures, apparently, but
some of the more fun programming fonts have at least a hundred, I think.
And there's fonts that turn things like :warning: into its own glyph,
using the ligature mechanism, and who knows how many of those there may
be.

> So I think we should first consider which ligatures we'd like to
> support, and whether the answer differs depending on the major mode.
> For example, ligatures like ==> etc. are relevant to prog-mode modes,
> but probably not to text modes.  By contrast, "ff" and "fi" are
> relevant only to text mode, I think.

I think we want to support all ligatures the font supports (in general).
But as you say, we may not want to switch all the ligatures on in all
modes -- but I'm not really sure about that, either.  If a person is
using the Firabase font (or whatever it was called), they presumably do
so because they want to get all the funny glyphs everywhere (that they
use that font).

But it's certainly possible -- having a switch to tweak which ligatures
should be active on a per-mode/buffer basis is something we'd need to
look at.  I think we can look at that after we actually have ligatures
displayed at all, though.  😀

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Lars Ingebrigtsen]

I see that the harfbuzz people say that we should just run all the text
through hb_shape instead of doing it selectively like we do it now:

---
Full text shaping is the only way to get this right. Everything else
is a hack, and piling hacks on top of hacks is just storing
maintenance problems up for yourself.
---

How big a performance issue would it be to just run all the text through
hb_shape?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 05 Nov 2021 16:56:00 +0100
> 
> I see that the harfbuzz people say that we should just run all the text
> through hb_shape instead of doing it selectively like we do it now:
> 
> ---
> Full text shaping is the only way to get this right. Everything else
> is a hack, and piling hacks on top of hacks is just storing
> maintenance problems up for yourself.
> ---

Yes, I know.  But how large chunk of the text they need to see to do
the job?  Emacs's display engine examines the text one character at a
time, so passing large chunks through the shaping engine is out of the
question.  The want at least words, possibly complete lines.

> How big a performance issue would it be to just run all the text through
> hb_shape?

You can try it by setting composition-function-table slots for every
character.  E.g., start by doing that for all ASCII and see what
happens.

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: gregory@heytings.org,  mattiase@acm.org,  emacs-devel@gnu.org,
>   schwab@linux-m68k.org,  stefankangas@gmail.com,  raman@google.com
> Date: Fri, 05 Nov 2021 16:37:50 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How many ligatures are there in the best of all fonts?  If the number
> > of the ligatures is not too large, then passing them to the shaper may
> > not be such a terrible idea.
> 
> I don't really know -- my test font has 27 ligatures, apparently, but
> some of the more fun programming fonts have at least a hundred, I think.
> And there's fonts that turn things like :warning: into its own glyph,
> using the ligature mechanism, and who knows how many of those there may
> be.

If the sequences of characters that are converted to ligatures are not
too frequent, that is still not a catastrophe.

> > So I think we should first consider which ligatures we'd like to
> > support, and whether the answer differs depending on the major mode.
> > For example, ligatures like ==> etc. are relevant to prog-mode modes,
> > but probably not to text modes.  By contrast, "ff" and "fi" are
> > relevant only to text mode, I think.
> 
> I think we want to support all ligatures the font supports (in general).

All the time? really??

For example, the mode line can show stuff like "=-", and the font
could have a ligature for that -- do we really want that ligature on
the mode line?

Or the font could have a ligature for "ffi" -- do we want a variable
named "efficient" be displayed with that ligature?  I'd be surprised.

> But as you say, we may not want to switch all the ligatures on in all
> modes -- but I'm not really sure about that, either.  If a person is
> using the Firabase font (or whatever it was called), they presumably do
> so because they want to get all the funny glyphs everywhere (that they
> use that font).

Most of them use prettify-symbols-mode or similar stuff, and that's
different from how character compositions work: you must actually put
a special text property to make that happen.  So there's is not much
for us to learn from that.

> But it's certainly possible -- having a switch to tweak which ligatures
> should be active on a per-mode/buffer basis is something we'd need to
> look at.  I think we can look at that after we actually have ligatures
> displayed at all, though.  😀

Displaying them is easy, just set up composition-function-table
accordingly (I can show you the code if you cannot figure that out).
That's the least of our problems.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> For example, the mode line can show stuff like "=-", and the font
> could have a ligature for that -- do we really want that ligature on
> the mode line?

That's a good point.  On the other hand, perhaps we want to use a
different font on the mode line when using one of these special fonts.

> Or the font could have a ligature for "ffi" -- do we want a variable
> named "efficient" be displayed with that ligature?  I'd be surprised.

Possibly, but probably not.  But I don't think people will choose to use
fonts for programming that have fonts like that.  (Are there even any
monospace fonts that have such ligatures?)  But when rendering document
using a variable-pitch font, then yes.

>> But it's certainly possible -- having a switch to tweak which ligatures
>> should be active on a per-mode/buffer basis is something we'd need to
>> look at.  I think we can look at that after we actually have ligatures
>> displayed at all, though.  😀
>
> Displaying them is easy, just set up composition-function-table
> accordingly (I can show you the code if you cannot figure that out).

Yes, please -- I've been poking at that, but didn't find the right
incantation.

> That's the least of our problems.

Well, it's one problem, and I think it's the one that has to be solved
first.  (I've now sent a message to the harfbuzz list to see whether
they have any input in how to get harfbuff to cough up this list of
strings based on the font.)

For instance, see bug#51385 where they have a weirdo font that maps
"[TRACE]" (and 260 other strings) to different glyphs.  We need that
list, because we certainly don't want to be passing "[TRACE]" to
harfbuzz when not using that font.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Yuri Khan]

On Fri, 5 Nov 2021 at 23:47, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> > Or the font could have a ligature for "ffi" -- do we want a variable
> > named "efficient" be displayed with that ligature?  I'd be surprised.
>
> Possibly, but probably not.  But I don't think people will choose to use
> fonts for programming that have fonts like that.  (Are there even any
> monospace fonts that have such ligatures?)

Andale Mono
Bitstream Vera Sans Mono
Courier New
Cousine
DejaVu Sans Mono
FreeMono
JetBrains Mono
Liberation Mono
Nimbus Mono PS
Noto Mono
Noto Sans Mono
Ubuntu Mono

are some monospaced fonts that I’ve got installed on my machine and
that have ligatures “fi” and “fl”. And yes, Cousine is currently my
programming font of choice. And no, I wouldn’t like it to use the fi
ligature, because it’s drawn with a one cell advance, i.e. it will
both break alignment and disrupt reading pace.

Re: Ligature support

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> are some monospaced fonts that I’ve got installed on my machine and
> that have ligatures “fi” and “fl”. And yes, Cousine is currently my
> programming font of choice. And no, I wouldn’t like it to use the fi
> ligature, because it’s drawn with a one cell advance, i.e. it will
> both break alignment and disrupt reading pace.

Do other programs make these into ligatures when you use these fonts?  I
tried:

hb-view /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf "afib"  -o /tmp/deja.png


[-- Attachment #2: Type: image/png, Size: 5074 bytes --]

[-- Attachment #3: Type: text/plain, Size: 85 bytes --]


And as expected, no ligaturing here.  The non-mono font does the
ligature, though:


[-- Attachment #4: Type: image/png, Size: 4963 bytes --]

[-- Attachment #5: Type: text/plain, Size: 106 bytes --]




-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>
> Date: Fri, 05 Nov 2021 18:16:24 +0100
> 
> Do other programs make these into ligatures when you use these fonts?  I
> tried:
> 
> hb-view /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf "afib"  -o /tmp/deja.png
> 
> And as expected, no ligaturing here.  The non-mono font does the
> ligature, though:

That has nothing to do with fonts being monospaced, all it tells you
is that DejaVuSansMono doesn't support ligatures.  But other
monospaced fonts do.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That has nothing to do with fonts being monospaced, all it tells you
> is that DejaVuSansMono doesn't support ligatures.  But other
> monospaced fonts do.

That font was one of the fonts Yuri listed as having fi ligatures.  And
indeed: 

        <LigatureSubst index="0">
          <LigatureSet glyph="f">
            <Ligature components="l" glyph="fl"/>
            <Ligature components="i" glyph="fi"/>
          </LigatureSet>
        </LigatureSubst>

But HarfBuzz is still choosing not to use a ligature here, apparently,
for reasons that escape me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 05 Nov 2021 23:17:58 +0100
> Cc: emacs-devel@gnu.org, yuri.v.khan@gmail.com
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That has nothing to do with fonts being monospaced, all it tells you
> > is that DejaVuSansMono doesn't support ligatures.  But other
> > monospaced fonts do.
> 
> That font was one of the fonts Yuri listed as having fi ligatures.  And
> indeed: 
> 
>         <LigatureSubst index="0">
>           <LigatureSet glyph="f">
>             <Ligature components="l" glyph="fl"/>
>             <Ligature components="i" glyph="fi"/>
>           </LigatureSet>
>         </LigatureSubst>
> 
> But HarfBuzz is still choosing not to use a ligature here, apparently,
> for reasons that escape me.

Before or after the setting of composition-function-table that I
posted?  If before, it's expected.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> That font was one of the fonts Yuri listed as having fi ligatures.  And
>> indeed: 
>> 
>>         <LigatureSubst index="0">
>>           <LigatureSet glyph="f">
>>             <Ligature components="l" glyph="fl"/>
>>             <Ligature components="i" glyph="fi"/>
>>           </LigatureSet>
>>         </LigatureSubst>
>> 
>> But HarfBuzz is still choosing not to use a ligature here, apparently,
>> for reasons that escape me.
>
> Before or after the setting of composition-function-table that I
> posted?  If before, it's expected.

Those screenshots were from hb-view, not Emacs, so they should be
totally authoritative in how HarfBuzz is meant to display that run of
characters.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Yuri Khan]

On Sat, 6 Nov 2021 at 00:16, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Do other programs make these into ligatures when you use these fonts?

Looks like they don’t. So, the fonts have a glyph for the ligature
code point, but it’s opt-in.

I can see how people could want to only use a subset of ligatures
supported by a font. Fira Code has a ton of ligatures for
multi-character operators in various programming languages, but one
might prefer to enable ligatures only for the operators that actually
exist in the language one is working with. Like, in HTML, --> is a
closing comment delimiter, but in C “while (n --> 0)” only works
because it’s interpreted as “decrement n and check if its old value
was greater than zero”.

Re: Ligature support

[Lars Ingebrigtsen]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> Looks like they don’t. So, the fonts have a glyph for the ligature
> code point, but it’s opt-in.

Do you happen to know what the opt-in mechanism in HarfBuzz is?

> I can see how people could want to only use a subset of ligatures
> supported by a font. Fira Code has a ton of ligatures for
> multi-character operators in various programming languages, but one
> might prefer to enable ligatures only for the operators that actually
> exist in the language one is working with. Like, in HTML, --> is a
> closing comment delimiter, but in C “while (n --> 0)” only works
> because it’s interpreted as “decrement n and check if its old value
> was greater than zero”.

Good point.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Yuri Khan]

On Sat, 6 Nov 2021 at 05:20, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> > Looks like they don’t. So, the fonts have a glyph for the ligature
> > code point, but it’s opt-in.
>
> Do you happen to know what the opt-in mechanism in HarfBuzz is?

No, sorry.

Re: Ligature support

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On Fri, Nov 05, 2021 at 05:45:09PM +0100, Lars Ingebrigtsen wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > For example, the mode line can show stuff like "=-", and the font
> > could have a ligature for that -- do we really want that ligature on
> > the mode line?
> 
> That's a good point.  On the other hand, perhaps we want to use a
> different font on the mode line when using one of these special fonts.
> 
> > Or the font could have a ligature for "ffi" -- do we want a variable
> > named "efficient" be displayed with that ligature?  I'd be surprised.
> 
> Possibly, but probably not.  But I don't think people will choose to use
> fonts for programming that have fonts like that.  (Are there even any
> monospace fonts that have such ligatures?)  But when rendering document
> using a variable-pitch font, then yes.

Thing is you sometimes want the ligature and sometimes you don't.
In languages which tend to slap words together, making a ligature
across the word junction (morpheme boundary) actually hinders
legibility. English has those too: the 'fl' in chiefly, the 'ff'
in shelfful, you get the idea.

I wonder whether the HarfBuzz engine takes that into consideration;
it would have to know (or guess?) the language it is treating.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ligature support

[Eli Zaretskii]

> Date: Fri, 5 Nov 2021 18:13:56 +0100
> From: <tomas@tuxteam.de>
> 
> Thing is you sometimes want the ligature and sometimes you don't.
> In languages which tend to slap words together, making a ligature
> across the word junction (morpheme boundary) actually hinders
> legibility. English has those too: the 'fl' in chiefly, the 'ff'
> in shelfful, you get the idea.
> 
> I wonder whether the HarfBuzz engine takes that into consideration;
> it would have to know (or guess?) the language it is treating.

We do pass the language to HarfBuzz when we think we know it, but the
problem is Emacs itself has no good notion of the "current language".
Such a notion is problematic in a multilingual editor such as Emacs.
It is something we still need to figure out, and after that implement
the necessary infrastructure.  What we have now is rudimentary and
very insufficient.

Re: Ligature support

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1539 bytes --]

On Fri, Nov 05, 2021 at 09:14:32PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 5 Nov 2021 18:13:56 +0100
> > From: <tomas@tuxteam.de>
> > 
> > Thing is you sometimes want the ligature and sometimes you don't.
> > [depends on language]

[...]

> > it would have to know (or guess?) the language it is treating.
> 
> We do pass the language to HarfBuzz when we think we know it, but the
> problem is Emacs itself has no good notion of the "current language".

This is what I was pointing at. I don't think this is a problem which
can be solved in general. You have homographs (words that write the
same) within one language, you have them across languages.

If the text itself is multilingual, your best bet is to ask the user
and your second-best bet is to do some statistical heuristics, which
only will "work" for a longer stretch of text.

If you press me, I think I can find two German homographs where the
one would take a ligature and the other not >:-)

> Such a notion is problematic in a multilingual editor such as Emacs.
> It is something we still need to figure out, and after that implement
> the necessary infrastructure.  What we have now is rudimentary and
> very insufficient.

I think that will always be an approximation. AFAIK Mozilla has (had?)
a library for guessing a text's (human) language (this is useful for
other things: capitalisation is language-dependent too, e.g. the Turkish
dotless i).

But it will always be something which fails in edge cases, I think.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ligature support

[Werner LEMBERG]

> If you press me, I think I can find two German homographs where the
> one would take a ligature and the other not >:-)

Indeed.  Regarding an 'st' ligature (which is mandatory for Fraktur,
by the way), there are some examples.

  Schiffstau:
    Schiffs-tau (ship rope) vs. Schiff-stau (ship traffic jam)

  Wachstube:
    Wachs-tube (wax tube) vs. Wach-stube (guard room)


      Werner

Re: Ligature support

[tomas]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

On Fri, Nov 05, 2021 at 08:16:08PM +0000, Werner LEMBERG wrote:
> 
> > If you press me, I think I can find two German homographs where the
> > one would take a ligature and the other not >:-)
> 
> Indeed.  Regarding an 'st' ligature (which is mandatory for Fraktur,
> by the way), there are some examples.
> 
>   Schiffstau:
>     Schiffs-tau (ship rope) vs. Schiff-stau (ship traffic jam)
> 
>   Wachstube:
>     Wachs-tube (wax tube) vs. Wach-stube (guard room)

Oh, nice :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ligature support

[Eli Zaretskii]

> Date: Fri, 05 Nov 2021 20:16:08 +0000 (UTC)
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> 
> > If you press me, I think I can find two German homographs where the
> > one would take a ligature and the other not >:-)
> 
> Indeed.  Regarding an 'st' ligature (which is mandatory for Fraktur,
> by the way), there are some examples.
> 
>   Schiffstau:
>     Schiffs-tau (ship rope) vs. Schiff-stau (ship traffic jam)
> 
>   Wachstube:
>     Wachs-tube (wax tube) vs. Wach-stube (guard room)

You guys are splitting hair when we don't even have a reliable means
to tell HarfBuzz the ligature is required for German and not, say, for
English.  Currently, composition-function-table is _global_, i.e. it
affects all the buffers in the session.  We have a lot of turf to
cover before the above fine-tuning will be relevant.

Volunteers and patches are most welcome.

Re: Ligature support

[Eli Zaretskii]

> Date: Fri, 5 Nov 2021 20:52:45 +0100
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > it would have to know (or guess?) the language it is treating.
> > 
> > We do pass the language to HarfBuzz when we think we know it, but the
> > problem is Emacs itself has no good notion of the "current language".
> 
> This is what I was pointing at.

Well, don't just point to the obvious: better sit down and code some
features that we can use to be smarter ;-)

> If the text itself is multilingual, your best bet is to ask the user

Asking the user during redisplay is a non-starter.

> and your second-best bet is to do some statistical heuristics, which
> only will "work" for a longer stretch of text.

That's a waste of CPU cycles: when we don't know the language, we ask
HarfBuzz to guess, and I trust HarfBuzz that it can guess as well or
better as we can.

> > Such a notion is problematic in a multilingual editor such as Emacs.
> > It is something we still need to figure out, and after that implement
> > the necessary infrastructure.  What we have now is rudimentary and
> > very insufficient.
> 
> I think that will always be an approximation.

Maybe, maybe not.  I Hope at least sometimes we could do better.
there are various hints in the form of the encoding, the source of the
text, etc.  We just need to figure out which means we have for
gleaning the language that is not obvious from the characters
themselves (because HarfBuzz does the latter already), and provide the
features for Lisp programs and users to use them.

Re: Ligature support

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3340 bytes --]

On Fri, Nov 05, 2021 at 10:30:47PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 5 Nov 2021 20:52:45 +0100
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > > it would have to know (or guess?) the language it is treating.
> > > 
> > > We do pass the language to HarfBuzz when we think we know it, but the
> > > problem is Emacs itself has no good notion of the "current language".
> > 
> > This is what I was pointing at.
> 
> Well, don't just point to the obvious: better sit down and code some
> features that we can use to be smarter ;-)
> 
> > If the text itself is multilingual, your best bet is to ask the user
> 
> Asking the user during redisplay is a non-starter.

:-)

More constructively, that's what happens while typesetting text.
That's what TeX has \/ for.

We have two classes of language: the ones, where ligatures are
essential (Arabic, Hangul -- I must admit that I know very little
about the latter). For those, there is no choice.

Then we have those where ligatures are rather a "decoration", an
accident of old handwriting further fashioned by the introduction
of movable type.

And a decoration you sometimes downright don't want (in TeX, last
time I looked, most German writers just disabled ligatures: the
"wrong ligatures" are so much more disturbing, and the prospect
of proofreading the thing for wrong ligatures and sprinkling your
source with \/ just isn't worth it).

In short, there are languages where "asking the user" is just the
only option; that means that the feature only makes sense while
typesetting (where you /can/ ask the user) and not while rendering
dynamically (the "redisplay" case we are treating here).

The problem is composed with TeX's legacy, which used its ligature
mechanism for things which strictly aren't, think -- for an em
dash. It's a nice hack, and people perceive that as a ligature,
too (you can see that elsewhere in this huge thread) but it ain't.

I still think: there isn't a general solution. Me? I'd prefer to
disable all ligatures unless I'm writing Arabic.

> 
> > and your second-best bet is to do some statistical heuristics, which
> > only will "work" for a longer stretch of text.
> 
> That's a waste of CPU cycles: when we don't know the language, we ask
> HarfBuzz to guess, and I trust HarfBuzz that it can guess as well or
> better as we can.

I haven't looked into it, but I wonder what magic it uses, if it
isn't some variation of "maximum likelihood over n-gram statistics".

> 
> > > Such a notion is problematic in a multilingual editor such as Emacs.
> > > It is something we still need to figure out, and after that implement
> > > the necessary infrastructure.  What we have now is rudimentary and
> > > very insufficient.
> > 
> > I think that will always be an approximation.
> 
> Maybe, maybe not.  I Hope at least sometimes we could do better.
> there are various hints in the form of the encoding, the source of the
> text, etc.  We just need to figure out which means we have for
> gleaning the language that is not obvious from the characters
> themselves (because HarfBuzz does the latter already), and provide the
> features for Lisp programs and users to use them.

Some day I'll peek into HarfBuzz's source code. Perhaps next year.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ligature support

[Eli Zaretskii]

> Date: Sat, 6 Nov 2021 10:16:25 +0100
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > We do pass the language to HarfBuzz when we think we know it, but the
> > > > problem is Emacs itself has no good notion of the "current language".
> > > 
> > > This is what I was pointing at.
> > 
> > Well, don't just point to the obvious: better sit down and code some
> > features that we can use to be smarter ;-)
> > 
> > > If the text itself is multilingual, your best bet is to ask the user
> > 
> > Asking the user during redisplay is a non-starter.
> [...]
> In short, there are languages where "asking the user" is just the
> only option; that means that the feature only makes sense while
> typesetting (where you /can/ ask the user) and not while rendering
> dynamically (the "redisplay" case we are treating here).

I thought you were suggesting to ask the user about the language
pertinent to a particular chunk of text.

Regarding ligatures in general: of course, the user should be able to
say whether the ligatures are wanted, and be able to select which of
the ligatures are and which aren't.  The issue at hand was that
ligatures are language sensitive, and we don't have a good notion of
the language of the text.

> > > and your second-best bet is to do some statistical heuristics, which
> > > only will "work" for a longer stretch of text.
> > 
> > That's a waste of CPU cycles: when we don't know the language, we ask
> > HarfBuzz to guess, and I trust HarfBuzz that it can guess as well or
> > better as we can.
> 
> I haven't looked into it, but I wonder what magic it uses, if it
> isn't some variation of "maximum likelihood over n-gram statistics".

Nothing of the kind, AFAIU: it just looks at the script of the
characters.

But it sounds strange to me to have Emacs use n-gram statistics during
display.  We should collect the relevant data from other sources.

Re: Ligature support

[Stefan Kangas]

tomas@tuxteam.de writes:

>> We do pass the language to HarfBuzz when we think we know it, but the
>> problem is Emacs itself has no good notion of the "current language".
>
> This is what I was pointing at. I don't think this is a problem which
> can be solved in general. You have homographs (words that write the
> same) within one language, you have them across languages.

Right, detecting a language is hard and is AFAIK discussed at the
frontlines of language technology.  We don't need to attempt that, I
think.  Other software just let you set the current language or
languages manually.

I would suggest a variable `current-language' (or something) that
controls this on a buffer level.

In word processors, you can indicate the current language down to the
individual word or perhaps even character level.  Perhaps we could also
have a text property for it?

Re: Ligature support

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 5 Nov 2021 13:43:31 -0700
> Cc: emacs-devel@gnu.org
> 
> I would suggest a variable `current-language' (or something) that
> controls this on a buffer level.
> 
> In word processors, you can indicate the current language down to the
> individual word or perhaps even character level.  Perhaps we could also
> have a text property for it?

The challenge here is to provide the means for users and Lisp programs
to specify the language of a chunk of text, not just of the entire
buffer, including reasonable defaults that use the locale and other
relevant meta-data.  And yes, text properties are one way of storing
the information, like we already do with charsets.

Re: Ligature support

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Well, it's one problem, and I think it's the one that has to be solved
> first.  (I've now sent a message to the harfbuzz list to see whether
> they have any input in how to get harfbuff to cough up this list of
> strings based on the font.)

And if HarfBuzz doesn't have the functions to spit out the info (i.e.,
if it only has lookups, but not introspection functionality), it should
be to difficult to write a little .el to parse the otf files:

https://docs.microsoft.com/en-us/typography/opentype/spec/otff

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 05 Nov 2021 18:18:30 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Well, it's one problem, and I think it's the one that has to be solved
> > first.  (I've now sent a message to the harfbuzz list to see whether
> > they have any input in how to get harfbuff to cough up this list of
> > strings based on the font.)
> 
> And if HarfBuzz doesn't have the functions to spit out the info (i.e.,
> if it only has lookups, but not introspection functionality), it should
> be to difficult to write a little .el to parse the otf files:
> 
> https://docs.microsoft.com/en-us/typography/opentype/spec/otff

If we have to.  But I still don't understand why would we care.

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 05 Nov 2021 17:45:09 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > For example, the mode line can show stuff like "=-", and the font
> > could have a ligature for that -- do we really want that ligature on
> > the mode line?
> 
> That's a good point.  On the other hand, perhaps we want to use a
> different font on the mode line when using one of these special fonts.

It doesn't matter, because composition-function-table is currently
global and doesn't depend on the font.

> > Or the font could have a ligature for "ffi" -- do we want a variable
> > named "efficient" be displayed with that ligature?  I'd be surprised.
> 
> Possibly, but probably not.  But I don't think people will choose to use
> fonts for programming that have fonts like that.

Oh yes, they do.

> (Are there even any monospace fonts that have such ligatures?)

Yes, definitely.

> > Displaying them is easy, just set up composition-function-table
> > accordingly (I can show you the code if you cannot figure that out).
> 
> Yes, please -- I've been poking at that, but didn't find the right
> incantation.

The following should display the ligatures starting with 'f' if the
font supports them:

  (aset composition-function-table
	?f
	'(["f[fhijlt]" 0 font-shape-gstring]))

> Well, it's one problem, and I think it's the one that has to be solved
> first.

See above: there's no problem here that isn't already solved.

> For instance, see bug#51385 where they have a weirdo font that maps
> "[TRACE]" (and 260 other strings) to different glyphs.  We need that
> list, because we certainly don't want to be passing "[TRACE]" to
> harfbuzz when not using that font.

I think we rather need to have a (minor) mode which uses those, and
under that mode we do want to pass these strings to HarfBuzz.  If the
default font doesn't support the corresponding ligatures, we display
the literal strings.  It is then up to the user to install the right
font and tell Emacs to use it (via some face, I guess).

Re: Ligature support

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> It doesn't matter, because composition-function-table is currently
> global and doesn't depend on the font.

Yes, and I think that has to change.  (Probably.)

> The following should display the ligatures starting with 'f' if the
> font supports them:
>
>   (aset composition-function-table
> 	?f
> 	'(["f[fhijlt]" 0 font-shape-gstring]))

Thanks!  That was both easier and less expressive than I thought, in a
way...  With my fancy test font.  Before:


[-- Attachment #2: Type: image/png, Size: 13355 bytes --]

[-- Attachment #3: Type: text/plain, Size: 81 bytes --]


Then

(aset composition-function-table ?# '(["#0[0-3]" 0 font-shape-gstring]))


[-- Attachment #4: Type: image/png, Size: 14241 bytes --]

[-- Attachment #5: Type: text/plain, Size: 114 bytes --]


Heh; there's some glitches here -- the leftmost swoosh is so extensive
that it doesn't paint it correctly, and:


[-- Attachment #6: Type: image/png, Size: 3505 bytes --]

[-- Attachment #7: Type: text/plain, Size: 1065 bytes --]


Entering some spaces at the start doesn't clear up all the artefacts.

>> For instance, see bug#51385 where they have a weirdo font that maps
>> "[TRACE]" (and 260 other strings) to different glyphs.  We need that
>> list, because we certainly don't want to be passing "[TRACE]" to
>> harfbuzz when not using that font.
>
> I think we rather need to have a (minor) mode which uses those, and
> under that mode we do want to pass these strings to HarfBuzz.  If the
> default font doesn't support the corresponding ligatures, we display
> the literal strings.  It is then up to the user to install the right
> font and tell Emacs to use it (via some face, I guess).

Yes and no.  There's no limit to the number of these strings that fonts
support, so if this hypothetical minor mode were to "list them all",
it'd basically be the same as running all text through harfbuzz.  Which
we don't want to do in general.  Ergo the need to have a per-font
ligature table.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> the same as running all text through harfbuzz.  Which we don't want to
> do in general.

But what's the effect of that, then, I wonder?  Uhm...

(aset composition-function-table
	?\s
	'([" [^ \n\t]+" 0 font-shape-gstring]))

Is this the correct incantation for "send (almost) all words to
harfbuzz"?  (I.e., a space and then a non-space sequence.)

In xdisp.c with this:

(benchmark-run 1 (while (not (eobp)) (forward-page 1) (redisplay t)))

Without: 1.2s
With: 1.3s

Sending all words to harfbuzz is probably the right solution in a
variable-pitch font buffer anyway (to get proper kerning), but it
probably doesn't make that much sense in a monospace-font buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Sending all words to harfbuzz is probably the right solution in a
> variable-pitch font buffer anyway (to get proper kerning), but it
> probably doesn't make that much sense in a monospace-font buffer.

The space-based thing was wrong, but here's another stab at testing the
speed.

(dolist (c (append (number-sequence ?A ?Z)
 	           (number-sequence ?a ?z)))
  (aset composition-function-table
	c
	(list (vector (concat (string c) "[^ \n\t]+") 0 #'font-shape-gstring))))

Without:


[-- Attachment #2: Type: image/png, Size: 3994 bytes --]

[-- Attachment #3: Type: text/plain, Size: 8 bytes --]


With:


[-- Attachment #4: Type: image/png, Size: 3936 bytes --]

[-- Attachment #5: Type: text/plain, Size: 233 bytes --]


So that fixes the kerning (this is with the proportional Deja Vu font).
And with that, scrolling through xdisp.c is 30% slower.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Nov 2021 00:02:57 +0100
> 
> (dolist (c (append (number-sequence ?A ?Z)
>  	           (number-sequence ?a ?z)))
>   (aset composition-function-table
> 	c
> 	(list (vector (concat (string c) "[^ \n\t]+") 0 #'font-shape-gstring))))
> 
> So that fixes the kerning (this is with the proportional Deja Vu font).
> And with that, scrolling through xdisp.c is 30% slower.

30% slowdown is already a bad sign.  However, scrolling through
xdisp.c is not representative enough, since most of that is source
code which will not ligate given the above rules.  If you want to make
a realistic comparison for xdisp.c, you need ligature rules for
program symbols, not for ASCII letters.

I tried the above with the King James Bible text, which you can
download from here:

  http://corpus.canterbury.ac.nz/resources/large.tar.gz

It's a 30K line, 4MB file.  With that, running the scrolling benchmark
whose code is shown below produces:

  without ligatures: 30.4 sec and 28 GC cycles
  with ligatures:    91.1 sec and 91 GC cycles

(This is an unoptimized build of Emacs 29; with optimized builds
and/or faster CPUs you will see shorter times, but the ratio should be
similar.)

Are we willing to make our redisplay 3 times slower with plain ASCII
text?

I think the current design of character compositions in Emacs is
inappropriate for sending all the text via the shaping engine.  It was
designed on the assumption that composed characters will be rare in
"normal" usage, and will only be massive in some specific scripts
where this is unavoidable, like Arabic and Hangul.  If we want to call
the shaper for everything we display, we should radically change how
this is designed and implemented.

Here's the function I used for the benchmark:

(defun scroll-up-benchmark ()
  (interactive)
  (let ((oldgc gcs-done)
        (oldtime (float-time)))
    (condition-case nil (while t (scroll-up) (redisplay))
      (error (message "GCs: %d Elapsed time: %f seconds"
                      (- gcs-done oldgc) (- (float-time) oldtime))))))

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> 30% slowdown is already a bad sign.

Oh yeah, it means that we can't just do that.  If we want to send
everything through hb_shape, we'd have to change our display to optimise
for that case.  (Which I'm guessing would be a lot of work.)

Today it's doing a lot of work to pick and choose those bits.

> Are we willing to make our redisplay 3 times slower with plain ASCII
> text?

Nope.

> I think the current design of character compositions in Emacs is
> inappropriate for sending all the text via the shaping engine.  It was
> designed on the assumption that composed characters will be rare in
> "normal" usage, and will only be massive in some specific scripts
> where this is unavoidable, like Arabic and Hangul.  If we want to call
> the shaper for everything we display, we should radically change how
> this is designed and implemented.

Yes, indeed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Nov 2021 19:08:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > 30% slowdown is already a bad sign.
> 
> Oh yeah, it means that we can't just do that.  If we want to send
> everything through hb_shape, we'd have to change our display to optimise
> for that case.  (Which I'm guessing would be a lot of work.)

We basically need to rethink the entire design of the display engine.
It starts with the basic premise in the design that buffer text is
examined one character at a time, and all the layout decisions for
that character are made before we proceed to the next character.  This
already required quite esoteric jumps through hoops to add character
composition (which needs to examine sequences of characters, and when
composable characters are detected, skip them all after processing
them).  If we want to pass most or all of the text through the shaping
engine all that needs to go away, and we need something very
different.

Re: Ligature support

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> There's no limit to the number of these strings that fonts support, so
> if this hypothetical minor mode were to "list them all", it'd
> basically be the same as running all text through harfbuzz.  Which we
> don't want to do in general.  Ergo the need to have a per-font
> ligature table.

Well, that doesn't follow.  Once we have a way to dump the ligature
tables, we know what ligatures the fonts the user has loaded have (in
total).  A typical user won't be loading more than a handful of fonts,
so the number of ligatures at any point won't be more than a few
hundred.

And since it's harmless (just takes a bit more time) to give a sequence
to hb_shape for a font that doesn't have the ligature, we don't need a
per-font thing.

But the user may want to switch ligatures off completely, or per a
buffer/mode basis, so we do need some extra support for that.  The
primary thing here, though, is to get at the ligature data in an
efficient manner.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Nov 2021 06:23:27 +0100
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > There's no limit to the number of these strings that fonts support, so
> > if this hypothetical minor mode were to "list them all", it'd
> > basically be the same as running all text through harfbuzz.  Which we
> > don't want to do in general.  Ergo the need to have a per-font
> > ligature table.
> 
> Well, that doesn't follow.  Once we have a way to dump the ligature
> tables, we know what ligatures the fonts the user has loaded have (in
> total).  A typical user won't be loading more than a handful of fonts,
> so the number of ligatures at any point won't be more than a few
> hundred.
> 
> And since it's harmless (just takes a bit more time) to give a sequence
> to hb_shape for a font that doesn't have the ligature, we don't need a
> per-font thing.

I think this is a premature conclusion.  For starters, it sounds like
you are thinking only about ASCII ligatures?  Non-ASCII scripts have
ligatures as well; in fact, in some of them we already pass all the
text to the shaping engine, because that's the only way of producing
display that users of those scripts will consider valid and legible.

And why assume people will not want to disable some ligatures for some
fonts, for example if they look ugly?

More generally, we don't yet have a complete enough idea about use
patterns and wishes of users wrt ligatures in various contexts.  At
least two separate contexts should be considered: source code (where
ligatures are normally limited to operators and symbols) and
human-readable text (where people may want the typographical
ligatures).

As for "a bit more time", I think that conclusion is also premature;
I've just shown a benchmark where it is more like "a lot more time".

> The primary thing here, though, is to get at the ligature data in an
> efficient manner.

I guess nothing I can say at this point will convince you this is NOT
the most important part, so I'll just shut up ;-)

Re: Ligature support

[chad]

[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]

On Fri, Nov 5, 2021 at 10:24 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> [...] A typical user won't be loading more than a handful of fonts,
> so the number of ligatures at any point won't be more than a few
> hundred.
>

I don't know if you're familiar with this or not, but there's a popular bit
of "eye candy" for emacs called all-the-icons:

  https://github.com/domtronn/all-the-icons.el

This package (and it's helpers) adds a bunch of icons to emacs in various
contexts (mode-line, dired, completions, etc.). It gets those icons from
several fonts (currently: the file icons from the Atom editor, the
FontAwesome icons, the GitHub "Octicons", Google's Material Design icons,
and a set of weather icons maintained by a group on github). Most of these
fonts are created and maintained with an eye towards use on the web, but
all-the-icons packages them up for easy-ish use by emacs. Aside: I recall
that some people dislike this approach to bringing icons into Emacs as a
hack, which it undoubtedly is. Until something like Stefan Kangas' icons.el
project bears fruit, it's difficult to imagine an alternative that will
provide a reasonable number of easily-scalable icons of the sizes desired
by most of the users of all-the-icons.

I bring this up because most of these fonts have ligatures, and they seem
to be using them for various "special effects" that I could imagine wanting
to use inside Emacs, and also because when I use this package inside emacs,
I don't see those fonts listed in list-fontsets, so it's possible that they
might also be "hiding" when you check to see how many fonts might be
scanned by harfbuzz.

~Chad

[-- Attachment #2: Type: text/html, Size: 2214 bytes --]

Re: Ligature support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This package (and it's helpers) adds a bunch of icons to emacs in various
  > contexts (mode-line, dired, completions, etc.). It gets those icons from
  > several fonts (currently: the file icons from the Atom editor, the
  > FontAwesome icons, the GitHub "Octicons", Google's Material Design icons,
  > and a set of weather icons maintained by a group on github).

1. Are all these fonts free, without exceptions?

2. How many fonts are involved?

3. Is it any practical problem to use that many fonts?
Would there be an advantage to repacking the icons that are useful in Emacs
into one font?  Do the licenses of these fonts permit that?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Ligature support

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

>   > This package (and it's helpers) adds a bunch of icons to emacs in various
>   > contexts (mode-line, dired, completions, etc.). It gets those icons from
>   > several fonts (currently: the file icons from the Atom editor, the
>   > FontAwesome icons, the GitHub "Octicons", Google's Material Design icons,
>   > and a set of weather icons maintained by a group on github).
>
> 1. Are all these fonts free, without exceptions?

Yes.

> 2. How many fonts are involved?

Six, AFAICT:

- Atom File Icons Plugin  MIT LICENSE
- FontAwesome Icons       SIL OFL LICENSE
- GitHub OctIcons         SIL OFL LICENSE
- Weather Icons           SIL OFL LICENSE
- Material Icons          APACHE LICENSE v2.0
- Custom Made Font        MIT LICENSE

> 3. Is it any practical problem to use that many fonts?
> Would there be an advantage to repacking the icons that are useful in Emacs
> into one font?  Do the licenses of these fonts permit that?

The fonts are just packaged up SVG image files from the above named icon
sets (e.g. Material Icons).  The idea with icons.el that I'm working on
is to skip the fonts and distribute the SVG files directly.

Re: Ligature support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for checking the licenses.  Here are some comments on what you
found.

  > - Atom File Icons Plugin  MIT LICENSE

Actually the term "MIT license" is a confusion for two similar
but not identical license.  One is the X11 license and the other is the
Expat license.  Would you please distinguish them?
See https://gnu.org/licenses/license-list.html.

  > The fonts are just packaged up SVG image files from the above named icon
  > sets (e.g. Material Icons).

I can see various meanings for "package up SVG files", and it can make
a difference regarding copyright.

One meaning is that some entity A released SVG files
and another entity B packaged them as fonts.  Is that what you mean?
If so, which one put the license on the fonts?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Ligature support

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

>   > - Atom File Icons Plugin  MIT LICENSE
>
> Actually the term "MIT license" is a confusion for two similar
> but not identical license.  One is the X11 license and the other is the
> Expat license.  Would you please distinguish them?
> See https://gnu.org/licenses/license-list.html.

In both cases where the overview said "MIT License", it is actually the
Expat license:

https://github.com/file-icons/atom/blob/master/LICENSE.md
https://github.com/domtronn/all-the-icons.el/blob/master/LICENSE

>   > The fonts are just packaged up SVG image files from the above named icon
>   > sets (e.g. Material Icons).
>
> I can see various meanings for "package up SVG files", and it can make
> a difference regarding copyright.
>
> One meaning is that some entity A released SVG files
> and another entity B packaged them as fonts.  Is that what you mean?
> If so, which one put the license on the fonts?

1. In this case, the fonts were released with the same license that was
   originally used for the SVG files:

   https://google.github.io/material-design-icons/#licensing

2. In two cases, the icons were actually originally released both as SVG
   files, and as fonts using the previously stated license:

   https://github.com/FortAwesome/Font-Awesome/blob/master/LICENSE.txt
   https://github.com/erikflowers/weather-icons#licensing
   https://github.com/file-icons/atom

3. In this case, the original files used expat:

   https://github.com/primer/octicons/blob/main/LICENSE

   But the font is released using SIL Open Font License.  From some
   cursory research online, it seems like this may also have been
   released by the original copyright holder using the SIL license at
   some point, but I'm not sure.

4. Finally, the custom made font uses the Expat license:

   https://github.com/domtronn/all-the-icons.el/blob/master/LICENSE

Re: Ligature support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I'm sure we can use this one way or another.

Is all the material available in a way that avoids the SIL font license?
It is better to avoid that if possible.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Ligature support

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> Is all the material available in a way that avoids the SIL font license?
> It is better to avoid that if possible.

It looks like that, yes.  I'll make sure to specifically avoid the SIL
font license when possible in my in-development library icons.el.

Re: Ligature support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is all the material available in a way that avoids the SIL font license?
  > > It is better to avoid that if possible.

  > It looks like that, yes.  I'll make sure to specifically avoid the SIL
  > font license when possible in my in-development library icons.el.

It is desirable to avoid the SIL font license, but it isn't crucial.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Ligature support

[chad]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Fri, Nov 19, 2021 at 11:39 PM Richard Stallman <rms@gnu.org> wrote:

> Is all the material available in a way that avoids the SIL font license?
> It is better to avoid that if possible.


 Out of curiosity, what makes the SIL licence something to be avoided? The
GNU project licenses list (https://www.gnu.org/licenses/license-list.en.html)
doesn't mention anything that makes me think it should be avoided, and in
fact it suggests to me that the SIL license is one of the better choices.

(This is largely academic to me at this point, but in my other work-life,
we almost released some content-specific icons as part of a font, so I
looked a bit to make sure that there was reasonable libre license, and SIL
seemed to be a reasonable choice. If something changed or I missed
something, I'm interested.)

Thanks in advance,
~Chad

[-- Attachment #2: Type: text/html, Size: 1305 bytes --]

Re: Ligature support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >  Out of curiosity, what makes the SIL licence something to be avoided?

The problem is described explicily in the item in
https://gnu.org/licenses/license-list.html.  It is not disastrous, but
it is annoying.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Ligature support

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: larsi@gnus.org, eliz@gnu.org, emacs-devel@gnu.org
> Date: Sun, 14 Nov 2021 23:51:50 -0500
> 
>   > This package (and it's helpers) adds a bunch of icons to emacs in various
>   > contexts (mode-line, dired, completions, etc.). It gets those icons from
>   > several fonts (currently: the file icons from the Atom editor, the
>   > FontAwesome icons, the GitHub "Octicons", Google's Material Design icons,
>   > and a set of weather icons maintained by a group on github).
> 
> 1. Are all these fonts free, without exceptions?
> 
> 2. How many fonts are involved?
> 
> 3. Is it any practical problem to use that many fonts?
> Would there be an advantage to repacking the icons that are useful in Emacs
> into one font?  Do the licenses of these fonts permit that?

Just to make sure we all are on the same page: the above is a tangent
that has no direct relation to the ligature support, the original
subject of this thread.

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 05 Nov 2021 23:38:12 +0100
> 
> Heh; there's some glitches here -- the leftmost swoosh is so extensive
> that it doesn't paint it correctly, and:
> 
> Entering some spaces at the start doesn't clear up all the artefacts.

Probably some display glitches because we don't expect characters to
reach so far to the left.

Note, however, that displaying this correctly will slow down
redisplay, because any change in the "swish" area will need to redraw
all the characters of that word.  This will be in particular painful
when the only change is that cursor moved in the same screen line:
we'd probably need to disable one of the more powerful redisplay
optimizations we have.

> > I think we rather need to have a (minor) mode which uses those, and
> > under that mode we do want to pass these strings to HarfBuzz.  If the
> > default font doesn't support the corresponding ligatures, we display
> > the literal strings.  It is then up to the user to install the right
> > font and tell Emacs to use it (via some face, I guess).
> 
> Yes and no.  There's no limit to the number of these strings that fonts
> support, so if this hypothetical minor mode were to "list them all",
> it'd basically be the same as running all text through harfbuzz.  Which
> we don't want to do in general.  Ergo the need to have a per-font
> ligature table.

I don't necessarily agree with the conclusion.  The list of ligatures
will have to be customizable, that's a given.  Thus, the user will
always be able to add/remove ligatures.  Whether we should have a
built-in tool to show all the ligatures supported by a font is an
orthogonal question, because there are (external) tools out there to
do that, and we don't have to duplicate their functionality.  (I also
think that the list of ligatures depends on the script and the
language, so the response to such a query is not necessarily a single
list.)

This is not, and should not be, the area of our expertise.  We
delegate that to a shaping engine for a good reason.

Re: Ligature support

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Probably some display glitches because we don't expect characters to
> reach so far to the left.

It's a pretty extreme font.

> This is not, and should not be, the area of our expertise.  We
> delegate that to a shaping engine for a good reason.

But we don't do that -- we pick and choose what we send to the shaping
engine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Ligature support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Nov 2021 19:05:48 +0100
> 
> > This is not, and should not be, the area of our expertise.  We
> > delegate that to a shaping engine for a good reason.
> 
> But we don't do that -- we pick and choose what we send to the shaping
> engine.

We choose what text we would like to ligate, not whether and how it
will ligate.  We also don't query the font about specific ligatures or
glyphs, we leave that to the shaper.

Re: Ligature support

[Benjamin Riefenstahl]

> Eli Zaretskii <eliz@gnu.org> writes:
>> Or the font could have a ligature for "ffi" -- do we want a variable
>> named "efficient" be displayed with that ligature?  I'd be surprised.

Lars Ingebrigtsen writes:
> Possibly, but probably not.  But I don't think people will choose to use
> fonts for programming that have fonts like that.  (Are there even any
> monospace fonts that have such ligatures?)  But when rendering document
> using a variable-pitch font, then yes.

That is what font "features" are for, as I understand the concept.  I
think Emacs supports those already.  Features enable or disable stuff
like certain ligatures.  The user can activate or deactivate features in
the font specification to choose which ligatures to use.  The idea would
be that the defaults are conservative, so I expect monospaced fonts
would usually disable most ligatures, but the user can still enable them
when and where s/he wants them.

Just my 2 cent,
benny

Re: Ligature support

[Eli Zaretskii]

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Date: Sat, 06 Nov 2021 14:25:11 +0100
> 
> > Possibly, but probably not.  But I don't think people will choose to use
> > fonts for programming that have fonts like that.  (Are there even any
> > monospace fonts that have such ligatures?)  But when rendering document
> > using a variable-pitch font, then yes.
> 
> That is what font "features" are for, as I understand the concept.  I
> think Emacs supports those already.  Features enable or disable stuff
> like certain ligatures.  The user can activate or deactivate features in
> the font specification to choose which ligatures to use.

We have no such mechanism in Emacs.  We pass a sequence of characters
to the shaping engine, and expect it to DTRT vis-a-vis the font and
return to use the font glyphs to display that sequence.  The only
control Lisp programs and users have here is via
composition-function-table, which determines when we use the shaping
engine for displaying one or more characters.

Re: Ligature support

[Benjamin Riefenstahl]

>> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
>> That is what font "features" are for, as I understand the concept.  I
>> think Emacs supports those already.  Features enable or disable stuff
>> like certain ligatures.  The user can activate or deactivate features in
>> the font specification to choose which ligatures to use.

Eli Zaretskii writes:
> We have no such mechanism in Emacs.  We pass a sequence of characters
> to the shaping engine, and expect it to DTRT vis-a-vis the font and
> return to use the font glyphs to display that sequence.  The only
> control Lisp programs and users have here is via
> composition-function-table, which determines when we use the shaping
> engine for displaying one or more characters.

Right, I did some experiments and I could not make it work in Emacs.

To elaborate, the idea here is to take ideas and concepts from

  https://www.freedesktop.org/software/fontconfig/fontconfig-devel/x19.html
  https://wiki.archlinux.org/title/Font_configuration/Examples#Disable_ligatures_for_monospaced_fonts
  
The way that this works at the basic level can be demonstrated using the
Nimbus font with the difference in output for

  $ hb-view .../NimbusMonoPS-Regular.otf ffi
  $ hb-view .../NimbusMonoPS-Regular.otf --features=liga=off ffi

(On Debian this font is in the package "fonts-urw-base35")

Maybe in Emacs the user could specifiy the features s/he wants like

  (set-fontset-font nil 'ascii "Nimbus-16:fontfeatures=liga=off" (selected-frame))

Although, as discussed, we probably want different configurations for
program code and normal prose, IOW in different buffers.

I have not read the code, but Emacs probably already passes these
parameters to Fontconfig which finds the font and than the feature list
is ignored, given that it does not make a difference for the question
which font file to use.  I'm guessing the features are sitting somewhere
in the data returned by Fontconfig.  They should be present there,
because that they can also be configured in Fontconfig's files, so it
can't be the job of the application itself to parse them.

Or maybe I could just not get it to work, because I had not configured
composition-function-table right (I tried:-().

Re: Ligature support

[Eli Zaretskii]

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Nov 2021 18:03:26 +0100
> 
> I have not read the code, but Emacs probably already passes these
> parameters to Fontconfig which finds the font and than the feature list
> is ignored, given that it does not make a difference for the question
> which font file to use.  I'm guessing the features are sitting somewhere
> in the data returned by Fontconfig.  They should be present there,
> because that they can also be configured in Fontconfig's files, so it
> can't be the job of the application itself to parse them.

That's not entirely accurate: we do ask for fonts with certain
features, but only request features related to language and script
support.

Re: Entering emojis

[Gregory Heytings]

>
> The layout in the vertical direction is completely incorrect in all of 
> the images you show.
>

Note that the image you should look at is the third one, named 
aegyptus.png.  I provided the other ones for comparison.

Re: Entering emojis

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 19:23:20 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: mattiase@acm.org, emacs-devel@gnu.org, schwab@linux-m68k.org, 
>     stefankangas@gmail.com, raman@google.com
> 
> Note that the image you should look at is the third one, named 
> aegyptus.png.  I provided the other ones for comparison.

I know.  But none of them shows the name of the Egyptian language
correctly.

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 446 bytes --]

P.S.  Emojis as a solution looking for a problem:
When they became popular in places like Japan in late 90's /
early-2000's they did solve a problem, namely the need to send  "more
than plain text" in a very band-width limited environment; so sending
characters that could map to glyphs that were "picturesque" did add
value of sorts.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Wed, 27 Oct 2021 10:27:32 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> It would be useful to have a 'literate-mode' where they would be replaced with the empty string, a space, U+FFFD, a suitable hex escape, or something else. A compromise would be to show them as monochrome glyphs that respect the typographic metrics of the surrounding text.

You can have this if you customize your fontset to use a font for the
'emoji' script that has no glyphs for the Emoji codepoints.

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 15:09:13 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Mattias Engdegård <mattiase@acm.org>
    >> Date: Wed, 27 Oct 2021 10:27:32 +0200
    >> Cc: Emacs developers <emacs-devel@gnu.org>
    >> 
    >> It would be useful to have a 'literate-mode' where they would be
    >> replaced with the empty string, a space, U+FFFD, a suitable hex
    >> escape, or something else. A compromise would be to show them as
    >> monochrome glyphs that respect the typographic metrics of the
    >> surrounding text.

    Eli> You can have this if you customize your fontset to use a font for the
    Eli> 'emoji' script that has no glyphs for the Emoji codepoints.

And if you use 'Symbola' or similar you'll get monochrome glyphs.

Robert
-- 

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 26 Oct 2021 23:12:12 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Gregory Heytings <gregory@heytings.org>, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > (Unrelatedly, almost no smileys display on my macOS machine, but I
> > believe this is a known issue.)
> 
> Why exactly are emojis broken on macOS?

What exactly is broken?  You say "almost no smileys", but don't
provide any details: which ones work and which ones don't?  How about
submitting a bug report with all the details, so we could investigate
and try to fix that before the release?

> I'm surprised to learn that I have to say:
> 
>     (set-fontset-font t 'emoji
>                       '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> Is that correct?

Not sure, because I think other users of macOS said they get this
automatically.  Is this in "emacs -Q"?  (Once again, a detailed bug
report is sorely needed.)

> Why can't we just do that automatically?

I don't know why the macOS font backend doesn't find this font in your
case, it's something that should be investigated.

If you are asking why we don't put the above in Emacs explicitly, then
it's for the same reason we don't have such an explicit setting with
Segoe UI Emoji for MS-Windows, nor mention any other proprietary fonts
in our sources: we don't want to promote those proprietary fonts.

> I think that most users will just assume that emojis are broken and move
> on.  Maybe some will soldier on and find the fix.  I very nearly didn't.

I hope we will indeed be able to find the reason(s) and fix whatever
needs fixing, if we can.  Please help by providing the details.

> The NEWS entry on this is also not very useful, as it gives:
> 
>     (set-fontset-font t 'emoji
>                       '("My New Emoji Font" . "iso10646-1") nil 'prepend)
> 
> But there is of course no such font "My New Emoji Font".

Of course.  Why should you think this fake font name is a name of a
real font?

> Also, and I'm sorry in advance, but can we please change the text to
> something understandable?
> 
>      ** New character script 'emoji' has been created.
>      Various blocks of codepoints have been split out of the 'symbol'
>      script into their own 'emoji' script to allow easier specification of
>      their treatment.  Which codepoints are treated as emoji is derived
>      from the Unicode specifications.
> 
> Uh, what?  I have no idea what practical difference any of the words in
> the above would make.  Blocks of codepoints?  What is a 'symbol' script?
> Am I a horrible programmer and human being if I don't know what any of
> this means?
> 
> Maybe I have a general concept of some of these things, but not enough
> to know how this affects me as a user.  Did emojis already work, but
> they are now just better?  Are emojis an entirely new feature in Emacs
> 28?  Do they work even a little bit in Emacs 27?  Do they work, but are
> half-broken?  Do they work perfectly?
> 
> The entry after start talking about Zero Width Joiners and like,
> seriously people, can we just *really* dumb this down to something like:
> 
>     *** Emojis now display nicely under X Windows and macOS.
> 
> Or like:
> 
>     *** Emacs can display 50 new Emojis and also with skin tones.
> 
> Or whatever the above is supposed to mean.  You can then of course go
> into whatever low-level technical details you want, but please at least
> explain first in casual terms what this even is.
> 
> Unless of course this is only of interest to actual Unicode experts, in
> which case ignore me and carry on.

Frankly, I think the level of sarcasm here is a bit overboard.  The
issue is indeed highly technical, and if one pretends not to know what
Emoji are, nor how their support before Emacs 28 was lacking, and if
you never tried to customize your fontsets to support Emoji (or any
other character) better, then yes, it's easy to conclude that the
above is useless.

Anyway, I tried to clarify that entry as best I could, I hope it's an
improvement.

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 14:48:37 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> I'm surprised to learn that I have to say:
    >> 
    >> (set-fontset-font t 'emoji
    >> '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
    >> 
    >> Is that correct?

    Eli> Not sure, because I think other users of macOS said they get this
    Eli> automatically.  Is this in "emacs -Q"?  (Once again, a detailed bug
    Eli> report is sorely needed.)

They do? When using one of the other ports, maybe.

    >> Why can't we just do that automatically?

    Eli> I don't know why the macOS font backend doesn't find this font in your
    Eli> case, it's something that should be investigated.

I have a vague memory that thereʼs code in the macOS font backend that
will only allow color fonts if explicitly requested, and there being a
good reason for it. Iʼll trawl through my archives.

    >> Also, and I'm sorry in advance, but can we please change the text to
    >> something understandable?

I understand it perfectly :-)

    >> ** New character script 'emoji' has been created.
    >> Various blocks of codepoints have been split out of the 'symbol'
    >> script into their own 'emoji' script to allow easier specification of
    >> their treatment.  Which codepoints are treated as emoji is derived
    >> from the Unicode specifications.
    >> 
    >> Uh, what?  I have no idea what practical difference any of the words in
    >> the above would make.  Blocks of codepoints?  What is a 'symbol' script?
    >> Am I a horrible programmer and human being if I don't know what any of
    >> this means?

You'll be telling me next you've not read tr-51 even once.

    Eli> Frankly, I think the level of sarcasm here is a bit overboard.  The
    Eli> issue is indeed highly technical, and if one pretends not to know what
    Eli> Emoji are, nor how their support before Emacs 28 was lacking, and if
    Eli> you never tried to customize your fontsets to support Emoji (or any
    Eli> other character) better, then yes, it's easy to conclude that the
    Eli> above is useless.

This is why Eli is the maintainer: he can express these things clearly
and without rancour (I tend to fail at the latter portion, and end up
having to delete what Iʼve written, which I did several times in this
reply)

    Eli> Anyway, I tried to clarify that entry as best I could, I hope it's an
    Eli> improvement.

LGTM

If I were to nitpick, Iʼd drop this hunk:

+where "My New Emoji Font" should be replaced by the actual name of the
+font you want to use.

because to me thatʼs obvious, but itʼs a minor point.

Robert
-- 

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 14:36:44 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Wed, 27 Oct 2021 14:48:37 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >>> Why can't we just do that automatically?

    Eli> I don't know why the macOS font backend doesn't find this font in your
    Eli> case, it's something that should be investigated.

    Robert> I have a vague memory that thereʼs code in the macOS font backend that
    Robert> will only allow color fonts if explicitly requested, and there being a
    Robert> good reason for it. Iʼll trawl through my archives.

Yes, src/macfont.m:2419

          /* Don't use a color bitmap font unless its family is
             explicitly specified.  */
          if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
            continue;

which I asked Yamamoto-san about last year, and the response was:
<https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00520.html>

which does a good job of explaining *what* was done for the Mac port
(which is a separate port) but not why.

Anyway, if I delete the above, Apple Color Emoji is used for emoji.

Iʼd push it to emacs-28, but I have no idea of the repercussions.

Robert
-- 

Re: Entering emojis

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Iʼd push it to emacs-28, but I have no idea of the repercussions.

Try pushing it to master, at least -- if nobody reports problems, we can
consider cherry-picking it for emacs-28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 15:09:44 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> Iʼd push it to emacs-28, but I have no idea of the repercussions.

    Lars> Try pushing it to master, at least -- if nobody reports problems, we can
    Lars> consider cherry-picking it for emacs-28.

Sure. Eli?

commit d3c78393b1
Author:     Robert Pluim <rpluim@gmail.com>
AuthorDate: Wed Oct 27 15:24:48 2021 +0200
Commit:     Robert Pluim <rpluim@gmail.com>
CommitDate: Wed Oct 27 15:24:48 2021 +0200

    Don't skip color fonts on macOS
    
    This allows emoji to be displayed using a Color Emoji font without
    the user having to use 'set-fontset-font'.
    
    * src/macfont.m (macfont_list): Don't skip color fonts.

diff --git a/src/macfont.m b/src/macfont.m
index d86f09f485..4291375f3c 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -2414,11 +2414,6 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
                   != (spacing >= FONT_SPACING_MONO)))
             continue;
 
-          /* Don't use a color bitmap font unless its family is
-             explicitly specified.  */
-          if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
-            continue;
-
           if (j > 0
               && !macfont_supports_charset_and_languages_p (desc, charset,
                                                             chars, languages))

Re: Entering emojis

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  gregory@heytings.org,  Stefan Kangas
>  <stefankangas@gmail.com>,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 15:26:29 +0200
> 
> >>>>> On Wed, 27 Oct 2021 15:09:44 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:
> 
>     Lars> Robert Pluim <rpluim@gmail.com> writes:
>     >> Iʼd push it to emacs-28, but I have no idea of the repercussions.
> 
>     Lars> Try pushing it to master, at least -- if nobody reports problems, we can
>     Lars> consider cherry-picking it for emacs-28.
> 
> Sure. Eli?

I'd prefer to have a fix on emacs-28, and asked a question with that
in mind.

Re: Entering emojis

[chad]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On Wed, Oct 27, 2021 at 6:53 AM Robert Pluim <rpluim@gmail.com> wrote:

>
> commit d3c78393b1
> Author:     Robert Pluim <rpluim@gmail.com>
> AuthorDate: Wed Oct 27 15:24:48 2021 +0200
> Commit:     Robert Pluim <rpluim@gmail.com>
> CommitDate: Wed Oct 27 15:24:48 2021 +0200
>
>     Don't skip color fonts on macOS
>
>     This allows emoji to be displayed using a Color Emoji font without
>     the user having to use 'set-fontset-font'.
>
>     * src/macfont.m (macfont_list): Don't skip color fonts.
>
> diff --git a/src/macfont.m b/src/macfont.m
> index d86f09f485..4291375f3c 100644
> --- a/src/macfont.m
> +++ b/src/macfont.m
> @@ -2414,11 +2414,6 @@ So we use
> CTFontDescriptorCreateMatchingFontDescriptor (no
>                    != (spacing >= FONT_SPACING_MONO)))
>              continue;
>
> -          /* Don't use a color bitmap font unless its family is
> -             explicitly specified.  */
> -          if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
> -            continue;
> -
>            if (j > 0
>                && !macfont_supports_charset_and_languages_p (desc, charset,
>                                                              chars,
> languages))
>
>
(Still catching up from several weeks away.)
If memory serves, this ability was added to the ns and mac ports back
before it was functional in GNUstep or typical GNU/Linux distributions, and
so someone (another memory says "RMS") asked that it not be enabled by
default, to avoid making the macOS version of Emacs more featureful than
the free versions.
Since that's no longer true, if my memory is correct, this change should be
a nice addition.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 2442 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Mon, 8 Nov 2021 11:52:24 -0800
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Gregory Heytings <gregory@heytings.org>,
>  Stefan Kangas <stefankangas@gmail.com>, Eli Zaretskii <eliz@gnu.org>,
>  EMACS development team <emacs-devel@gnu.org>
> 
>  -          /* Don't use a color bitmap font unless its family is
>  -             explicitly specified.  */
>  -          if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
>  -            continue;
>  -
>             if (j > 0
>                 && !macfont_supports_charset_and_languages_p (desc, charset,
>                                                               chars, languages))
> 
> (Still catching up from several weeks away.) 
> If memory serves, this ability was added to the ns and mac ports back before it was functional in GNUstep
> or typical GNU/Linux distributions, and so someone (another memory says "RMS") asked that it not be
> enabled by default, to avoid making the macOS version of Emacs more featureful than the free versions. 
> Since that's no longer true, if my memory is correct, this change should be a nice addition.

I think you are confusing two different issues.  but in any case, we
already installed a safer variant of the above.

Re: Entering emojis

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  gregory@heytings.org,  Stefan Kangas
>  <stefankangas@gmail.com>,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 15:07:49 +0200
> 
>           /* Don't use a color bitmap font unless its family is
>              explicitly specified.  */
>           if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
>             continue;
> 
> which I asked Yamamoto-san about last year, and the response was:
> <https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00520.html>
> 
> which does a good job of explaining *what* was done for the Mac port
> (which is a separate port) but not why.
> 
> Anyway, if I delete the above, Apple Color Emoji is used for emoji.

Do we have the 'emoji' script in AREF (spec, FONT_EXTRA_INDEX) in that
function, when Emacs is looking for a font to display Emoji?  If so,
we could refrain from the above condition only when a font for Emoji
is being sought, and that could be safe enough for the release branch.

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 16:34:58 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> Anyway, if I delete the above, Apple Color Emoji is used for emoji.

    Eli> Do we have the 'emoji' script in AREF (spec, FONT_EXTRA_INDEX) in that
    Eli> function, when Emacs is looking for a font to display Emoji?  If so,
    Eli> we could refrain from the above condition only when a font for Emoji
    Eli> is being sought, and that could be safe enough for the release branch.

Yes. This seems to work so far.

diff --git a/src/macfont.m b/src/macfont.m
index d86f09f485..78ed5d53f3 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -2415,8 +2415,12 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
             continue;
 
           /* Don't use a color bitmap font unless its family is
-             explicitly specified.  */
-          if ((sym_traits & kCTFontTraitColorGlyphs) && NILP (family))
+             explicitly specified or we're looking for a font for
+             emoji.  */
+          if ((sym_traits & kCTFontTraitColorGlyphs)
+              && NILP (family)
+              && !EQ (CDR_SAFE (assq_no_quit (QCscript, AREF (spec, FONT_EXTRA_INDEX))),
+                      Qemoji))
             continue;
 
           if (j > 0

Robert
-- 

Re: Entering emojis

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  gregory@heytings.org,  stefankangas@gmail.com,
>   emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 17:12:16 +0200
> 
> >>>>> On Wed, 27 Oct 2021 16:34:58 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> Anyway, if I delete the above, Apple Color Emoji is used for emoji.
> 
>     Eli> Do we have the 'emoji' script in AREF (spec, FONT_EXTRA_INDEX) in that
>     Eli> function, when Emacs is looking for a font to display Emoji?  If so,
>     Eli> we could refrain from the above condition only when a font for Emoji
>     Eli> is being sought, and that could be safe enough for the release branch.
> 
> Yes. This seems to work so far.

Then I'm okay with installing this on the release branch.

Thanks.

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 19:08:47 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: larsi@gnus.org,  gregory@heytings.org,  stefankangas@gmail.com,
    >> emacs-devel@gnu.org
    >> Date: Wed, 27 Oct 2021 17:12:16 +0200
    >> 
    >> >>>>> On Wed, 27 Oct 2021 16:34:58 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    >> >> Anyway, if I delete the above, Apple Color Emoji is used for emoji.
    >> 
    Eli> Do we have the 'emoji' script in AREF (spec, FONT_EXTRA_INDEX) in that
    Eli> function, when Emacs is looking for a font to display Emoji?  If so,
    Eli> we could refrain from the above condition only when a font for Emoji
    Eli> is being sought, and that could be safe enough for the release branch.
    >> 
    >> Yes. This seems to work so far.

    Eli> Then I'm okay with installing this on the release branch.

Done as e3171e7e86

Robert
-- 

Re: Entering emojis

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  larsi@gnus.org,
>   gregory@heytings.org,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 14:36:44 +0200
> 
>     Eli> Anyway, I tried to clarify that entry as best I could, I hope it's an
>     Eli> improvement.
> 
> LGTM
> 
> If I were to nitpick, Iʼd drop this hunk:
> 
> +where "My New Emoji Font" should be replaced by the actual name of the
> +font you want to use.
> 
> because to me thatʼs obvious, but itʼs a minor point.

I can add "(although this should be obvious)" there.

Re: Entering emojis

[Robert Pluim]

>>>>> On Wed, 27 Oct 2021 16:22:22 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Stefan Kangas <stefankangas@gmail.com>,  larsi@gnus.org,
    >> gregory@heytings.org,  emacs-devel@gnu.org
    >> Date: Wed, 27 Oct 2021 14:36:44 +0200
    >> 
    Eli> Anyway, I tried to clarify that entry as best I could, I hope it's an
    Eli> improvement.
    >> 
    >> LGTM
    >> 
    >> If I were to nitpick, Iʼd drop this hunk:
    >> 
    >> +where "My New Emoji Font" should be replaced by the actual name of the
    >> +font you want to use.
    >> 
    >> because to me thatʼs obvious, but itʼs a minor point.

    Eli> I can add "(although this should be obvious)" there.

Now who's being sarcastic? ;-)

Robert
-- 

Re: Entering emojis

[Stefan Kangas]

Robert Pluim <rpluim@gmail.com> writes:

>     >> If I were to nitpick, Iʼd drop this hunk:
>     >>
>     >> +where "My New Emoji Font" should be replaced by the actual name of the
>     >> +font you want to use.
>     >>
>     >> because to me thatʼs obvious, but itʼs a minor point.
>
>     Eli> I can add "(although this should be obvious)" there.

I think we can just drop it.

My point was not "I don't understand that this is an example font name"
but "why not just put the font I need to unbreak this so I don't have to
specifically look around for it".

If we can't name the font to use for political reasons, the example is
as good as it gets and needs no further explanation.

> Now who's being sarcastic? ;-)

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 27 Oct 2021 14:06:50 +0000
> Cc: larsi@gnus.org, gregory@heytings.org, emacs-devel@gnu.org
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> >     >> If I were to nitpick, Iʼd drop this hunk:
> >     >>
> >     >> +where "My New Emoji Font" should be replaced by the actual name of the
> >     >> +font you want to use.
> >     >>
> >     >> because to me thatʼs obvious, but itʼs a minor point.
> >
> >     Eli> I can add "(although this should be obvious)" there.
> 
> I think we can just drop it.

Done.

> My point was not "I don't understand that this is an example font name"
> but "why not just put the font I need to unbreak this so I don't have to
> specifically look around for it".

Look at the latest text, it no longer makes it sound like this
fictitious font is needed for Emoji to display correctly.  Or at least
that's what I wanted the text to imply.

Re: Entering emojis

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> How about submitting a bug report with all the details, so we could
> investigate and try to fix that before the release?

Will do.

> Frankly, I think the level of sarcasm here is a bit overboard.

Sarcasm was not my intention.  I was trying to express my slight
frustration, this is true, but also how this could be perceived if you
are not familiar with the topic.  In fact, I was convinced we had a good
and exciting feature on our hands, and was eager to help us demonstrate
that in a clear way to our users.

However, it seems like I communicated other things in addition to the
ones I was trying to communicate.  Communicating complex sentiments
in writing is hard, but obviously I didn't try hard enough here.

So I apologize, especially to Robert and you, if it was a bit overboard;
in particular if my reply stirred up any negative emotions.  I don't
want to contribute to a negative climate if I can avoid it.

> Anyway, I tried to clarify that entry as best I could, I hope it's an
> improvement.

Thank you!  The new text is very good.

Re: Entering emojis

[Michael Albinus]

Stefan Kangas <stefankangas@gmail.com> writes:

> However, it seems like I communicated other things in addition to the
> ones I was trying to communicate.  Communicating complex sentiments
> in writing is hard, but obviously I didn't try hard enough here.

You shall use emojis :-)

> Thank you!  The new text is very good.

SCNR. Best regards, Michael.

Re: Entering emojis

[Jonas Bernoulli]

Stefan Kangas <stefankangas@gmail.com> writes:

> I tested it, and I like it.  Transient seems like a good choice for
> something this.  Another idea is to have a grid where you can pick one
> using arrow keys and RET, which might feel more familiar to some users
> (I'm not sure if transient supports this).  Or perhaps a combination
> of the two?

Using transient would work here but currently enabling such navigation
using (setq transient-enable-popup-navigation t) affects all transients.
A prefix slot that controls this behavior could be added of course.

Also the transient buffer's content is recreated after each command
(which isn't noticeable when using magit's transients but here it might)
and navigation also has some limitations.

So all in all I would stick with what you have implemented already.

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Using transient would work here but currently enabling such navigation
> using (setq transient-enable-popup-navigation t) affects all transients.
> A prefix slot that controls this behavior could be added of course.

Hm...  doesn't seem to work here -- perhaps it's because how these
transients are constructed.  That is, <down> takes me to the first
choice, but then nothing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
>> Using transient would work here but currently enabling such navigation
>> using (setq transient-enable-popup-navigation t) affects all transients.
>> A prefix slot that controls this behavior could be added of course.
>
> Hm...  doesn't seem to work here -- perhaps it's because how these
> transients are constructed.  That is, <down> takes me to the first
> choice, but then nothing.

Somewhere else I mentioned that the top group heading appears
out of order.

I've figured that out now; this was caused by `transient--pixel-width'.
Moving save-window-excursion outside of `with-temp-buffer fixed' it.  I
have also dropped the `set-window-dedicated-p', which doesn't seem to
make a difference either way.

This change might very well fix the above issue as well.  T.V and I have
also noted navigational issues (though all three of us noticed different
ones).  The one I noticed is fixed by this, I expect the others are too.

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas Bernoulli <jonas@bernoul.li> writes:

> I've figured that out now; this was caused by `transient--pixel-width'.
> Moving save-window-excursion outside of `with-temp-buffer fixed' it.  I
> have also dropped the `set-window-dedicated-p', which doesn't seem to
> make a difference either way.

It makes a difference if a user has made a window dedicated (since we're
switching windows, and that may not be allowed if the user has done that).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1656 bytes --]

Jonas Bernoulli <jonas@bernoul.li> writes:


The other "interesting bug" that becomes apparent with Emacspeak:

Background: Emacspeak attaches a minibuffer setup hook that plays a
sound when the minibuffer opens.

It appears that each operation via transient opens the minibuffer twice
-- since I hear the sound twice, rather  than just once as one would
expect.

Might be worth debugging since it likely has a performance hit when
navigating large trees.
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Jonas Bernoulli <jonas@bernoul.li> writes:
>>
>>> Using transient would work here but currently enabling such navigation
>>> using (setq transient-enable-popup-navigation t) affects all transients.
>>> A prefix slot that controls this behavior could be added of course.
>>
>> Hm...  doesn't seem to work here -- perhaps it's because how these
>> transients are constructed.  That is, <down> takes me to the first
>> choice, but then nothing.
>
> Somewhere else I mentioned that the top group heading appears
> out of order.
>
> I've figured that out now; this was caused by `transient--pixel-width'.
> Moving save-window-excursion outside of `with-temp-buffer fixed' it.  I
> have also dropped the `set-window-dedicated-p', which doesn't seem to
> make a difference either way.
>
> This change might very well fix the above issue as well.  T.V and I have
> also noted navigational issues (though all three of us noticed different
> ones).  The one I noticed is fixed by this, I expect the others are too.
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Gregory Heytings]

>
> Do you know an example input method that has a lot of variants?
>

I think what Eli has in mind is more something like chinese-py.

Re: Entering emojis

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> I think what Eli has in mind is more something like chinese-py.

Ah, right.

Looking at https://unicode.org/emoji/charts/emoji-list.html, it divides
the emojis up into groups like "Food & Drink", which seems very useful,
and it says that this data is from the CLDR.  It doesn't seem that we've
imported that data into Emacs?  At least grepping around in
admin/unidata/ doesn't seem to give any promising matches.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Looking at https://unicode.org/emoji/charts/emoji-list.html, it divides
> the emojis up into groups like "Food & Drink", which seems very useful,
> and it says that this data is from the CLDR.  It doesn't seem that we've
> imported that data into Emacs?  At least grepping around in
> admin/unidata/ doesn't seem to give any promising matches.

Looks like it's in a file we haven't imported:
common/annotations/en.xml from the CLDR repo.  It has entries like

		<annotation cp="🥝">food | fruit | kiwi</annotation>
		<annotation cp="🥝" type="tts">kiwi fruit</annotation>
		<annotation cp="🍅">fruit | tomato | vegetable</annotation>
		<annotation cp="🍅" type="tts">tomato</annotation>
		<annotation cp="🫒">food | olive</annotation>
		<annotation cp="🫒" type="tts">olive</annotation>

which seems promising, if somewhat confusing.  I mean, one of these
three things isn't like the other: "food | fruit | kiwi".  So I guess it
can't be used to create an automatic segmentation, but after creating
some categories manually, it can be used to look up stuff when
populating the UI.  Probably.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Probably.

No, I found the proper file to use:

[☘🌱-🌵🌾-🍃]                  ;	Animals & Nature	;	plant-other
[🍅🍇-🍓🥝🥥🥭]                ;	Food & Drink	;	food-fruit
[🌰🌶🌽🍄🍆🥑🥒🥔🥕🥜🥦]       ;	Food & Drink	;	food-vegetable
[🥬]                           ;	Food & Drink	;	food-vegetable
[🌭-🌯🍔-🍗🍞🍟🍲🍳🍿🥐🥓]     ;	Food & Drink	;	food-prepared
[🥖-🥚🥞🥣🥨-🥫🥯🧀🧂]         ;	Food & Drink	;	food-prepared
[🍘-🍝🍠-🍥🍱🥟-🥡🥮]          ;	Food & Drink	;	food-asian
[🍦-🍰🎂🥧🧁]                  ;	Food & Drink	;	food-sweet

It's the common/properties/labels.txt file from the CLDR zip file.

So we could have an interface that goes Food & Drink->food->vegetable->
and then all the vegetables.  With completion at every point showing the
matching emojis...

These don't have any variants, though, but I think the same goes for the
ones with variants.  Possibly.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Matthias Meulien]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

If we use an input method, will I have to switch between input methods when
writing french prose? Is it possible to have multiple input methods
activated at the same time?

What I am used to, looks more like abbrevs:

When using Gitlab flavored markdown in a web browser, emojis are entered by
pressing : (colon) then the name of the emoji as displayed on
https://www.webfx.com/tools/emoji-cheat-sheet/, and a popup shows matching
emojis when the user enters chars.

The same method applies to github, slack, etc.

Le lun. 25 oct. 2021 à 20:46, Eli Zaretskii <eliz@gnu.org> a écrit :

> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Date: Mon, 25 Oct 2021 20:32:24 +0200
> >
> > Emacs can display such nice glyphs as 🏴‍☠️, 🧝🏾‍♂️ and  👨🏾‍🦰 now,
> but has anybody
> > made a package to compose them?  Or rather -- to navigate the Emoji
> > Space Of Possibilities?
> >
> > I don't really have any idea of what that would look like in an Emacs
> > context, though.  Perhaps something with transient.el?
>
> Rather an input method, I'd say.
>
>

[-- Attachment #2: Type: text/html, Size: 1722 bytes --]

Re: Entering emojis

[Eli Zaretskii]

> From: Matthias Meulien <orontee@gmail.com>
> Date: Mon, 25 Oct 2021 22:36:38 +0200
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel <emacs-devel@gnu.org>
> 
> If we use an input method, will I have to switch between input methods when writing french prose? Is it
> possible to have multiple input methods activated at the same time?

Emacs 28 has a new feature: transient input method.  Look it up in
NEWS.

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 860 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:
The following is a start and is what I use, it doesn't  do anything for
composing emojis though.


I use
C-; u runs the command list-unicode-display
list-unicode-display is an autoloaded interactive Lisp function in
¡®list-unicode-display.el¡¯.

It is bound to C-; u, C-x @ h u.

(list-unicode-display &optional REGEXP)

Display a list of unicode characters with names matching REGEXP.
If no regexp is supplied, all characters are shown.  This takes
some time.

> made a package to compose them?  Or rather -- to navigate the Emoji
> Space Of Possibilities?
>
> I don't really have any idea of what that would look like in an Emacs
> context, though.  Perhaps something with transient.el?

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Lars Ingebrigtsen]

This is now totally and utterly feature complete, so give it a whirl if
you're into emojis.  🤑  The branch is scratch/emoji, and the commands
are `M-x emoji-insert', `C-u M-x emoji-insert' and `M-x emoji-list'.

(Cue discussions about whether these should be called `list-emoji' and
`insert-emoji'.)

I've only tested this stuff on Debian/bookworm, where emojis work out of
the box.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> (Cue discussions about whether these should be called `list-emoji' and
> `insert-emoji'.)

How about having the cake and eating it too?

    (defalias 'insert-emoji #'emoji-insert)

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> How about having the cake and eating it too?
>
>     (defalias 'insert-emoji #'emoji-insert)

Sure -- aliases for normal functions are a pain (because when reading
code, the reader will be puzzled when encountering the variations), but
these are user commands, so it's probably fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Sure -- aliases for normal functions are a pain (because when reading
> code, the reader will be puzzled when encountering the variations), but
> these are user commands, so it's probably fine.

Yeah, I think there is some logic to it in the particular cases
"list-*", "insert-*", "describe-*" and maybe a handful of others.

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 26 Oct 2021 23:20:53 +0200
> 
> This is now totally and utterly feature complete, so give it a whirl if
> you're into emojis.  🤑  The branch is scratch/emoji, and the commands
> are `M-x emoji-insert', `C-u M-x emoji-insert' and `M-x emoji-list'.

Would it make sense to install this on the release branch?  Since it's
a new orthogonal feature, it cannot possibly break anything else, just
be broken by itself, right?

(And why is the file in lisp/play/?  Typing Emoji is nowadays not a
game at all.  Can we move this to lisp/textmodes/?)

Thanks.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is now totally and utterly feature complete, so give it a whirl if
>> you're into emojis.  🤑  The branch is scratch/emoji, and the commands
>> are `M-x emoji-insert', `C-u M-x emoji-insert' and `M-x emoji-list'.
>
> Would it make sense to install this on the release branch?  Since it's
> a new orthogonal feature, it cannot possibly break anything else, just
> be broken by itself, right?

It requires changes to transient.el to work, unfortunately.

> (And why is the file in lisp/play/?  Typing Emoji is nowadays not a
> game at all.  Can we move this to lisp/textmodes/?)

I thought we put all the fun stuff in lisp/play/, whether it's a game or
not.  Like handwrite.el and morse.el.  And it's not a mode, so textmodes
seemed like an odd place to put it?  And it didn't seem to fit in any of
the other categories, either.

But I have no opinion here, really -- putting it anywhere's fine by me.

These commands need to have a keystroke, though.  I think people who use
them will be using them a lot, so there should be a default keymap
placement for them.  There's three commands now -- emoji-insert
(graphical/category choosing), emoji-search (textual based on names) and
emoli-list (one big buffer of emojis).

So...  er...  off the top of my head...

C-x 8 e e
C-x 8 e s
C-x 8 e l

`C-x 8' because that's where `C-x 8 RET' is.  Or...  is there a more
likely place for insertion commands?  Uhm...  `C-x g'.  🤔💥

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 14:28:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> This is now totally and utterly feature complete, so give it a whirl if
> >> you're into emojis.  🤑  The branch is scratch/emoji, and the commands
> >> are `M-x emoji-insert', `C-u M-x emoji-insert' and `M-x emoji-list'.
> >
> > Would it make sense to install this on the release branch?  Since it's
> > a new orthogonal feature, it cannot possibly break anything else, just
> > be broken by itself, right?
> 
> It requires changes to transient.el to work, unfortunately.

I've seen them, but they, too, seem orthogonal, or somewhat safe, or
both?

Jonas, what is your opinion about the safety of the changes in
transient.el that Lars added recently?

> These commands need to have a keystroke, though.  I think people who use
> them will be using them a lot, so there should be a default keymap
> placement for them.  There's three commands now -- emoji-insert
> (graphical/category choosing), emoji-search (textual based on names) and
> emoli-list (one big buffer of emojis).
> 
> So...  er...  off the top of my head...
> 
> C-x 8 e e
> C-x 8 e s
> C-x 8 e l
> 
> `C-x 8' because that's where `C-x 8 RET' is.  Or...  is there a more
> likely place for insertion commands?  Uhm...  `C-x g'.  🤔💥

Why not just "C-x 8 e"?  It's available.

Re: Entering emojis

[Jonas Bernoulli]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Would it make sense to install this on the release branch?  Since it's
>> > a new orthogonal feature, it cannot possibly break anything else, just
>> > be broken by itself, right?
>> 
>> It requires changes to transient.el to work, unfortunately.
>
> I've seen them, but they, too, seem orthogonal, or somewhat safe, or
> both?
>
> Jonas, what is your opinion about the safety of the changes in
> transient.el that Lars added recently?

Very safe.  If there's a bug it should only affect new transients that
set the new `variable-pitch' slot.

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I thought we put all the fun stuff in lisp/play/, whether it's a game or
> not.  Like handwrite.el and morse.el.  And it's not a mode, so textmodes
> seemed like an odd place to put it?  And it didn't seem to fit in any of
> the other categories, either.

Whatever we may think of Emojis, many people find them a very useful
tool for communication.  So I agree with Eli that they belong in
"textmodes" rather than "play".  Or perhaps "mail" or "net"?  But also
consider that we could just put them in the top-level "lisp" directory
together with a lot of other random stuff.

BTW, morse.el is an outlier in the "play" category, IMO.  I don't know
why it was put there when rot13.el was not; rot13.el is a more obvious
candidate for "play" as it can really only be used for joking around,
while morse code is a serious and useful tool for communication.

(I'd be in favour of "git mv lisp/rot13.el lisp/play".)

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Whatever we may think of Emojis, many people find them a very useful
> tool for communication.  So I agree with Eli that they belong in
> "textmodes" rather than "play".  Or perhaps "mail" or "net"?  But also
> consider that we could just put them in the top-level "lisp" directory
> together with a lot of other random stuff.

I'd rather not put more stuff at the top level if I can help it...  So
perhaps "textmodes" is the most likely thing?  Even if it doesn't really
have anything in particular to do with text-mode derivatives, really.

> BTW, morse.el is an outlier in the "play" category, IMO.  I don't know
> why it was put there when rot13.el was not; rot13.el is a more obvious
> candidate for "play" as it can really only be used for joking around,
> while morse code is a serious and useful tool for communication.
>
> (I'd be in favour of "git mv lisp/rot13.el lisp/play".)

I'm in favour of not moving anything.  :-)  But otherwise I agree.

Anyway, nobody's come up with any alternative ideas for key bindings, so
I'm tentatively putting these commands on the `C-x 8 e' submap.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Move shorthands.el to lisp/emacs-lisp/?

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'd rather not put more stuff at the top level if I can help it...

BTW, shouldn't lisp/shorthands.el be in lisp/emacs-lisp/shorthands.el?

Re: Move shorthands.el to lisp/emacs-lisp/?

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> I'd rather not put more stuff at the top level if I can help it...
>
> BTW, shouldn't lisp/shorthands.el be in lisp/emacs-lisp/shorthands.el?

Yes, it should be.  And it hasn't been in emacs-28 that long, so moving
it sooner rather than later is a good idea, I think.

Eli, what do you think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Move shorthands.el to lisp/emacs-lisp/?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  João Távora
>  <joaotavora@gmail.com>
> Date: Thu, 28 Oct 2021 23:28:50 +0200
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Lars Ingebrigtsen <larsi@gnus.org> writes:
> >
> >> I'd rather not put more stuff at the top level if I can help it...
> >
> > BTW, shouldn't lisp/shorthands.el be in lisp/emacs-lisp/shorthands.el?
> 
> Yes, it should be.  And it hasn't been in emacs-28 that long, so moving
> it sooner rather than later is a good idea, I think.
> 
> Eli, what do you think?

I don't mind moving it now.  But please be sure to us "git mv", not
the "manual" method, so that at least some of the Git commands could
work across the renaming boundary.

Re: Move shorthands.el to lisp/emacs-lisp/?

[Yuri Khan]

On Fri, 29 Oct 2021 at 13:09, Eli Zaretskii <eliz@gnu.org> wrote:

> I don't mind moving it now.  But please be sure to us "git mv", not
> the "manual" method, so that at least some of the Git commands could
> work across the renaming boundary.

“git mv” is not magical though; it does not do anything that a manual
move + staging both the deletion and the addition of the moved file
does not do.

What matters for reliable move detection is that the file contents
should not be modified in the same commit as the move.

Re: Move shorthands.el to lisp/emacs-lisp/?

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Fri, 29 Oct 2021 16:45:19 +0700
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, João Távora <joaotavora@gmail.com>, 
> 	Stefan Kangas <stefankangas@gmail.com>, Emacs developers <emacs-devel@gnu.org>
> 
> On Fri, 29 Oct 2021 at 13:09, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I don't mind moving it now.  But please be sure to us "git mv", not
> > the "manual" method, so that at least some of the Git commands could
> > work across the renaming boundary.
> 
> “git mv” is not magical though; it does not do anything that a manual
> move + staging both the deletion and the addition of the moved file
> does not do.

You misunderstood what I meant by "manual".  I meant "git rm" followed
by "git add".

And if you want to say that the latter also does the same as "git mv",
then that's not my experience.

P.S. And I don't actually understand why you decided to chime in:
would something I said cause sub-optimal results?

Re: Move shorthands.el to lisp/emacs-lisp/?

[Yuri Khan]

On Fri, 29 Oct 2021 at 17:10, Eli Zaretskii <eliz@gnu.org> wrote:

> > “git mv” is not magical though; it does not do anything that a manual
> > move + staging both the deletion and the addition of the moved file
> > does not do.
>
> You misunderstood what I meant by "manual".  I meant "git rm" followed
> by "git add".
>
> And if you want to say that the latter also does the same as "git mv",
> then that's not my experience.

In my understanding and experience, that should have the same effect,
provided that the file content hash does not change.

> P.S. And I don't actually understand why you decided to chime in:
> would something I said cause sub-optimal results?

Just trying to dispel myths. There are a lot of those around Git, and
I find that suboptimal. Nothing personal and I’m sorry if you perceive
it as nitpicking on you.

People may acquire an anxiety like “if you meant to ‘git mv’ a file
but you just did an F6 Enter in a file manager, you have to move it
back and do it correctly”. That is not the case; one just needs to
stage both the deletion and the addition.

Alternatively, they may get an idea that “as long as you use ‘git mv’,
Git will track the rename”. This is also not the case; if one modifies
the moved file before committing the move, exact move detection will
not work and Git will fall back to heuristic-based move detection.

Re: Move shorthands.el to lisp/emacs-lisp/?

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Fri, 29 Oct 2021 17:31:07 +0700
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>, João Távora <joaotavora@gmail.com>, 
> 	Stefan Kangas <stefankangas@gmail.com>, Emacs developers <emacs-devel@gnu.org>
> 
> > P.S. And I don't actually understand why you decided to chime in:
> > would something I said cause sub-optimal results?
> 
> Just trying to dispel myths. There are a lot of those around Git, and
> I find that suboptimal. Nothing personal and I’m sorry if you perceive
> it as nitpicking on you.

I don't perceive it as nitpicking, I perceive it as raising the noise
level here, something we hardly need.

> People may acquire an anxiety like “if you meant to ‘git mv’ a file
> but you just did an F6 Enter in a file manager, you have to move it
> back and do it correctly”. That is not the case; one just needs to
> stage both the deletion and the addition.
> 
> Alternatively, they may get an idea that “as long as you use ‘git mv’,
> Git will track the rename”. This is also not the case; if one modifies
> the moved file before committing the move, exact move detection will
> not work and Git will fall back to heuristic-based move detection.

All I asked was to use "git mv", which AFAIK is _the_ recommended
method of renaming files under Git.  Any thoughts about Git-related
anxieties are neither intended nor implied.

This is not a Git mailing list, so we don't really need to discuss
every possible way of doing an operation with Git.  One that works is
enough.

Re: Move shorthands.el to lisp/emacs-lisp/?

[Dmitry Gutov]

On 29.10.2021 13:31, Yuri Khan wrote:
> On Fri, 29 Oct 2021 at 17:10, Eli Zaretskii<eliz@gnu.org>  wrote:
> 
>>> “git mv” is not magical though; it does not do anything that a manual
>>> move + staging both the deletion and the addition of the moved file
>>> does not do.
>> You misunderstood what I meant by "manual".  I meant "git rm" followed
>> by "git add".
>>
>> And if you want to say that the latter also does the same as "git mv",
>> then that's not my experience.
> In my understanding and experience, that should have the same effect,
> provided that the file content hash does not change.

As long as they're done in the same commit, of course.

Re: Move shorthands.el to lisp/emacs-lisp/?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't mind moving it now.  But please be sure to us "git mv", not
> the "manual" method, so that at least some of the Git commands could
> work across the renaming boundary.

I've now done this in the emacs-28 branch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 27 Oct 2021 14:38:05 +0000
> Cc: emacs-devel@gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > I thought we put all the fun stuff in lisp/play/, whether it's a game or
> > not.  Like handwrite.el and morse.el.  And it's not a mode, so textmodes
> > seemed like an odd place to put it?  And it didn't seem to fit in any of
> > the other categories, either.
> 
> Whatever we may think of Emojis, many people find them a very useful
> tool for communication.  So I agree with Eli that they belong in
> "textmodes" rather than "play".  Or perhaps "mail" or "net"?  But also
> consider that we could just put them in the top-level "lisp" directory
> together with a lot of other random stuff.

Or in lisp/international/ ?

> BTW, morse.el is an outlier in the "play" category, IMO.  I don't know
> why it was put there when rot13.el was not; rot13.el is a more obvious
> candidate for "play" as it can really only be used for joking around,
> while morse code is a serious and useful tool for communication.
> 
> (I'd be in favour of "git mv lisp/rot13.el lisp/play".)

No, rot13 is not a game.

Re: Entering emojis

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> (I'd be in favour of "git mv lisp/rot13.el lisp/play".)
>
> No, rot13 is not a game.

Lrg vg pna pregnvayl ragregnva.

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Wed, 27 Oct 2021 09:27:07 -0700
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (I'd be in favour of "git mv lisp/rot13.el lisp/play".)
> >
> > No, rot13 is not a game.
> 
> Lrg vg pna pregnvayl ragregnva.

Gung vg pna.

Re: Entering emojis

[Stefan Monnier]

Stefan Kangas [2021-10-27 14:38:05] wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> I thought we put all the fun stuff in lisp/play/, whether it's a game or
>> not.  Like handwrite.el and morse.el.  And it's not a mode, so textmodes
>> seemed like an odd place to put it?  And it didn't seem to fit in any of
>> the other categories, either.
>
> Whatever we may think of Emojis, many people find them a very useful
> tool for communication.  So I agree with Eli that they belong in
> "textmodes" rather than "play".  Or perhaps "mail" or "net"?  But also
> consider that we could just put them in the top-level "lisp" directory
> together with a lot of other random stuff.

lisp/international is also a fairly natural place for it, since that's
where we handle most of the non-ASCII issues.


        Stefan

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> lisp/international is also a fairly natural place for it, since that's
> where we handle most of the non-ASCII issues.

And Eli suggested that too, and I think it's fine, so I think we have a
winner.  :-)  I'll copy it over there when applying the patches from the
scratch/emoji branch.

But I'm thinking that I'd rather this go to master than to emacs-28.
Sure, it's self contained, but I'm absolutely confident that there's a
mass of bugs in the code (since it's 526 lines of brand new code,
statistically speaking there should be at least 53 serious bugs in it).

So including it in emacs-28 is likely to make the Emacs 28.1 release be
pushed back.

But I don't have a strong opinion here, so if the consensus is to put it
on emacs-28, I won't protest.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Kangas <stefan@marxist.se>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 18:06:09 +0200
> 
> But I'm thinking that I'd rather this go to master than to emacs-28.
> Sure, it's self contained, but I'm absolutely confident that there's a
> mass of bugs in the code (since it's 526 lines of brand new code,
> statistically speaking there should be at least 53 serious bugs in it).

Without it, there's no convenient way to type Emoji at all, so a buggy
command cannot be worse.  But I won't insist.

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But I'm thinking that I'd rather this go to master than to emacs-28.
> Sure, it's self contained, but I'm absolutely confident that there's a
> mass of bugs in the code (since it's 526 lines of brand new code,
> statistically speaking there should be at least 53 serious bugs in it).
>
> So including it in emacs-28 is likely to make the Emacs 28.1 release be
> pushed back.

How about just adding to NEWS (and maybe elsewhere): "emoji.el is
included as an experimental package, use at your own risk and report
bugs to ...  A stable version of this package will almost certainly be
included in Emacs 29.1."  Or is that very ugly/bad?

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> How about just adding to NEWS (and maybe elsewhere): "emoji.el is
> included as an experimental package, use at your own risk and report
> bugs to ...  A stable version of this package will almost certainly be
> included in Emacs 29.1."  Or is that very ugly/bad?

Sacha Chua reminded me that GNU ELPA exists.  :-)  I think the best way
forward here is to put the emoji stuff in the trunk, but also put it on
GNU ELPA as a package.  It'll only work for people in Emacs 28 and
newer, though, but that way we can continue ironing out problems/adding
features without getting in the way of releasing Emacs 28.1.

emoji.el is basically ready to go, but I still haven't heard from Jonas
about the transient.el changes.  Is the email address above the correct
one?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> emoji.el is basically ready to go, but I still haven't heard from Jonas
> about the transient.el changes.  Is the email address above the correct
> one?

Not only correct but perfect really. 😝
                   my favorite ------^

"Learn to mail" just got bumped up on my TODO list. 🤪
                      another favorite -------------^

> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

This needs updating now, at least temporarily:

 (domestic 🐈 only, the antidote for overdose, 🍼.)

Okay okay, I'll stop.  Makes you wonder though; was this a good idea?
In the wrong hands...

Anyway I told Lars that I plan to push his commit within a week, but
really I can do that tomorrow, if you want me to.

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1644 bytes --]

Jonas Bernoulli <jonas@bernoul.li> writes:


And both those emojis produced perfectly meaningful spoken utterances --
I suspect Emacs+Emacspeak may well be the only tool that knows to do
that:-)


So quoting from the Emacspeak Press Releases over the years:
<cite>
On this theme, when once challenged by a proponent of a crash-prone
but well-marketed mousetrap with the assertion "Emacs is a system from
the 70's", the creator of Emacspeak evinced surprise at the unusual
candor manifest in the assertion that it would take popular
idiot-proven interfaces until the year 2070 to catch up to where the
Emacspeak audio desktop is today.
</cite>

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> emoji.el is basically ready to go, but I still haven't heard from Jonas
>> about the transient.el changes.  Is the email address above the correct
>> one?
>
> Not only correct but perfect really. •05
>                    my favorite ------^
>
> "Learn to mail" just got bumped up on my TODO list. •0Ï6
>                       another favorite -------------^
>
>> (domestic pets only, the antidote for overdose, milk.)
>>    bloggy blog: http://lars.ingebrigtsen.no
>
> This needs updating now, at least temporarily:
>
>  (domestic ”9Ê2 only, the antidote for overdose, ”9¼2.)
>
> Okay okay, I'll stop.  Makes you wonder though; was this a good idea?
> In the wrong hands...
>
> Anyway I told Lars that I plan to push his commit within a week, but
> really I can do that tomorrow, if you want me to.
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Anyway I told Lars that I plan to push his commit within a week, but
> really I can do that tomorrow, if you want me to.

Sooner is better.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

Hey Lars,

Lars Ingebrigtsen <larsi@gnus.org> writes:

>                      There's three commands now -- emoji-insert
> (graphical/category choosing), emoji-search (textual based on names) and
> emoli-list (one big buffer of emojis).

Great work on the emoji branch; looking forward to be able to reach 🤪
with "C-x 8 e s zany" instead of "C-x 8 RET one big M-DEL large" 🙌

My current UX for emoji browsing is C-x 8 RET *PATTERN; with
icomplete-vertical-mode, it feels pretty similar to your emoji-search
AFAICT.  One nifty thing about C-x 8 RET is that it shows the actual
characters next to the candidate completions; cf. read-char-by-name and
mule--ucs-names-affixation.

Would it make sense to do something similar with emoji-search?

(I've tried to tweak the call to completing-read in emoji--choose-emoji
to get something similar to read-char-by-name's annotations, but my
command over the completion machinery is as yet insufficient 😵‍💫)

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> My current UX for emoji browsing is C-x 8 RET *PATTERN; with
> icomplete-vertical-mode, it feels pretty similar to your emoji-search
> AFAICT.  One nifty thing about C-x 8 RET is that it shows the actual
> characters next to the candidate completions; cf. read-char-by-name and
> mule--ucs-names-affixation.
>
> Would it make sense to do something similar with emoji-search?

Oh yeah, I meant to add that, but I forgot.  I'll do so now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Oh yeah, I meant to add that, but I forgot.  I'll do so now.

Now pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Now pushed.

C-x 8 e s ok C-j
C-x 8 e s *bowing C-j

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> C-x 8 e s ok C-j
> C-x 8 e s *bowing C-j

Do you mean that it's not requiring a match?  That should have been
fixed some minutes ago.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> C-x 8 e s ok C-j
>> C-x 8 e s *bowing C-j
>
> Do you mean that it's not requiring a match?  That should have been
> fixed some minutes ago.

Apologies for the confusion; I wrote C-j out of habit, because it's what
icomplete uses for "use first completion candidate"; RET means "ignore
candidates; just use what I typed" when REQUIRE-MATCH is nil.

As of 2021-10-28 "Fix build warning" (8c43fe0940), RET and C-j both
bring up the "choose a variation" transient, so everything is as
expected IIUC!

Btw, should the variations be presented as transient arguments, instead
of transient commands?  transient.el allows saving arguments with C-x
C-s, so it'd make sense IMO to turn the first-insertion UX into
something like

1. start looking for emoji,
2. apply variations as transient arguments,
3. C-x C-s those arguments for frequently used variations,
4. pick emoji,

and later insertions could just be steps 1 and 4, with an occasional
variation tweak when needed.

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Btw, should the variations be presented as transient arguments, instead
> of transient commands?  transient.el allows saving arguments with C-x
> C-s, so it'd make sense IMO to turn the first-insertion UX into
> something like
>
> 1. start looking for emoji,
> 2. apply variations as transient arguments,
> 3. C-x C-s those arguments for frequently used variations,
> 4. pick emoji,
>
> and later insertions could just be steps 1 and 4, with an occasional
> variation tweak when needed.

I've never used transient before this, so I don't quite understand what
you're saying here.  🤨  But recently used emojis land in the "Recent"
transient (which can also be accessed from `C-x 8 e r' now).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've never used transient before this, so I don't quite understand what
> you're saying here.  🤨

I have used transient a lot in the context of magit, where each
transient command gets a bunch of "arguments" that you can save
permanently, thereby saving you the trouble of giving common options
such as "--graph", "--decorate" and the like everytime you want to run
"git log".

Here, lemme try something…

*bangs head on keyboard for an hour*

OK, see attached.  With M-x emoji-pick, you can now

- hit "-s" to pick your favourite skin tone,
- hit "C-x C-s" to save it (to ~/.emacs.d/transient/values.el),
- insert emoji,
- on subsequent M-x emoji-pick, you no longer need to pick a skin tone,
- you can still hit "-s" occasionally to unset the tone, and a second
  time to pick a different one,
- you get minibuffer history with M-p.

Note that I have never written transients in my life before (not with
that level of complexity anyway), so I'm sure my implementation is all
manners of uncouth.


[-- Attachment #2: transient-arguments.el --]
[-- Type: text/x-emacs-lisp, Size: 1553 bytes --]

(defun emoji-pick-read-affixation (names)
  (mapcar
   (lambda (name)
     (list name (char-to-string
                 (char-from-name
                  (concat "EMOJI MODIFIER FITZPATRICK TYPE-" name)))))
   names))

(defun emoji-pick-read-tone (prompt initial-input history)
  (let* ((names '("1-2" "3" "4" "5" "6"))
         (str (completing-read
               prompt
               (lambda (string pred action)
                 (if (eq action 'metadata)
                     `(metadata
		       (affixation-function . ,'emoji-pick-read-affixation))
                   (complete-with-action action names string pred)))
               nil t initial-input history)))
    (char-to-string
     (char-from-name (concat "EMOJI MODIFIER FITZPATRICK TYPE-" str)))))

(transient-define-infix emoji-pick:--skin-tone ()
  :description "Skin tone"
  :class 'transient-option
  :shortarg "-s"
  :argument ""
  :prompt "Skin tone: "
  :reader 'emoji-pick-read-tone)

(transient-define-prefix emoji-pick ()
  ["Variations"
   (emoji-pick:--skin-tone)]
  [["Emoji"
    ("s" "🤷" emoji-shrug)
    ("y" "👍" emoji-yes)]]
  (interactive)
  (transient-setup 'emoji-pick))

(defun emoji-shrug (variation)
  (interactive (or (transient-args 'emoji-pick)
                   (list nil)))
  (insert ?🤷)
  (when variation
    (insert variation)))

(defun emoji-yes (variation)
  (interactive (or (transient-args 'emoji-pick)
                   (list nil)))
  (insert ?👍)
  (when variation
    (insert variation)))

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> OK, see attached.  With M-x emoji-pick, you can now
>
> - hit "-s" to pick your favourite skin tone,
> - hit "C-x C-s" to save it (to ~/.emacs.d/transient/values.el),
> - insert emoji,
> - on subsequent M-x emoji-pick, you no longer need to pick a skin tone,
> - you can still hit "-s" occasionally to unset the tone, and a second
>   time to pick a different one,
> - you get minibuffer history with M-p.

Oh, that sounds pretty nice...

> Note that I have never written transients in my life before (not with
> that level of complexity anyway), so I'm sure my implementation is all
> manners of uncouth.

Unfortunately, it doesn't seem to work.  I just get

transient-infix-read: Lisp nesting exceeds ‘max-lisp-eval-depth’

when evalling (emoji-pick:--skin-tone).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> OK, see attached.  With M-x emoji-pick, you can now
>>
>> - hit "-s" to pick your favourite skin tone,
>> - hit "C-x C-s" to save it (to ~/.emacs.d/transient/values.el),
>> - insert emoji,
>> - on subsequent M-x emoji-pick, you no longer need to pick a skin tone,
>> - you can still hit "-s" occasionally to unset the tone, and a second
>>   time to pick a different one,
>> - you get minibuffer history with M-p.
>
> Oh, that sounds pretty nice...

Yes, and pretty orthogonal to emoji-recent AFAICT: saving the transient
argument allows it to be applied by default for emoji that you haven't
used yet.

>> Note that I have never written transients in my life before (not with
>> that level of complexity anyway), so I'm sure my implementation is all
>> manners of uncouth.
>
> Unfortunately, it doesn't seem to work.  I just get
>
> transient-infix-read: Lisp nesting exceeds ‘max-lisp-eval-depth’
>
> when evalling (emoji-pick:--skin-tone).

Mmm.  I've tweaked the file; with the tip of the master branch
(2021-10-29 "Add some gnus-short-group-name tests" (8ada213b87)), things
seem to work on my end?  I'm starting Emacs with

emacs -Q -l transient-arguments.el

then hitting M-x emoji-pick.


[-- Attachment #2: transient-arguments.el --]
[-- Type: text/x-emacs-lisp, Size: 1477 bytes --]

(require 'transient)

(defun emoji-pick-read-affixation (names)
  (mapcar
   (lambda (name)
     (list name (char-to-string
                 (char-from-name
                  (concat "EMOJI MODIFIER FITZPATRICK TYPE-" name)))))
   names))

(defun emoji-pick-read-tone (prompt initial-input history)
  (let* ((names '("1-2" "3" "4" "5" "6"))
         (str (completing-read
               prompt
               (lambda (string pred action)
                 (if (eq action 'metadata)
                     `(metadata
		       (affixation-function . ,'emoji-pick-read-affixation))
                   (complete-with-action action names string pred)))
               nil t initial-input history)))
    (char-to-string
     (char-from-name (concat "EMOJI MODIFIER FITZPATRICK TYPE-" str)))))

(transient-define-infix emoji-pick:--skin-tone ()
  :description "Skin tone"
  :class 'transient-option
  :shortarg "-s"
  :argument ""
  :prompt "Skin tone: "
  :reader 'emoji-pick-read-tone)

(transient-define-prefix emoji-pick ()
  ["Variations"
   (emoji-pick:--skin-tone)]
  [["Emoji"
    ("s" "🤷" emoji-shrug)
    ("y" "👍" emoji-yes)]]
  (interactive)
  (transient-setup 'emoji-pick))

(defun emoji-shrug (&optional variation)
  (interactive (transient-args 'emoji-pick))
  (insert (concat "🤷" variation)))

(defun emoji-yes (&optional variation)
  (interactive (transient-args 'emoji-pick))
  (insert (concat "👍" variation)))

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Mmm.  I've tweaked the file; with the tip of the master branch
> (2021-10-29 "Add some gnus-short-group-name tests" (8ada213b87)), things
> seem to work on my end?  I'm starting Emacs with
>
> emacs -Q -l transient-arguments.el
>
> then hitting M-x emoji-pick.

Thanks, that works for me.

I'm not sure we should have such a thing in the interface, though.
Being able to set a default skin tone/gender sounds very fraught
politically.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Kévin Le Gouguec]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'm not sure we should have such a thing in the interface, though.
> Being able to set a default skin tone/gender sounds very fraught
> politically.

AFAICT this feature would merely let each user, individually, pick
default variations that make sense for them personally.  I struggle to
see a politically controversial side to that, but maybe there are
implications I fail to imagine.

I personally don't use variations 🤷 so the main appeal of this feature
would be not having to pick one every time I insert an emoji 😉

Re: Entering emojis

[Lars Ingebrigtsen]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> AFAICT this feature would merely let each user, individually, pick
> default variations that make sense for them personally.  I struggle to
> see a politically controversial side to that, but maybe there are
> implications I fail to imagine.
>
> I personally don't use variations 🤷

Well, that's a political statement all on its own.  😀

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> AFAICT this feature would merely let each user, individually, pick
>> default variations that make sense for them personally.  I struggle to
>> see a politically controversial side to that, but maybe there are
>> implications I fail to imagine.
>>
>> I personally don't use variations 🤷
>
> Well, that's a political statement all on its own.  😀

As is not adding this feature. 😱

I would be annoyed if I had to every time go through the same extra
steps, so that I could express my feelings using an emoji that looked
like me.

Another reason I would like this feature added is that its an area where
transient shines.  For now this would have to be done using something
involving completing-read as Kévin has done.  It would be nice to use a
sub-prefix to select the color/gender/... but currently a prefix (or
rather one of its suffixes) cannot return a value to the parent prefix.
But that was actually already one of the features I plan to implement
next.  Until the approach using completing-read also works.

Re: Entering emojis

[Jonas Bernoulli]

Jonas Bernoulli <jonas@bernoul.li> writes:

> I would be annoyed if I had to every time go through the same extra
> steps, so that I could express my feelings using an emoji that looked
> like me.

After actually having used this a few times:

What you have implemented makes a lot of sense to.  It's not an extra
step because everyone has to do it, even the Simpsons.  Also it isn't
really inconvenient because its just one additional key press.  That
being said I still think it would nice to have the ability to set
defaults, but it's not a must have.

Re: Entering emojis

[Lars Ingebrigtsen]

Gah.  I've been using the wrong file as the base file to get the
categories of emojis -- it was last updated in 2016.

This curiously named file seems to be the base of that file, but is
up-to-date, so it has all the new emojis added the last five years:

https://unicode.org/Public/emoji/14.0/emoji-test.txt

So I'll be rewriting that bit tomorrow.  That file also seems to allow
me to easy make the derivation groups without parsing all the other
files, so it should be faster, too.  So...  better all around, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> https://unicode.org/Public/emoji/14.0/emoji-test.txt
>
> So I'll be rewriting that bit tomorrow.  That file also seems to allow
> me to easy make the derivation groups without parsing all the other
> files, so it should be faster, too.  So...  better all around, I think.

Yup.  This is now implemented, and it's way less hacky, and I seem to be
able to get all the derivations I'd expect (and I don't see any false
positives).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yup.  This is now implemented, and it's way less hacky, and I seem to be
> able to get all the derivations I'd expect (and I don't see any false
> positives).

The `M-x emoji-insert' interface here is strictly better than what I see
on my phone (which is presumably state of the art).  Perhaps we could
add arrow keys/TAB/C-n/C-p/etc.+RET navigation, though that would just
be the cherry on top.

I did notice that there is a noticeable delay when using the command for
the first time.  Is that expected?

In addition to "C-g", perhaps it would be good to give a standard key
like "q" the meaning "go back to the previous screen" and maybe "Q" to
mean "I changed my mind, close everything and insert no emoji"?

One final observation is that it would be nice if we could click the
emojis to insert them, on all screens (including the overview ones).

BTW, when I insert this emoji "🧙🏿‍♀️" and then hit DEL repeatedly, the
buffer text first turns into "🧙🏿‍♀" and then "🧙🏿", and then "🧙🏿"
and then "🧙".  Is that a known issue?

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> The `M-x emoji-insert' interface here is strictly better than what I see
> on my phone (which is presumably state of the art).

:-)

> Perhaps we could add arrow keys/TAB/C-n/C-p/etc.+RET navigation,
> though that would just be the cherry on top.

In addition to the letter input thing?

> I did notice that there is a noticeable delay when using the command for
> the first time.  Is that expected?

Funny you should ask -- I've just finished (I think) making the Emacs
build process output the resulting structures to
international/emoji-labels.el, so that we don't have to parse the file
run-time.  And this can also be used by `describe-char' (or something
else) to give the user the name of the displayed grapheme cluster:

(gethash "🇺🇾" emoji--names)
=> "flag: Uruguay"

I haven't hooked that up yet, though -- it should only load that file if
we're describing an emoji grapheme cluster.  So I guess the thing to
test is whether script is `emoji'?

> In addition to "C-g", perhaps it would be good to give a standard key
> like "q" the meaning "go back to the previous screen" and maybe "Q" to
> mean "I changed my mind, close everything and insert no emoji"?

This is the first time I've used transient.el -- do transients normally
bind `q'?

> One final observation is that it would be nice if we could click the
> emojis to insert them, on all screens (including the overview ones).

Yes, that's a good idea.  I don't know whether Transient supports that,
though.  Anybody know, and if it does, what incantation do I have to use
to make that work?

> BTW, when I insert this emoji "🧙🏿‍♀️" and then hit DEL repeatedly, the
> buffer text first turns into "🧙🏿‍♀" and then "🧙🏿", and then "🧙🏿"
> and then "🧙".  Is that a known issue?

That's how it's supposed to work, I think?  It's a grapheme cluster
consisting of three Unicode characters (and two zero-width joiners).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Funny you should ask -- I've just finished (I think) making the Emacs
> build process output the resulting structures to
> international/emoji-labels.el, so that we don't have to parse the file
> run-time.

Well, that didn't really make `C-x 8 e e' (the first time) a whole lot
faster.  I mean, it's twice as fast, but I thought it was going to make
it instantaneous, and it's not.  And it's because I'm going over all the
emojis doing `char-displayable-p' on them all to weed out the ones we
don't have a font for, and that's apparently very slow?  (I mean, it
takes 0.2s to do them all, so it's not...  beyond the pale, but it's
still annoying.)

Is there a faster way?  I mean, in this context we know the script and
stuff, so perhaps there's a shortcut we could take?  Anybody know?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 28 Oct 2021 13:19:19 +0200
> Cc: emacs-devel@gnu.org
> 
> Well, that didn't really make `C-x 8 e e' (the first time) a whole lot
> faster.  I mean, it's twice as fast, but I thought it was going to make
> it instantaneous, and it's not.  And it's because I'm going over all the
> emojis doing `char-displayable-p' on them all to weed out the ones we
> don't have a font for, and that's apparently very slow?  (I mean, it
> takes 0.2s to do them all, so it's not...  beyond the pale, but it's
> still annoying.)

Why do you need to call char-displayable-p in this case?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do you need to call char-displayable-p in this case?

To see whether the font has support for the emoji (and not display it).
Unicode is adding new characters all the time, and not even Noto Color
Emoji has them all.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 14:56:17 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why do you need to call char-displayable-p in this case?
> 
> To see whether the font has support for the emoji (and not display it).

Why not display them?  They won't appear as Emoji if the font doesn't
support them, but why does it matter for the feature we are
discussing?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why not display them?  They won't appear as Emoji if the font doesn't
> support them, but why does it matter for the feature we are
> discussing?

It's not very useful to display the tofu boxes in the graphical display.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 15:54:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why not display them?  They won't appear as Emoji if the font doesn't
> > support them, but why does it matter for the feature we are
> > discussing?
> 
> It's not very useful to display the tofu boxes in the graphical display.

Why not? it will at least tell the user the font doesn't support
that.  More importantly, it will allow the user to choose a sequence
even if his/her Emacs cannot display it, because those of the
addressees (assuming this is in an email message, for example) could.
And as a nice bonus, it makes the code much faster.

So I'd say it's a net win.  But if you are still unconvinced, how
about a user option to control whether undisplay-able emoji sequences
are filtered out?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why not? it will at least tell the user the font doesn't support
> that.  More importantly, it will allow the user to choose a sequence
> even if his/her Emacs cannot display it, because those of the
> addressees (assuming this is in an email message, for example) could.

But this is in the graphical display -- the user has no way of knowing
what they're entering (it's just the glyph, not the name).

The other interface (the text-based one) lists all glyphs, even the ones
that Emacs can't display, so that'd be the way to insert such emojis.

> And as a nice bonus, it makes the code much faster.

I've poked at the problem some more, and it seems like
`char-displayable-p' is really fast on characters it can actually
display, but slow on characters it doesn't know about?  Perhaps this
makes it trigger loading more fonts or something?  (I haven't actually
read the code yet, just done some testing.)  If this is the case, is
there a way to say "don't try to load anything, but just see if you have
the glyph in the current set"?

> So I'd say it's a net win.  But if you are still unconvinced, how
> about a user option to control whether undisplay-able emoji sequences
> are filtered out?

I don't think it's necessary, but if people request it, I don't object
to adding it.  (But I don't think anybody will.  🙃)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 23:49:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why not? it will at least tell the user the font doesn't support
> > that.  More importantly, it will allow the user to choose a sequence
> > even if his/her Emacs cannot display it, because those of the
> > addressees (assuming this is in an email message, for example) could.
> 
> But this is in the graphical display -- the user has no way of knowing
> what they're entering (it's just the glyph, not the name).

I assumed you display the names as well.  Why don't you? it could be
important, because the appearance is not always everything.  Sometimes
the selection of an Emoji needs that you understand what it expresses,
in words.

> I've poked at the problem some more, and it seems like
> `char-displayable-p' is really fast on characters it can actually
> display, but slow on characters it doesn't know about?  Perhaps this
> makes it trigger loading more fonts or something?  (I haven't actually
> read the code yet, just done some testing.)

Of course, that function could involve looking for a suitable font, if
none of those already loaded supports the character.  Emacs does there
the same as it does when it needs to display the character.

> If this is the case, is there a way to say "don't try to load
> anything, but just see if you have the glyph in the current set"?

But that makes no sense, because there's no guarantee the fonts
already loaded support the character you ask about, and no guarantee
there isn't some other font which does.  And in any case, AFAIU the
delay of loading fonts is not wasted if Emacs does succeed to find a
font, because then actually displaying the character will find the
required font already loaded.  It's only the failure to find a font
that causes multiple failed attempts for the next character or
sequence.

When this search is slow, how many sequences fail to display?  Or is
it slow even if all the sequences can be displayed?  If the latter,
there could be some issue here that doesn't meet the eye, and we
should start by profiling.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I assumed you display the names as well.  Why don't you? it could be
> important, because the appearance is not always everything.  Sometimes
> the selection of an Emoji needs that you understand what it expresses,
> in words.

Choosing emojis graphically is the popular way of choosing them, so
Emacs should have such a method.  If you don't want to use that,
nobody's forcing you -- use the method that displays with names instead.

> But that makes no sense, because there's no guarantee the fonts
> already loaded support the character you ask about, and no guarantee
> there isn't some other font which does.

It doesn't make sense in general, but for emojis you're unlikely to have
many separate fonts that cover different sets of emojis.  So once we've
established that you have an emoji font, we don't have to try to keep
loading fonts for the emojis that this font doesn't have.

> When this search is slow, how many sequences fail to display?  Or is
> it slow even if all the sequences can be displayed?  If the latter,
> there could be some issue here that doesn't meet the eye, and we
> should start by profiling.

The slowness is linear with the number of emojis that we don't have a
font for.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 14:46:26 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I assumed you display the names as well.  Why don't you? it could be
> > important, because the appearance is not always everything.  Sometimes
> > the selection of an Emoji needs that you understand what it expresses,
> > in words.
> 
> Choosing emojis graphically is the popular way of choosing them, so
> Emacs should have such a method.  If you don't want to use that,
> nobody's forcing you -- use the method that displays with names instead.

No, I meant to show both.

> > When this search is slow, how many sequences fail to display?  Or is
> > it slow even if all the sequences can be displayed?  If the latter,
> > there could be some issue here that doesn't meet the eye, and we
> > should start by profiling.
> 
> The slowness is linear with the number of emojis that we don't have a
> font for.

Yes, but for the cases you see on your system, what is the actual
number?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Choosing emojis graphically is the popular way of choosing them, so
>> Emacs should have such a method.  If you don't want to use that,
>> nobody's forcing you -- use the method that displays with names instead.
>
> No, I meant to show both.

That's what the `C-x 8 e s' method does.  The filtering is for the `C-x
8 e e' command, which only shows the glyphs.

> Yes, but for the cases you see on your system, what is the actual
> number?

I think it was about forty?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 15:38:37 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, but for the cases you see on your system, what is the actual
> > number?
> 
> I think it was about forty?

Can you show a fully-expanded profile of such a run?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you show a fully-expanded profile of such a run?

I don't know how.  Here's the things that don't have a glyph in my
version of Noto Color Emoji:

(benchmark-run 1
  (dolist (char '(129008 129706 129767 129659 129660 129707 129705 129708 128735 128734 128733 129753 129751 129752 129722 129721 129719 129720 129484 129732 129731 129733 129766 129782 129781 129776 129780 129779 129778 129777 129401 129764 129765 129761 129763 129762 129760))
    (char-displayable-p char)))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>     (char-displayable-p char)))

The thing that takes time is way down in

fontset_find_font (Lisp_Object fontset, int c, struct face *face,

This bit:

	  /* Find a font best-matching with the spec without checking
	     the support of the character C.  That checking is costly,
	     and even without the checking, the found font supports C
	     in high possibility.  */
	  font_entity = font_find_for_lface (f, face->lface,
					     FONT_DEF_SPEC (font_def), -1);
	  if (NILP (font_entity))
	    {
	      /* Record that no font matches the spec.  */
	      RFONT_DEF_SET_FACE (rfont_def, -1);
	      continue;
	    }

So might it be an idea to introduce a variable that could be bound to
avoid this part?  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So might it be an idea to introduce a variable that could be bound to
> avoid this part?  

I gave it a try, and that does indeed fix the slight pause the first
time I use `C-x 8 e e'.  I'm pushing the change to scratch/emoji for
consideration.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I gave it a try, and that does indeed fix the slight pause the first
> time I use `C-x 8 e e'.  I'm pushing the change to scratch/emoji for
> consideration.

And I should have tested that more in a fresh Emacs -- as you said, it's
just not clear under what circumstances we already have the font.  It
seems like even the emoji font is "spread out".

So I guess we'll just have to live with the delay...  or redo the
transient interface to not be all defined "up front".  Postponing the
checks until the glyphs are actually displayed would also get rid of the
pause.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 16:53:16 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > I gave it a try, and that does indeed fix the slight pause the first
> > time I use `C-x 8 e e'.  I'm pushing the change to scratch/emoji for
> > consideration.
> 
> And I should have tested that more in a fresh Emacs -- as you said, it's
> just not clear under what circumstances we already have the font.  It
> seems like even the emoji font is "spread out".
> 
> So I guess we'll just have to live with the delay...  or redo the
> transient interface to not be all defined "up front".  Postponing the
> checks until the glyphs are actually displayed would also get rid of the
> pause.

Let's not jump to conclusions yet, and see what that profile tells us.

Do you really see more than one font being used for the Emoji display
on your system?  If so, how many fonts, and which ones are they?

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Do you really see more than one font being used for the Emoji display
> on your system?  If so, how many fonts, and which ones are they?

As far as I can tell, there's only one emoji font file...  but disabling
that loading thing seemed to give me random responses to
char-displayable-p.

That is, I implemented it to always ask for the first emoji "properly"
(i.e., loading fonts), but suppressed it for the next run of emojis.
This seemed to lead to char-displayable-p not returning a font for
... arbitrary glyphs?

Looking at font-log, I see:

 (list "-*-Noto Color Emoji-*-iso10646-1"
       ["-GOOG-Noto Color Emoji-medium-normal-normal-*-m-0-iso10646-1"])
 (list "-PfEd-Noto Color Emoji-*-iso10646-1" nil)
 (default\ fontset:\ font\ for 127462 nil)
 (list "-*-Noto Color Emoji-*-ascii-0"
       ["-GOOG-Noto Color Emoji-medium-normal-normal-*-m-0-iso10646-1"])
 (list "-*-Noto Color Emoji-*-iso8859-1" nil)
 (list "-PfEd-Noto Color Emoji-*-ascii-0" nil)
 (list "-PfEd-Noto Color Emoji-*-iso8859-1" nil)
 (default\ fontset:\ font\ for 127987 nil)
 (list "-*-Noto Color Emoji-*-iso10646-1"
       ["-GOOG-Noto Color Emoji-medium-normal-normal-*-m-0-iso10646-1"])
 (list "-PfEd-Noto Color Emoji-*-iso10646-1" nil)
 (default\ fontset:\ font\ for 127988 nil)
 (list "-*-Noto Color Emoji-*-iso10646-1"
       ["-GOOG-Noto Color Emoji-medium-normal-normal-*-m-0-iso10646-1"])
 (list "-PfEd-Noto Color Emoji-*-iso10646-1" nil)
 (default\ fontset:\ font\ for 9726 nil)
 (list "-*-Noto Color Emoji-*-iso10646-1"
       ["-GOOG-Noto Color Emoji-medium-normal-normal-*-m-0-iso10646-1"])

But I don't know how to interpret that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 16:30:52 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> >     (char-displayable-p char)))
> 
> The thing that takes time is way down in
> 
> fontset_find_font (Lisp_Object fontset, int c, struct face *face,
> 
> This bit:
> 
> 	  /* Find a font best-matching with the spec without checking
> 	     the support of the character C.  That checking is costly,
> 	     and even without the checking, the found font supports C
> 	     in high possibility.  */
> 	  font_entity = font_find_for_lface (f, face->lface,
> 					     FONT_DEF_SPEC (font_def), -1);
> 	  if (NILP (font_entity))
> 	    {
> 	      /* Record that no font matches the spec.  */
> 	      RFONT_DEF_SET_FACE (rfont_def, -1);
> 	      continue;
> 	    }
> 
> So might it be an idea to introduce a variable that could be bound to
> avoid this part?  

I'd prefer not to do it on such a low level.

I'd still would like to see a profile.  Here's the recipe:

  emacs -Q
  M-x load-file RET lisp/play/emoji.el RET
  M-x profiler-start RET RET

Then run the command you say is slow.  When it finishes, do this:

  M-x profiler-report RET

Then go to the first line in the profile that shows "+" at its
beginning and type "C-u RET".  This should expand the profile in its
entirety; post the result here.

Note that I specifically asked to load emoji.el, not .elc, because
that will make the profile more detailed.

Thanks.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I'd still would like to see a profile.  Here's the recipe:
>
>   emacs -Q
>   M-x load-file RET lisp/play/emoji.el RET
>   M-x profiler-start RET RET
>
> Then run the command you say is slow.  When it finishes, do this:
>
>   M-x profiler-report RET

         133  78% - ...
          87  51%  - let
          87  51%   - while
          87  51%    - let
          87  51%     - emoji--adjust-displayable
          87  51%      - if
          87  51%       - setcdr
          87  51%        - seq-filter
          87  51%         - seq-map
          87  51%          - apply
          87  51%           - #<compiled 0x1843caf3bb7f29b4>
          87  51%            - mapcar
          87  51%             - #<compiled 0x91a61154359fe3f>
          87  51%              - #<lambda -0x17fb383fa8059ce8>
          87  51%               - not
          87  51%                - symbolp
          87  51%                   char-displayable-p
          34  20%  - minibuffer-complete
          34  20%   - completion-in-region
          34  20%    - completion--in-region
          34  20%     - #<compiled -0xf1bbde4b0103a2>
          34  20%      - apply
          34  20%       - #<compiled 0x13ba59660bb71ecd>
          34  20%        - completion--in-region-1
          34  20%         - completion--do-completion
          34  20%          - completion-try-completion
          34  20%           - completion--nth-completion
          34  20%            - completion--some
          34  20%             - #<compiled 0x1d99e92c356afe75>
          34  20%              - completion-basic-try-completion
          34  20%               - try-completion
          34  20%                - #<compiled -0x1ca7c2c7aa49bfc8>
          34  20%                   complete-with-action
          12   7%    Automatic GC


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 18:34:31 +0200
> 
> >   emacs -Q
> >   M-x load-file RET lisp/play/emoji.el RET
> >   M-x profiler-start RET RET
> >
> > Then run the command you say is slow.  When it finishes, do this:
> >
> >   M-x profiler-report RET
> 
>          133  78% - ...
>           87  51%  - let
>           87  51%   - while
>           87  51%    - let
>           87  51%     - emoji--adjust-displayable
>           87  51%      - if
>           87  51%       - setcdr
>           87  51%        - seq-filter
>           87  51%         - seq-map
>           87  51%          - apply
>           87  51%           - #<compiled 0x1843caf3bb7f29b4>
>           87  51%            - mapcar
>           87  51%             - #<compiled 0x91a61154359fe3f>
>           87  51%              - #<lambda -0x17fb383fa8059ce8>
>           87  51%               - not
>           87  51%                - symbolp
>           87  51%                   char-displayable-p

Please repeat this after "M-x load-file Ret mule.el RET", so that we
see which parts of char-displayable-p take the lion's share of the
time.  Sorry I didn't think about this earlier.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Please repeat this after "M-x load-file Ret mule.el RET", so that we
> see which parts of char-displayable-p take the lion's share of the
> time.  Sorry I didn't think about this earlier.

Doesn't seem to be significantly more detailed...  The time spent is in
internal-char-font (i.e., in C).

         160  79% - command-execute
         160  79%  - call-interactively
         107  52%   - funcall-interactively
         102  50%    - eval-last-sexp
         102  50%     - elisp--eval-last-sexp
          92  45%      - eval
          92  45%       - progn
          88  43%        - require
          88  43%         - load-with-code-conversion
          88  43%          - if
          88  43%           - let
          84  41%            - while
          84  41%             - let
          84  41%              - emoji--adjust-displayable
          84  41%               - if
          84  41%                - setcdr
          84  41%                 - seq-filter
          84  41%                  - seq-map
          84  41%                   - apply
          84  41%                    - #<compiled 0x1843b3846bb8a9b4>
          84  41%                     - mapcar
          84  41%                      - #<compiled 0x91a61154973e1ff>
          84  41%                       - #<lambda -0x17f97417128c1ee8>
          81  40%                        - not
          81  40%                         - symbolp
          81  40%                          - char-displayable-p
          81  40%                           - cond
          81  40%                              let
           3   1%                        - let*
           3   1%                         - if
           3   1%                            let*
           4   1%            - unwind-protect
           4   1%             - let
           4   1%              - let
           4   1%                 eval-buffer
          10   4%      - macroexpand-all
          10   4%       - macroexp--expand-all
          10   4%        - #<compiled -0x1b8122fb0766dcec>
          10   4%         - macroexp--all-forms
          10   4%          - macroexp--expand-all
          10   4%             macroexp-macroexpand
           5   2%    - execute-extended-command
           5   2%     - command-execute
           5   2%      - call-interactively
           5   2%       - funcall-interactively
           5   2%          profiler-report
          53  26%   - byte-code
          53  26%    - read-extended-command
          53  26%     - completing-read
          53  26%      - completing-read-default
          22  10%       - read-from-minibuffer
           1   0%        - redisplay_internal (C function)
           1   0%         - #<compiled 0xea9a01304f8c1ee>
           1   0%          - apply
           1   0%           - redisplay--pre-redisplay-functions
           1   0%            - run-hook-with-args
           1   0%             - redisplay--update-region-highlight
           1   0%                #<compiled 0x1aa0cae987b1b988>


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 20:23:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please repeat this after "M-x load-file Ret mule.el RET", so that we
> > see which parts of char-displayable-p take the lion's share of the
> > time.  Sorry I didn't think about this earlier.
> 
> Doesn't seem to be significantly more detailed...  The time spent is in
> internal-char-font (i.e., in C).

OK, that's what I thought it will say.

So how about the following  alternative strategy:

 . First, call (internal-char-font nil #x1f300).  This will return a
   cons cell whose car is a font object for the font that supports
   that codepoint.  If it returns nil, the system doesn't have a font
   for Emoji.

 . Then, for each character you want to test for being displayable,
   do this:

      (font-get-glyphs FONT-OBJECT 0 1 '[CODEPOINT])

   where FONT-OBJECT is the car of the value returned by
   internal-char-font, and CODEPOINT is the character you want to test
   for being displayable, for example #x1f600.  If this returns nil,
   that character is not supported by that font.

This should be faster, since it only checks a single font, and the
expensive call is outside the loop.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>  . Then, for each character you want to test for being displayable,
>    do this:
>
>       (font-get-glyphs FONT-OBJECT 0 1 '[CODEPOINT])
>
>    where FONT-OBJECT is the car of the value returned by
>    internal-char-font, and CODEPOINT is the character you want to test
>    for being displayable, for example #x1f600.  If this returns nil,
>    that character is not supported by that font.
>
> This should be faster, since it only checks a single font, and the
> expensive call is outside the loop.

Great!  But plugging that into the code seems to yield no difference, so
I tried various strategies to time this.  And in my synthetic tests,
it's way faster.  This takes 0.3s, which is many orders of magnitude
faster than just calling internal-char-font:

(benchmark-run 1
  (let ((font (car (internal-char-font nil ?😀))))
    (dotimes (i 50000)
      (font-get-glyphs font 0 1 (vector i)))))

So I've been trying to profile the code, and with this wrapper:

(defun wrap (a b c d)
  (font-get-glyphs a b c d))

I get this trace:

         114  61% - ...
         103  55%  - emoji--adjust-displayable
         103  55%   - let
         103  55%    - emoji--adjust-displayable-1
         103  55%     - if
         103  55%      - let
         103  55%       - while
         103  55%        - let
         103  55%         - emoji--adjust-displayable-1
         103  55%          - if
          96  52%           - let
          96  52%            - while
          96  52%             - let
          96  52%              - emoji--adjust-displayable-1
          96  52%               - if
          96  52%                - while
          96  52%                 - let
          88  47%                  - if
          88  47%                   - let
          88  47%                    - if
          88  47%                     - elt
          88  47%                        wrap

Which seems to say that all the time is indeed spent in font-get-glyphs.
So I'm scratching my head and poking at this, because something has to
be...  wrong...  I'll be back tomorrow with further results.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'll be back tomorrow with further results.  :-/

Hey, it's almost tomorrow.

I made emoji.el spit out the actual code points it's checking, and I've
made a big micro test from that (included below).

And indeed, for that set of code points, internal-char-font and
font-get-glyphs take almost exactly the same time, which is very odd.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Attachment #2: ftest.el --]
[-- Type: application/emacs-lisp, Size: 12316 bytes --]

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> And indeed, for that set of code points, internal-char-font and
> font-get-glyphs take almost exactly the same time, which is very odd.

Or perhaps not!  That function does quite a lot of work both when it
gets a match and not (allocating vectors and stuff).

As a test, I made it return t/nil if fed a character:

diff --git a/src/font.c b/src/font.c
index 5e761abc5e..3b0309088b 100644
--- a/src/font.c
+++ b/src/font.c
@@ -5067,6 +5067,14 @@ DEFUN ("font-get-glyphs", Ffont_get_glyphs, Sfont_get_glyphs, 3, 4, 0,
 	}
       chars = aref_addr (object, ifrom);
     }
+  else if (INTEGERP (object))
+    {
+      int c = XFIXNAT (object);
+      if (font->driver->encode_char (font, c) == FONT_INVALID_CODE)
+	return Qnil;
+      else
+	return Qt;
+    }
   else
     wrong_type_argument (Qarrayp, object);
 
And that made it really, really fast -- like 100x faster in the test
case.  :-)

Should I fix this up and document it?  It's perhaps not an ideal
interface...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Should I fix this up and document it?  It's perhaps not an ideal
> interface...

To put it mildly.  So here's a new function that has one clear purpose:

diff --git a/src/font.c b/src/font.c
index 5e761abc5e..1bde062e87 100644
--- a/src/font.c
+++ b/src/font.c
@@ -4977,6 +4977,21 @@ DEFUN ("query-font", Fquery_font, Squery_font, 1, 1, 0,
 		 : Qnil));
 }
 
+DEFUN ("font-has-glyph-p", Ffont_has_glyph_p, Sfont_has_glyph_p, 2, 2, 0,
+       doc:
+       /* Say whether FONT has a glyph for CHAR.  */)
+  (Lisp_Object font_object, Lisp_Object character)
+{
+  struct font *font = CHECK_FONT_GET_OBJECT (font_object);
+  CHECK_CHARACTER (character);
+
+  int c = XFIXNAT (character);
+  if (font->driver->encode_char (font, c) == FONT_INVALID_CODE)
+    return Qnil;
+  else
+    return Qt;
+}
+
 DEFUN ("font-get-glyphs", Ffont_get_glyphs, Sfont_get_glyphs, 3, 4, 0,
        doc:
        /* Return a vector of FONT-OBJECT's glyphs for the specified characters.
@@ -5548,6 +5563,7 @@ syms_of_font (void)
   defsubr (&Sclose_font);
   defsubr (&Squery_font);
   defsubr (&Sfont_get_glyphs);
+  defsubr (&Sfont_has_glyph_p);
   defsubr (&Sfont_match_p);
   defsubr (&Sfont_at);
 #if 0


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 23:21:28 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Should I fix this up and document it?  It's perhaps not an ideal
> > interface...
> 
> To put it mildly.  So here's a new function that has one clear purpose:
> 
> diff --git a/src/font.c b/src/font.c
> index 5e761abc5e..1bde062e87 100644
> --- a/src/font.c
> +++ b/src/font.c
> @@ -4977,6 +4977,21 @@ DEFUN ("query-font", Fquery_font, Squery_font, 1, 1, 0,
>  		 : Qnil));
>  }
>  
> +DEFUN ("font-has-glyph-p", Ffont_has_glyph_p, Sfont_has_glyph_p, 2, 2, 0,

Please call this font-has-char, since that's what it tests.

> +       doc:
> +       /* Say whether FONT has a glyph for CHAR.  */)

Not FONT, FONT-OBJECT.  They are different beasts.

> +  (Lisp_Object font_object, Lisp_Object character)
> +{
> +  struct font *font = CHECK_FONT_GET_OBJECT (font_object);
> +  CHECK_CHARACTER (character);
> +
> +  int c = XFIXNAT (character);
> +  if (font->driver->encode_char (font, c) == FONT_INVALID_CODE)
> +    return Qnil;
> +  else
> +    return Qt;
> +}

Some font backends have a has_char method, which should be used if
available in preference to encode_char.  So I think a better
implementation for this is to call font_has_char (in which case
FONT-OBJECT could also be a font-entity, something that could be
useful in some cases).

Re: Entering emojis

[Andreas Schwab]

On Okt 30 2021, Eli Zaretskii wrote:

>> +DEFUN ("font-has-glyph-p", Ffont_has_glyph_p, Sfont_has_glyph_p, 2, 2, 0,
>
> Please call this font-has-char, since that's what it tests.
>
>> +       doc:
>> +       /* Say whether FONT has a glyph for CHAR.  */)
>
> Not FONT, FONT-OBJECT.  They are different beasts.

And CHAR needs to be CHARACTER, since that's how the arguement is
spelled.

>> +  (Lisp_Object font_object, Lisp_Object character)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Entering emojis

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  stefankangas@gmail.com,
>   emacs-devel@gnu.org
> Date: Sat, 30 Oct 2021 09:06:35 +0200
> 
> On Okt 30 2021, Eli Zaretskii wrote:
> 
> >> +DEFUN ("font-has-glyph-p", Ffont_has_glyph_p, Sfont_has_glyph_p, 2, 2, 0,
> >
> > Please call this font-has-char, since that's what it tests.
> >
> >> +       doc:
> >> +       /* Say whether FONT has a glyph for CHAR.  */)
> >
> > Not FONT, FONT-OBJECT.  They are different beasts.
> 
> And CHAR needs to be CHARACTER, since that's how the arguement is
> spelled.

Yes, I've changed that not long ago.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> +DEFUN ("font-has-glyph-p", Ffont_has_glyph_p, Sfont_has_glyph_p, 2, 2, 0,
>
> Please call this font-has-char, since that's what it tests.

Sure.  The really logical name would be font-has-glyph-for-char-p, but
that's a mouthful.

>> +       doc:
>> +       /* Say whether FONT has a glyph for CHAR.  */)
>
> Not FONT, FONT-OBJECT.  They are different beasts.

Fixed.

> Some font backends have a has_char method, which should be used if
> available in preference to encode_char.  So I think a better
> implementation for this is to call font_has_char (in which case
> FONT-OBJECT could also be a font-entity, something that could be
> useful in some cases).

Yup.  This seems to make the check about 15% faster.  Now pushed to the
branch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Robert Pluim]

>>>>> On Thu, 28 Oct 2021 16:32:52 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
    >> Date: Thu, 28 Oct 2021 14:56:17 +0200
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > Why do you need to call char-displayable-p in this case?
    >> 
    >> To see whether the font has support for the emoji (and not display it).

    Eli> Why not display them?  They won't appear as Emoji if the font doesn't
    Eli> support them, but why does it matter for the feature we are
    Eli> discussing?

I guess because if thereʼs no font at all, you'll end up whatever
glyphless-char-display is configured to do. I donʼt know how likely
that is, donʼt most people have some maybe-ugly-but-functional font
installed like Symbola?

Robert
-- 

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Perhaps we could add arrow keys/TAB/C-n/C-p/etc.+RET navigation,
>> though that would just be the cherry on top.
>
> In addition to the letter input thing?

My idea is that it could be implemented right on top, yes.  I don't know
if transient.el supports this, though.

>> In addition to "C-g", perhaps it would be good to give a standard key
>> like "q" the meaning "go back to the previous screen" and maybe "Q" to
>> mean "I changed my mind, close everything and insert no emoji"?
>
> This is the first time I've used transient.el -- do transients normally
> bind `q'?

Magit only has `C-g', but I haven't used other transient.el UIs much.

My idea was in analogy with the fact that in any help buffer I can hit
"q" to quit.  OTOH maybe transient.el feels more like a minibuffer thing
than a special buffer?  So maybe we could just list the "C-g" keybinding?

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> My idea is that it could be implemented right on top, yes.  I don't know
> if transient.el supports this, though.

I don't know transient innards well, but I think it's basically just
doing a read-char-like thing.  I.e., it's modal.  So cursor movement
would be counter to the general gist of how it's supposed to work.

Possibly.  Perhaps somebody else with more Transient experience can
chime in.

> My idea was in analogy with the fact that in any help buffer I can hit
> "q" to quit.  OTOH maybe transient.el feels more like a minibuffer thing
> than a special buffer?  So maybe we could just list the "C-g" keybinding?

It kinda looks like a special buffer, but that's not the general UX of
it, I think.

Does Magit list the `C-g' binding?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I don't know transient innards well, but I think it's basically just
> doing a read-char-like thing.  I.e., it's modal.  So cursor movement
> would be counter to the general gist of how it's supposed to work.

My idea is not that you should put point in the transient window and
navigate it as a regular buffer, but that the arrow keys would highlight
different emojis somehow, and that you can then press RET to insert the
highlighted one.  In this scenario, point would stay in the window you
ran `emoji-insert' from, just as before.  If that makes sense.

So I don't think it necessarily runs contrary to the fact that
transient.el is supposed to be modal.  Or maybe I'm missing something.

> It kinda looks like a special buffer, but that's not the general UX of
> it, I think.
>
> Does Magit list the `C-g' binding?

Nope.  So it might just be me, but I can see some users not generalizing
the knowledge of "C-g works everywhere" to "surely it works at this
screen that looks quite different to everything else, too".  Maybe?
I'm not sure.

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> My idea is not that you should put point in the transient window and
> navigate it as a regular buffer, but that the arrow keys would highlight
> different emojis somehow, and that you can then press RET to insert the
> highlighted one.  In this scenario, point would stay in the window you
> ran `emoji-insert' from, just as before.  If that makes sense.

Ah, yes, that sounds like a very intuitive interface, and I don't think
adding something like that to transient.el would be hard.

>> Does Magit list the `C-g' binding?
>
> Nope.  So it might just be me, but I can see some users not generalizing
> the knowledge of "C-g works everywhere" to "surely it works at this
> screen that looks quite different to everything else, too".  Maybe?
> I'm not sure.

I was surprised, too, that `C-g' was the interface here to go "up" in
the menu hierarchy, but then I got used to it very quickly -- and I
discovered it naturally while using it.  So while it's an innovation, it
feels natural.  (And I guess the experience from Magit shows that people
like it.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> My idea is not that you should put point in the transient window and
>> navigate it as a regular buffer, but that the arrow keys would highlight
>> different emojis somehow, and that you can then press RET to insert the
>> highlighted one.  In this scenario, point would stay in the window you
>> ran `emoji-insert' from, just as before.  If that makes sense.
>
> Ah, yes, that sounds like a very intuitive interface, and I don't think
> adding something like that to transient.el would be hard.

It's already supported after (setq transient-popup-navigation-help t),
but as I mentioned in an earlier reply from today, that still has some
wards that need ironing out.

>>> Does Magit list the `C-g' binding?
>>
>> Nope.

Many of the commands that are available in all transients begin with the
"C-x" prefix.  If you press that, then those bindings are shown at the
bottom of the transient.  If you want to always show them, then you can
do that by using one of them: "C-x t".

"C-g" doesn't have a prefix because it is more important than those with
a prefix.  Despite that "C-x" shows this binding and a few other related
"sticky" commands as well.  They are called "sticky" because they are
available even after typing an incomplete key sequence, including but
not limited to "C-g".  So if you type "-" to change some argument and
then change your mind, you can press "C-g" to only quit the prefix key
but not the complete transient prefix command.

As for always showing the "C-g" binding (and the other shared bindings);
we did that in the past, but when I learned that even very experienced
users did not disable that and continued to waste some screen every time
a transient (well a "magit popup" at the time), I decided to not show it
by default anymore.

>> So it might just be me, but I can see some users not generalizing
>> the knowledge of "C-g works everywhere" to "surely it works at this
>> screen that looks quite different to everything else, too".  Maybe?
>> I'm not sure.
>
> I was surprised, too, that `C-g' was the interface here to go "up" in
> the menu hierarchy, but then I got used to it very quickly -- and I
> discovered it naturally while using it.  So while it's an innovation, it
> feels natural.  (And I guess the experience from Magit shows that people
> like it.)

That seems to be generally true, but I surely heard from a handful of
people who didn't like it at all when I switched from "q" (as used by
the predecessor (magit-popup.el) to "C-g".  Anyway the reason is this:

I wanted *all* alphabetic characters to be available for used by
prefix-specific suffixes, without their authors having to worry about
shadowing a binding that is supposed to be available in all transients,
especially not such an important one.

In the case of emoji it would be weird if only [a-pr-z] could be used.
Also some prefixes might have some suffix that is about "quitting"
something that hasn't anything to do with the transient interface but
with its specific functionality.  The magit-blame transient prefix for
example binds "q" to "quit blaming", i.e. it removes blame information
from the current buffer.

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 28 Oct 2021 06:33:55 -0400
> 
> BTW, when I insert this emoji "🧙🏿‍♀️" and then hit DEL repeatedly, the
> buffer text first turns into "🧙🏿‍♀" and then "🧙🏿", and then "🧙🏿"
> and then "🧙".  Is that a known issue?

It's hard to tell.  You show the sequences of codepoints, and
presumably expect everyone who reads the mail to see the same display
as on your machine?  But that is not a reliable assumption, especially
if there's some bug involved.  So images would be much better.

In general, we allow to use DEL to delete the codepoints that were
composed into a single grapheme cluster, one codepoint at a time, and
that's a feature, because otherwise it would be very cumbersome to
convert one sequence into a similar but different one: you'd need to
delete everything and retype from scratch.  If that is what you see,
then yes, it's expected and intentional behavior, not a bug.

Re: Entering emojis

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> In general, we allow to use DEL to delete the codepoints that were
> composed into a single grapheme cluster, one codepoint at a time, and
> that's a feature, because otherwise it would be very cumbersome to
> convert one sequence into a similar but different one: you'd need to
> delete everything and retype from scratch.  If that is what you see,
> then yes, it's expected and intentional behavior, not a bug.

Aha.  Yes, that does sound like what I see.

I fully trust that you know more than me about what makes sense in
general, and I have no opinion about that.

But in the specific case of emojis, I would have expected that the
emojis just get deleted in one go, as that is what happens in other
software where I use emojis.  (Mostly chat programs on my phone.)

Would it be possible and make sense to handle emojis differently from
other grapheme clusters in that regard?  The desire not to retype them
from scratch might be less of a concern now that we have `emoji-insert'.

Re: Entering emojis

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> But in the specific case of emojis, I would have expected that the
> emojis just get deleted in one go, as that is what happens in other
> software where I use emojis.  (Mostly chat programs on my phone.)

Yes, I think it would make sense for commands like this to treat these
grapheme clusters like a single object.  (It might not make sense for
other grapheme clusters, though?)

I just checked what Macos does -- I opened the Notepad app and found the
emoji input method (after googling; it's on control-command-space).  I
inserted one of the grapheme cluster ones, and DEL deleted the entire
thing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 29 Oct 2021 01:48:36 +0200
> 
> I just checked what Macos does -- I opened the Notepad app and found the
> emoji input method (after googling; it's on control-command-space).  I
> inserted one of the grapheme cluster ones, and DEL deleted the entire
> thing.

Since when do we want to be bug-compatible to silly programs like
Notepad?

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 28 Oct 2021 16:37:55 -0700
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> But in the specific case of emojis, I would have expected that the
> emojis just get deleted in one go, as that is what happens in other
> software where I use emojis.  (Mostly chat programs on my phone.)
> 
> Would it be possible and make sense to handle emojis differently from
> other grapheme clusters in that regard?  The desire not to retype them
> from scratch might be less of a concern now that we have `emoji-insert'.

First, we don't yet have emoji-insert, it's only on master (which I
think is a mistake, given the significant improvements in this are we
will have in Emacs 28; but I digress).

And second, the current behavior could be quite useful in the Emoji
case as well.  For example, consider the case where you typed the male
version and you want to change it to the female version instead.  Or
when you copy-paste the Emoji from some other text, then want to
change it in small ways.

The behavior you suggest could be a user option, by default off, and
not specific to Emoji.  Changing the behavior unconditionally, or even
making that the default of that option, makes no sense to me, since
the current behavior is very old, and I never heard any complaints
about it.  making it specific to Emoji is possible, but sub-optimal,
because it would require the user to remember what kind of composition
sequence he/she is typing; and many users would have a difficulty to
know whether a given composed sequence is considered Emoji or not,
given that more and more non-Emoji characters can have "Emoji
presentation form" requested by appending VS-16.

And I submit that, as in many other situations in Emacs development,
we are jumping too fast to conclusions.  Here's a new feature, fresh
out of the oven, not even 100% complete yet, and we bump into behavior
we never noticed before, and immediately want to change it because it
surprises us.  Why not hold our horses until we have _some_ experience
with the new feature, and then consider this and other aspects with
some perspective in mind?  What's the rush to make changes in
unrelated functionalities just because we were surprised by a TIL-like
phenomenon, with hardly a few keystrokes of experience under our
belts?

Re: Entering emojis

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> First, we don't yet have emoji-insert, it's only on master

I suggest a change on master.

> And second, the current behavior could be quite useful in the Emoji
> case as well.  For example, consider the case where you typed the male
> version and you want to change it to the female version instead.  Or
> when you copy-paste the Emoji from some other text, then want to
> change it in small ways.
[...]
> The behavior you suggest could be a user option, by default off, and
> not specific to Emoji.  Changing the behavior unconditionally, or even
> making that the default of that option, makes no sense to me, since
> the current behavior is very old, and I never heard any complaints
> about it.

Please note my complaint then.  Although I'd like to not only complain
but also more constructively propose a solution: I think that
`delete-backward-char' should treat emojis as single characters by
default.

I find the current behavior far from ideal, and note that it is
different from all other software I have seen.  I also think it makes no
sense.  Having to type DEL five times to remove an emoji is to my mind
clearly not the interface that most users will want.

The UI we have now, and that you claim is superior, is much more fiddly
and error-prone to get right than just running `DEL C-x 8 e' again.
Emojis are similar enough to be tricky to tell apart that I can't see
myself ever being interested in trying to do detailed modification of an
emoji grapheme cluster by hand.  YMMV.

> And I submit that, as in many other situations in Emacs development,
> we are jumping too fast to conclusions.

I agree with the logic in your reasoning as a general proposition, but
the above points still stand in this particular case.

> What's the rush to make changes in unrelated functionalities just
> because we were surprised by a TIL-like phenomenon, with hardly a few
> keystrokes of experience under our belts?

I see no particular rush to fix this.

As soon as emoji-insert starts being more widely used, I expect that
someone will report a bug about this (or complain on IRC and on
Reddit, and perhaps write up a third-party package
"fix-delete-emoji.el", etc.).

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 29 Oct 2021 07:43:05 -0700
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> > The behavior you suggest could be a user option, by default off, and
> > not specific to Emoji.  Changing the behavior unconditionally, or even
> > making that the default of that option, makes no sense to me, since
> > the current behavior is very old, and I never heard any complaints
> > about it.
> 
> Please note my complaint then.  Although I'd like to not only complain
> but also more constructively propose a solution: I think that
> `delete-backward-char' should treat emojis as single characters by
> default.

Would it be good enough to use C-d (or <Delete>) to do that?  Those
already delete the entire composed sequence.  The behavior of DEL is
special and intentional.

> As soon as emoji-insert starts being more widely used, I expect that
> someone will report a bug about this (or complain on IRC and on
> Reddit, and perhaps write up a third-party package
> "fix-delete-emoji.el", etc.).

Let's wait and see, okay?  I'm not saying I know there will be no
complaints, I'm saying we will be able to devise a better solution
if/when we have those complaints, because hopefully they will provide
additional information.

Re: Entering emojis

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Would it be good enough to use C-d (or <Delete>) to do that?  Those
> already delete the entire composed sequence.  The behavior of DEL is
> special and intentional.
[...]
> Let's wait and see, okay?  I'm not saying I know there will be no
> complaints, I'm saying we will be able to devise a better solution
> if/when we have those complaints, because hopefully they will provide
> additional information.

Sure, that's fine by me.  Thanks.

PS. I'm seeing some other (IMHO) unexpected decomposition when using C-d
    to delete "🧙🏿‍♀️".  But if we intend to look into all this later, it
    should be fine to leave that for later too.

Re: Entering emojis

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 29 Oct 2021 10:01:46 -0700
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Would it be good enough to use C-d (or <Delete>) to do that?  Those
> > already delete the entire composed sequence.  The behavior of DEL is
> > special and intentional.
> [...]
> > Let's wait and see, okay?  I'm not saying I know there will be no
> > complaints, I'm saying we will be able to devise a better solution
> > if/when we have those complaints, because hopefully they will provide
> > additional information.
> 
> Sure, that's fine by me.  Thanks.

It turned out we never had a feature like that, so I have now
installed a change whereby the <Delete> function key (but not C-d nor
DEL!) deletes by grapheme clusters when used to delete forward,
i.e. either without any prefix arg or with a positive prefix arg.

Re: Entering emojis

[Daniel Martín]

Stefan Kangas <stefankangas@gmail.com> writes:

>
>> And second, the current behavior could be quite useful in the Emoji
>> case as well.  For example, consider the case where you typed the male
>> version and you want to change it to the female version instead.  Or
>> when you copy-paste the Emoji from some other text, then want to
>> change it in small ways.
> [...]
>> The behavior you suggest could be a user option, by default off, and
>> not specific to Emoji.  Changing the behavior unconditionally, or even
>> making that the default of that option, makes no sense to me, since
>> the current behavior is very old, and I never heard any complaints
>> about it.
>
> Please note my complaint then.  Although I'd like to not only complain
> but also more constructively propose a solution: I think that
> `delete-backward-char' should treat emojis as single characters by
> default.
>
> I find the current behavior far from ideal, and note that it is
> different from all other software I have seen.  I also think it makes no
> sense.  Having to type DEL five times to remove an emoji is to my mind
> clearly not the interface that most users will want.

On one hand, the Unicode spec suggests that Emojis should be treated as
single grapheme clusters, not only for cursor movement, but for editing
operations as well.  From https://www.unicode.org/reports/tr51:

"A supported emoji modifier sequence should be treated as a single
grapheme cluster for editing purposes (cursor moment, deletion, and so
on); word break, line break, and so on."

On the other hand, one of the most popular text editors, VSCode, doesn't
follow these suggestions and follows the Emacs behavior: DEL deletes a
code point, and <Delete> deletes the entire grapheme cluster.
LibreOffice also deletes emojis like Emacs.

Re: Entering emojis

[Lars Ingebrigtsen]

Daniel Martín <mardani29@yahoo.es> writes:

> On the other hand, one of the most popular text editors, VSCode, doesn't
> follow these suggestions and follows the Emacs behavior: DEL deletes a
> code point, and <Delete> deletes the entire grapheme cluster.
> LibreOffice also deletes emojis like Emacs.

So perhaps we should just keep the current behaviour (or have a user
option to change it).

Two more data points: Pages (the Word equivalent in Macos) deletes the
entire cluster when you hit the key named "delete", and so does the
editor in XCode.  But then again, those machines don't have two separate
deletion keys.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So perhaps we should just keep the current behaviour (or have a user
> option to change it).

A more ticklish question is what we should do with out string primitives
(if anything).  For instance:

(string-limit "Hello, 👨🏽‍❤️‍💋‍👨🏾" 8)
=> "Hello, 👨"

and

(truncate-string-to-width "👨🏽‍❤️‍💋‍👨🏾" 2 nil t)
=> "👨"

which is...  uhm...  In a way, this grapheme cluster thing is slightly
like it was during the shift to utf-8, when not all string primitives
worked on characters, but bytes instead.  Less dramatic, of course, but.

I think we'll be seeing many amusing display glitches in this area.  🥲

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stefan Kangas <stefankangas@gmail.com>,
>   emacs-devel@gnu.org
> Date: Sat, 30 Oct 2021 00:16:39 +0200
> 
> A more ticklish question is what we should do with out string primitives
> (if anything).  For instance:
> 
> (string-limit "Hello, 👨🏽‍❤️‍💋‍👨🏾" 8)
> => "Hello, 👨"
> 
> and
> 
> (truncate-string-to-width "👨🏽‍❤️‍💋‍👨🏾" 2 nil t)
> => "👨"

Nothing, they should "just work", barring bugs.

What does string-width return for this string on your system?

> which is...  uhm...  In a way, this grapheme cluster thing is slightly
> like it was during the shift to utf-8, when not all string primitives
> worked on characters, but bytes instead.  Less dramatic, of course, but.
> 
> I think we'll be seeing many amusing display glitches in this area.  🥲

We shouldn't, because string-width already supports composed text.
There's always one more bug, of course, but there are no design
problems here, AFAICT.  Emoji sequences are just one special case of
character composition, that's all.  So this is nothing like a switch
to multibyte characters, at least not in theory.

Re: Entering emojis

[Eli Zaretskii]

> Date: Sat, 30 Oct 2021 09:19:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, stefankangas@gmail.com, mardani29@yahoo.es
> 
> > (truncate-string-to-width "👨🏽‍❤️‍💋‍👨🏾" 2 nil t)
> > => "👨"
> 
> Nothing, they should "just work", barring bugs.
> 
> What does string-width return for this string on your system?
> 
> > which is...  uhm...  In a way, this grapheme cluster thing is slightly
> > like it was during the shift to utf-8, when not all string primitives
> > worked on characters, but bytes instead.  Less dramatic, of course, but.
> > 
> > I think we'll be seeing many amusing display glitches in this area.  🥲
> 
> We shouldn't, because string-width already supports composed text.
> There's always one more bug, of course, but there are no design
> problems here, AFAICT.

And I see that there is, indeed, a bug (or a missing feature) in
truncate-string-to-width: its algorithm assumes that string-width
returns a number that is the sum of char-width values for its
constituent characters, which is not necessarily true when
character-composition is involved.  It needs instead to consider
string-width values on subsequent substrings.  It also cannot assume
that string-width is monotonically increasing in the number of
characters in the substring, as that, too, could be false when
character-composition is involved.

Again, this is nothing specific to Emoji, this can happen with any
composed text, for example any ligature that produces a single glyph
from 2 or more characters.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> And I see that there is, indeed, a bug (or a missing feature) in
> truncate-string-to-width: its algorithm assumes that string-width
> returns a number that is the sum of char-width values for its
> constituent characters, which is not necessarily true when
> character-composition is involved.

(I should have read the entire thread before responding to the previous
email, I see.)

> It needs instead to consider string-width values on subsequent
> substrings.  It also cannot assume that string-width is monotonically
> increasing in the number of characters in the substring, as that, too,
> could be false when character-composition is involved.

Yup.

> Again, this is nothing specific to Emoji, this can happen with any
> composed text, for example any ligature that produces a single glyph
> from 2 or more characters.

I was idly speculating whether the current truncate-string-to-width
behaviour is "right" in some sense, and we'd have to introduce some new
primitives here, but I guess not.

But perhaps we should have one new primitive here -- something like
string-glyph-split -- that would split a string into a list of strings
where each string represents a complete glyph.  I think that might be
useful for people who are composing displays by chopping things off the
start/end of strings, for instance.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org,  stefankangas@gmail.com,  mardani29@yahoo.es
> Date: Sat, 30 Oct 2021 14:25:47 +0200
> 
> But perhaps we should have one new primitive here -- something like
> string-glyph-split -- that would split a string into a list of strings
> where each string represents a complete glyph.

You mean, a complete grapheme cluster, I presume.

That could be done, I think, by using the information returned by the
composition stuff; see find-composition.

Re: Entering emojis

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> But perhaps we should have one new primitive here -- something like
>> string-glyph-split -- that would split a string into a list of strings
>> where each string represents a complete glyph.
>
> You mean, a complete grapheme cluster, I presume.

Well, the grapheme cluster represents a glyph.  :-)

> That could be done, I think, by using the information returned by the
> composition stuff; see find-composition.

Yup.  And then we'd change truncate-string-to-width to be based on that
new function, or change truncate-string-to-width in a similar way?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org,  stefankangas@gmail.com,  mardani29@yahoo.es
> Date: Sat, 30 Oct 2021 14:34:10 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That could be done, I think, by using the information returned by the
> > composition stuff; see find-composition.
> 
> Yup.  And then we'd change truncate-string-to-width to be based on that
> new function, or change truncate-string-to-width in a similar way?

Something like that, I didn't invest enough thought in a possible
implementation.

Re: Entering emojis

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> (string-limit "Hello, 👨🏽‍❤️‍💋‍👨🏾" 8)
>> => "Hello, 👨"
>> 
>> and
>> 
>> (truncate-string-to-width "👨🏽‍❤️‍💋‍👨🏾" 2 nil t)
>> => "👨"
>
> Nothing, they should "just work", barring bugs.

Well...  they don't "just work" now, as the example shows.  But perhaps
you're not seeing the correct glyphs here?  Here's a screenshot.


[-- Attachment #2: Type: image/png, Size: 6824 bytes --]

[-- Attachment #3: Type: text/plain, Size: 191 bytes --]



> What does string-width return for this string on your system?

2 for both strings.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stefan Kangas <stefankangas@gmail.com>,
>   emacs-devel@gnu.org
> Date: Sat, 30 Oct 2021 00:16:39 +0200
> 
> (string-limit "Hello, 👨🏽‍❤️‍💋‍👨🏾" 8)
> => "Hello, 👨"

string-limit cannot be used with strings that have wide characters and
characters that are composed on display, if the result is supposed to
be correct on display.  string-limit assumes each character occupies 1
column on display, which is not true in those two cases.  It's the
same problem as with using 'length' when you want to know how many
columns will a string occupy on display.

Re: Entering emojis

[Howard Melman]

> Two more data points: Pages (the Word equivalent in Macos) deletes the
> entire cluster when you hit the key named "delete", and so does the
> editor in XCode.  But then again, those machines don't have two separate
> deletion keys.

FWIW, mac laptops only have one key labeled "delete" (in a
position many other keyboards label "backspace"), but,
full-sized mac external keyboards (like I use with my iMac)
have two deletion keys, both labeled "delete" but one also
has an icon suggesting forward deletion.  I believe on mac
laptops you can emulate the forward delete via fn-delete.

For me in Pages both keys delete the entire cluster.

-- 

Howard

Re: Entering emojis

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1112 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:


Nice!

Perhaps we already get what I am about to say below from what you have,
but given that there are 256 variation selectors, it might be useful to
have a bigram/trigram model for chars that can meaningfully compose as
an emoji?

Once we had such a bigram/trigram model, the choices could be
progressively filtered without creating a giant list of choices  ---
this would permit the user to focus in on the   emojis that make sense
as an emoji.
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> https://unicode.org/Public/emoji/14.0/emoji-test.txt
>>
>> So I'll be rewriting that bit tomorrow.  That file also seems to allow
>> me to easy make the derivation groups without parsing all the other
>> files, so it should be faster, too.  So...  better all around, I think.
>
> Yup.  This is now implemented, and it's way less hacky, and I seem to be
> able to get all the derivations I'd expect (and I don't see any false
> positives).

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Entering emojis

[Lars Ingebrigtsen]

"T.V Raman" <raman@google.com> writes:

> Perhaps we already get what I am about to say below from what you have,
> but given that there are 256 variation selectors, it might be useful to
> have a bigram/trigram model for chars that can meaningfully compose as
> an emoji?

That's not really how the emojis work -- Unicode publishes a document
that says which characters (should) have glyphs for various variations
(mostly in the skin tone/gender area).  The interface lets you navigate
these things.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 07:19:55 -0700
> 
> Perhaps we already get what I am about to say below from what you have,
> but given that there are 256 variation selectors, it might be useful to
> have a bigram/trigram model for chars that can meaningfully compose as
> an emoji?

AFAIK, only 4 variation selectors actually have any effect in the
available fonts (plus VS-15 and VS-16, which just select text or Emoji
appearance).

Re: Entering emojis

[Jonas Bernoulli]

Lars,

I just saw your direct message and have indeed not seen this thread
before.  Seems I still don't know how to do email...

This thread is very much in line with what I have been doing the last
weeks.  The plan was to:

1. Take care of all the longstanding task not related to transient.el
2. Address longstanding issues in transient.el
3. Add new transient features I wanted to implement for a long time
4. Add new (slightly less important) transient features people are
   asking for

But a lot of new feature and support requests related to transient.el
are coming in right now, so I ended up working that list from the back,
while trying to throw in a few of the important changes/fixes along the
way. :P    <--- I would like another transient for ascii smileys too.

I have now skimmed the messages in this thread that contain the string
"transient" and am going to reply soon, though probably not until
tomorrow evening.

     Cheers,
     Jonas

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas Bernoulli <jonas@bernoul.li> writes:

> But a lot of new feature and support requests related to transient.el
> are coming in right now, so I ended up working that list from the back,
> while trying to throw in a few of the important changes/fixes along the
> way. :P    <--- I would like another transient for ascii smileys too.

😄

> I have now skimmed the messages in this thread that contain the string
> "transient" and am going to reply soon, though probably not until
> tomorrow evening.

Sure, no problem.  I'm just itching to push the emoji stuff to master,
but it depends on the transient change.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Entering emojis

[Jonas Bernoulli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> 😄

👍🏼 Like it a lot!

Lately I was actually occasionally wishing this existed.

Let's start with the nitpicking:
- The section description is in some cases displayed at the bottom.
- That's it.

It's great to see something added to Emacs that uses Transient.
Thanks for that!  And for the variable-pitch functionality.

Transient has some support for dynamically generating the suffixes of
some prefix using the `setup-children' function, see notmuch-transient
for an example (https://git.sr.ht/~tarsius/notmuch-transient).
the result.

You maybe could use that approach too, at least for the top-level prefix
commands.  I haven't tried using this for sub-prefixes though.  In the
notmuch case the suffixes are recreated every time but you could of
course cache the result.

Ah, actually I ran into one more issue.  When I used describe-function
to jump to transient-define-prefix to compare it to the code you copied
from there I got a lot of new completion candidates for various
generated suffix commands that are not intended to invoked directly (and
whose name begin with "transient:", which is transient's convention for
"anonymous" suffixes but weird I admit).

Transient actually tries to prevent that using a *supper weird* hack (I
would rather not explain in detail), but you bypass when you generate
the suffixes.  This hack isn't actually necessary anymore starting with
Emacs 28, but I haven't implemented the new approach yet, though it
should be trivial:

I intend to use (put SUFFIX 'modes '(transient-no-mode)) to do the M-x
hiding going forward, which reminds me: it might be a good idea to
officially support and document some value that indicates that a command
should *never* be shown by M-x.  Careful readers might be confused by
the use of a mode that doesn't actually exist and it would be good if
not every package author who wants to do this has to define some "ceci
n'est pas une mode" variable and/or function to have a doc-string to
explain what is going on.

And since I am going a bit off-topic anyway, let's take it all the way
with a question that is only related in that it came up in the context
of another package that also uses transient:

Is there some generic way to turn a function that reads a single value
in the minibuffer into one that reads multiple values, similar to what
completing-read-multiple does but which isn't limited to a set of known
choices.  The function I have in mind is read-file-name-multiple (which
is supposed to go beyond merely supporting wildcards, supporting input
like "foo.txt,~/bin/bar.sh").

Magit actually uses completing-read-multiple for its "--" argument used
to limit logs and diffs to one or more files.  It can use do that
because the set of choices is known and limited to files tracked in the
current repository.  This other package needs to read multiple files
that are scattered all across the file system.

  🛌 time! Signing off.
  Jonas

Reading multiple files (was: Entering emojis)

[Stefan Monnier]

> Is there some generic way to turn a function that reads a single value
> in the minibuffer into one that reads multiple values, similar to what
> completing-read-multiple does but which isn't limited to a set of known
> choices.  The function I have in mind is read-file-name-multiple (which
> is supposed to go beyond merely supporting wildcards, supporting input
> like "foo.txt,~/bin/bar.sh").

Have you tried (completing-read-multiple "Foo: " #'completion-file-name-table)?


        Stefan

Re: Reading multiple files (was: Entering emojis)

[Jonas Bernoulli]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Is there some generic way to turn a function that reads a single value
>> in the minibuffer into one that reads multiple values, similar to what
>> completing-read-multiple does but which isn't limited to a set of known
>> choices.  The function I have in mind is read-file-name-multiple (which
>> is supposed to go beyond merely supporting wildcards, supporting input
>> like "foo.txt,~/bin/bar.sh").
>
> Have you tried (completing-read-multiple "Foo: " #'completion-file-name-table)?

Two reactions:
🥴 Duh! 
🤩 Isn't Emacs just wonderful?!

     Have a 🍩
     Jonas

Re: Entering emojis

[Lars Ingebrigtsen]

Jonas Bernoulli <jonas@bernoul.li> writes:

> Ah, actually I ran into one more issue.  When I used describe-function
> to jump to transient-define-prefix to compare it to the code you copied
> from there I got a lot of new completion candidates for various
> generated suffix commands that are not intended to invoked directly (and
> whose name begin with "transient:", which is transient's convention for
> "anonymous" suffixes but weird I admit).

Yes, I just cargo-culted stuff there, and it should be cleaned up.  Do
we really have to put these "commands" into the obarray at all?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Problem with some new emoji in Emacs

[Jean Louis]

I am delighted with the new built in feature of emoji.

Until now I have always used `emojify-mode' which worked well.

I am encountering a disturbing problem, this could be some error,
maybe not. I cannot know.

Here is the screen with new emoji feature:
https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png

When I turn on the emojify-mode then I get this problem here:
https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png

and I will try to insert it here:
     3 🗂️ ║ Central Files
     10  🖍️ 

As I have never encountered this problem before in emojify-mode, I
think this is error. The error will be seen when emojify-mode is
turned on. I am using latest development Emacs.

Maybe it is problem of emojify-mode, maybe it is problem of this
latest Emacs changes.

The error does not appear to all emojis.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Problem with some new emoji in Emacs

[Eli Zaretskii]

> Date: Sat, 13 Nov 2021 22:43:41 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: emacs-devel@gnu.org
> 
> I am delighted with the new built in feature of emoji.
> 
> Until now I have always used `emojify-mode' which worked well.
> 
> I am encountering a disturbing problem, this could be some error,
> maybe not. I cannot know.
> 
> Here is the screen with new emoji feature:
> https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png
> 
> When I turn on the emojify-mode then I get this problem here:
> https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png
> 
> and I will try to insert it here:
>      3 🗂️ ║ Central Files
>      10  🖍️ 
> 

Looks like emojify-mode is incompatible with Emacs 28 and later
display of Emoji.  It uses static compositions, whereas the built-in
Emoji support uses automatic compositions, which is a different
feature.  Does emojify-mode support Variation Selectors?  That's what
gives birth to those squares you see on the second screenshot.

Why do you need emojify-mode?  What does it do that Emacs cannot do
without it?

Re: Problem with some new emoji in Emacs

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-11-13 23:17]:
> > Date: Sat, 13 Nov 2021 22:43:41 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: emacs-devel@gnu.org
> > 
> > I am delighted with the new built in feature of emoji.
> > 
> > Until now I have always used `emojify-mode' which worked well.
> > 
> > I am encountering a disturbing problem, this could be some error,
> > maybe not. I cannot know.
> > 
> > Here is the screen with new emoji feature:
> > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png
> > 
> > When I turn on the emojify-mode then I get this problem here:
> > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png
> > 
> > and I will try to insert it here:
> >      3 🗂️ ║ Central Files
> >      10  🖍️ 
> > 
> 
> Looks like emojify-mode is incompatible with Emacs 28 and later
> display of Emoji.  It uses static compositions, whereas the built-in
> Emoji support uses automatic compositions, which is a different
> feature.  Does emojify-mode support Variation Selectors?  That's what
> gives birth to those squares you see on the second screenshot.

It may be, I cannot know about it. I will inform also Iqbal, who is
author of it.

;; Author: Iqbal Ansari
;; URL: https://github.com/iqbalansari/emacs-emojify

> Why do you need emojify-mode?  What does it do that Emacs cannot do
> without it?

To get emoji working as in emojify mode, I had to replace some of
them. This new emoji mode in Emacs did not support everything to what
I was used to in emojify mode, so I had to replace some symbols
looking similar to each other but being different, or I have to drop
using them. Example is HEAVY CHECK MARK ✔ which appears larger and
more significant in emojify mode.

When I use describe-char I can see that ✔ and ✔️ are same now on my
screen but they don't activate, so the first one I see smaller, second
one emojified.

This is because I use C-x 8 RET HEAVY CHECK MARK to inser the first
one.

In emojify-mode when I insert the char, it becomes automatically
recognized. In this new feature I have to use different C-X 8 e i
notion to insert it.

I have closed this file, opened it again, and again I see difference
between ✔ and ✔️ here visually, but describe-char tells me about
different fonts used.

Thus largest difference is that emojify-mode is automatic. And I
cannot understand why is this char not recognized in new feature of
Emacs, though it tells it is same char.

Screenshot:
https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-23:49:09.png


First one being:

             position: 2201 of 3363 (65%), column: 8
            character: ✔ (displayed as ✔) (codepoint 10004, #o23424, #x2714)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x2714
               script: symbol
               syntax: w 	which means: word
             category: .:Base
             to input: type "C-x 8 RET 2714" or "C-x 8 RET HEAVY CHECK MARK"
          buffer code: #xE2 #x9C #x94
            file code: #xE2 #x9C #x94 (encoded by coding system utf-8-unix)
              display: by this font (glyph code):
    ftcrhb:-PfEd-DejaVu Sans Mono-medium-normal-normal-*-18-*-*-*-m-0-iso10646-1 (#xAFC)

Character code properties: customize what to show
  name: HEAVY CHECK MARK
  general-category: So (Symbol, Other)
  decomposition: (10004) ('✔')

There are text properties here:
  fontified            t
  pabbrev-added        t

[back]

and second

             position: 2207 of 2285 (97%), column: 14
            character: ✔ (displayed as ✔) (codepoint 10004, #o23424, #x2714)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x2714
               script: symbol
               syntax: w 	which means: word
             category: .:Base
             to input: type "C-x 8 RET 2714" or "C-x 8 RET HEAVY CHECK MARK"
          buffer code: #xE2 #x9C #x94
            file code: #xE2 #x9C #x94 (encoded by coding system utf-8-unix)
              display: composed to form "✔️" (see below)

Composed with the following character(s) "️" using this font:
  ftcrhb:-GOOG-Noto Color Emoji-medium-normal-normal-*-18-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 10004 152 22 0 23 17 5 nil]
with these character(s):
  ️ (#xfe0f) VARIATION SELECTOR-16

Character code properties: customize what to show
  name: HEAVY CHECK MARK
  general-category: So (Symbol, Other)
  decomposition: (10004) ('✔')

There are text properties here:
  fontified            t
  pabbrev-added        t

[back]


Jean

Re: Problem with some new emoji in Emacs

[Eli Zaretskii]

> Date: Sat, 13 Nov 2021 23:49:35 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> When I use describe-char I can see that ✔ and ✔️ are same now on my
> screen but they don't activate

What do you mean by "don't activate"?

> so the first one I see smaller, second one emojified.

Are you sure this is unrelated to emojify-mode?  The second one
requests an Emoji-style presentation, the first one doesn't.

> In emojify-mode when I insert the char, it becomes automatically
> recognized.

What do you mean by "recognized"?

> I have closed this file, opened it again, and again I see difference
> between ✔ and ✔️ here visually, but describe-char tells me about
> different fonts used.

That's what should happen: the U+FE0F VARIATION SELECTOR 16 character
requests the preceding character to be displayed as Emoji.  So what
you see is Emacs 28 and later behaving as designed, and as the Unicode
Standard says it should behave.

Re: Problem with some new emoji in Emacs

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-11-14 09:36]:
> > Date: Sat, 13 Nov 2021 23:49:35 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: larsi@gnus.org, emacs-devel@gnu.org
> > 
> > When I use describe-char I can see that ✔ and ✔️ are same now on my
> > screen but they don't activate
> 
> What do you mean by "don't activate"?

They don't show up, even those I look now above don't show up same,
left one is smaller check mark, right one is emojified.

This has been explained in bug report that there is char FE0F after ✔
and that is where it becomes recognized and emojified ✔ and ✔️ are not
same as I have done C-x 8 RET FE0F right after ✔ and have understood
how and why it works.

> That's what should happen: the U+FE0F VARIATION SELECTOR 16 character
> requests the preceding character to be displayed as Emoji.  So what
> you see is Emacs 28 and later behaving as designed, and as the Unicode
> Standard says it should behave.

Thanks! 👏


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Problem with some new emoji in Emacs

[Stefan Monnier]

> Here is the screen with new emoji feature:
> https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png
>
> When I turn on the emojify-mode then I get this problem here:
> https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png

Does this look worse than if you use Emacs-27 to view the exact same text?


        Stefan

Re: Problem with some new emoji in Emacs

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2021-11-13 23:38]:
> > Here is the screen with new emoji feature:
> > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png
> >
> > When I turn on the emojify-mode then I get this problem here:
> > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png
> 
> Does this look worse than if you use Emacs-27 to view the exact same
> text?

I never had any problem until this new feature with emoji came. I am
trying to switch to the new built in feature. But I have problems
understanding the difference between "✔" and "✔️" as I see them
different on screen and describe-char tells me they are some, opening
file again shows me different. emojify-mode shows me error after the
second heavy check mark.

Opening the file in different editor shows me emoji on the right side,
no emoji on left side, just as in Emacs. This is probably expected,
but I cannot inspect it to see the difference.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Problem with some new emoji in Emacs

[Jean Louis]

* Jean Louis <bugs@gnu.support> [2021-11-13 23:58]:
> * Stefan Monnier <monnier@iro.umontreal.ca> [2021-11-13 23:38]:
> > > Here is the screen with new emoji feature:
> > > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:19.png
> > >
> > > When I turn on the emojify-mode then I get this problem here:
> > > https://gnu.support/files/tmp/2021-11-13/Screenshot-2021-11-13-22:33:36.png
> > 
> > Does this look worse than if you use Emacs-27 to view the exact same
> > text?
> 
> I never had any problem until this new feature with emoji came. I am
> trying to switch to the new built in feature. But I have problems
> understanding the difference between "✔" and "✔️" as I see them
> different on screen and describe-char tells me they are some, opening
> file again shows me different. emojify-mode shows me error after the
> second heavy check mark.

I got it now, after reading the bug issue.



Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/