bug#46240: Sorting order of read-char-by-name

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

Tags: patch

After typing e.g. 'C-x 8 RET GREEK TAB' completions are sorted in an
non-alphabetical order:

Ͳ	GREEK CAPITAL LETTER ARCHAIC SAMPI
Β	GREEK CAPITAL LETTER BETA
Χ	GREEK CAPITAL LETTER CHI
Ϯ	GREEK CAPITAL LETTER DEI
Δ	GREEK CAPITAL LETTER DELTA

where the 22nd letter of the Greek alphabet CHI is between BETA and DELTA.
This is because currently completions are sorted by English names.

The following patch sorts them by their Unicode order that mostly follows
the alphabetical order, and at least makes more sense to be consistent
with Unicode tables where characters are grouped more logically.
Then the above example is sorted alphabetically:

Α	GREEK CAPITAL LETTER ALPHA
Β	GREEK CAPITAL LETTER BETA
Γ	GREEK CAPITAL LETTER GAMMA
Δ	GREEK CAPITAL LETTER DELTA
Ε	GREEK CAPITAL LETTER EPSILON

More examples with better sorting is 'C-x 8 RET SUBSCRIPT TAB'
where instead of sorting by names of numbers:

₈	SUBSCRIPT DIGIT EIGHT
₅	SUBSCRIPT DIGIT FIVE
₄	SUBSCRIPT DIGIT FOUR
₉	SUBSCRIPT DIGIT NINE
₁	SUBSCRIPT DIGIT ONE
₇	SUBSCRIPT DIGIT SEVEN
₆	SUBSCRIPT DIGIT SIX

will be sorted by numerical order:

₀	SUBSCRIPT DIGIT ZERO
₁	SUBSCRIPT DIGIT ONE
₂	SUBSCRIPT DIGIT TWO
₃	SUBSCRIPT DIGIT THREE
₄	SUBSCRIPT DIGIT FOUR
₅	SUBSCRIPT DIGIT FIVE

etc.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mule--ucs-names-sort-by-char.patch --]
[-- Type: text/x-diff, Size: 1096 bytes --]

diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 5dc3de4422..0593d72279 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -3083,6 +3083,12 @@ mule--ucs-names-affixation
               (list name (concat (if char (format "%c" char) " ") "\t") "")))
           names))
 
+(defun mule--ucs-names-sort-by-char (names)
+  (let* ((chars-and-names
+          (mapcar (lambda (name) (cons (gethash name ucs-names) name)) names))
+         (sorted (sort chars-and-names (lambda (a b) (< (car a) (car b))))))
+    (mapcar #'cdr sorted)))
+
 (defun char-from-name (string &optional ignore-case)
   "Return a character as a number from its Unicode name STRING.
 If optional IGNORE-CASE is non-nil, ignore case in STRING.
@@ -3132,6 +3138,7 @@ read-char-by-name
 	     (if (eq action 'metadata)
 		 '(metadata
 		   (affixation-function . mule--ucs-names-affixation)
+		   (display-sort-function . mule--ucs-names-sort-by-char)
 		   (category . unicode-name))
 	       (complete-with-action action (ucs-names) string pred)))))
 	 (char

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Mon, 01 Feb 2021 19:23:41 +0200
> 
> After typing e.g. 'C-x 8 RET GREEK TAB' completions are sorted in an
> non-alphabetical order:
> 
> Ͳ	GREEK CAPITAL LETTER ARCHAIC SAMPI
> Β	GREEK CAPITAL LETTER BETA
> Χ	GREEK CAPITAL LETTER CHI
> Ϯ	GREEK CAPITAL LETTER DEI
> Δ	GREEK CAPITAL LETTER DELTA
> 
> where the 22nd letter of the Greek alphabet CHI is between BETA and DELTA.
> This is because currently completions are sorted by English names.
> 
> The following patch sorts them by their Unicode order that mostly follows
> the alphabetical order, and at least makes more sense to be consistent
> with Unicode tables where characters are grouped more logically.

This has 2 disadvantages:

  . the user needs to know the order of characters within a script
    he/she doesn't necessarily read
  . the user needs to know the order _between_ scripts, if the
    candidates include characters from different Unicode blocks

If the user doesn't know this order, he/she might be unable to find
the required character quickly, if the list of candidates is long
enough.

The current order, while it doesn't follow the order of the characters
within the script, makes it very easy to find the character for anyone
who knows English (more generally, Latin) alphabet.  So I'm not sure
the proposed change is necessarily for the better, at least not in all
the use cases.

bug#46240: [External] : bug#46240: Sorting order of read-char-by-name

[Drew Adams]

> makes more sense to be consistent
> with Unicode tables where characters are grouped more logically.

> More examples with better sorting 

It's not "better" sorting; it's different sorting.

Different sort orders are useful in different ways.

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The current order, while it doesn't follow the order of the characters
> within the script, makes it very easy to find the character for anyone
> who knows English (more generally, Latin) alphabet.  So I'm not sure
> the proposed change is necessarily for the better, at least not in all
> the use cases.

I agree -- sorting by the Unicode character names is better here, I
think.

The command is for inserting characters you don't use constantly
(because if you did, you'd use a different input method that made those
characters more readily available), which means that you probably aren't
super-familiar with them.  So you'd be looking at the (English)
descriptions of the characters to find the one you want.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

> This has 2 disadvantages:
>
>   . the user needs to know the order of characters within a script
>     he/she doesn't necessarily read

In this case usually the names of characters of an unknown script
say nothing too, so sorting them by their names doesn't help much.

>   . the user needs to know the order _between_ scripts, if the
>     candidates include characters from different Unicode blocks

IMHO, this is not a disadvantage, but an advantage, because
it helps to group the matched characters by their Unicode ranges.
So when the user is not interested in certain scripts, then the user
can simply scroll down the groups of uninteresting ranges, and
look for characters only in needed Unicode blocks of scripts.

> If the user doesn't know this order, he/she might be unable to find
> the required character quickly, if the list of candidates is long
> enough.

I never rely on the current sorting alphabetically by names.
When the list of candidates is long, I need to use isearch
to search in the necessary block whose characters are scattered
currently in almost random order.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> The current order, while it doesn't follow the order of the characters
>> within the script, makes it very easy to find the character for anyone
>> who knows English (more generally, Latin) alphabet.  So I'm not sure
>> the proposed change is necessarily for the better, at least not in all
>> the use cases.
>
> I agree -- sorting by the Unicode character names is better here, I
> think.
>
> The command is for inserting characters you don't use constantly
> (because if you did, you'd use a different input method that made those
> characters more readily available), which means that you probably aren't
> super-familiar with them.  So you'd be looking at the (English)
> descriptions of the characters to find the one you want.

Indeed, the English descriptions help to identify the characters.

But the problem is that currently characters of one Unicode block
are scattered among the whole list.

For example, in 'C-x 8 RET SUPER TAB' the character SUPERHERO
from the Unicode block 'Supplemental Symbols and Pictographs'
currently is displayed at the beginning of the long list of
SUPERSCRIPT characters from the block 'Superscripts and Subscripts',
but its counterpart character SUPERVILLAIN is displayed
at the end of the long list of SUPERSCRIPT characters.

Whereas while sorting them by their codepoints, characters
of the same Unicode block come together:

🦸	SUPERHERO
🦹	SUPERVILLAIN

bug#46240: [External] : bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> makes more sense to be consistent
>> with Unicode tables where characters are grouped more logically.
>
>> More examples with better sorting
>
> It's not "better" sorting; it's different sorting.
>
> Different sort orders are useful in different ways.

This means there should be a customizable option?

bug#46240: [External] : bug#46240: Sorting order of read-char-by-name

[Drew Adams]

> >> More examples with better sorting
> >
> > It's not "better" sorting; it's different sorting.
> >
> > Different sort orders are useful in different ways.
> 
> This means there should be a customizable option?

Not necessarily an option.  But a variable,
yes.

Emacs has a policy (misguided, IMO) that
code should never bind a user option.

Given that policy, making this (only) a
user option means code can't override
the option value.

What's really needed (for this and other
stuff) is BOTH a way for users to express
a general, default, preference (defcustom)
AND a way for user code and other code to
bind a variable for this to any (allowed)
value.

E.g., a user or a library should be able
to create a command that does completion
against ucs-names with a particular sort
order, which might be different from
whatever the user (defcustom) chose as
the option value.
___

(I know that in Isearch you chose to have
both a defcustom and a defvar for some
things, which is fine as one solution.
My own preference would be to relax the
policy of never allowing an option to be
bound.)

bug#46240: [External] : bug#46240: Sorting order of read-char-by-name

[Drew Adams]

> But the problem is that currently characters of one Unicode block
> are scattered among the whole list.

I agree 100% that different sort orders
can be useful.
___

I think Emacs needs a _general_ way for
users and code to change sort orders.

In particular, users should be able to
change sort order on the fly, during
completion.

And code should be able to provide, for
any given completion choosing (e.g.
`completing-read'), the allowable sort
orders for a user to choose from.
___

(FWIW, I provide that with Icicles.)

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 46240@debbugs.gnu.org
> Date: Tue, 02 Feb 2021 19:13:13 +0200
> 
> > This has 2 disadvantages:
> >
> >   . the user needs to know the order of characters within a script
> >     he/she doesn't necessarily read
> 
> In this case usually the names of characters of an unknown script
> say nothing too, so sorting them by their names doesn't help much.

It isn't black and white.  For example, I'm familiar with quite a few
scripts and their characters, but have no idea about the alphabetical
order of most of them, with the exception of only two or three (end
even there I don't always remember the order of every character).

> >   . the user needs to know the order _between_ scripts, if the
> >     candidates include characters from different Unicode blocks
> 
> IMHO, this is not a disadvantage, but an advantage, because
> it helps to group the matched characters by their Unicode ranges.

Think of a case when there are more than 2 or three scripts involved.
You will have them take several window-fulls, and it could then be a
problem to find the script you are looking for.  This is similar to
having a directory with many files whose names use several scripts:
when these files are sorted alphabetically, you may not immediately
know whether file names that belong to some script are close to the
beginning or end of the list, because the relative ordering of scripts
in Unicode codepoint order is not necessarily something we remember.
(And then there are some scripts that use more than a single Unicode
block.)

> > If the user doesn't know this order, he/she might be unable to find
> > the required character quickly, if the list of candidates is long
> > enough.
> 
> I never rely on the current sorting alphabetically by names.
> When the list of candidates is long, I need to use isearch
> to search in the necessary block whose characters are scattered
> currently in almost random order.

Well, I guess there are also people like you, then.  So maybe we need
this to be an opt-in behavior.

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  46240@debbugs.gnu.org
> Date: Tue, 02 Feb 2021 19:16:00 +0200
> 
> But the problem is that currently characters of one Unicode block
> are scattered among the whole list.
> 
> For example, in 'C-x 8 RET SUPER TAB' the character SUPERHERO
> from the Unicode block 'Supplemental Symbols and Pictographs'
> currently is displayed at the beginning of the long list of
> SUPERSCRIPT characters from the block 'Superscripts and Subscripts',
> but its counterpart character SUPERVILLAIN is displayed
> at the end of the long list of SUPERSCRIPT characters.

There are such use cases as well, yes.  And there are others.  But the
point is that even if some Unicode block gets scattered (when the part
the user typed natches too many character names), the ordering by name
allows an easier orientation in the long list.

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>> If the user doesn't know this order, he/she might be unable to find
>> the required character quickly, if the list of candidates is long
>> enough.
>
> I never rely on the current sorting alphabetically by names.
> When the list of candidates is long, I need to use isearch
> to search in the necessary block whose characters are scattered
> currently in almost random order.

I'm not opposed to adding a variable to control the sorting order, but I
still don't quite understand when it would be useful to sort by Unicode
order.

Why does the current (English name) sorting result in an almost random
order?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> For example, in 'C-x 8 RET SUPER TAB' the character SUPERHERO
> from the Unicode block 'Supplemental Symbols and Pictographs'
> currently is displayed at the beginning of the long list of
> SUPERSCRIPT characters from the block 'Superscripts and Subscripts',
> but its counterpart character SUPERVILLAIN is displayed
> at the end of the long list of SUPERSCRIPT characters.
>
> Whereas while sorting them by their codepoints, characters
> of the same Unicode block come together:
>
> 🦸	SUPERHERO
> 🦹	SUPERVILLAIN

Oh, I see.  Yes, in that case sorting the English character names is,
indeed, not very optimal.

Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 80 bytes --]

> So maybe we need this to be an opt-in behavior.

Here is an option for this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: read-char-by-name-sort-function.patch --]
[-- Type: text/x-diff, Size: 2430 bytes --]

diff --git a/etc/NEWS b/etc/NEWS
index 7cdb9d9430..0ff78e58e1 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -850,6 +850,11 @@ iso-transl RET', it supports the same key sequences as 'C-x 8',
 so e.g. like 'C-x 8 [' inserts a left single quotation mark,
 'C-x \ [' does the same.
 
+---
+*** New user option 'read-char-by-name-sort-function'.
+It can enable sorting the characters of completion from
+'C-x 8 RET TAB' by codepoints instead of character names.
+
 ---
 *** Improved language transliteration in Malayalam input methods.
 Added a new Mozhi scheme.  The inapplicable ITRANS scheme is now
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 5dc3de4422..6e1e045a4a 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -3083,6 +3083,12 @@ mule--ucs-names-affixation
               (list name (concat (if char (format "%c" char) " ") "\t") "")))
           names))
 
+(defun mule--ucs-names-sort-by-code (names)
+  (let* ((codes-and-names
+          (mapcar (lambda (name) (cons (gethash name ucs-names) name)) names))
+         (sorted (sort codes-and-names (lambda (a b) (< (car a) (car b))))))
+    (mapcar #'cdr sorted)))
+
 (defun char-from-name (string &optional ignore-case)
   "Return a character as a number from its Unicode name STRING.
 If optional IGNORE-CASE is non-nil, ignore case in STRING.
@@ -3104,6 +3110,16 @@ char-from-name
                                            ignore-case))
                 code)))))))
 
+(defcustom read-char-by-name-sort-function nil
+  "Function to sort characters displayed by `read-char-by-name' completion."
+  :type '(choice
+          (const :tag "Sort by character names" nil)
+          (const :tag "Sort by character codepoints"
+                 mule--ucs-names-sort-by-code)
+          (function :tag "Custom function"))
+  :group 'mule
+  :version "28.1")
+
 (defun read-char-by-name (prompt)
   "Read a character by its Unicode name or hex number string.
 Display PROMPT and read a string that represents a character by its
@@ -3130,8 +3146,9 @@ read-char-by-name
 	   prompt
 	   (lambda (string pred action)
 	     (if (eq action 'metadata)
-		 '(metadata
+		 `(metadata
 		   (affixation-function . mule--ucs-names-affixation)
+		   (display-sort-function . ,read-char-by-name-sort-function)
 		   (category . unicode-name))
 	       (complete-with-action action (ucs-names) string pred)))))
 	 (char

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 46240@debbugs.gnu.org
> Date: Wed, 03 Feb 2021 19:27:54 +0200
> 
> Here is an option for this:

Thanks, but why does the value of the defcustom _have_ to be a
function?  Such defcustoms make it harder for users to customize them,
because not everyone can write a sorting function.

How about allowing simple values there, and if you really think we
should allow a function, make that one of the possible values, but not
require that the value be a function?

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Whereas while sorting them by their codepoints, characters
>> of the same Unicode block come together:
>>
>> 🦸	SUPERHERO
>> 🦹	SUPERVILLAIN
>
> Oh, I see.  Yes, in that case sorting the English character names is,
> indeed, not very optimal.
>
> Hm...

Here's a random idea -- perhaps the output could be formatted
differently than today (as an option)?  That is, with sections and
headings?

So

EMOJI
=====
🦸	SUPERHERO
🦹	SUPERVILLAIN
MATH
====
⁸	SUPERSCRIPT DIGIT EIGHT
⁴	SUPERSCRIPT DIGIT FOUR 
¹	SUPERSCRIPT DIGIT ONE 

or something?  (I don't know whether we actually have the data to do a
display like this, though.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  46240@debbugs.gnu.org
> Date: Wed, 03 Feb 2021 19:02:39 +0100
> 
> EMOJI
> =====
> 🦸	SUPERHERO
> 🦹	SUPERVILLAIN
> MATH
> ====
> ⁸	SUPERSCRIPT DIGIT EIGHT
> ⁴	SUPERSCRIPT DIGIT FOUR 
> ¹	SUPERSCRIPT DIGIT ONE 

Won't this produce multiple sections with identical headings,
scattered all over the list?  And if so, would that be helpful?

> (I don't know whether we actually have the data to do a display like
> this, though.)

We have the names of blocks in Blocks.txt, and could produce a
char-table out of it.

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> EMOJI
>> =====
>> 🦸	SUPERHERO
>> 🦹	SUPERVILLAIN
>> MATH
>> ====
>> ⁸	SUPERSCRIPT DIGIT EIGHT
>> ⁴	SUPERSCRIPT DIGIT FOUR 
>> ¹	SUPERSCRIPT DIGIT ONE 
>
> Won't this produce multiple sections with identical headings,
> scattered all over the list?  And if so, would that be helpful?

I was thinking that this would be an alternative sorting -- first by
code block, and then by character name in each block.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: juri@linkov.net,  46240@debbugs.gnu.org
> Date: Wed, 03 Feb 2021 19:21:26 +0100
> 
> I was thinking that this would be an alternative sorting -- first by
> code block, and then by character name in each block.

That could be useful, indeed, if the list is not too long.  For
example, this:

  C-x 8 RET *LETTER RET

produces a very long list where subdivision by block will not be very
useful, IMO.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

> Here's a random idea -- perhaps the output could be formatted
> differently than today (as an option)?  That is, with sections and
> headings?
>
> So
>
> EMOJI
> =====
> 🦸	SUPERHERO
> 🦹	SUPERVILLAIN
> MATH
> ====
> ⁸	SUPERSCRIPT DIGIT EIGHT
> ⁴	SUPERSCRIPT DIGIT FOUR 
> ¹	SUPERSCRIPT DIGIT ONE 
>
> or something?  (I don't know whether we actually have the data to do a
> display like this, though.)

I've implemented this in
https://lists.gnu.org/archive/html/emacs-devel/2020-11/msg00884.html
but the implementation was not too compact,
so I'm not sure if it's worth adding as an option.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> Here is an option for this:
>
> Thanks, but why does the value of the defcustom _have_ to be a
> function?  Such defcustoms make it harder for users to customize them,
> because not everyone can write a sorting function.
>
> How about allowing simple values there, and if you really think we
> should allow a function, make that one of the possible values, but not
> require that the value be a function?

I agree.  If there is a need to add more values later,
such as grouping by Unicode blocks, then it would be
easier to add new values, not functions.

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> I've implemented this in
> https://lists.gnu.org/archive/html/emacs-devel/2020-11/msg00884.html
> but the implementation was not too compact,
> so I'm not sure if it's worth adding as an option.

Looks nice!  Adding it as an option sounds like a good idea to me...
but would this need another variable in addition to the other variable
you proposed to just alter the sorting, or could these things somehow be
the same variable?

Adding these headers would only make sense if the user is sorting by
code instead of name...  so could the `read-char-by-name-sort-function'
variable instead be, say, `read-char-by-name-display' with values
`names', `code', `sections' (where `names' would be the current one,
`code' just sort by code, and `sections' would sort by code, and display
headings)?  Or something along those lines?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

>> I've implemented this in
>> https://lists.gnu.org/archive/html/emacs-devel/2020-11/msg00884.html
>> but the implementation was not too compact,
>> so I'm not sure if it's worth adding as an option.
>
> Looks nice!  Adding it as an option sounds like a good idea to me...
> but would this need another variable in addition to the other variable
> you proposed to just alter the sorting, or could these things somehow be
> the same variable?

Better to try adding all options to the same variable
to reduce the number of knobs.

> Adding these headers would only make sense if the user is sorting by
> code instead of name...  so could the `read-char-by-name-sort-function'
> variable instead be, say, `read-char-by-name-display' with values
> `names', `code', `sections' (where `names' would be the current one,
> `code' just sort by code, and `sections' would sort by code, and display
> headings)?  Or something along those lines?

Yep.  The only difference that this patch (that contains previous
implementation of grouping by blocks) uses `nil' instead of `names'
since this is the default value.

Oops, I noticed that my previous implementation sorted by names
inside each block, not by code.  I'm not sure if this makes sense?
Definitely, sorting by code inside blocks should be implemented
as the primary feature, but should an additional option with previous
implementation be retained to sort inside blocks by names too?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: read-char-by-name-display.patch --]
[-- Type: text/x-diff, Size: 3277 bytes --]

diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 5dc3de4422..465fd1bf53 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -3083,6 +3083,43 @@ mule--ucs-names-affixation
               (list name (concat (if char (format "%c" char) " ") "\t") "")))
           names))
 
+(defun mule--ucs-names-by-code (names)
+  (let* ((codes-and-names
+          (mapcar (lambda (name) (cons (gethash name ucs-names) name)) names))
+         (sorted (sort codes-and-names (lambda (a b) (< (car a) (car b))))))
+    (mapcar #'cdr sorted)))
+
+(defun mule--ucs-names-by-group (names)
+  (let* ((names-chars
+          (mapcar (lambda (name) (cons name (gethash name ucs-names))) names))
+         (groups-names
+          (seq-group-by
+           (lambda (name-char)
+             (let ((script (aref char-script-table (cdr name-char))))
+               (if script (symbol-name script) "ungrouped")))
+           names-chars))
+         names-headers header)
+    (dolist (group groups-names)
+      (setq header t)
+      (dolist (name-char (cdr group))
+        (push (list (car name-char)
+                    (concat
+                     ;; header
+                     (if header
+                         (progn
+                           (setq header nil)
+                           (concat "\n" (propertize
+                                         (format "* %s\n" (car group))
+                                         'face 'header-line)))
+                       "")
+                     ;; prefix
+                     (if (cdr name-char) (format "%c" (cdr name-char)) " ")
+                     " ")
+                    ;; suffix
+                    "")
+              names-headers)))
+    (nreverse names-headers)))
+
 (defun char-from-name (string &optional ignore-case)
   "Return a character as a number from its Unicode name STRING.
 If optional IGNORE-CASE is non-nil, ignore case in STRING.
@@ -3104,6 +3141,15 @@ char-from-name
                                            ignore-case))
                 code)))))))
 
+(defcustom read-char-by-name-display nil
+  "How to display characters by `read-char-by-name' completion."
+  :type '(choice
+          (const :tag "Sort by character names" nil)
+          (const :tag "Sort by character codepoints" code)
+          (const :tag "Group by Unicode blocks" sections))
+  :group 'mule
+  :version "28.1")
+
 (defun read-char-by-name (prompt)
   "Read a character by its Unicode name or hex number string.
 Display PROMPT and read a string that represents a character by its
@@ -3130,8 +3176,14 @@ read-char-by-name
 	   prompt
 	   (lambda (string pred action)
 	     (if (eq action 'metadata)
-		 '(metadata
-		   (affixation-function . mule--ucs-names-affixation)
+		 `(metadata
+		   (affixation-function
+                    . ,(if (eq read-char-by-name-display 'sections)
+                           'mule--ucs-names-by-group
+                         'mule--ucs-names-affixation))
+		   (display-sort-function
+                    . ,(when (eq read-char-by-name-display 'code)
+                         'mule--ucs-names-by-code))
 		   (category . unicode-name))
 	       (complete-with-action action (ucs-names) string pred)))))
 	 (char

bug#46240: Sorting order of read-char-by-name

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> Oops, I noticed that my previous implementation sorted by names
> inside each block, not by code.  I'm not sure if this makes sense?
> Definitely, sorting by code inside blocks should be implemented
> as the primary feature, but should an additional option with previous
> implementation be retained to sort inside blocks by names too?

Having the option to sort by names within each block sounds nice to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> Oops, I noticed that my previous implementation sorted by names
>> inside each block, not by code.  I'm not sure if this makes sense?
>> Definitely, sorting by code inside blocks should be implemented
>> as the primary feature, but should an additional option with previous
>> implementation be retained to sort inside blocks by names too?
>
> Having the option to sort by names within each block sounds nice to me.

Oh, then sorting order of sections would need own option.  Currently
sections are sorted by section names (i.e. mostly by script names
alphabetically, e.g. "adlam", "aegean-number", "ahom", etc.),
but a new option could sort them by their boundary codepoints
(i.e. "basic-latin", "latin-supplement", "latin-extended"),
so now options are going out of control :)

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  46240@debbugs.gnu.org
> Date: Fri, 05 Feb 2021 00:34:17 +0200
> 
> > Having the option to sort by names within each block sounds nice to me.
> 
> Oh, then sorting order of sections would need own option.  Currently
> sections are sorted by section names (i.e. mostly by script names
> alphabetically, e.g. "adlam", "aegean-number", "ahom", etc.),
> but a new option could sort them by their boundary codepoints
> (i.e. "basic-latin", "latin-supplement", "latin-extended"),
> so now options are going out of control :)

I think we can get away with only one sorting order for sections:
alphabetically.  Most tools I use that show large regions of Unicode
space do that, and I find it very convenient for quickly finding the
block I need without having to remember its place in the codepoint
order (which is quite random).

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

>> > Having the option to sort by names within each block sounds nice to me.
>>
>> Oh, then sorting order of sections would need own option.  Currently
>> sections are sorted by section names (i.e. mostly by script names
>> alphabetically, e.g. "adlam", "aegean-number", "ahom", etc.),
>> but a new option could sort them by their boundary codepoints
>> (i.e. "basic-latin", "latin-supplement", "latin-extended"),
>> so now options are going out of control :)
>
> I think we can get away with only one sorting order for sections:
> alphabetically.  Most tools I use that show large regions of Unicode
> space do that, and I find it very convenient for quickly finding the
> block I need without having to remember its place in the codepoint
> order (which is quite random).

Then customization will be much simpler with just 2 variables
'read-char-by-name-sort' and 'read-char-by-name-group':


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: read-char-by-name-group.patch --]
[-- Type: text/x-diff, Size: 4123 bytes --]

diff --git a/etc/NEWS b/etc/NEWS
index fb77688470..ed04bcdf13 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -851,6 +851,16 @@ iso-transl RET', it supports the same key sequences as 'C-x 8',
 so e.g. like 'C-x 8 [' inserts a left single quotation mark,
 'C-x \ [' does the same.
 
+---
+*** New user option 'read-char-by-name-sort'.
+It can enable sorting the characters of completion from
+'C-x 8 RET TAB' by codepoints instead of character names.
+
+---
+*** New user option 'read-char-by-name-group'.
+It groups the characters of completion from 'C-x 8 RET TAB'
+by Unicode blocks.
+
 ---
 *** Improved language transliteration in Malayalam input methods.
 Added a new Mozhi scheme.  The inapplicable ITRANS scheme is now
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 5dc3de4422..0df410987e 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -3083,6 +3083,42 @@ mule--ucs-names-affixation
               (list name (concat (if char (format "%c" char) " ") "\t") "")))
           names))
 
+(defun mule--ucs-names-group (names)
+  (let* ((codes-and-names
+          (mapcar (lambda (name) (cons (gethash name ucs-names) name)) names))
+         (grouped
+          (seq-group-by
+           (lambda (code-name)
+             (let ((script (aref char-script-table (car code-name))))
+               (if script (symbol-name script) "ungrouped")))
+           codes-and-names))
+         names-with-header header)
+    (dolist (group (sort grouped (lambda (a b) (string< (car a) (car b)))))
+      (setq header t)
+      (dolist (code-name (cdr group))
+        (push (list
+               (cdr code-name)
+               (concat
+                (if header
+                    (progn
+                      (setq header nil)
+                      (concat "\n" (propertize
+                                    (format "* %s\n" (car group))
+                                    'face 'header-line)))
+                  "")
+                ;; prefix
+                (if (car code-name) (format "%c" (car code-name)) " ") "\t")
+               ;; suffix
+               "")
+              names-with-header)))
+    (nreverse names-with-header)))
+
+(defun mule--ucs-names-sort-by-code (names)
+  (let* ((codes-and-names
+          (mapcar (lambda (name) (cons (gethash name ucs-names) name)) names))
+         (sorted (sort codes-and-names (lambda (a b) (< (car a) (car b))))))
+    (mapcar #'cdr sorted)))
+
 (defun char-from-name (string &optional ignore-case)
   "Return a character as a number from its Unicode name STRING.
 If optional IGNORE-CASE is non-nil, ignore case in STRING.
@@ -3104,6 +3140,22 @@ char-from-name
                                            ignore-case))
                 code)))))))
 
+(defcustom read-char-by-name-sort nil
+  "How to sort characters for `read-char-by-name' completion."
+  :type '(choice
+          (const :tag "Sort by character names" nil)
+          (const :tag "Sort by character codepoints" code))
+  :group 'mule
+  :version "28.1")
+
+(defcustom read-char-by-name-group nil
+  "How to group characters for `read-char-by-name' completion.
+When non-nil, split characters to sections of Unicode blocks
+sorted alphabetically."
+  :type 'boolean
+  :group 'mule
+  :version "28.1")
+
 (defun read-char-by-name (prompt)
   "Read a character by its Unicode name or hex number string.
 Display PROMPT and read a string that represents a character by its
@@ -3130,8 +3182,14 @@ read-char-by-name
 	   prompt
 	   (lambda (string pred action)
 	     (if (eq action 'metadata)
-		 '(metadata
-		   (affixation-function . mule--ucs-names-affixation)
+		 `(metadata
+		   (affixation-function
+                    . ,(if read-char-by-name-group
+                           'mule--ucs-names-group
+                         'mule--ucs-names-affixation))
+		   (display-sort-function
+                    . ,(when (eq read-char-by-name-sort 'code)
+                         'mule--ucs-names-sort-by-code))
 		   (category . unicode-name))
 	       (complete-with-action action (ucs-names) string pred)))))
 	 (char

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: larsi@gnus.org,  46240@debbugs.gnu.org
> Date: Sat, 06 Feb 2021 21:35:18 +0200
> 
> +*** New user option 'read-char-by-name-sort'.

I suggest to name this read-char-by-name-sort-order.

> +*** New user option 'read-char-by-name-group'.

I suggest to name this read-char-by-name-group-by-block.

Thanks.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> +*** New user option 'read-char-by-name-sort'.
>
> I suggest to name this read-char-by-name-sort-order.

I always strive to shorten variable names as much as possible.
Overly long names cause problems, especially in completions,
and in any list of variables.  There are docstrings that can
explain in more words about the variable purpose.

>> +*** New user option 'read-char-by-name-group'.
>
> I suggest to name this read-char-by-name-group-by-block.

The intention of this shorter name was to allow more
options to be added to the same variable later.

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: larsi@gnus.org,  46240@debbugs.gnu.org
> Date: Sun, 07 Feb 2021 20:56:57 +0200
> 
> >> +*** New user option 'read-char-by-name-sort'.
> >
> > I suggest to name this read-char-by-name-sort-order.
> 
> I always strive to shorten variable names as much as possible.
> Overly long names cause problems, especially in completions,
> and in any list of variables.  There are docstrings that can
> explain in more words about the variable purpose.
> 
> >> +*** New user option 'read-char-by-name-group'.
> >
> > I suggest to name this read-char-by-name-group-by-block.
> 
> The intention of this shorter name was to allow more
> options to be added to the same variable later.

<Shrug> those were suggestions, not requests.  The names you proposed
sounded not descriptive enough to me, so I looked for better ones.
feel free to ignore me.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

>> The intention of this shorter name was to allow more
>> options to be added to the same variable later.
>
> <Shrug> those were suggestions, not requests.  The names you proposed
> sounded not descriptive enough to me, so I looked for better ones.
> feel free to ignore me.

Sorry, I tried hard to find shortest names,
while being more verbose in the docstrings,
so I updated the docstrings to describe them
more thoroughly.

But I found a more serious problem: while testing I noticed
that some similar characters in the same group have
different syntax, for example:

SUBSCRIPT FOUR ₄ has syntax: _ 	which means: symbol
SUBSCRIPT FIVE ₅ has syntax: w 	which means: word

Many other SUBSCRIPT characters have different syntax.
The difference is noticeable because word motion commands stop
at different places: some stop at such characters, but some
SUBSCRIPT/SUPERSCRIPT characters are skipped by forward-word.

bug#46240: Sorting order of read-char-by-name

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: larsi@gnus.org,  46240@debbugs.gnu.org
> Date: Tue, 09 Feb 2021 20:13:09 +0200
> 
> But I found a more serious problem: while testing I noticed
> that some similar characters in the same group have
> different syntax, for example:
> 
> SUBSCRIPT FOUR ₄ has syntax: _ 	which means: symbol
> SUBSCRIPT FIVE ₅ has syntax: w 	which means: word

I think I fixed this now.

> Many other SUBSCRIPT characters have different syntax.

Please tell which ones.  I only found 9 SUBSCRIPT and SUPERSCRIPT
characters that had the wrong syntax, so "many" does sound surprising.

bug#46240: Sorting order of read-char-by-name

[Juri Linkov]

tags 46240 fixed
close 46240 28.0.50
thanks

>> But I found a more serious problem: while testing I noticed
>> that some similar characters in the same group have
>> different syntax, for example:
>> 
>> SUBSCRIPT FOUR ₄ has syntax: _ 	which means: symbol
>> SUBSCRIPT FIVE ₅ has syntax: w 	which means: word
>
> I think I fixed this now.

Confirmed.

>> Many other SUBSCRIPT characters have different syntax.
>
> Please tell which ones.  I only found 9 SUBSCRIPT and SUPERSCRIPT
> characters that had the wrong syntax, so "many" does sound surprising.

Sorry, actually it was much less than many - only 9.