bug#13177: 24.3.50; doc of `read-char-by-name'

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

Emacs-Lisp `read*' functions generally return what they say
they read.  This function does not necessarily return a character,
which is what its doc says it does.  It can return nil.  The doc
needs to mention this behavior, especially as it is pretty
exceptional.
 
There's nothing wrong with such behavior, but I wonder a bit whether
this function shouldn't perhaps use `completing-read' with a non-nil
REQUIRE-MATCH argument.  IOW, perhaps it should ignore names that
are unknown.  In that case it would always do what the doc currently
says it does: return a character (by name).  At least it's another
possibility.

Dunno just what the intention was for this function.  If it really
was to return a recognized Unicode character then this is a product
bug and reading should not end until the user enters matching input
(or hits `C-g').  If the intention was really to do what the
function does now, then this is a doc bug: mention a possible nil
return value and what it means.
 
Alternatively, we might add another function, say
`read-char-by-name-exclusive', that always returns a named char.
That would be somewhat analogous to how `read-char-exclusive'
ignores non-chars, while `read-char' signals an error for them.
(Note though that neither of those functions simply accepts and
returns a non-char event.)
 
Anyway, for this bug report and for my part, it would be enough
to correct the doc to say that if the user enters a name that
is unknown then nil is returned.
 
In GNU Emacs 24.3.50.1 (i386-mingw-nt5.1.2600)
 of 2012-12-13 on ODIEONE
Bzr revision: 111211 eggert@cs.ucla.edu-20121213021749-eyqqen0ewhn2hogq
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
 -IC:/Devel/emacs/build/include -Wall -Wextra -Wno-sign-compare
 -Wno-type-limits -Wno-missing-field-initializers -Wno-pointer-sign
 -Wdeclaration-after-statement --ldflags -LC:/Devel/emacs/build/lib'
 

bug#13177: 24.3.50; doc of `read-char-by-name'

[Juri Linkov]

> Dunno just what the intention was for this function.  If it really
> was to return a recognized Unicode character then this is a product
> bug and reading should not end until the user enters matching input
> (or hits `C-g').

As its docstring says, it also accepts a hexadecimal number
of Unicode code whose input can't use completion.

> Anyway, for this bug report and for my part, it would be enough
> to correct the doc to say that if the user enters a name that
> is unknown then nil is returned.

Then this patch would be enough:

=== modified file 'lisp/international/mule-cmds.el'
--- lisp/international/mule-cmds.el	2012-10-11 20:05:47 +0000
+++ lisp/international/mule-cmds.el	2012-12-15 11:09:03 +0000
@@ -2944,7 +2944,10 @@ (defun read-char-by-name (prompt)
 
 This function also accepts a hexadecimal number of Unicode code
 point or a number in hash notation, e.g. #o21430 for octal,
-#x2318 for hex, or #10r8984 for decimal."
+#x2318 for hex, or #10r8984 for decimal.
+
+When input is neither a known Unicode name nor a hex number string,
+return nil."
   (let ((input
          (completing-read
           prompt

bug#13177: 24.3.50; doc of `read-char-by-name'

[Stefan Monnier]

>> Dunno just what the intention was for this function.  If it really
>> was to return a recognized Unicode character then this is a product
>> bug and reading should not end until the user enters matching input
>> (or hits `C-g').
> As its docstring says, it also accepts a hexadecimal number
> of Unicode code whose input can't use completion.

That does not preclude using require-match.


        Stefan

bug#13177: 24.3.50; doc of `read-char-by-name'

[Juri Linkov]

>>> Dunno just what the intention was for this function.  If it really
>>> was to return a recognized Unicode character then this is a product
>>> bug and reading should not end until the user enters matching input
>>> (or hits `C-g').
>> As its docstring says, it also accepts a hexadecimal number
>> of Unicode code whose input can't use completion.
>
> That does not preclude using require-match.

I guess you mean adding a predicate that will check input
for a valid hex number?  It could act as an union of
character name completions and a predicate to check hex numbers.
But I don't imagine how to combine them without adding
a large list of all possible hex numbers to the completion list.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Stefan Monnier]

> I guess you mean adding a predicate that will check input
> for a valid hex number?  It could act as an union of
> character name completions and a predicate to check hex numbers.
> But I don't imagine how to combine them without adding
> a large list of all possible hex numbers to the completion list.

The completion table can be a function.


        Stefan

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> >> Dunno just what the intention was for this function.  If it really
> >> was to return a recognized Unicode character then this is a product
> >> bug and reading should not end until the user enters matching input
> >> (or hits `C-g').
>
> > As its docstring says, it also accepts a hexadecimal number
> > of Unicode code whose input can't use completion.
> 
> That does not preclude using require-match.

Yes.

The problem for me is that I don't know what the intention (design) was or why.
Maybe someone has a good reason to return nil or to return a number that does
not correspond to a recognized code point?

The design should be clarified (and perhaps decided/changed, if appropriate)
before we try to fit the doc to it.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Stefan Monnier]

> The problem for me is that I don't know what the intention (design)
> was or why.  Maybe someone has a good reason to return nil or to
> return a number that does not correspond to a recognized code point?

It's called "read-char-by-name", so the intention is pretty clear: it
should return a char.


        Stefan

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> > The problem for me is that I don't know what the intention (design)
> > was or why.  Maybe someone has a good reason to return nil or to
> > return a number that does not correspond to a recognized code point?
> 
> It's called "read-char-by-name", so the intention is pretty clear: it
> should return a char.

Great.  That's what I would prefer also.  And that was exactly what I said in
the original bug report: that's what the other `read*' functions do: return the
thing their names say they read.

So we start with a code bug - make sure it always returns a char.  Then we fix
the doc.

However, this function has been around for a while, so there might be some code
that expects it to sometimes return nil or sometimes a non code-point number.
Hopefully, any code expecting nil is only trying to workaround this same bug...

Anyway, FWIW you've got my vote in favor of fixing the code to always return a
char.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Juri Linkov]

> Great.  That's what I would prefer also.  And that was exactly what
> I said in the original bug report: that's what the other `read*'
> functions do: return the thing their names say they read.

`read-char' says it reads a character but it returns 0 (^@) for
invalid characters.  I see no reason why 0 would be better than nil.
Using an arbitrary character ^@ for invalid characters makes no sense.

Try to eval

  (insert-char (read-char))

and type `C-SPC'.  It inserts the character ^@ with the value 0.

Typing `C-M-SPC' reports:

  (wrong-type-argument characterp 134217728)

Clicking a mouse button reports in the echo area:

  "Non-character input-event"

This exhibits three different types of error processing for invalid
characters.  I think that displaying "Non-character input-event"
in the echo area is the most reasonable and user-friendly.

> So we start with a code bug - make sure it always returns a char.
> Then we fix the doc.
> Anyway, FWIW you've got my vote in favor of fixing the code
> to always return a char.

Also in bug#13195 the same request:

> Seems like `read-char-by-name' should always return something
> that `insert-char' can use, i.e., something that passes `characterp'.

Yes, to always return a valid char or nil, we could check for `characterp'
like in the patch below.

So when you type `C-x 8 RET #10r1111111111 RET' it will report
in the echo area:

  "You did not specify a valid character"

IMO, this is a sufficient error processing, no?

=== modified file 'lisp/international/mule-cmds.el'
--- lisp/international/mule-cmds.el	2012-12-15 13:07:58 +0000
+++ lisp/international/mule-cmds.el	2012-12-16 09:06:59 +0000
@@ -2944,7 +2944,10 @@ (defun read-char-by-name (prompt)
 
 This function also accepts a hexadecimal number of Unicode code
 point or a number in hash notation, e.g. #o21430 for octal,
-#x2318 for hex, or #10r8984 for decimal."
+#x2318 for hex, or #10r8984 for decimal.
+
+When input is neither a known Unicode name nor a hex number
+that specifies a valid character, return nil."
   (let* ((enable-recursive-minibuffers t)
 	 (input
 	  (completing-read
@@ -2953,14 +2956,16 @@ (defun read-char-by-name (prompt)
 	     (let ((completion-ignore-case t))
 	       (if (eq action 'metadata)
 		   '(metadata (category . unicode-name))
-		 (complete-with-action action (ucs-names) string pred)))))))
-    (cond
-     ((string-match-p "\\`[0-9a-fA-F]+\\'" input)
-      (string-to-number input 16))
-     ((string-match-p "\\`#" input)
-      (read input))
-     (t
-      (cdr (assoc-string input (ucs-names) t))))))
+		 (complete-with-action action (ucs-names) string pred))))))
+	 (character
+	  (cond
+	   ((string-match-p "\\`[0-9a-fA-F]+\\'" input)
+	    (string-to-number input 16))
+	   ((string-match-p "\\`#" input)
+	    (read input))
+	   (t
+	    (cdr (assoc-string input (ucs-names) t))))))
+    (and (characterp character) character)))
 
 (define-obsolete-function-alias 'ucs-insert 'insert-char "24.3")
 (define-key ctl-x-map "8\r" 'insert-char)

bug#13177: 24.3.50; doc of `read-char-by-name'

[Andreas Schwab]

Juri Linkov <juri@jurta.org> writes:

> Try to eval
>
>   (insert-char (read-char))
>
> and type `C-SPC'.  It inserts the character ^@ with the value 0.

That's the correct value for C-SPC.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#13177: 24.3.50; doc of `read-char-by-name'

[Juri Linkov]

>>   (insert-char (read-char))
>>
>> and type `C-SPC'.  It inserts the character ^@ with the value 0.
>
> That's the correct value for C-SPC.

Yes, I was confused by the docstring of `read-char' that says:

  If the character has modifiers, they are resolved and reflected to the
  character code if possible (e.g. C-SPC -> 0).

But nevertheless `read-char' can return non-characters numbers unacceptable
for `insert-char'.  Currently `read-char-by-name' does the same. e.g.:

  M-: (insert-char (read-char)) RET C-M-SPC

reports:

  (wrong-type-argument characterp 134217728)

and

  M-: (insert-char (read-char-by-name "Insert character (Unicode name or hex): "))
  RET #x8000000 RET

reports the same:

  (wrong-type-argument characterp 134217728)

So we are trying to fix a non-problem.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

>   M-: (insert-char (read-char)) RET C-M-SPC
> reports:
>   (wrong-type-argument characterp 134217728)
> 
> So we are trying to fix a non-problem.

No.  Let's please concentrate on `read-char-by-name', not `read-char'.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> > Great.  That's what I would prefer also.  And that was exactly what
> > I said in the original bug report: that's what the other `read*'
> > functions do: return the thing their names say they read.
> 
> `read-char' says it reads a character but it returns 0 (^@) for
> invalid characters.  I see no reason why 0 would be better than nil.
> Using an arbitrary character ^@ for invalid characters makes no sense.

OK, perhaps my analogy was too black & white.  Let's try to deal with
`read-char' exceptions to returning a char in a separate thread.  Keep in mind,
though, that `read-char' has been around even longer (a lot longer) than
`read-char-by-name', so changes would like affect more 3rd-party code.

Also, `read-char' returns non-char events, which is why we also have
`read-char-exclusive'.  Anyway, let's not let my analogy get in the way too
much; let's concentrate here on `read-char-by-name' and try to figure out what
the best behavior (and then doc) would be.

> > So we start with a code bug - make sure it always returns a char.
> > Then we fix the doc.  Anyway, FWIW you've got my vote in favor of
> > fixing the code to always return a char.
> 
> Also in bug#13195 the same request:
> 
> > Seems like `read-char-by-name' should always return something
> > that `insert-char' can use, i.e., something that passes 
> > `characterp'.
> 
> Yes, to always return a valid char or nil, we could check for 
> `characterp' like in the patch below.

Why return nil ever?  That's still part of the discussion, AFAIK.  nil is not a
character.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The problem for me is that I don't know what the intention (design)
>> was or why.  Maybe someone has a good reason to return nil or to
>> return a number that does not correspond to a recognized code point?
>
> It's called "read-char-by-name", so the intention is pretty clear: it
> should return a char.

Agreed.  I changed read-char-by-name on the trunk to signal an error for
invalid input.  (This also avoids other wonky cases, e.g. entering ##
giving the empty symbol because `read-char-by-name' uses `read'...)

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> > It's called "read-char-by-name", so the intention is pretty 
> > clear: it should return a char.
> 
> Agreed.  I changed read-char-by-name on the trunk to signal 
> an error for invalid input.  (This also avoids other wonky cases,
> e.g. entering ## giving the empty symbol because `read-char-by-name'
> uses `read'...)

I don't understand why you did that, if you agree with Stefan.  He proposed
making `completing-read' require input that represents a valid char.  Why let
`completing-read' accept invalid input and then raise an error saying the input
is invalid?

Sorry, I don't get it - seems not very helpful.

The function should require the user to provide matching input (name or valid
code) or use `C-g' or `C-]' to cancel reading.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> > Dunno just what the intention was for this function.  If it really
> > was to return a recognized Unicode character then this is a product
> > bug and reading should not end until the user enters matching input
> > (or hits `C-g').
> 
> As its docstring says, it also accepts a hexadecimal number
> of Unicode code whose input can't use completion.

Yes, I know.  But the doc string also says that the hex numbers accepted are
those corresponding to Unicode code points.  And it mentions hash notation for
numbers, but in that case it does not say that those numbers too must correspond
to Unicode code points.

The question I posed is whether the intention was to always return a recognized
Unicode char.  If that is the case then the function should never return nil or
anything else (e.g. a non code-point number) other than a Unicode char.  It's an
open question.

If you are answering for the design intention, stating that the intention is not
to do what the doc currently says (always return a Unicode char), then fine.  In
that case, all that's needed is a doc-string patch, to point out that exception
to its general statement.

However (see below), that exception needs to be clarified in the case of hash
notation input that does not correspond to a Unicode char.

> +When input is neither a known Unicode name nor a hex number string,
> +return nil."

It is not enough, according to the existing doc string, for input to be  any old
hex number - it must be a "hexadecimal number of Unicode code point" - or
presumably we return nil (?).  The additional info should say this (or
equivalent):

"Return nil if the input does not correspond to a Unicode character."

The doc string should also be corrected to not give the impression that hash
notation input somehow escapes this need for the number to represent a
character, i.e., to be numerically equivalent to "a hexadecimal number of
Unicode code point".

So we should replace the last paragraph of the doc string with something like
this:

"Besides a Unicode character name, input can represent a Unicode character
numerically.  It can be a hexadecimal number or a number in hash notation, e.g.
#o21430 for octal, #x2318 for hex, or #10r8984 for decimal."

Follow that with the statement above that _any other_ input, i.e., any input
that does not correspond to a recognized Unicode char, means return nil.  Then
things will be clear.

Otherwise, we are not saying anything about what is returned if the input hash
notation does not correspond to a Unicode char - a doc bug.

If, on the other hand, it is not the case that the function returns nil for hash
notation input that does not correspond to a known character, then the doc
should say that clearly.

In that case, the function has two exceptions to returning a Unicode character:

1. Return nil if a name is entered that is not a recognized name.

2. Return a number if hash notation is entered that does not match a Unicode
code point.

I cannot speak for what the function is supposed to do (design), so I cannot say
whether there is a code bug or a doc bug here.  I can only outline the
possibilities.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 15 Dec 2012 07:52:51 -0800
> Cc: 13177@debbugs.gnu.org
> 
> > As its docstring says, it also accepts a hexadecimal number
> > of Unicode code whose input can't use completion.
> 
> Yes, I know.  But the doc string also says that the hex numbers accepted are
> those corresponding to Unicode code points.  And it mentions hash notation for
> numbers, but in that case it does not say that those numbers too must correspond
> to Unicode code points.

#xNNNNN can also specify 8-bit characters whose codepoints are outside
Unicode.

I have no idea whether this distinction is relevant to this function,
but since it talks about "Unicode code points", people who deal with
the documentation need to be aware of this subtlety.

bug#13177: 24.3.50; doc of `read-char-by-name'

[Drew Adams]

> #xNNNNN can also specify 8-bit characters whose codepoints are outside
> Unicode.
> 
> I have no idea whether this distinction is relevant to this function,
> but since it talks about "Unicode code points", people who deal with
> the documentation need to be aware of this subtlety.

Precisely.  Is `read-char-by-name' intended to just return such a non code-point
number or should it always return a Unicode char as it name suggests?

Once we know what the intention is, if the code doesn't fit it we can change the
code.  Once we know what the correct code does, we can fit the doc to it.