bug#32562: 26; `read-char(-exclusive)' and `characterp'

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

Perhaps there is no bug here, other than incomplete or insufficiently
clear doc.  Dunno.

emacs -Q
M-: (setq foo (read-char "Char: "))
M-:

shows: 134217786 (#o1000000072, #x800003a), a number too large
to be `characterp'.  And M-: (characterp foo) ; returns nil.

Same thing for `read-char-exclusive'.

Something seems wrong.  The doc for `read-char(-exclusive)' talks about
reading a character. For `read-char' it says that it raises an error for
a non-char event.  And for `read-char-exclusive' it says that it ignores
non-char events that it reads.

But they both seem to read the event from `M-:' and return its value
(a number). And it is not a character, according to `characterp'.

I'm confused.  Please consider fixing this, at least by clarifying the doc.

Why should a function that reads a character, and either ignores a
non-char or raises an error for a non-char, return a non-character?
And what function reads a `characterp' character, either ignoring
anything else or raising an error for anything else?

In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-30
Repository revision: 07f8f9bc5a51f5aa94eb099f3e15fbe0c20ea1ea
Windowing system distributor `Microsoft Corp.', version 10.0.16299
Configured using:
 `configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Noam Postavsky]

forcemerge 13599 32562
quit

On 28 August 2018 at 16:39, Drew Adams <drew.adams@oracle.com> wrote:

> Something seems wrong.  The doc for `read-char(-exclusive)' talks about
> reading a character. For `read-char' it says that it raises an error for
> a non-char event.  And for `read-char-exclusive' it says that it ignores
> non-char events that it reads.
>
> But they both seem to read the event from `M-:' and return its value
> (a number). And it is not a character, according to `characterp'.

This is bug#13599/1042

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> forcemerge 13599 32562
> quit
> 
> > Something seems wrong.  The doc for `read-char(-exclusive)' talks about
> > reading a character. For `read-char' it says that it raises an error for
> > a non-char event.  And for `read-char-exclusive' it says that it ignores
> > non-char events that it reads.
> >
> > But they both seem to read the event from `M-:' and return its value
> > (a number). And it is not a character, according to `characterp'.
> 
> This is bug#13599/1042

Hm. Maybe it is. But is `read-char' necessarily about chars that satisfy `characterp'?

I would think so, but maybe not. And there are other "character" functions, which accept chars as args - `text-char-description', for example. (That's what I was passing the result of `read-char' to.) 

The doc string of `text-char-description' says "file-character CHARACTER". What's a "file character"? A character that can appear in a file name? Same thing as what `max-char' should recognize? Same thing as what `read-char' should consider a char? The Elisp manual (node Describing Characters) says it is "characters that appear in text—like ‘single-key-description’, except that..." It says nothing about "file characters". The doc is not too clear.

This stuff is not clear more generally, I think - beyond the max value of `max-char'. Do we have or want to have different kinds of "characters" returned from or passed as args to different "character" functions? Why (or why not)?

Wrt my original problem: taking a value of `M-:' from `read-char' and passing it to `text-char-description', Emacs has a regression of sorts. Older Emacs versions "work", whereas recent versions raise an error. E.g. Emacs 20 `read-char' returns -134217670, and passing that to `text-char-description' gives "\272". Whatever `read-char' can read, it seems, `text-char-description' can describe (perhaps imperfectly?).

I think that "the bug" is not so clear cut as just adjusting `max-char'. I think the behavior of several functions that expect or return chars might deserve adjustment, and the doc might deserve some clarification/fixes. We might need to decide what the right behaviors are first.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Noam Postavsky]

Drew Adams <drew.adams@oracle.com> writes:

> I think that "the bug" is not so clear cut as just adjusting
> `max-char'.

No, of course not.  It's discussed at length Bug#13599.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Tue, 28 Aug 2018 13:39:20 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> Perhaps there is no bug here, other than incomplete or insufficiently
> clear doc.  Dunno.

It's a documentation bug.

> emacs -Q
> M-: (setq foo (read-char "Char: "))
> M-:
> 
> shows: 134217786 (#o1000000072, #x800003a), a number too large
> to be `characterp'.  And M-: (characterp foo) ; returns nil.
> 
> Same thing for `read-char-exclusive'.
> 
> Something seems wrong.  The doc for `read-char(-exclusive)' talks about
> reading a character. For `read-char' it says that it raises an error for
> a non-char event.  And for `read-char-exclusive' it says that it ignores
> non-char events that it reads.
> 
> But they both seem to read the event from `M-:' and return its value
> (a number). And it is not a character, according to `characterp'.

What the documentation failed to make clear is that the value returned
by these functions is an _event_, not a character.  For ASCII
characters, the event and the character are identical, but modifier
bits that cannot be resolved to yield an ASCII character are left in
the returned value, and that makes the value not a valid character
code.

I have now clarified this subtle point in the documentation (on the
emacs-26 branch).

> And what function reads a `characterp' character, either ignoring
> anything else or raising an error for anything else?

What do you mean by "reading a 'characterp'"?  E.g., if the user types
C-% or M-a, what 'characterp' code would you expect to see?  Also,
does "reading a 'characterp'" include supporting input methods, if
any?

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Tue, 28 Aug 2018 14:24:36 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 32562@debbugs.gnu.org
> 
> is `read-char' necessarily about chars that satisfy `characterp'?

No.  It returns a character event, not a character.

> The doc string of `text-char-description' says "file-character CHARACTER". What's a "file character"? A character that can appear in a file name?

No, text-char-description accepts only valid character codes, those
which satisfy the 'characterp' test.  This is unlike
single-key-description, which accepts _events_, and thus will happily
process character input events that are not valid character codes,
i.e. fail the 'characterp' test.  I've now made that clear in the
respective doc strings.

> This stuff is not clear more generally, I think - beyond the max value of `max-char'. Do we have or want to have different kinds of "characters" returned from or passed as args to different "character" functions? Why (or why not)?

The basic difference is between a character code and a character input
event.

> Wrt my original problem: taking a value of `M-:' from `read-char' and passing it to `text-char-description', Emacs has a regression of sorts. Older Emacs versions "work", whereas recent versions raise an error. E.g. Emacs 20 `read-char' returns -134217670, and passing that to `text-char-description' gives "\272". Whatever `read-char' can read, it seems, `text-char-description' can describe (perhaps imperfectly?).

It's not a regression: text-char-description wants a valid character
code.

I'm closing this bug, as I think this is a documentation issue which
is now fixed.

Thanks.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > Perhaps there is no bug here, other than incomplete or insufficiently
> > clear doc.  Dunno.
> 
> It's a documentation bug.
>
> > But they both seem to read the event from `M-:' and return its value
> > (a number). And it is not a character, according to `characterp'.
> 
> What the documentation failed to make clear is that the value returned
> by these functions is an _event_, not a character.  For ASCII
> characters, the event and the character are identical, but modifier
> bits that cannot be resolved to yield an ASCII character are left in
> the returned value, and that makes the value not a valid character
> code.
> 
> I have now clarified this subtle point in the documentation (on the
> emacs-26 branch).

Thank you.

> > And what function reads a `characterp' character, either ignoring
> > anything else or raising an error for anything else?
> 
> What do you mean by "reading a 'characterp'"?

I said "reads a `characterp' character", meaning reads an event
that satisfies `characterp'. Apparently the functions whose names
or doc suggests that they read a character do not necessarily
return something that satisfies `characterp'.

> E.g., if the user types
> C-% or M-a, what 'characterp' code would you expect to see?

I was asking for a function that insists on getting an event that
satisfies `characterp', i.e., that keeps reading events until it does.

> Also, does "reading a 'characterp'" include supporting input methods, if
> any?

Sorry, can't answer that. I know nothing about input methods.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Mon, 10 Sep 2018 06:43:07 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 32562@debbugs.gnu.org
> 
> I was asking for a function that insists on getting an event that
> satisfies `characterp', i.e., that keeps reading events until it does.

Not sure we have such a beast, but wouldn't calling read-char in a
loop until you get something that passes the 'characterp' test do?

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Andreas Schwab]

On Sep 10 2018, Drew Adams <drew.adams@oracle.com> wrote:

> I said "reads a `characterp' character", meaning reads an event
> that satisfies `characterp'. Apparently the functions whose names
> or doc suggests that they read a character do not necessarily
> return something that satisfies `characterp'.

A character with modifiers doesn't satisfy characterp, but it is still a
character (which read-char can return).

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > I was asking for a function that insists on getting an event that
> > satisfies `characterp', i.e., that keeps reading events until it does.
> 
> Not sure we have such a beast, but wouldn't calling read-char in a
> loop until you get something that passes the 'characterp' test do?

I think that's what I'm doing now, but I don't recall now just
which code I was using this for. ;-)

If the doc has been clarified that's a help. Thx.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > I said "reads a `characterp' character", meaning reads an event
> > that satisfies `characterp'. Apparently the functions whose names
> > or doc suggests that they read a character do not necessarily
> > return something that satisfies `characterp'.
> 
> A character with modifiers doesn't satisfy characterp, but it is still a
> character (which read-char can return).

Yes, well, that's what I found a bit confusing wrt the
doc and function names. It wasn't (isn't?) too clear
just what we mean by "character", in that case.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Mon, 10 Sep 2018 08:01:40 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 32562@debbugs.gnu.org
> 
> > A character with modifiers doesn't satisfy characterp, but it is still a
> > character (which read-char can return).
> 
> Yes, well, that's what I found a bit confusing wrt the
> doc and function names. It wasn't (isn't?) too clear
> just what we mean by "character", in that case.

I made the documentation say "character event" where it matters.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > > A character with modifiers doesn't satisfy characterp, but it is still a
> > > character (which read-char can return).
> >
> > Yes, well, that's what I found a bit confusing wrt the
> > doc and function names. It wasn't (isn't?) too clear
> > just what we mean by "character", in that case.
> 
> I made the documentation say "character event" where it matters.

I imagine that you made it clearer (I don't have the new doc).

But see what I replied to Andreas. It seems like we talk in some
places about reading a character even if what gets read might
not be `characterp'. "Character event" doesn't clarify that, for me.

Seems like the doc (but I don't have time now to search it)
doesn't distinguish "character" meaning something that is
`characterp' from other meanings of it (not sure what those
are, but apparently `read-char' can return such things).

It's not clear to me, at least. Why isn't `chararcterp' necessary
and sufficient for testing whether something is a character?

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Mon, 10 Sep 2018 12:48:22 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: schwab@suse.de, 32562@debbugs.gnu.org
> 
> > > > A character with modifiers doesn't satisfy characterp, but it is still a
> > > > character (which read-char can return).
> > >
> > > Yes, well, that's what I found a bit confusing wrt the
> > > doc and function names. It wasn't (isn't?) too clear
> > > just what we mean by "character", in that case.
> > 
> > I made the documentation say "character event" where it matters.
> 
> I imagine that you made it clearer (I don't have the new doc).

I hope so.

> But see what I replied to Andreas. It seems like we talk in some
> places about reading a character even if what gets read might
> not be `characterp'. "Character event" doesn't clarify that, for me.
> 
> Seems like the doc (but I don't have time now to search it)
> doesn't distinguish "character" meaning something that is
> `characterp' from other meanings of it (not sure what those
> are, but apparently `read-char' can return such things).

The changes I made can be seen here:

  http://git.savannah.gnu.org/cgit/emacs.git/commit/?h=emacs-26&id=5cf282d65f10f59f7efa63359dfd2b2e124943da

Please read them (in their context of surrounding manual text, of
course), and then see if something is still unclear.  Otherwise, this
discussion is too theoretical to be useful.

> It's not clear to me, at least. Why isn't `chararcterp' necessary
> and sufficient for testing whether something is a character?

It is.  But it is not sufficient for testing whether something is a
character input event, because an input event can include modifiers,
and valid character codepoints don't include modifiers.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> The changes I made can be seen here:
> 
>   https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__git.savannah.gnu.org_cgit_emacs.git_commit_-3Fh-3Demacs-2D26-26id-
> 3D5cf282d65f10f59f7efa63359dfd2b2e124943da&d=DwIBAg&c=RoP1YumCXCg
> aWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=kI3P6ljGv6CTHIKju0jqInF6AOwMCYR
> DQUmqX22rJ98&m=jioWGqzV72URRu3VKa5ldIhRy-
> eUITHjY2CjCiAMVEw&s=gVdGs2AlmqeIsl7dbZM0_9ulTQ_PPe0vAnT5MUx8AN
> M&e=

Thanks, that helps.

> Please read them (in their context of surrounding manual text, of
> course), and then see if something is still unclear.  

I can't easily read them in the context of the surrounding material,
but I read the diff, at least.

I see "character event" referred to but not explicitly defined, as
such. The inline explanation is clear enough, I think. Why not
treat the first occurrence of "character event" as a defined term
(however that's done in texinfo, so that it ends up in Info quoted,
as "character event"), and create an index entry for it (plus an
entry "event, character")? (Perhaps you did that; I can't tell.)

I think this might be the right occurrence to treat as defining
"character event":

+keys, and function keys.  Ordinary keys correspond to (possibly
+modified) characters; the events they generate are represented in Lisp
+as characters.  The event type of a character event is the character
                                     ^^^^^^^^^^^^^^^
+itself (an integer), which might have some modifier bits set; see

> > It's not clear to me, at least. Why isn't `chararcterp' necessary
> > and sufficient for testing whether something is a character?
> 
> It is.  But it is not sufficient for testing whether something is a
> character input event, because an input event can include modifiers,
> and valid character codepoints don't include modifiers.

Got it. Thx.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Tue, 11 Sep 2018 08:30:56 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 32562@debbugs.gnu.org
> 
> I can't easily read them in the context of the surrounding material,
> but I read the diff, at least.

The context is given by the corresponding *.texi files.

> I see "character event" referred to but not explicitly defined, as
> such. The inline explanation is clear enough, I think. Why not
> treat the first occurrence of "character event" as a defined term
> (however that's done in texinfo, so that it ends up in Info quoted,
> as "character event"), and create an index entry for it (plus an
> entry "event, character")? (Perhaps you did that; I can't tell.)
> 
> I think this might be the right occurrence to treat as defining
> "character event":
> 
> +keys, and function keys.  Ordinary keys correspond to (possibly
> +modified) characters; the events they generate are represented in Lisp
> +as characters.  The event type of a character event is the character
>                                      ^^^^^^^^^^^^^^^
> +itself (an integer), which might have some modifier bits set; see

That's the definition you wanted, right there: it says that events can
represent characters, and then talks about "character events" thus
defined.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > I think this might be the right occurrence to treat as defining
> > "character event":
> >
> > +keys, and function keys.  Ordinary keys correspond to (possibly
> > +modified) characters; the events they generate are represented in Lisp
> > +as characters.  The event type of a character event is the character
> >                                      ^^^^^^^^^^^^^^^
> > +itself (an integer), which might have some modifier bits set; see
> 
> That's the definition you wanted, right there: it says that events can
> represent characters, and then talks about "character events" thus
> defined.

Thats what I said: 'I think this might be the right occurrence to treat as defining "character event"'. I was asking that that defining occurrence get double quotes and index entries, to make clear that it is a term definition.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Tue, 11 Sep 2018 10:22:21 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 32562@debbugs.gnu.org
> 
> Thats what I said: 'I think this might be the right occurrence to treat as defining "character event"'. I was asking that that defining occurrence get double quotes and index entries, to make clear that it is a term definition.

Added.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Drew Adams]

> > Thats what I said: 'I think this might be the right occurrence to treat as
> defining "character event"'. I was asking that that defining occurrence get
> double quotes and index entries, to make clear that it is a term definition.
> 
> Added.

Thanks. Feel free to close this bug. I didn't close it, as I don't know if you intended to do something more for it.

bug#32562: 26; `read-char(-exclusive)' and `characterp'

[Eli Zaretskii]

> Date: Tue, 11 Sep 2018 11:58:31 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 32562@debbugs.gnu.org
> 
> Feel free to close this bug.

I already did.