Re: 23.0.60; end-of-sentence and non-breaking space

Re: 23.0.60; end-of-sentence and non-breaking space

[Chong Yidong]

From bug#1726 and bug#1727:

> forward-sentence does not treat non-breaking space as a space for
> purposes of sentence ends.
...
> When I type C-x = at a non-breaking space, it tells me that it
> has code 160, hex a0.  But when I execute (insert "\xa0"),
> it inserts something that displays as `\240' and for which C-x =
> displays this:
>
>    Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
>    (93%) column=5
>
> Is that a bug?  It seems quite confusing to me.

ISTR that there was an extended discussion about classifying
non-breaking spaces on this list a while back.  But I can't find it now.
Does anyone remember the details?

Re: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    > When I type C-x = at a non-breaking space, it tells me that it
    > has code 160, hex a0.  But when I execute (insert "\xa0"),
    > it inserts something that displays as `\240' and for which C-x =
    > displays this:

    >    Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
    >    (93%) column=5
    >
    > Is that a bug?  It seems quite confusing to me.

    ISTR that there was an extended discussion about classifying
    non-breaking spaces on this list a while back.  But I can't find it now.
    Does anyone remember the details?

I am not sure we are talking about the same question.
The issue I am raising is not one of classifying it,
it is that these two different character codes get used
and I don't see an explanation of what's going on.

RE: bug#1727: 23.0.60; end-of-sentence and non-breaking space

[Drew Adams]

> ISTR that there was an extended discussion about classifying
> non-breaking spaces on this list a while back.  But I can't 
> find it now. Does anyone remember the details?

Dunno if this is what you were thinking of, but there was this discussion about
treating (classifying) nonbreaking space as whitespace:

http://lists.gnu.org/archive/html/emacs-devel/2007-06/msg01089.html

Re: 23.0.60; end-of-sentence and non-breaking space

[Stefan Monnier]

>> has code 160, hex a0.  But when I execute (insert "\xa0"),
>> it inserts something that displays as `\240' and for which C-x =
>> displays this:
>> 
>> Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
>> (93%) column=5
>> 
>> Is that a bug?  It seems quite confusing to me.

This raw-byte char is what used to be called an eight-bit-control (or
eight-bit-graphic depending on the actual value) char.

I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
non-breaking space).


        Stefan

Re: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    This raw-byte char is what used to be called an eight-bit-control (or
    eight-bit-graphic depending on the actual value) char.

    I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
    eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
    non-breaking space).

1. Is that the right thing for \xa0 in a string to mean?
Or should it mean the character with code xa0?

2. I find it hard to think about that question since I don't see any
documentation explaining how this ought to work.  That documentation
is essential.

Re: 23.0.60; end-of-sentence and non-breaking space

[Stefan Monnier]

>     This raw-byte char is what used to be called an eight-bit-control (or
>     eight-bit-graphic depending on the actual value) char.

>     I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
>     eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
>     non-breaking space).

> 1. Is that the right thing for \xa0 in a string to mean?
> Or should it mean the character with code xa0?

> 2. I find it hard to think about that question since I don't see any
> documentation explaining how this ought to work.  That documentation
> is essential.

Good point.  Especially because I think this changed from Emacs-20 to
Emacs-21, and I think it also changed now from Emacs-22 to Emacs-23.

IIUC if you want the character with code #xa0, then using \u00a0 would
seem like the most unambiguous option (I notice that "\ua0" gives
a weird error "Non-hex digit used for Unicode escape").

Not sure what \NNN or \xMM should do.


        Stefan

Re: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Jan 2009 22:06:21 -0500
> Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
> Good point.  Especially because I think this changed from Emacs-20 to
> Emacs-21, and I think it also changed now from Emacs-22 to Emacs-23.

I think the change in Emacs 23 is OK, but it needs to be consistent in
characters and strings.

> IIUC if you want the character with code #xa0, then using \u00a0 would
> seem like the most unambiguous option

Agreed.  Using \uNNNN is an unambiguous way of saying you want a Unicode
character whose codepoint is NNNN in hex.

> Not sure what \NNN or \xMM should do.

I think they should insert a raw byte with that code, and I think they
should do that both in characters and in strings, so that the
inconsistent behavior reported by Richard will become consistent.

Re: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    IIUC if you want the character with code #xa0, then using \u00a0 would
    seem like the most unambiguous option (I notice that "\ua0" gives
    a weird error "Non-hex digit used for Unicode escape").

I expected \xa0 to give me that character.  It still seems strange
that it would do anything else.

When I read the documentation of \u, I thought it meant "unicode" as
opposed to "Emacs's internal code".  Since I knew that Emacs now
follows unicode for these characters, I saw no reason to consider
using \u.

Re: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> Date: Sat, 03 Jan 2009 10:21:58 -0500
> Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
>     IIUC if you want the character with code #xa0, then using \u00a0 would
>     seem like the most unambiguous option (I notice that "\ua0" gives
>     a weird error "Non-hex digit used for Unicode escape").
> 
> I expected \xa0 to give me that character.  It still seems strange
> that it would do anything else.

We need some way of inserting raw 8-bit bytes, because otherwise code
that encodes and decodes text in Lisp will not work.  For inserting
characters, we have the \u alternative; but I don't think there's
alternative for raw bytes except insert \xNN.

Re: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    We need some way of inserting raw 8-bit bytes, because otherwise code
    that encodes and decodes text in Lisp will not work.  For inserting
    characters, we have the \u alternative; but I don't think there's
    alternative for raw bytes except insert \xNN.

Naybe that is a valid reason for the current behavior, but that
doesn't alter the need for the manual to document the behavior.

Meanwhile, the Chinese and Chinese-derived character codes
do not follow Unicode.  So you can't enter them with \u.
What is the way to enter them?

Re: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> CC: cyd@stupidchicken.com, emacs-devel@gnu.org,
> 	monnier@iro.umontreal.ca, 1726@emacsbugs.donarmstrong.com
> Date: Sat, 03 Jan 2009 21:16:21 -0500
> 
>     We need some way of inserting raw 8-bit bytes, because otherwise code
>     that encodes and decodes text in Lisp will not work.  For inserting
>     characters, we have the \u alternative; but I don't think there's
>     alternative for raw bytes except insert \xNN.
> 
> Naybe that is a valid reason for the current behavior, but that
> doesn't alter the need for the manual to document the behavior.

That was an attempt at explaining the reasons, not telling they don't
need to be documented.

> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.  So you can't enter them with \u.
> What is the way to enter them?

The problem at hand exists only for codes that are less than FF hex.

Re: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    > Meanwhile, the Chinese and Chinese-derived character codes
    > do not follow Unicode.  So you can't enter them with \u.
    > What is the way to enter them?

    The problem at hand exists only for codes that are less than FF hex.

Maybe, but isn't there a similar problem for Chinese-derived
characters?  How does one specify these codes in a string constant?
Shouldn't there be some way?

Maybe there is never a need to do it; maybe we don't need to add a
feature for it.  But if we don't, we should document that there is
currently no way.

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Jason Rumney]

Richard M Stallman wrote:
> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.

They do in Emacs 23, though I think if you enter \x1234, it will be 
treated the same as \u1234, as characters with more than 8 bits are 
clearly not eight-bit raw bytes.

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> Date: Sun, 04 Jan 2009 12:29:05 +0800
> From: Jason Rumney <jasonr@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, cyd@stupidchicken.com, emacs-devel@gnu.org
> 
> Richard M Stallman wrote:
> > Meanwhile, the Chinese and Chinese-derived character codes
> > do not follow Unicode.
> 
> They do in Emacs 23

Not all of them, I think.

Re: 23.0.60; end-of-sentence and non-breaking space

[Kenichi Handa]

In article <E1LJIXN-0008Vg-Pe@fencepost.gnu.org>, Richard M Stallman <rms@gnu.org> writes:

>     We need some way of inserting raw 8-bit bytes, because otherwise code
>     that encodes and decodes text in Lisp will not work.  For inserting
>     characters, we have the \u alternative; but I don't think there's
>     alternative for raw bytes except insert \xNN.

> Naybe that is a valid reason for the current behavior, but that
> doesn't alter the need for the manual to document the behavior.

> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.  So you can't enter them with \u.
> What is the way to enter them?

Most of Chinese and Chinese-derived character codes are
unified into Unicode area.  Only a few codes can't be
unified with Unicode, and thus decoded into the character
space over #x110000.  But, in that sense, Chinese and
Chinese-derived character codes are not special.  There
exist several non-Chinese character sets (e.g. tibetan)
containing characters that doesn't exist in Unicode, and
they are decoded into the character space over #x110000 too.

But, all of them can be accessed by "\U00XXXXXX".

---
Kenichi Handa
handa@m17n.org

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

      There
    exist several non-Chinese character sets (e.g. tibetan)
    containing characters that doesn't exist in Unicode, and
    they are decoded into the character space over #x110000 too.

    But, all of them can be accessed by "\U00XXXXXX".

Can you please document this (and the rest of what we have discussed
in this thread)?

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> Date: Mon, 05 Jan 2009 19:01:02 -0500
> Cc: cyd@stupidchicken.com, bug-submit-list@donarmstrong.com,
> 	bug-gnu-emacs@gnu.org, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
>       There
>     exist several non-Chinese character sets (e.g. tibetan)
>     containing characters that doesn't exist in Unicode, and
>     they are decoded into the character space over #x110000 too.
> 
>     But, all of them can be accessed by "\U00XXXXXX".
> 
> Can you please document this (and the rest of what we have discussed
> in this thread)?

You already asked me to do this, and it's on my TODO.

Re: 23.0.60; end-of-sentence and non-breaking space

[Kenichi Handa]

In article <utz8gmj9t.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Richard M Stallman <rms@gnu.org>
> > Date: Sat, 03 Jan 2009 10:21:58 -0500
> > Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> > 
> >     IIUC if you want the character with code #xa0, then using \u00a0 would
> >     seem like the most unambiguous option (I notice that "\ua0" gives
> >     a weird error "Non-hex digit used for Unicode escape").
> > 
> > I expected \xa0 to give me that character.  It still seems strange
> > that it would do anything else.

> We need some way of inserting raw 8-bit bytes, because otherwise code
> that encodes and decodes text in Lisp will not work.  For inserting
> characters, we have the \u alternative; but I don't think there's
> alternative for raw bytes except insert \xNN.

I modified read_escape to treat "\xXX" as a raw-byte code
but treat "\xXXX.." as a character code U+XXX...  As far as
I remember, this is to keep backward compatibility.

And, we have the alternative for raw bytes.  That is to use
octal form, something like "\240".

---
Kenichi Handa
handa@m17n.org