bug#1726: 23.0.60; end-of-sentence and non-breaking space

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    This raw-byte char is what used to be called an eight-bit-control (or
    eight-bit-graphic depending on the actual value) char.

    I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
    eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
    non-breaking space).

1. Is that the right thing for \xa0 in a string to mean?
Or should it mean the character with code xa0?

2. I find it hard to think about that question since I don't see any
documentation explaining how this ought to work.  That documentation
is essential.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Stefan Monnier]

>     This raw-byte char is what used to be called an eight-bit-control (or
>     eight-bit-graphic depending on the actual value) char.

>     I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
>     eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
>     non-breaking space).

> 1. Is that the right thing for \xa0 in a string to mean?
> Or should it mean the character with code xa0?

> 2. I find it hard to think about that question since I don't see any
> documentation explaining how this ought to work.  That documentation
> is essential.

Good point.  Especially because I think this changed from Emacs-20 to
Emacs-21, and I think it also changed now from Emacs-22 to Emacs-23.

IIUC if you want the character with code #xa0, then using \u00a0 would
seem like the most unambiguous option (I notice that "\ua0" gives
a weird error "Non-hex digit used for Unicode escape").

Not sure what \NNN or \xMM should do.


        Stefan

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Jan 2009 22:06:21 -0500
> Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
> Good point.  Especially because I think this changed from Emacs-20 to
> Emacs-21, and I think it also changed now from Emacs-22 to Emacs-23.

I think the change in Emacs 23 is OK, but it needs to be consistent in
characters and strings.

> IIUC if you want the character with code #xa0, then using \u00a0 would
> seem like the most unambiguous option

Agreed.  Using \uNNNN is an unambiguous way of saying you want a Unicode
character whose codepoint is NNNN in hex.

> Not sure what \NNN or \xMM should do.

I think they should insert a raw byte with that code, and I think they
should do that both in characters and in strings, so that the
inconsistent behavior reported by Richard will become consistent.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    IIUC if you want the character with code #xa0, then using \u00a0 would
    seem like the most unambiguous option (I notice that "\ua0" gives
    a weird error "Non-hex digit used for Unicode escape").

I expected \xa0 to give me that character.  It still seems strange
that it would do anything else.

When I read the documentation of \u, I thought it meant "unicode" as
opposed to "Emacs's internal code".  Since I knew that Emacs now
follows unicode for these characters, I saw no reason to consider
using \u.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> Date: Sat, 03 Jan 2009 10:21:58 -0500
> Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
>     IIUC if you want the character with code #xa0, then using \u00a0 would
>     seem like the most unambiguous option (I notice that "\ua0" gives
>     a weird error "Non-hex digit used for Unicode escape").
> 
> I expected \xa0 to give me that character.  It still seems strange
> that it would do anything else.

We need some way of inserting raw 8-bit bytes, because otherwise code
that encodes and decodes text in Lisp will not work.  For inserting
characters, we have the \u alternative; but I don't think there's
alternative for raw bytes except insert \xNN.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    We need some way of inserting raw 8-bit bytes, because otherwise code
    that encodes and decodes text in Lisp will not work.  For inserting
    characters, we have the \u alternative; but I don't think there's
    alternative for raw bytes except insert \xNN.

Naybe that is a valid reason for the current behavior, but that
doesn't alter the need for the manual to document the behavior.

Meanwhile, the Chinese and Chinese-derived character codes
do not follow Unicode.  So you can't enter them with \u.
What is the way to enter them?

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Kenichi Handa]

In article <utz8gmj9t.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Richard M Stallman <rms@gnu.org>
> > Date: Sat, 03 Jan 2009 10:21:58 -0500
> > Cc: cyd@stupidchicken.com, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> > 
> >     IIUC if you want the character with code #xa0, then using \u00a0 would
> >     seem like the most unambiguous option (I notice that "\ua0" gives
> >     a weird error "Non-hex digit used for Unicode escape").
> > 
> > I expected \xa0 to give me that character.  It still seems strange
> > that it would do anything else.

> We need some way of inserting raw 8-bit bytes, because otherwise code
> that encodes and decodes text in Lisp will not work.  For inserting
> characters, we have the \u alternative; but I don't think there's
> alternative for raw bytes except insert \xNN.

I modified read_escape to treat "\xXX" as a raw-byte code
but treat "\xXXX.." as a character code U+XXX...  As far as
I remember, this is to keep backward compatibility.

And, we have the alternative for raw bytes.  That is to use
octal form, something like "\240".

---
Kenichi Handa
handa@m17n.org

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> CC: cyd@stupidchicken.com, emacs-devel@gnu.org,
> 	monnier@iro.umontreal.ca, 1726@emacsbugs.donarmstrong.com
> Date: Sat, 03 Jan 2009 21:16:21 -0500
> 
>     We need some way of inserting raw 8-bit bytes, because otherwise code
>     that encodes and decodes text in Lisp will not work.  For inserting
>     characters, we have the \u alternative; but I don't think there's
>     alternative for raw bytes except insert \xNN.
> 
> Naybe that is a valid reason for the current behavior, but that
> doesn't alter the need for the manual to document the behavior.

That was an attempt at explaining the reasons, not telling they don't
need to be documented.

> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.  So you can't enter them with \u.
> What is the way to enter them?

The problem at hand exists only for codes that are less than FF hex.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Jason Rumney]

Richard M Stallman wrote:
> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.

They do in Emacs 23, though I think if you enter \x1234, it will be 
treated the same as \u1234, as characters with more than 8 bits are 
clearly not eight-bit raw bytes.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> Date: Sun, 04 Jan 2009 12:29:05 +0800
> From: Jason Rumney <jasonr@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, cyd@stupidchicken.com, emacs-devel@gnu.org
> 
> Richard M Stallman wrote:
> > Meanwhile, the Chinese and Chinese-derived character codes
> > do not follow Unicode.
> 
> They do in Emacs 23

Not all of them, I think.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Kenichi Handa]

In article <E1LJIXN-0008Vg-Pe@fencepost.gnu.org>, Richard M Stallman <rms@gnu.org> writes:

>     We need some way of inserting raw 8-bit bytes, because otherwise code
>     that encodes and decodes text in Lisp will not work.  For inserting
>     characters, we have the \u alternative; but I don't think there's
>     alternative for raw bytes except insert \xNN.

> Naybe that is a valid reason for the current behavior, but that
> doesn't alter the need for the manual to document the behavior.

> Meanwhile, the Chinese and Chinese-derived character codes
> do not follow Unicode.  So you can't enter them with \u.
> What is the way to enter them?

Most of Chinese and Chinese-derived character codes are
unified into Unicode area.  Only a few codes can't be
unified with Unicode, and thus decoded into the character
space over #x110000.  But, in that sense, Chinese and
Chinese-derived character codes are not special.  There
exist several non-Chinese character sets (e.g. tibetan)
containing characters that doesn't exist in Unicode, and
they are decoded into the character space over #x110000 too.

But, all of them can be accessed by "\U00XXXXXX".

---
Kenichi Handa
handa@m17n.org

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

      There
    exist several non-Chinese character sets (e.g. tibetan)
    containing characters that doesn't exist in Unicode, and
    they are decoded into the character space over #x110000 too.

    But, all of them can be accessed by "\U00XXXXXX".

Can you please document this (and the rest of what we have discussed
in this thread)?

Re: bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Eli Zaretskii]

> From: Richard M Stallman <rms@gnu.org>
> Date: Mon, 05 Jan 2009 19:01:02 -0500
> Cc: cyd@stupidchicken.com, bug-submit-list@donarmstrong.com,
> 	bug-gnu-emacs@gnu.org, 1726@emacsbugs.donarmstrong.com, emacs-devel@gnu.org
> 
>       There
>     exist several non-Chinese character sets (e.g. tibetan)
>     containing characters that doesn't exist in Unicode, and
>     they are decoded into the character space over #x110000 too.
> 
>     But, all of them can be accessed by "\U00XXXXXX".
> 
> Can you please document this (and the rest of what we have discussed
> in this thread)?

You already asked me to do this, and it's on my TODO.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    > Meanwhile, the Chinese and Chinese-derived character codes
    > do not follow Unicode.  So you can't enter them with \u.
    > What is the way to enter them?

    The problem at hand exists only for codes that are less than FF hex.

Maybe, but isn't there a similar problem for Chinese-derived
characters?  How does one specify these codes in a string constant?
Shouldn't there be some way?

Maybe there is never a need to do it; maybe we don't need to add a
feature for it.  But if we don't, we should document that there is
currently no way.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

    > When I type C-x = at a non-breaking space, it tells me that it
    > has code 160, hex a0.  But when I execute (insert "\xa0"),
    > it inserts something that displays as `\240' and for which C-x =
    > displays this:

    >    Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
    >    (93%) column=5
    >
    > Is that a bug?  It seems quite confusing to me.

    ISTR that there was an extended discussion about classifying
    non-breaking spaces on this list a while back.  But I can't find it now.
    Does anyone remember the details?

I am not sure we are talking about the same question.
The issue I am raising is not one of classifying it,
it is that these two different character codes get used
and I don't see an explanation of what's going on.

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Stefan Monnier]

>> has code 160, hex a0.  But when I execute (insert "\xa0"),
>> it inserts something that displays as `\240' and for which C-x =
>> displays this:
>> 
>> Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
>> (93%) column=5
>> 
>> Is that a bug?  It seems quite confusing to me.

This raw-byte char is what used to be called an eight-bit-control (or
eight-bit-graphic depending on the actual value) char.

I.e. "\xa0" is treated as a string that contains the \xa0 byte (i.e. an
eight-bit-* (aka raw-byte) char) rather than the \xa0 char (a latin-1
non-breaking space).


        Stefan

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Chong Yidong]

From bug#1726 and bug#1727:

> forward-sentence does not treat non-breaking space as a space for
> purposes of sentence ends.
...
> When I type C-x = at a non-breaking space, it tells me that it
> has code 160, hex a0.  But when I execute (insert "\xa0"),
> it inserts something that displays as `\240' and for which C-x =
> displays this:
>
>    Char:   (4194208, #o17777640, #x3fffa0, raw-byte) point=198 of 211
>    (93%) column=5
>
> Is that a bug?  It seems quite confusing to me.

ISTR that there was an extended discussion about classifying
non-breaking spaces on this list a while back.  But I can't find it now.
Does anyone remember the details?

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Richard M Stallman]

forward-sentence does not treat non-breaking space as a space
for purposes of sentence ends.

I will fix this as soon as I know a way to put non-breaking space
into a string constant.



In GNU Emacs 23.0.60.15 (mipsel-unknown-linux-gnu, GTK+ Version 2.12.11)
 of 2008-12-22 on lemote-yeeloong
configured using `configure  'CFLAGS=-O0 -g -Wno-pointer-sign' 'mipsel-unknown-linux-gnu' 'build_alias=mipsel-unknown-linux-gnu' 'host_alias=mipsel-unknown-linux-gnu' 'target_alias=mipsel-unknown-linux-gnu''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Mail

Minor modes in effect:
  gpm-mouse-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
r e SPC s h o u l d SPC b e SPC a SPC C-x C-f ESC p 
ESC DEL ESC DEL l r e a d . c RET C-s C-j r e a d _ 
s c DEL DEL e s c C-v ESC v C-v C-v C-v C-v C-v C-v 
C-u C-x m C-a C-k P e r h a p s SPC \ N SPC s h o u 
d l SPC g i v e ESC b C-b C-b C-t C-e SPC n o n - b 
r e a k i n g SPC s o p a c e . ESC b C-f C-d C-c C-c 
C-x b o u t g TAB RET g C-p C-p C-n C-n R o u t - 2 
5 RET C-p C-p e C-u C-u C-n C-u C-u C-n C-p C-p C-o 
C-o I SPC n e e d SPC t h i s SPC ESC DEL t o SPC s 
o l v e SPC t h i s SPC p r o b l e m SPC i n SPC o 
r d e r SPC t o SPC i DEL f i x SPC p a r a g r a p 
h s . e l RET t o SPC h a b d l e C-b C-b C-b DEL n 
C-p C-e ESC DEL ESC DEL f o r w a r d - s e n t e n 
c e C-n SPC n o n ESC / ESC SPC ESC / . C-x C-s C-a 
C-p C-p C-@ C-n C-n C-n C-w C-x C-s ESC x r e p o r 
t SPC e m a v s SPC DEL DEL c s SPC b u g RET

Recent messages:
Auto save file for draft message exists; consider M-x mail-recover
Sending...
Wrote /home/rms/outgoing/out-24
Sending...done
Move: 1 of 1
Move: 1 file
Wrote /home/rms/outgoing/out-24
Mark set
Auto-saving...done
Wrote /home/rms/outgoing/out-24

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Lars Magne Ingebrigtsen]

Richard M Stallman <rms@gnu.org> writes:

> forward-sentence does not treat non-breaking space as a space
> for purposes of sentence ends.
>
> I will fix this as soon as I know a way to put non-breaking space
> into a string constant.

The discussion then turned to the various string literal syntaxes.

Has this issue been resolved so that the bug report can be closed?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#1726: 23.0.60; end-of-sentence and non-breaking space

[Glenn Morris]

Version: 23.1

Lars Magne Ingebrigtsen wrote:

> Richard M Stallman <rms@gnu.org> writes:
>
>> forward-sentence does not treat non-breaking space as a space
>> for purposes of sentence ends.
[...]
> The discussion then turned to the various string literal syntaxes.
>
> Has this issue been resolved so that the bug report can be closed?

I see the issue in 22.3 but not 23.1 and later, so I assume it was fixed
ages ago.

2009-01-16  Richard M Stallman  <rms at gnu.org>

 * textmodes/paragraphs.el (sentence-end): Accept non-break space.