bug#10576: Subject: 23.4; char class [:lower:] misses latin small letter sharp s

bug#10576: Subject: 23.4; char class [:lower:] misses latin small letter sharp s

[Andreas Röhler]

Hi,

in scratch-buffer, emacs -Q,

cursor at end of line at LATIN SMALL LETTER SHARP S,

evaluating the form below results in "0", i.e. no move

(skip-chars-forward "[:lower:]")ß

AFAIU [:lower:] doesn't recognise that char

C-u C-x =
says:

         character: ß (223, #o337, #xdf)
preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
        code point: 0xDF
            syntax: w 	which means: word
          category: .:Base, h:Korean, j:Japanese, l:Latin
       buffer code: #xC3 #x9F
         file code: #xC3 #x9F (encoded by coding system utf-8-unix)
           display: by this font (glyph code)
     xft:-unknown-DejaVu Sans 
Mono-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1 (#xA1)

Character code properties: customize what to show
   name: LATIN SMALL LETTER SHARP S
   general-category: Ll (Letter, Lowercase)

There are text properties here:
   fontified            t

;;;;;;;;;;;

thanks all,

Andreas

In GNU Emacs 23.4.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
  of 2012-01-20
Windowing system distributor `The X.Org Foundation', version 11.0.10903000
configured using `configure  'CFLAGS=-g -O''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: de_DE.UTF-8
   value of $XMODIFIERS: @im=local
   locale-coding-system: utf-8-unix
   default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
( s k i p - c h a r s - f o r w a r d SPC " [ [ : l
o w e r : ] ] " ) a s d f <left> <left> <left> <left>
C-x C-e <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <left> C-d <right> <right> <right> <right> <right>
<right> <right> <right> <right> C-d <right> <right>
C-x C-e <left> <right> <backspace> <backspace> <backspace>
<backspace> ? <left> C-d ß <left> C-x C-e C-u C-c C-r
C-u C-x C-e <backspace> <right> <return> <return> <up>
<up> <up> <return> e v a l u a t i n g SPC t h e SPC
f o r m SPC b e l w o <backspace> <backspace> o w SPC
r e s u l t s SPC i n SPC " 0 " , SPC i . e . SPC <return>
<left> n o SPC m o v e <down> <down> <return> <return>
<up> <left> <left> C-x C-e M-x r e p o r t - e a c
<backspace> <backspace> m a c s - b u <tab> <retur
n>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
4 [2 times]
0
0

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader
gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 hex-util hashcash mail-utils emacsbug tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd font-setting
tool-bar dnd fontset image fringe lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mldrag mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting x-toolkit x multi-tty emacs)

bug#10576: Subject: 23.4; char class [:lower:] misses latin small letter sharp s

[Andreas Schwab]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

> evaluating the form below results in "0", i.e. no move
>
> (skip-chars-forward "[:lower:]")ß
>
> AFAIU [:lower:] doesn't recognise that char

That's because ?ß has no upper case, so lowercasep in src/buffer.h
return 0.

(Actually in Unicode, ?ß does have a capital variant, ?ẞ, but while
lowercasing ?ẞ maps to ?ß, uppercasing ?ß actually maps to "SS".)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Jorgen Schaefer]

Hi!
The following seems like a bug:

(string-match "[[:lower:]]" "ß") => nil

`describe-char' for this says:

  name: LATIN SMALL LETTER SHARP S
  general-category: Ll (Letter, Lowercase)
  decomposition: (223) ('ß')

Not sure why it would not be considered a lower-case letter. Umlauts
like ä, ö and ü are matched correctly.

Regards,
        -- Jorgen

Configured using:
 `configure --without-x'

Important settings:
  value of $LC_ALL: 
  value of $LC_COLLATE: de_DE.UTF-8
  value of $LC_CTYPE: de_DE.UTF-8
  value of $LC_MESSAGES: POSIX
  value of $LC_MONETARY: POSIX
  value of $LC_NUMERIC: POSIX
  value of $LC_TIME: POSIX
  value of $LANG: POSIX
  locale-coding-system: utf-8-unix

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Glenn Morris]

Jorgen Schaefer wrote:

> Not sure why it would not be considered a lower-case letter. Umlauts
> like ä, ö and ü are matched correctly.

See http://debbugs.gnu.org/10576

(I have no idea whether this is an Emacs bug or not.)

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Andreas Röhler]

Am 12.02.2014 18:55, schrieb Glenn Morris:
> Jorgen Schaefer wrote:
>
>> Not sure why it would not be considered a lower-case letter. Umlauts
>> like ä, ö and ü are matched correctly.
>
> See http://debbugs.gnu.org/10576
>
> (I have no idea whether this is an Emacs bug or not.)
>
>
>
>

IMO the answer given at link is not valid. Indeed the implementation in buffer.h does check --&& upcase1 (c)-- and expects a result, i.e. ignores the fact, some characters 
might not have an upcase variant.

When seeing there is a downcase-table, the check probably should be done against this.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> Date: Wed, 12 Feb 2014 20:31:20 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> 
> > See http://debbugs.gnu.org/10576
> >
> > (I have no idea whether this is an Emacs bug or not.)
> >
> 
> IMO the answer given at link is not valid.

It accurately describes what happens in the code, so it's definitely
valid.

> When seeing there is a downcase-table, the check probably should be done against this.

Not sure what you mean by that, please elaborate.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Andreas Röhler]

Am 12.02.2014 20:49, schrieb Eli Zaretskii:
>> Date: Wed, 12 Feb 2014 20:31:20 +0100
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>>
>>> See http://debbugs.gnu.org/10576
>>>
>>> (I have no idea whether this is an Emacs bug or not.)
>>>
>>
>> IMO the answer given at link is not valid.
>
> It accurately describes what happens in the code, so it's definitely
> valid.
>
>> When seeing there is a downcase-table, the check probably should be done against this.
>
> Not sure what you mean by that, please elaborate.
>
>

See buffer.h
IIUC the mentioned lowercasep is implemented as !uppercasep (c) && upcase1 (c) != c;
upcase1 (c) must fail, as there is no upcased of this char.

While upcase1 can't succeed, downcase should - if "ß" is a member of downcase_table.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> Date: Wed, 12 Feb 2014 21:10:57 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> CC: 16731@debbugs.gnu.org
> 
> While upcase1 can't succeed, downcase should - if "ß" is a member of downcase_table.

But which character do you want to downcase in this case?

This whole logic works only for _pairs_ of characters (and the
char-table used here is populated by calls to set-case-syntax-pair).
Such machinery cannot possibly work when there's no pair.

The only way I can see out of this conundrum is to consult the
Lowercase Unicode property of the character as fallback, assuming that
won't slow down regex search too much.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Andreas Röhler]

Am 12.02.2014 21:16, schrieb Eli Zaretskii:
>> Date: Wed, 12 Feb 2014 21:10:57 +0100
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>> CC: 16731@debbugs.gnu.org
>>
>> While upcase1 can't succeed, downcase should - if "ß" is a member of downcase_table.
>
> But which character do you want to downcase in this case?
>
> This whole logic works only for _pairs_ of characters (and the
> char-table used here is populated by calls to set-case-syntax-pair).

So populate it differently, resp. allow empty slots.

> Such machinery cannot possibly work when there's no pair.
>
> The only way I can see out of this conundrum is to consult the
> Lowercase Unicode property of the character as fallback, assuming that
> won't slow down regex search too much.
>
>

You can do (downcase "d") for example, which results in "d".

Instead of

upcase1 (c) != c

what about

downcase (c) == c

?

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Juanma Barranquero]

On Wed, Feb 12, 2014 at 9:33 PM, Andreas Röhler
<andreas.roehler@easy-emacs.de> wrote:

> what about
>
> downcase (c) == c

Won't that be true for characters that have no upcase/downcase
difference, like digits?

   J

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> Date: Wed, 12 Feb 2014 21:33:31 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> CC: 16731@debbugs.gnu.org
> 
> Am 12.02.2014 21:16, schrieb Eli Zaretskii:
> >> Date: Wed, 12 Feb 2014 21:10:57 +0100
> >> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> >> CC: 16731@debbugs.gnu.org
> >>
> >> While upcase1 can't succeed, downcase should - if "ß" is a member of downcase_table.
> >
> > But which character do you want to downcase in this case?
> >
> > This whole logic works only for _pairs_ of characters (and the
> > char-table used here is populated by calls to set-case-syntax-pair).
> 
> So populate it differently, resp. allow empty slots.

How will we then be able to distinguish between lower-case characters
that have no upcase variant and characters that are not lower-case
characters at all?

> You can do (downcase "d") for example, which results in "d".
> 
> Instead of
> 
> upcase1 (c) != c
> 
> what about
> 
> downcase (c) == c
> 
> ?

The same is true for any non-letter, like punctuation.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Andreas Röhler]

Am 13.02.2014 04:46, schrieb Eli Zaretskii:
>> Date: Wed, 12 Feb 2014 21:33:31 +0100
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>> CC: 16731@debbugs.gnu.org
>>
>> Am 12.02.2014 21:16, schrieb Eli Zaretskii:
>>>> Date: Wed, 12 Feb 2014 21:10:57 +0100
>>>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>>>> CC: 16731@debbugs.gnu.org
>>>>
>>>> While upcase1 can't succeed, downcase should - if "ß" is a member of downcase_table.
>>>
>>> But which character do you want to downcase in this case?
>>>
>>> This whole logic works only for _pairs_ of characters (and the
>>> char-table used here is populated by calls to set-case-syntax-pair).
>>
>> So populate it differently, resp. allow empty slots.
>
> How will we then be able to distinguish between lower-case characters
> that have no upcase variant and characters that are not lower-case
> characters at all?
>
>> You can do (downcase "d") for example, which results in "d".
>>
>> Instead of
>>
>> upcase1 (c) != c
>>
>> what about
>>
>> downcase (c) == c
>>
>> ?
>
> The same is true for any non-letter, like punctuation.
>
>

Okay, right.

So it seems upcase_table is populated wrongly with "ß"?

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

> How will we then be able to distinguish between lower-case characters
> that have no upcase variant and characters that are not lower-case
> characters at all?

Right: to handle this, we need to distinguish characters that are
lower-case without an uppercase variant from characters which are
neither lowercase nor uppercase.

We could do that by saying that the upcase table should return nil or -1
for ß, to indicate that the upcase version is "missing".  But such
a change will probably require carefully revising "all" the code that
uses those tables.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> Date: Thu, 13 Feb 2014 09:27:43 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> CC: 16731@debbugs.gnu.org, Juanma Barranquero <lekktu@gmail.com>
> 
> So it seems upcase_table is populated wrongly with "ß"?

I see nothing wrong with it: its entry is the character itself, like
any other character that has no up-case variant.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Andreas Röhler <andreas.roehler@easy-emacs.de>,
>   16731@debbugs.gnu.org
> Date: Thu, 13 Feb 2014 08:37:45 -0500
> 
> > How will we then be able to distinguish between lower-case characters
> > that have no upcase variant and characters that are not lower-case
> > characters at all?
> 
> Right: to handle this, we need to distinguish characters that are
> lower-case without an uppercase variant from characters which are
> neither lowercase nor uppercase.
> 
> We could do that by saying that the upcase table should return nil or -1
> for ß, to indicate that the upcase version is "missing".  But such
> a change will probably require carefully revising "all" the code that
> uses those tables.

Right.  I can instead suggest a much less intrusive change below.  Its
only disadvantage is that if some user or Lisp program overrides the
standard case tables, and actually _wants_ some lower-case characters
behave as if they weren't, looking at the Unicode tables will undo
such customizations.  If this is a concern, perhaps we could compare
the case table with the standard value, and only use the Unicode
attributes when they are equal?

If the approach below is accepted, a related question is how to treat
letters whose category is Lt, i.e. "titlecase" -- do we consider such
letters upper case or don't we?

--- src/buffer.h~0	2014-01-01 09:46:07.000000000 +0200
+++ src/buffer.h	2014-02-13 18:27:32.225839000 +0200
@@ -1349,7 +1349,19 @@ downcase (int c)
 }
 
 /* True if C is upper case.  */
-INLINE bool uppercasep (int c) { return downcase (c) != c; }
+INLINE bool uppercasep (int c)
+{
+  Lisp_Object val;
+
+  if (downcase (c) != c)
+    return true;
+
+  if (NILP (Vunicode_category_table))
+    return false;
+
+  val = CHAR_TABLE_REF (Vunicode_category_table, c);
+  return INTEGERP (val) && XINT (val) == UNICODE_CATEGORY_Lu;
+}
 
 /* Upcase a character C known to be not upper case.  */
 INLINE int
@@ -1364,7 +1376,16 @@ upcase1 (int c)
 INLINE bool
 lowercasep (int c)
 {
-  return !uppercasep (c) && upcase1 (c) != c;
+  Lisp_Object val;
+
+  if (!uppercasep (c) && upcase1 (c) != c)
+    return true;
+
+  if (NILP (Vunicode_category_table))
+    return false;
+
+  val = CHAR_TABLE_REF (Vunicode_category_table, c);
+  return INTEGERP (val) && XINT (val) == UNICODE_CATEGORY_Ll;
 }
 
 /* Upcase a character C, or make no change if that cannot be done.  */

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

>  /* True if C is upper case.  */
> -INLINE bool uppercasep (int c) { return downcase (c) != c; }
> +INLINE bool uppercasep (int c)
> +{
> +  Lisp_Object val;
> +
> +  if (downcase (c) != c)
> +    return true;
> +
> +  if (NILP (Vunicode_category_table))
> +    return false;
> +
> +  val = CHAR_TABLE_REF (Vunicode_category_table, c);
> +  return INTEGERP (val) && XINT (val) == UNICODE_CATEGORY_Lu;
> +}
 
Doesn't sound too bad.  But it does beg the question: why check
(downcase (c) != c) at all, then?


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Thu, 13 Feb 2014 12:10:49 -0500
> 
> >  /* True if C is upper case.  */
> > -INLINE bool uppercasep (int c) { return downcase (c) != c; }
> > +INLINE bool uppercasep (int c)
> > +{
> > +  Lisp_Object val;
> > +
> > +  if (downcase (c) != c)
> > +    return true;
> > +
> > +  if (NILP (Vunicode_category_table))
> > +    return false;
> > +
> > +  val = CHAR_TABLE_REF (Vunicode_category_table, c);
> > +  return INTEGERP (val) && XINT (val) == UNICODE_CATEGORY_Lu;
> > +}
>  
> Doesn't sound too bad.  But it does beg the question: why check
> (downcase (c) != c) at all, then?

Because it's faster, and for most characters will do the job.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Juanma Barranquero]

On Thu, Feb 13, 2014 at 5:33 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> If the approach below is accepted, a related question is how to treat
> letters whose category is Lt, i.e. "titlecase" -- do we consider such
> letters upper case or don't we?

No Unicode expert, but this suggest they are uppercase, sort of:

http://www.unicode.org/faq/casemap_charprop.html

"Q: What is titlecase? How is it different from uppercase?

A: Titlecase takes its name from the case format used when forming a
title, in which the initial letter in a word is capitalized and the
rest are not. Titlecase is also used in forming a sentence by
capitalizing the first word, and for forming proper names. The
titlecase mapping in the Unicode Standard is the mapping applied to
the initial character in a word.

The titlecase mapping in Unicode differs from the uppercase mapping in
that a number of characters require special handling. These are
chiefly ligatures and digraphs such as 'fl', 'dz', and 'lj', plus a
number of polytonic Greek characters. For example, U+01C7 (LJ) maps to
U+01C8 (Lj) rather than to U+01C9 (lj)."

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Andreas Röhler]

Am 13.02.2014 18:39, schrieb Eli Zaretskii:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
>> Date: Thu, 13 Feb 2014 12:10:49 -0500
>>
>>>   /* True if C is upper case.  */
>>> -INLINE bool uppercasep (int c) { return downcase (c) != c; }
>>> +INLINE bool uppercasep (int c)
>>> +{
>>> +  Lisp_Object val;
>>> +
>>> +  if (downcase (c) != c)
>>> +    return true;
>>> +
>>> +  if (NILP (Vunicode_category_table))
>>> +    return false;
>>> +
>>> +  val = CHAR_TABLE_REF (Vunicode_category_table, c);
>>> +  return INTEGERP (val) && XINT (val) == UNICODE_CATEGORY_Lu;
>>> +}
>>
>> Doesn't sound too bad.  But it does beg the question: why check
>> (downcase (c) != c) at all, then?
>
> Because it's faster, and for most characters will do the job.
>

Maybe I'm missing the point: all change needed is not to store "ß" into the uppercase-table.
Why not store nil there instead?

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

>> Doesn't sound too bad.  But it does beg the question: why check
>> (downcase (c) != c) at all, then?
> Because it's faster,

Is it?  Both lookups look like CHAR_TABLE_REF to me.

> and for most characters will do the job.

But we'll check the unicode table at least for more than half the
characters (i.e. for all the lowercase and non-case characters), so the
fast path can't give us more than a factor of 2 speed up anyway, and the
slow path is made slower by unnecessarily looking up the case table.

I guess what I mean is that without actual measurements it's not obvious
at all that speed is a good justification.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Thu, 13 Feb 2014 13:10:02 -0500
> 
> >> Doesn't sound too bad.  But it does beg the question: why check
> >> (downcase (c) != c) at all, then?
> > Because it's faster,
> 
> Is it?  Both lookups look like CHAR_TABLE_REF to me.
> 
> > and for most characters will do the job.
> 
> But we'll check the unicode table at least for more than half the
> characters (i.e. for all the lowercase and non-case characters), so the
> fast path can't give us more than a factor of 2 speed up anyway, and the
> slow path is made slower by unnecessarily looking up the case table.
> 
> I guess what I mean is that without actual measurements it's not obvious
> at all that speed is a good justification.

What about custom buffer-local case tables?

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> Date: Thu, 13 Feb 2014 19:02:08 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> CC: 16731@debbugs.gnu.org
> 
> Maybe I'm missing the point: all change needed is not to store "ß" into the uppercase-table.
> Why not store nil there instead?

Because that's not what case tables are documented to hold.  We will
break back compatibility if we put nil there.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Thu, 13 Feb 2014 18:58:04 +0100
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 16731@debbugs.gnu.org
> 
> On Thu, Feb 13, 2014 at 5:33 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > If the approach below is accepted, a related question is how to treat
> > letters whose category is Lt, i.e. "titlecase" -- do we consider such
> > letters upper case or don't we?
> 
> No Unicode expert, but this suggest they are uppercase, sort of:
> 
> http://www.unicode.org/faq/casemap_charprop.html
> 
> "Q: What is titlecase? How is it different from uppercase?
> 
> A: Titlecase takes its name from the case format used when forming a
> title, in which the initial letter in a word is capitalized and the
> rest are not. Titlecase is also used in forming a sentence by
> capitalizing the first word, and for forming proper names. The
> titlecase mapping in the Unicode Standard is the mapping applied to
> the initial character in a word.
> 
> The titlecase mapping in Unicode differs from the uppercase mapping in
> that a number of characters require special handling. These are
> chiefly ligatures and digraphs such as 'fl', 'dz', and 'lj', plus a
> number of polytonic Greek characters. For example, U+01C7 (LJ) maps to
> U+01C8 (Lj) rather than to U+01C9 (lj)."

The question is whether we want [:upper:] to match titlecase letters.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Juanma Barranquero]

On Thu, Feb 13, 2014 at 7:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> The question is whether we want [:upper:] to match titlecase letters.

Yes, I understand. And I'm pointing out that, unless there's a
separate [:title:] matcher, matching them with [:upper:] is not
entirely unreasonable. Whether it is the right thing to do or not will
depend on the uses, I think.

    J

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Glenn Morris]

Eli Zaretskii wrote:

> The question is whether we want [:upper:] to match titlecase letters.

What does grep do?
(http://debbugs.gnu.org/16631 ?)

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

> What about custom buffer-local case tables?

That's what I meant by my question, yes.  Your change will break about half of
the uses of buffer-local case tables.  Using the unicode table all the
time will break them all.
Is it a real issue?  I really don't know.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Juanma Barranquero <lekktu@gmail.com>,  16731@debbugs.gnu.org
> Date: Thu, 13 Feb 2014 13:47:33 -0500
> 
> Eli Zaretskii wrote:
> 
> > The question is whether we want [:upper:] to match titlecase letters.
> 
> What does grep do?
> (http://debbugs.gnu.org/16631 ?)

Grep (like most of other programs) uses locale-dependent tables
provided by libc, so it's not really relevant for us what it does.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Thu, 13 Feb 2014 14:15:37 -0500
> 
> > What about custom buffer-local case tables?
> 
> That's what I meant by my question, yes.  Your change will break about half of
> the uses of buffer-local case tables.  Using the unicode table all the
> time will break them all.
> Is it a real issue?  I really don't know.

Neither do I.

How about if we use the unicode tables only if the corresponding
buffer's case-table is the standard one (Vascii_*_table)?

bug#16731: 24.3.50; , Latin small letter sharp s is not considered lower-case

[Paul Eggert]

Grep doesn't just use glibc's tables; it has its own dfa matcher (also 
shared by awk), and runs into problem in this area as well.  I'm working 
on fixes for this in my limited spare time.

If you want 'uppercasep' to match what glibc and grep mean by 
[[:upper:]], Emacs might need to check not merely for 
UNICODE_CATEGORY_Lu but also for other Unicode categories (mixed case, 
title case).  I haven't investigated the details.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

>> Is it a real issue?  I really don't know.
> Neither do I.

Maybe it's not a problem.  Someone(TM) should grep to try and figure it
out, and then try it out.

> How about if we use the unicode tables only if the corresponding
> buffer's case-table is the standard one (Vascii_*_table)?

That sounds kludgy.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Fri, 14 Feb 2014 12:22:35 -0500
> 
> > How about if we use the unicode tables only if the corresponding
> > buffer's case-table is the standard one (Vascii_*_table)?
> 
> That sounds kludgy.

Why kludgy?  If the tables were not customized, it is a sign that this
buffer is OK with the default properties, which is what the Unicode
properties are about.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

>> > How about if we use the unicode tables only if the corresponding
>> > buffer's case-table is the standard one (Vascii_*_table)?
>> That sounds kludgy.
> Why kludgy?

Because, if someone were to take the Vascii_*_table, make a little
change to them and use them in a buffer, he suddenly gets different
behavior for some chars he hasn't touched.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Fri, 14 Feb 2014 15:59:00 -0500
> 
> >> > How about if we use the unicode tables only if the corresponding
> >> > buffer's case-table is the standard one (Vascii_*_table)?
> >> That sounds kludgy.
> > Why kludgy?
> 
> Because, if someone were to take the Vascii_*_table

How could they? these variables are not exposed to Lisp.  Only
ascii-case-table is, which is not the one I had in mind.

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Stefan Monnier]

> How could they? these variables are not exposed to Lisp.  Only
> ascii-case-table is, which is not the one I had in mind.

Right, I was thinking of standard-case-table.  Still, same problem: take
that standard case table change it a bit, and suddenly other chars than
the ones you changed are affected.


        Stefan

bug#16731: 24.3.50; Latin small letter sharp s is not considered lower-case

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: andreas.roehler@easy-emacs.de,  16731@debbugs.gnu.org
> Date: Sun, 16 Feb 2014 22:09:32 -0500
> 
> > How could they? these variables are not exposed to Lisp.  Only
> > ascii-case-table is, which is not the one I had in mind.
> 
> Right, I was thinking of standard-case-table.  Still, same problem: take
> that standard case table change it a bit, and suddenly other chars than
> the ones you changed are affected.

But customizing case-tables is already a very special use case.  Why
can't we expect such users to deal with these issues?

The only alternative (besides leaving the original problem unsolved)
is to ignore buffer-local case tables.  Is this more acceptable?

bug#10576: Subject: 23.4; char class [:lower:] misses latin small letter sharp s

[Lars Ingebrigtsen]

Jorgen Schaefer <forcer@forcix.cx> writes:

> The following seems like a bug:
>
> (string-match "[[:lower:]]" "ß") => nil

This has been fixed in Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no