bug#24603: [RFC 15/18] Base lower- and upper-case tests on Unicode properties

[Eli Zaretskii <eliz@gnu.org>]

> From: Michal Nazarewicz <mina86@mina86.com>
> Date: Tue,  4 Oct 2016 03:10:38 +0200
> 
> +** 'upper' and 'lower' character classes are unaffected by case table
> +since they are now based purely on Unicode properties.

This is actually a backward-incompatible change, isn't it?  If so, it
should be in the corresponding section of NEWS.  More importantly,
there should be a way to get back the old behavior, i.e. to force
'upper' and 'lower' use the current case tables.

Better yet, can we use the Unicode properties only where case tables
are insufficient, like in the case of ligatures being broken up into
individual characters by case conversions?  That'd be
backward-compatible, so won't risk breaking existing code.

I'm also okay with a defcustom, by default off, to prefer the Unicode
data, as you did, so that we could in the future make this the default
behavior.  But doing this right now without any transition period and
no way of going back is too radical, I think.

Please also note that Unicode tables are global, very large, and in
many cases tricky to change from Lisp (as compared to simple
char-tables).  So customizing the case conversions that are based
solely on the Unicode tables is much harder and/or has global
implications, unlike the case tables.  With that in mind, I think we
should make the transition smoother, and we should probably add
convenience APIs for customizing the case conversions the Unicode way,
before we switch to that as the default.

Thanks.