bug#24425: [PATCH] Don’t cast Unicode to 8-bit when casing unibyte strings

[Eli Zaretskii <eliz@gnu.org>]

> From: Michal Nazarewicz <mina86@mina86.com>
> Date: Tue, 13 Sep 2016 00:46:07 +0200
> 
> Currently, when operating on unibyte strings and buffers, if casing
> ASCII character results in a Unicode character the result is forcefully
> converted to 8-bit by masking all but the eight least significant bits.
> This has awkward results such as:
> 
>         (let ((table (make-char-table 'case-table)))
>           (set-char-table-parent table (current-case-table))
>           (set-case-syntax-pair ?I ?ı table)
>           (set-case-syntax-pair ?İ ?i table)
>           (with-case-table table
>             (concat (upcase "istanabul") " " (downcase "IRMA"))))
>         => "0STANABUL 1rma"
> 
> Change the code so that ASCII characters being cased to Unicode
> characters are left unchanged when operating on unibyte data.  In other
> words, aforementioned example will produce:
> 
>         => "iSTANBUL "Irma"
> 
> Arguably this isn’t correct either but it’s less wrong and ther’s not
> much we can do when the strings are unibyte.

Thanks, but I don't think it's TRT to fix this in a way that produces
a semi-broken result.  Second-guessing what the user/caller means and
silently producing results that only make sense if the guess was
correct is about the worst thing we could do in these dark-corner
situations.

Currently, case changes in unibyte characters and strings are only
well defined for pure ASCII text; if the input or the result is not
pure ASCII, we produce "undefined behavior".  In particular, case
tables are not set at all for unibyte characters, because it's not
text, it's a byte stream.  Either we decide that we don't want to
support case changes in unibyte non-ASCII characters, and we stick to
the current behavior (or maybe even signal an error, except that I'm
afraid that would break too many things); or we decide we want to
support this use case, but then do it properly.  Properly means that
upcasing "istanbul" in the above example will produce "İSTANBUL", not
"iSTANBUL", and downcasing "IRMA" will produce "ırma".  Yes, these are
multibyte strings produced from unibyte input, but I think it's the
only result we can claim to be correct for a supported use case.
(Such a change could still break some code somewhere, but at least
it's a defendable breakage.)

> Note that casify_object had a ‘(c >= 0 && c < 256)’ condition but since
> CHAR_TO_BYTE8 (and thus MAKE_CHAR_UNIBYTE) happily casts Unicode
> characters to 8-bit (i.e. c & 0xFF), this never triggered for discussed
> case.

We could convert that condition into an eassert, if we are certain the
condition should never trigger.  But that's an aside.

Thanks.