bug#31679: 26.1; detect-coding-string does not detect UTF-16

[Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>]

Hi Eli,


>> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
>>   (detect-coding-string "h\0t\0m\0l\0")
>> 
>> And I was surprised that this does not detect UTF-16 but instead gives
>> (no-conversion).

Eli Zaretskii writes:
> First, you should lose the trailing null (or add one more), since
> UTF-16 strings must, by definition, have an even number of bytes.

Actually this string *has* 8 bytes, the last '\0' completes the 'l' to
form the two-byte character.

> Next, you should disable null byte detection by binding
> inhibit-null-byte-detection to a non-nil value, because otherwise
> Emacs's guesswork will prefer no-conversion, assuming this is binary
> data.

O.k. that is a good tip. 

> Why? because it is perfectly valid for a plain-ASCII string to include
> null bytes, so Emacs prefers to guess ASCII.

While NUL is a valid ASCII character according to the standard,
practically nobody uses it as a character.  So for a heuristic in this
context, it would be a bad decision to treat it just as another
character.

And indeed NUL bytes are treated as a strong indication of binary data,
it seems.  I tried to debug this.  The C routine detect_coding_utf_16
tries to distinguish between binary and UTF-16, but it is not called for
the string above.  That routine is called OTOH, when I add a non-ASCII
character as in "h\0t\0m\0l\0ü\0", but even than it decides that the
string is not UTF-16 (?).

> Morale: detecting an encoding in Emacs is based on heuristic
> _guesswork_, which is heavily biased to what is deemed to be the most
> frequent use cases.  And UTF-16 is quite infrequent, at least on Posix
> hosts.
>
> IOW, detecting encoding in Emacs is not as reliable as you seem to
> expect.  If you _know_ the text is in UTF-16, just tell Emacs to use
> that, don't let it guess.

My use-case is that I am trying to paste types other than UTF8_STRING
from the X11 clipboard, and have them handled as automatically as
possible.  While official clipboard types probably have a documented
encoding (and I have code for those), applications like Firefox also put
private formats there.  And Firefox seems to like UTF-16, even the
text/html format it puts there is UTF-16.

I have tried to debug the C routines that implement this (s.a.), but the
code is somewhat hairy.  I guess I'll have another look to see if I can
understand it better.


Thanks so far,
benny