Re: coding tags and utf-16

[David Kastrup <dak@gnu.org>]

Kenichi Handa <handa@m17n.org> writes:

> In article <m1psn61xim.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>  So, in any cases, a tag value itself is useless.  Then how
>>>  to detect utf-16 more reliably?  In the current Emacs
>>>  (i.e. Ver.22), I think we can use auto-coding-regexp-alist
>>>  or auto-coding-alist.  In the former case, we can register
>>>  BOM patterns and also something like "\\`\\(\0[\0-\177]\\)+"
>>>  for utf-16be.  In the latter case, you can use more
>>>  complicated heuristics in a registered function.
>
>> Can't it be somehow added to detect_coding_utf_16?
>
> Yes, but usually it has no effect if, for instance,
> iso-8859-1 is more preferred.  If only ASCII and Latin-1
> characters are encoded in utf-16, all bytes (including BOM)
> are valid for iso-8859-1.

I thought we had discussed this already.  The BOM-encodings should
have priority since the likelihood of a misdetection is negligible
(the character pair does not make sense at the start of a text in
latin-1 in any language): the only thing that can reasonably be
expected to happen is that a binary file is detected as utf-16.  Not
much of an issue, I'd say.

Of course, for the BOM-less utf-16 encodings, priority should depend
on the language environment.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum