Re: coding tags and utf-16

[Kenichi Handa <handa@m17n.org>]

In article <m3wtf7xt6z.fsf@seneca.benny.turtle-trading.net>, Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net> writes:

> Kenichi Handa writes:
>> For decoding UTF-8, we should not delete that BOM but treat it as
>> the content of the text.  For UTF-16, Unicode explicitly says that
>> "The BOM is not considered part of the content of the text", but for
>> UTF-8, it doesn't say such a thing.

> NOTEPAD.EXE (the basic MS Windows editor) adds a BOM when writing
> UTF-8 files.  When I saw that and tried to discuss it on their
> newsgroups, I learned that it seems to be Microsoft's POV that this is
> a good thing.

> Which means files like that exist.  Treating the BOM as content means
> that U+FEFF creeps into the regular content of documents through
> cut-and-paste and through components of template systems.  I have
> already seen that happening in real life and of course it leads to
> stupid bugs.  I think Emacs should do better.

But, it's simply a bug to delete the leading U+FEFF from the
content while decoding utf-8.  Perhaps we should add some
customizable flag to control that behavior after the
release.

>> utf-16-be [==] utf-16be-with-signature [!=] utf-16be

> ;-)

^.^;;;

---
Kenichi Handa
handa@m17n.org