Re: 23.0.60; [nxml] BOM and utf-8

["Stephen J. Turnbull" <stephen@xemacs.org>]

Miles Bader writes:

 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > >  > is the attached xml file (simple.xml) really invalid (as indicated by
 > >  > nxhtml) or is this a bug in nxhtml?
 > >
 > > Neither.  Emacs is (arguably) reading it incorrectly.
 > 
 > By "arguably" I presume you're referring to the "Microsoft does <random
 > stupid thing>, therefore everybody who doesn't do <random stupid thing>
 > is incorrect" tactic.

No, by "arguably" I'm referring to the fact that although the optional
UTF-8 signature has been part of ISO/IEC 10646-1 and Unicode for a
decade or so, not to mention Internet STD 63 (aka RFC 3269), I fully
expected somebody like you to pop up and argue about it.

It is a bad standard (see STD 63) and possibly Microsoft-induced, but
it *is* the standard and is showing no signs of going away; see
Section 16.8 of *The Unicode Standard*, v5.0.  In fact, the trend is
the other way around: the ancient RFCs 2044 and 2279 don't mention it
either way, but STD 63 found it necessary to *add* it.

 > In general, other apps that read such files are not expecting the
 > BOM, and won't be able to deal with it.  So Emacs wouldn't be doing
 > the user any favors by hiding the BOM from him.

So pop up a warning to the effect that the BOM was stripped per the
Unicode standard, and that if it needs to be preserved, set
UNICODE_ME_SOFTLY in the environment or bind `unicode-me-softly'
around the codec.

Alternatively, sabotage the Microsoft users by silently eating the BOM
on the way in, and writing the file in GNU substandard[1] format on the
way out.

Footnotes: 
[1]  A substandard is a standard with stupid optional features
subtracted. :-)