bug#20623: XML and HTML files with encoding/charset="utf-8" declaration loose BOM; Coding system is reset from utf-8-with-signature to utf-8 on save

[Eli Zaretskii <eliz@gnu.org>]

[Please don't remove the bug address from the CC list, so that this
discussion is recorded in the bug data base.]

> Date: Thu, 21 May 2015 22:49:47 +0200
> From: Simon Ledergerber <sledergerber@gmx.net>
> 
>  From the documentation I understand that utf-8 is without BOM and 
> utf-8-with-signature is with BOM. Maybe I am wrong and should rather 
> understand that utf-8 is auto-detect. But then there is something like 
> utf-8-without-signature missing to specify explicitly that no BOM is 
> desired.
> 
> In my opinion, it is correct when Emacs prefers utf-8 over 
> utf-8-with-signature when it opens a file without BOM that can still be 
> recognized as UTF-8.
> 
> However when a file is opened with a BOM already present, it should 
> stick to the utf-8-with-signature coding system, because the BOM "EF BB 
> BF" unambiguously marks the file as UTF-8. (For UTF-16 for example, 
> there is a different BOM byte pattern. There are other coding systems 
> which do not have a BOM at all.)

What do you mean by "stick to"?  When I try visiting an XML file that
is encoded with BOM, Emacs decodes the file correctly, and the value
of buffer-file-coding-system is utf-8-with-signature.  Isn't that what
you want?  If that's what you want, but it doesn't happen for you,
please try in "emacs -Q".  It's possible that the default you set:

  (setq-default buffer-file-coding-system 'utf-8-dos)

is the reason for what you see.  (I don't understand why you need such
a default, and it sounds like a bad idea to me.)

> By doing C-x <RET> f and then saving it with C-x C-s, I expect to be 
> able to change the coding system.  For example, if I specify utf-8-dos, 
> the BOM should be removed, if one was present, and CR LF should be 
> inserted for EOL. On the other side, if I choose 
> utf-8-with-signature-unix, a BOM should be written and LF be taken for 
> EOL. (The conversion between DOS and Unix works, just the BOM is the 
> problem.)
> 
> I have found a link, where this topic was already discussed, but it 
> didn't help me further:
> http://superuser.com/questions/41254/make-emacs-not-remove-the-bom-from-xml-files
> 
> In that post Vebjorn Ljosa asked exactly the question I have. Richard 
> Hoskins replies with the answer to change the coding system with C-x 
> <RET> r utf-8-with-signature. Unfortunately, it didn't work for me - 
> after doing a change in the file and saving, it got back to utf-8 
> automatically - that's why I have filed the bug.

That's not how you force a file to be saved in a specific encoding.
You should do this instead:

  C-x RET c utf-8-with-signature RET C-x C-s

The "C-x RET c" prefix forces the next Emacs operation to use the
specified encoding.  In this case, Emacs will ask for confirmation,
because the encoding you specified is different from what the XML
comment says.