Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Reiner Steib <4.uce.03.r.s@nurfuerspam.de>]

On Mon, Mar 31 2003, Jesper Harder wrote:

> FWIW, Oort Gnus and current CVS Emacs works as expected.  It used to
> exhibit the same bug you're describing, 

Yes, I remember seeing this on the Ding-List in some of Kai's
articles.  But Kai and me didn't recall whether the problem was in
Emacs CVS-HEAD or in Oort Gnus (and whether it was fixed or not).

> but I was fixed earlier this year -- probably by this change:
>
> 2003-01-03  Dave Love  <fx@gnu.org>
>
> 	* international/mule-cmds.el (sort-coding-systems):
> 	Adjust priority of utf-16 and x-ctext.

Yes, that's it, thanks for pointing this out.  After applying the
following patch to lisp/international/mule-cmds.el from Emacs 21.3.1
and evaluation `sort-coding-systems', I get utf-8 as expected.

--8<---------------cut here---------------start------------->8---
--- mule-cmds.el	26 Dec 2002 17:27:20 -0000	1.216
+++ mule-cmds.el	3 Jan 2003 20:16:11 -0000	1.217
@@ -425,9 +425,18 @@
 		    (let ((base (coding-system-base x)))
 		      (+ (if (eq base most-preferred) 64 0)
 			 (let ((mime (coding-system-get base 'mime-charset)))
+			   ;; Prefer coding systems corresponding to a
+			   ;; MIME charset.
 			   (if mime
-			       (if (string-match "^x-" (symbol-name mime))
-				   16 32)
+			       ;; Lower utf-16 priority so that we
+			       ;; normally prefer utf-8 to it, and put
+			       ;; x-ctext below that.
+			       (cond ((or (eq base 'mule-utf-16-le)
+					  (eq base 'mule-utf-16-be))
+				      16)
+				     ((string-match "^x-" (symbol-name mime))
+				      8)
+				     (t 32))
 			     0))
 			 (if (memq base lang-preferred) 8 0)
 			 (if (string-match "-with-esc$" (symbol-name base))
--8<---------------cut here---------------end--------------->8---

> Didn't it get applied to Emacs 21.3?

No.  If the EMACS_21_1_RC branch is still maintained, it should
probably be applied there as well.

On Mon, Mar 31 2003, Kenichi Handa wrote:

> Oops, I've just found that Emacs' coding systems utf-16-le and
> utf-16-be produce BOM (Byte Order Mark) which is a bug according to
> their specifications.  I've just installed a fix.

Does it make sense to apply it to EMACS_21_1_RC too?

>> Expected behavior:
>> - The article should be encoded with
>>   "Content-Type: text/plain; charset=utf-8".
> I don't know why GNUS prefers utf-16-X to utf-8.  At least,
> sort-coding-systems prefers utf-8.

Apparently not in Emacs 21.3, unless I misunderstood the
abovementioned patch to `sort-coding-systems'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/