Broken charset=utf-16be articles with Gnus and Emacs 21.3

Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Reiner Steib]

In GNU Emacs 21.3.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2003-03-19 on ni
configured using `configure  --prefix=/import/xtra/emacs/RC'
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: POSIX
  value of $LC_CTYPE: en_US.ISO_8859-1
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: C
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: t

Gnus v5.9.0
GNU Emacs 21.3.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2003-03-19 on ni
200 news.uni-ulm.de DNEWS Version  5.6f3,, S0, posting OK 

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

Actions to reproduce the bug:

- Start Gnus `M-x gnus RET' and compose an article `a'.

- Switch to TeX input method: `M-x set-input-method RET TeX RET'

- Enter some characters:

  \sigma ä (a with diaeresis) \omega
  (Note: \sigma and \omega (without ä) is not sufficient.)
  
  .. *or* ...

  \sigma \omega \alpha \o \int

- Preview `M-m P' (or `C-u M-m P') or send the message.

  The preview (and the outgoing message) will be encoded with
  "Content-Type: text/plain; charset=utf-16be" and it will not be
  readable with Gnus and most other MUAs or newsreaders.

  I've been told (-> Simon Krahnke) that the result isn't even correct
  UTF-16.

Expected behavior:

- The article should be encoded with
  "Content-Type: text/plain; charset=utf-8".

Note: I'm sending this message with a different version of Gnus, so
don't be confused when looking at my User-Agent header.  FWIW, the
same bug also happens with Oort Gnus from CVS.  With Emacs 21.1 / Gnus
5.9.0 [1] and with Emacs 21.2 / Oort Gnus v0.17 [2] we have the
expected behavior [1].  Therefore I think this is a bug in Emacs 21.3,
but I'm not sure (Cc-ing bugs@gnus.org).

Additional confusion: Replying to [1] (with full citation), I get
utf-8 [3], but copying the non-Latin charaters into this buffer
results in utf-16be in the preview.  I also get utf-16be, when I reply
to [2] and delete the lines containing the word "Blindtext".

Bye, Reiner.

[1] Reported by Simon Krahnke: <news:8765q3nwm9.fsf@xts.gnuu.de>
    (Newsgroups: de.comm.software.gnus)

[2] Reported by Mark Trettin:
    <news:dcsg.m3of3vcgwr.fsf@beldin.mt743742.dialup.rwth-aachen.de>
    (Newsgroups: de.comm.software.gnus)

[3] <news:v9d6kaig3n.fsf@marauder.physik.uni-ulm.de>
    (Newsgroups: de.comm.software.gnus)
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Kenichi Handa]

In article <v9adfeig1c.fsf@marauder.physik.uni-ulm.de>, Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:
> Actions to reproduce the bug:

> - Start Gnus `M-x gnus RET' and compose an article `a'.

> - Switch to TeX input method: `M-x set-input-method RET TeX RET'

> - Enter some characters:

>   \sigma ä (a with diaeresis) \omega
>   (Note: \sigma and \omega (without ä) is not sufficient.)
  
>   .. *or* ...

>   \sigma \omega \alpha \o \int

> - Preview `M-m P' (or `C-u M-m P') or send the message.

>   The preview (and the outgoing message) will be encoded with
>   "Content-Type: text/plain; charset=utf-16be" and it will not be
>   readable with Gnus and most other MUAs or newsreaders.

>   I've been told (-> Simon Krahnke) that the result isn't even correct
>   UTF-16.

Oops, I've just found that Emacs' coding systems utf-16-le
and utf-16-be produce BOM (Byte Order Mark) which is a bug
according to their specifications.  I've just installed a
fix.

> Expected behavior:

> - The article should be encoded with
>   "Content-Type: text/plain; charset=utf-8".

I don't know why GNUS prefers utf-16-X to utf-8.  At least,
sort-coding-systems prefers utf-8.

---
Ken'ichi HANDA
handa@m17n.org

Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Jesper Harder]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

>   The preview (and the outgoing message) will be encoded with
>   "Content-Type: text/plain; charset=utf-16be" and it will not be
>
> Note: I'm sending this message with a different version of Gnus, so
> don't be confused when looking at my User-Agent header.  FWIW, the
> same bug also happens with Oort Gnus from CVS.  With Emacs 21.1 / Gnus
> 5.9.0 [1] and with Emacs 21.2 / Oort Gnus v0.17 [2] we have the
> expected behavior [1].  Therefore I think this is a bug in Emacs 21.3,
> but I'm not sure (Cc-ing bugs@gnus.org).

FWIW, Oort Gnus and current CVS Emacs works as expected.  It used to
exhibit the same bug you're describing, but I was fixed earlier this
year -- probably by this change:


2003-01-03  Dave Love  <fx@gnu.org>

	* international/mule-cmds.el (sort-coding-systems):
	Adjust priority of utf-16 and x-ctext.

Didn't it get applied to Emacs 21.3?

Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Reiner Steib]

On Mon, Mar 31 2003, Jesper Harder wrote:

> FWIW, Oort Gnus and current CVS Emacs works as expected.  It used to
> exhibit the same bug you're describing, 

Yes, I remember seeing this on the Ding-List in some of Kai's
articles.  But Kai and me didn't recall whether the problem was in
Emacs CVS-HEAD or in Oort Gnus (and whether it was fixed or not).

> but I was fixed earlier this year -- probably by this change:
>
> 2003-01-03  Dave Love  <fx@gnu.org>
>
> 	* international/mule-cmds.el (sort-coding-systems):
> 	Adjust priority of utf-16 and x-ctext.

Yes, that's it, thanks for pointing this out.  After applying the
following patch to lisp/international/mule-cmds.el from Emacs 21.3.1
and evaluation `sort-coding-systems', I get utf-8 as expected.

--8<---------------cut here---------------start------------->8---
--- mule-cmds.el	26 Dec 2002 17:27:20 -0000	1.216
+++ mule-cmds.el	3 Jan 2003 20:16:11 -0000	1.217
@@ -425,9 +425,18 @@
 		    (let ((base (coding-system-base x)))
 		      (+ (if (eq base most-preferred) 64 0)
 			 (let ((mime (coding-system-get base 'mime-charset)))
+			   ;; Prefer coding systems corresponding to a
+			   ;; MIME charset.
 			   (if mime
-			       (if (string-match "^x-" (symbol-name mime))
-				   16 32)
+			       ;; Lower utf-16 priority so that we
+			       ;; normally prefer utf-8 to it, and put
+			       ;; x-ctext below that.
+			       (cond ((or (eq base 'mule-utf-16-le)
+					  (eq base 'mule-utf-16-be))
+				      16)
+				     ((string-match "^x-" (symbol-name mime))
+				      8)
+				     (t 32))
 			     0))
 			 (if (memq base lang-preferred) 8 0)
 			 (if (string-match "-with-esc$" (symbol-name base))
--8<---------------cut here---------------end--------------->8---

> Didn't it get applied to Emacs 21.3?

No.  If the EMACS_21_1_RC branch is still maintained, it should
probably be applied there as well.

On Mon, Mar 31 2003, Kenichi Handa wrote:

> Oops, I've just found that Emacs' coding systems utf-16-le and
> utf-16-be produce BOM (Byte Order Mark) which is a bug according to
> their specifications.  I've just installed a fix.

Does it make sense to apply it to EMACS_21_1_RC too?

>> Expected behavior:
>> - The article should be encoded with
>>   "Content-Type: text/plain; charset=utf-8".
> I don't know why GNUS prefers utf-16-X to utf-8.  At least,
> sort-coding-systems prefers utf-8.

Apparently not in Emacs 21.3, unless I misunderstood the
abovementioned patch to `sort-coding-systems'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Mark Trettin]

On Mon, 31 Mar 2003, Reiner Steib verbalised:
> On Mon, Mar 31 2003, Jesper Harder wrote:

[...]

>> but I was fixed earlier this year -- probably by this change:
>>
>> 2003-01-03  Dave Love  <fx@gnu.org>
>>
>> 	* international/mule-cmds.el (sort-coding-systems):
>> 	Adjust priority of utf-16 and x-ctext.
> 
> Yes, that's it, thanks for pointing this out.  After applying the
> following patch to lisp/international/mule-cmds.el from Emacs 21.3.1
> and evaluation `sort-coding-systems', I get utf-8 as expected.

[...]

>> Didn't it get applied to Emacs 21.3?
 
> No.  If the EMACS_21_1_RC branch is still maintained, it should
> probably be applied there as well.

Will there be some kind of "official" patch for Emacs 21.3.1 or do have to
apply the snipped patch?

[...]

Bis dann

	 Mark
-- 
Mark Trettin ------- *Aachen* -- Wo ist das? ------> N: 50°46' O: 06°05'
BOFH excuse #317:
Internet exceeded Luser level, please wait until a luser logs off before
attempting to log back on.