bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23

[Reiner Steib <reinersteib+gmane@imap.cc>]

On Sat, Nov 29 2008, Simon Josefsson wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>>>>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>>>>   ;; into multibyte chars.  --Stef
>>>>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>>>>   ;; that bad. --Simon
>>
>> Simon, could you please clarify why you reverted Stefan's change in
>> `nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
>> Emacs 23.
>
> I don't recall, but people should notice relatively quickly if there are
> problems in this area (wrong display of non-ascii) 

Hm, both changes happened 2004, but I don't recall any reports beside
the recent ones in 2008:

  http://thread.gmane.org/gmane.emacs.gnus.general/67112
  (bug#464, reported by James Cloos)
  http://thread.gmane.org/gmane.emacs.bugs/21524
  (bug#1174, reported by Frank Schmitt)
  
  ----------------------------
  revision 7.9
  date: 2004-09-13 13:52:48 +0200;  author: jas;  state: Exp;  lines: +5 -3
  (nnimap-demule): Revert 2004-08-30 change.
  ----------------------------

> so feel free to revert the patch or apply another patch instead.  It
> needs to be tested under Emacs 22 too, though,

I never saw this problem myself.  I cannot see any difference with a
few UTF-8 articles (C-T-E: 8bit, [1]), neither with Emacs 22 (with
current Gnus trunk) or Emacs trunk (Gnus 5.13 from there).  Stefan,
what are the "accidental emacs-mule-valid byte sequences" that trigger
this problem?  It would be good if someone could send me a problematic
article. [2]

I've just checked in (Gnus and Emacs) some code to debug this problem.
I'd like to ask those who saw the bug in Emacs 23 to test the articles
in question:

- With current Emacs 23 (Emacs CVS trunk)

- With Emacs 22 plus current Gnus CVS trunk (No Gnus)

- If you see wrong display: does it display correctly after evaluating
  the following:

  M-x gnus-backlog-shutdown RET

  (setq nnimap-demule-use-string-to-multibyte nil
        gnus-verbose 10)

  Check the *Messages* buffer for messages
  "nnimap-demule-use-string-to-multibyte: nil" to ensure that the
  article is decoded again with this setting.  You may need to
  re-enter the group.

> if it is installed in the Gnus CVS.

I don't want different code in Gnus and Emacs.  If all else fails, we
can make it conditional.

Bye, Reiner.

[1] Cc-s of the following articles (available on Gmane) from Aidan Kehoe:
    <18492.30425.377545.700503@parhasard.net>
    <18518.43672.183610.662699@parhasard.net>
    <18712.43474.265690.792714@parhasard.net>

    Non-ascii characters in the attribution line ("scríobh"), the
    signature ("¿Dónde estará ahora mi sobrino Yoghurtu Nghé ..."),
    and "’", "İ", "ı"

[2] (push '(utf-8 . 8bit) mm-body-charset-encoding-alist)
    --> trying to produce a problematic article here:
    AE-Ä OE-Ö UE-Ü ae-ä oe-ö ue-ü ss-ß

    Should be sent with: 
      Content-Type: text/plain; charset=utf-8
      Content-Transfer-Encoding: 8bit
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/