From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.bugs Subject: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23 Date: Mon, 01 Dec 2008 16:04:01 -0500 Message-ID: References: <871vyf6q40.fsf@marauder.physik.uni-ulm.de> <87k5ampwej.fsf@marauder.physik.uni-ulm.de> Reply-To: Stefan Monnier , 1174@emacsbugs.donarmstrong.com NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1228166055 15695 80.91.229.12 (1 Dec 2008 21:14:15 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 1 Dec 2008 21:14:15 +0000 (UTC) Cc: 1174@emacsbugs.donarmstrong.com, Frank Schmitt , ding@gnus.org To: Simon Josefsson Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Dec 01 22:15:19 2008 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1L7G6x-0002gY-3N for geb-bug-gnu-emacs@m.gmane.org; Mon, 01 Dec 2008 22:15:19 +0100 Original-Received: from localhost ([127.0.0.1]:55184 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1L7G5m-0006vN-N6 for geb-bug-gnu-emacs@m.gmane.org; Mon, 01 Dec 2008 16:14:06 -0500 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1L7G25-0003Q6-J3 for bug-gnu-emacs@gnu.org; Mon, 01 Dec 2008 16:10:17 -0500 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1L7G24-0003Os-O9 for bug-gnu-emacs@gnu.org; Mon, 01 Dec 2008 16:10:17 -0500 Original-Received: from [199.232.76.173] (port=45909 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1L7G24-0003OX-FE for bug-gnu-emacs@gnu.org; Mon, 01 Dec 2008 16:10:16 -0500 Original-Received: from rzlab.ucr.edu ([138.23.92.77]:37853) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1L7G23-0007p1-Un for bug-gnu-emacs@gnu.org; Mon, 01 Dec 2008 16:10:16 -0500 Original-Received: from rzlab.ucr.edu (rzlab.ucr.edu [127.0.0.1]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id mB1LAC0k024112; Mon, 1 Dec 2008 13:10:12 -0800 Original-Received: (from debbugs@localhost) by rzlab.ucr.edu (8.13.8/8.13.8/Submit) id mB1LA4Cb023453; Mon, 1 Dec 2008 13:10:04 -0800 X-Loop: don@donarmstrong.com Resent-From: Stefan Monnier Resent-To: bug-submit-list@donarmstrong.com Resent-CC: Emacs Bugs Resent-Date: Mon, 01 Dec 2008 21:10:04 +0000 Resent-Message-ID: Resent-Sender: don@donarmstrong.com X-Emacs-PR-Message: report 1174 X-Emacs-PR-Package: emacs X-Emacs-PR-Keywords: Original-Received: via spool by 1174-submit@emacsbugs.donarmstrong.com id=B1174.122816544921351 (code B ref 1174); Mon, 01 Dec 2008 21:10:04 +0000 Original-Received: (at 1174) by emacsbugs.donarmstrong.com; 1 Dec 2008 21:04:09 +0000 Original-Received: from pruche.dit.umontreal.ca (pruche.dit.umontreal.ca [132.204.246.22]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id mB1L44h3021345 for <1174@emacsbugs.donarmstrong.com>; Mon, 1 Dec 2008 13:04:05 -0800 Original-Received: from alfajor.home (vpn-132-204-232-120.acd.umontreal.ca [132.204.232.120]) by pruche.dit.umontreal.ca (8.14.1/8.14.1) with ESMTP id mB1L4cxf018284; Mon, 1 Dec 2008 16:04:38 -0500 Original-Received: by alfajor.home (Postfix, from userid 20848) id 0A4831C1EF; Mon, 1 Dec 2008 16:04:01 -0500 (EST) In-Reply-To: <87k5ampwej.fsf@marauder.physik.uni-ulm.de> (Reiner Steib's message of "Sat, 29 Nov 2008 13:08:20 +0100") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (gnu/linux) X-NAI-Spam-Score: 0 X-NAI-Spam-Rules: 1 Rules triggered RV3159=0 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 3) Resent-Date: Mon, 01 Dec 2008 16:10:17 -0500 X-BeenThere: bug-gnu-emacs@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:22823 Archived-At: > Simon, could you please clarify why you reverted Stefan's change in > `nnimap-demule'? It breaks reading UTF-8 articles via nnimap.el in > Emacs 23. Having looked at the code again, I'm more than ever confident that string-to-unibyte is the right thing to use. Maybe the code I installed back then failed to fallback to string-as-unibyte when string-to-unibyte was not available, which caused a bug for Simon? In any case the newly committed code has a prenthesis typo that makes it still use the old code and ignore the new config var nnimap-demule-use-string-to-multibyte. Also I recommend to just use the patch below instead. The first hunk removes an unnecessary use of nnimap-demule since the output will be inserted into a unibyte buffer. Stefan --- nnimap.el.~1.50.~ 2008-12-01 15:38:55.000000000 -0500 +++ nnimap.el 2008-12-01 15:49:53.000000000 -0500 @@ -608,12 +608,11 @@ (with-current-buffer nnimap-server-buffer (setq uid imap-current-message mbx imap-current-mailbox - headers (nnimap-demule - (if (imap-capability 'IMAP4rev1) + headers (if (imap-capability 'IMAP4rev1) ;; xxx don't just use car? alist doesn't contain ;; anything else now, but it might... (nth 2 (car (imap-message-get uid 'BODYDETAIL))) - (imap-message-get uid 'RFC822.HEADER))) + (imap-message-get uid 'RFC822.HEADER)) lines (imap-body-lines (imap-message-body imap-current-message)) chars (imap-message-get imap-current-message 'RFC822.SIZE))) (nnheader-insert-nov @@ -901,40 +900,17 @@ (when (nnimap-possibly-change-server server) (nnoo-status-message 'nnimap server))) -(defvar nnimap-demule-use-string-to-multibyte (fboundp 'string-to-multibyte) - "Temporary internal debug variable. -If you have problems (UTF-8 not decoded correctly on IMAP) with -the default value, please report it as a bug!") -;; FIXME: Clarify if we need to make this variable conditional on the Emacs -;; version (Emacs 22 vs. Emacs 23;Emacs 21 doesn't have `string-to-multibyte' -;; anyhow). --rsteib -;; -;; http://thread.gmane.org/gmane.emacs.gnus.general/67112 -;; (bug#464, reported by James Cloos) -;; http://thread.gmane.org/gmane.emacs.bugs/21524 -;; (bug#1174, reported by Frank Schmitt) - -(defun nnimap-demule (string) - ;; BEWARE: we used to use string-as-multibyte here which is braindead - ;; because it will turn accidental emacs-mule-valid byte sequences - ;; into multibyte chars. --Stef - ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be - ;; that bad. --Simon - (gnus-message 9 "nnimap-demule-use-string-to-multibyte: %s" - nnimap-demule-use-string-to-multibyte) - (if nnimap-demule-use-string-to-multibyte - ;; Stefan - (funcall (if (and (fboundp 'string-to-multibyte) - (subrp (symbol-function 'string-to-multibyte))) - 'string-to-multibyte - 'identity) - (or string ""))) - ;; Simon - (funcall (if (and (fboundp 'string-as-multibyte) - (subrp (symbol-function 'string-as-multibyte))) - 'string-as-multibyte - 'identity) - (or string ""))) +;; We used to use a string-as-multibyte here, but it is really incorrect. +;; This function is used when we're about to insert a unibyte string +;; into a potentially multibyte buffer. The string is either an article +;; header or body (or both?), undecoded. When Emacs is asked to convert +;; a unibyte string to multibyte, it may either use the equivalent of +;; nothing (e.g. non-Mule XEmacs), string-make-unibyte (i.e. decode using +;; locale), string-as-multibyte (decode using emacs-internal coding system) +;; or string-to-multibyte (keep the data undecoded as a sequence of bytes). +;; Only the last one preserves the data such that we can reliably later on +;; decode the text using the mime info. +(defalias 'nnimap-demule 'mm-string-to-multibyte) (defun nnimap-make-callback (article gnus-callback buffer) "Return a callback function."