From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 2F259431FBF for ; Sun, 26 Feb 2012 01:33:21 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -2.3 X-Spam-Level: X-Spam-Status: No, score=-2.3 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vHidQBY8fa39 for ; Sun, 26 Feb 2012 01:33:20 -0800 (PST) Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36]) by olra.theworths.org (Postfix) with ESMTP id 4150F431FBD for ; Sun, 26 Feb 2012 01:33:20 -0800 (PST) Received: from localhost (unknown [192.168.200.4]) by max.feld.cvut.cz (Postfix) with ESMTP id F314919F33AD; Sun, 26 Feb 2012 10:33:18 +0100 (CET) X-Virus-Scanned: IMAP AMAVIS Received: from max.feld.cvut.cz ([192.168.200.1]) by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new, port 10044) with ESMTP id 0D5y2YkqSFhu; Sun, 26 Feb 2012 10:33:17 +0100 (CET) Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34]) by max.feld.cvut.cz (Postfix) with ESMTP id 99AFD19F33A6; Sun, 26 Feb 2012 10:33:17 +0100 (CET) Received: from steelpick.2x.cz (cable-86-56-3-85.cust.telecolumbus.net [86.56.3.85]) (Authenticated sender: sojkam1) by imap.feld.cvut.cz (Postfix) with ESMTPSA id 60887660968; Sun, 26 Feb 2012 10:33:17 +0100 (CET) Received: from wsh by steelpick.2x.cz with local (Exim 4.77) (envelope-from ) id 1S1aTk-00017s-EQ; Sun, 26 Feb 2012 10:33:16 +0100 From: Michal Sojka To: Serge Z , notmuch@notmuchmail.org Subject: Re: Double decoded text/html parts (was: [PATCH] test: Add test for searching of uncommonly encoded messages) In-Reply-To: <20120225083600.17873.66388@localhost> References: <877gzd5axk.fsf@steelpick.2x.cz> <1330043595-22054-1-git-send-email-sojkam1@fel.cvut.cz> <20120224042925.2870.87924@localhost> <874nug67il.fsf@steelpick.2x.cz> <20120224075700.13214.28221@localhost> <874nugiq2g.fsf@steelpick.2x.cz> <20120225083600.17873.66388@localhost> User-Agent: Notmuch/0.11.1+239~g2e86bb7 (http://notmuchmail.org) Emacs/23.3.1 (x86_64-pc-linux-gnu) Date: Sun, 26 Feb 2012 10:33:16 +0100 Message-ID: <871upihrc3.fsf@steelpick.2x.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2012 09:33:21 -0000 On Sat, 25 Feb 2012, Serge Z wrote: > > Hi! > I've struck another problem: > > I've got an html/text email with body encoded with cp1251. > Its encoding is mentioned in both Content-type: email header and html > tag. So when the client tries to display it with external html2text converter, > The message is decoded twice: first by client, second by html2text (I > use w3m). Right. After my analysis of the problem (see below) it seems there is no trivial solution for this. > As I understand, notmuch (while indexing this message) decodes it once and > index it in the right way (though including html tags to index). But what if > the message contains no "charset" option in Content-Type email header but > contain content-type tag with charset noted? This should not happen. It violates RFC 2046, section 4.1.2. > Should such message be considered as being composed wrong or it should > be indexed with diving into html details (content-type)? I don't think it's wrongly composed and it should be even correctly indexed (with my patch). The problem is when you view such a message with an external HTML viewer. In my mailbox I can find two different types of text/html parts. First, the parts that contain complete HTML document including all headers and especially . Such parts could be passed to external HTML viewer without any decoding by notmuch. The second type is text/html part that does not contain any HTML headers. Passing such a part to an external HTML viewer undecoded would require it to guess the correct charset from the content. AFAIK Firefox users can set fallback charset (used for HTML documents with unknown charset) in the preferences, but I don't know what other browsers would do. In particular, do you know how w3m behaves when charset is not specified? In any way, if we want notmuch to do the right thing, we should analyze the content of text/html parts and decide whether to decode the part or not. Perhaps, a simple heuristic could be to search the content of the part for strings "charset=" and "encoding=" and if any is found, notmuch wouldn't decode that part. Otherwise it will decode it according to Content-Type header. As a curiosity, I found the following in one of my emails. Note that two different encodings (iso-8859-2 and windows-1250) are specified at the same time :) That's the reason why I think that fixing the problem won't be trivial. Content-Type: text/html; charset="iso-8859-2" Content-Transfer-Encoding: 8bit Cheers, -Michal