From: Michal Sojka <sojkam1@fel.cvut.cz>
To: Serge Z <triumhiz@yandex.ru>, notmuch@notmuchmail.org
Subject: Re: Double decoded text/html parts (was: [PATCH] test: Add test for searching of uncommonly encoded messages)
Date: Sun, 26 Feb 2012 10:33:16 +0100 [thread overview]
Message-ID: <871upihrc3.fsf@steelpick.2x.cz> (raw)
In-Reply-To: <20120225083600.17873.66388@localhost>
On Sat, 25 Feb 2012, Serge Z wrote:
>
> Hi!
> I've struck another problem:
>
> I've got an html/text email with body encoded with cp1251.
> Its encoding is mentioned in both Content-type: email header and html <meta>
> tag. So when the client tries to display it with external html2text converter,
> The message is decoded twice: first by client, second by html2text (I
> use w3m).
Right. After my analysis of the problem (see below) it seems there is no
trivial solution for this.
> As I understand, notmuch (while indexing this message) decodes it once and
> index it in the right way (though including html tags to index). But what if
> the message contains no "charset" option in Content-Type email header but
> contain <meta> content-type tag with charset noted?
This should not happen. It violates RFC 2046, section 4.1.2.
> Should such message be considered as being composed wrong or it should
> be indexed with diving into html details (content-type)?
I don't think it's wrongly composed and it should be even correctly
indexed (with my patch). The problem is when you view such a message
with an external HTML viewer.
In my mailbox I can find two different types of text/html parts. First,
the parts that contain complete HTML document including all headers and
especially <meta http-equiv="content-type" content="text/html; ...">.
Such parts could be passed to external HTML viewer without any decoding
by notmuch.
The second type is text/html part that does not contain any HTML
headers. Passing such a part to an external HTML viewer undecoded would
require it to guess the correct charset from the content.
AFAIK Firefox users can set fallback charset (used for HTML documents
with unknown charset) in the preferences, but I don't know what other
browsers would do. In particular, do you know how w3m behaves when
charset is not specified?
In any way, if we want notmuch to do the right thing, we should analyze
the content of text/html parts and decide whether to decode the part or
not. Perhaps, a simple heuristic could be to search the content of the
part for strings "charset=" and "encoding=" and if any is found, notmuch
wouldn't decode that part. Otherwise it will decode it according to
Content-Type header.
As a curiosity, I found the following in one of my emails. Note that two
different encodings (iso-8859-2 and windows-1250) are specified at the
same time :) That's the reason why I think that fixing the problem won't
be trivial.
Content-Type: text/html; charset="iso-8859-2"
Content-Transfer-Encoding: 8bit
<?xml version="1.0" encoding="windows-1250" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-2" />
Cheers,
-Michal
next prev parent reply other threads:[~2012-02-26 9:33 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-22 17:10 Searching through different charsets Serge Z
2012-02-24 0:31 ` Michal Sojka
2012-02-24 0:33 ` [PATCH] test: Add test for searching of uncommonly encoded messages Michal Sojka
2012-02-24 4:29 ` Serge Z
2012-02-24 7:00 ` Michal Sojka
2012-02-24 7:57 ` Serge Z
2012-02-24 8:38 ` Michal Sojka
2012-02-25 8:36 ` Serge Z
2012-02-26 9:33 ` Michal Sojka [this message]
2012-02-26 10:20 ` Double decoded text/html parts (was: [PATCH] test: Add test for searching of uncommonly encoded messages) Serge Z
2012-02-24 7:36 ` [PATCH 1/2] Convert non-UTF-8 parts to UTF-8 before indexing them Michal Sojka
2012-02-24 7:36 ` [PATCH 2/2] test: Remove 'broken' flag from encoding test Michal Sojka
2012-02-25 4:33 ` [PATCH 1/2] Convert non-UTF-8 parts to UTF-8 before indexing them Austin Clements
2012-02-29 11:55 ` David Bremner
2012-02-29 11:55 ` [PATCH] test: Add test for searching of uncommonly encoded messages David Bremner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=871upihrc3.fsf@steelpick.2x.cz \
--to=sojkam1@fel.cvut.cz \
--cc=notmuch@notmuchmail.org \
--cc=triumhiz@yandex.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).