From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 287DE6DE026D for ; Sat, 14 Jul 2018 05:50:50 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.011 X-Spam-Level: X-Spam-Status: No, score=-0.011 tagged_above=-999 required=5 tests=[SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qaK031-7gHAX for ; Sat, 14 Jul 2018 05:50:48 -0700 (PDT) X-Greylist: delayed 614 seconds by postgrey-1.36 at arlo; Sat, 14 Jul 2018 05:50:48 PDT Received: from smtp.eurecom.fr (smtp.eurecom.fr [193.55.113.210]) by arlo.cworth.org (Postfix) with ESMTP id 965BF6DE01F0 for ; Sat, 14 Jul 2018 05:50:48 -0700 (PDT) X-IronPort-AV: E=Sophos;i="5.51,352,1526335200"; d="scan'208";a="7852552" Received: from waha.eurecom.fr (HELO smtps.eurecom.fr) ([10.3.2.236]) by drago1i.eurecom.fr with ESMTP; 14 Jul 2018 14:40:29 +0200 Received: from archibald (dslb-084-056-082-241.084.056.pools.vodafone-ip.de [84.56.82.241]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtps.eurecom.fr (Postfix) with ESMTPSA id D201872B for ; Sat, 14 Jul 2018 14:40:28 +0200 (CEST) From: Sebastian Poeplau To: notmuch@notmuchmail.org Subject: Handling mislabeled emails encoded with Windows-1252 Date: Sat, 14 Jul 2018 14:40:28 +0200 Message-ID: <87lgaeat37.fsf@eurecom.fr> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Mailman-Approved-At: Sat, 14 Jul 2018 06:42:17 -0700 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2018 12:50:50 -0000 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, This email is to suggest a minor change in how notmuch handles text encoding when displaying emails. The motivation is the following: I keep receiving emails that are encoded with Windows-1252 but claim to be ISO=C2=A08859-1. The two character sets only differ in the range between 0x= 80 and 0x9F where Windows-1252 contains special characters (e.g. =E2=80=9Cquot= ation marks=E2=80=9D) while ISO=C2=A08859-1 only has non-printable ones. The misl= abeling thus causes some special characters in such emails to be displayed with a replacement symbol for non-printable characters. Of course, it would be best to fix the problem on the sender's side, making their mail client declare the encoding correctly. However, sometimes this is just not possible and we need to make do with what we receive. The change I would thus like to suggest is to always treat ISO=C2=A08859-1 as Windows-1252; since the former only contains non-printab= le characters in the range where the two differ, we would not lose any printable information. According to Wikipedia, this substitution is common in email clients and browsers because of the frequent mislabeling=C2=A0[1]. Attached you find a simple patch that illustrates my suggestion. While it works well for my limited use cases, it's obviously not entirely reliable. Does anyone have a good idea how to better handle the issue? I searched GMime for related functionality but didn't quite find what I was looking for. Do you feel that the issue should be raised with the GMime people instead? Best regards, Sebastian [1] https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Windows-1252 --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=charset.patch diff -ura notmuch-0.27/notmuch-show.c notmuch-0.27-patched/notmuch-show.c --- notmuch-0.27/notmuch-show.c 2018-06-13 03:42:34.000000000 +0200 +++ notmuch-0.27-patched/notmuch-show.c 2018-07-11 10:32:56.000456518 +0200 @@ -291,6 +291,8 @@ charset = g_mime_object_get_content_type_parameter (part, "charset"); if (charset) { GMimeFilter *charset_filter; + if (!strcmp(charset, "iso-8859-1")) + charset = "CP1252"; charset_filter = g_mime_filter_charset_new (charset, "UTF-8"); /* This result can be NULL for things like "unknown-8bit". * Don't set a NULL filter as that makes GMime print --=-=-=--