Re: Handling mislabeled emails encoded with Windows-1252

[Sebastian Poeplau <sebastian.poeplau@eurecom.fr>]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

Hi again,

>> Everyone's mail situation is unique, but I haven't noticed this
>> problem. Do you have a mechanical (e.g. scripted) way of detecting such
>> mails? I suppose it could just look for characters in the range 0x80 to
>> 0x95 in allegedly ISO_8859-1 messages. A census of the situation in my
>> own mail would help me think about this problem, I think.
>
> Yes, I guess that should be a good enough heuristic for detecting
> affected mail. I'll try to come up with a simple script and post it
> here.

Attached is a Python script that checks individual message files and
prints their name if it finds them to contain mislabeled Windows-1252
text. The heuristic seems to work well on my mail - let me know if you
encounter any issues!

Cheers,
Sebastian



[-- Attachment #2: find_mislabeled_cp1252.py --]
[-- Type: application/octet-stream, Size: 840 bytes --]

#!/usr/bin/env python3
from email.parser import BytesParser
import email.policy


mail_parser = BytesParser(policy=email.policy.default)


def check_message(filename):
    """
    Return True if the specified message contains mislabeled Windows-1252 text.
    """
    with open(filename, 'rb') as f:
        message = mail_parser.parse(f)

    for part in message.walk():
        if part.get_content_type() != 'text/plain' or \
           part.get_content_charset() != 'iso-8859-1':
            continue

        body = part.get_content()
        for char in body:
            code = ord(char)
            if code > 0x80 and code < 0xa0:
                return True

    return False


def main(args):
    for arg in args:
        if check_message(arg):
            print(arg)


if __name__ == '__main__':
    import sys
    main(sys.argv)