UnicodeDecodeError with python API

UnicodeDecodeError with python API

[Sebastian Fischmeister]

Hi,

I'm trying to use the python API for notmuch, and get the following
error:

---

Traceback (most recent call last):
  File "./test.py", line 66, in <module>
    print(type(y.get_part(1)))
  File "/usr/lib/python3.4/site-packages/notmuch/message.py", line 602, in get_part
    parts = self.get_message_parts()
  File "/usr/lib/python3.4/site-packages/notmuch/message.py", line 591, in get_message_parts
    email_msg = email.message_from_file(fp)
  File "/usr/lib/python3.4/email/__init__.py", line 56, in message_from_file
    return Parser(*args, **kws).parse(fp)
  File "/usr/lib/python3.4/email/parser.py", line 54, in parse
    data = fp.read(8192)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3447: ordinal not in range(128)

---

The code works for most messages though. How can I get around this
problem? Is it a problem in my code or the binding?

I'm using notmuch 0.19 with python 3.4.3.

Thanks,
  Sebastian

Re: UnicodeDecodeError with python API

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

On Sun, Mar 29, 2015 at 09:08:28AM -0400, Sebastian Fischmeister wrote:
> Traceback (most recent call last):
>   File "./test.py", line 66, in <module>
>     print(type(y.get_part(1)))
>   …
>   File "/usr/lib/python3.4/email/parser.py", line 54, in parse
>     data = fp.read(8192)
>   File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
>     return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3447: ordinal not in range(128)

My first guess is that the file's encoding doesn't match your locale.
Do you have a non-ASCII locale set?  You can check with:

  $ locale
  LANG=en_US.UTF-8
  LC_CTYPE="en_US.UTF-8"
  LC_NUMERIC="en_US.UTF-8"
  LC_TIME="en_US.UTF-8"
  LC_COLLATE=C
  LC_MONETARY="en_US.UTF-8"
  LC_MESSAGES="en_US.UTF-8"
  LC_PAPER="en_US.UTF-8"
  LC_NAME="en_US.UTF-8"
  LC_ADDRESS="en_US.UTF-8"
  LC_TELEPHONE="en_US.UTF-8"
  LC_MEASUREMENT="en_US.UTF-8"
  LC_IDENTIFICATION="en_US.UTF-8"
  LC_ALL=

Although you can obviously have a different preferred language or
Unicode-capable encoding.  What you don't want is:

  # locale
  LANG=
  LC_CTYPE="POSIX"
  LC_NUMERIC="POSIX"
  LC_TIME="POSIX"
  LC_COLLATE="POSIX"
  LC_MONETARY="POSIX"
  LC_MESSAGES="POSIX"
  LC_PAPER="POSIX"
  LC_NAME="POSIX"
  LC_ADDRESS="POSIX"
  LC_TELEPHONE="POSIX"
  LC_MEASUREMENT="POSIX"
  LC_IDENTIFICATION="POSIX"
  LC_ALL=

If that's not the problem, could you attach the troublesome message?
Or a minimal example derived from the troublesome message?

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: UnicodeDecodeError with python API

[Sebastian Fischmeister]

> My first guess is that the file's encoding doesn't match your locale.
> Do you have a non-ASCII locale set?  You can check with:

It seems to be more tricky than I thought. I didn't have a locale set.

When I set one, I can parse some emails with this:

export LANG=en_US.latin-1

Others with this:

export LANG=en_US.UTF-8

Others fail with either of the two.

I can display all messages correctly in emacs though.

  Sebastian

Re: UnicodeDecodeError with python API

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

On Sun, Mar 29, 2015 at 07:10:53PM -0400, Sebastian Fischmeister wrote:
> > My first guess is that the file's encoding doesn't match your
> > locale.  Do you have a non-ASCII locale set?  You can check with:
> 
> It seems to be more tricky than I thought. I didn't have a locale set.
> 
> When I set one, I can parse some emails with this:
> 
> export LANG=en_US.latin-1
> 
> Others with this:
> 
> export LANG=en_US.UTF-8
> 
> Others fail with either of the two.

Hmm, that's surprising.  In hindsight, the locale should only be
affecting the *output* (e.g., a non-Unicode locale might cause a
UnicodeEncodeError).  However, you're getting your errors on input.
I'd expect the files to be loaded and parsed as byte-streams, but
maybe there's a bug in Python's email parser.  It wouldn't be the
first time it's had trouble with bytes-vs-Unicode (see these old bugs
with similar tracebacks from the initial transition to 3.0 [1,2], or
search “unicode email” on http://bugs.python.org/).  I'd try to
reproduce this failure by calling email.message_from_file(…) directly
(getting notmuch out of the loop), and then file a bug against Python
once you have a pure-Python reproduction.

Cheers,
Trevor

[1]: http://bugs.python.org/issue1086
[2]: http://bugs.python.org/issue1258#msg56470

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]