bug#5794: 23.1; UnicodeEncodeError in python-shell

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Max Arnold]

When python shell invoked via C-c C-c, its stdout encoding is set to None and unicode
characters can not be printed. Simple test case (0xA9 should produce unicode symbol (c) ):

1. Invoke python shell manually:

M-x python-shell
>>> print u'\xA9'
©
>>>

2. And this is how to trigger the error: close python shell opened at step (1), create
new buffer containing the same print command, switch it to python-mode and press
C-c C-c (python-send-buffer). Spawned python shell will show a traceback:

>>> Traceback (most recent call last):
  File "/tmp/py5308nvC", line 1, in <module>
    print u'\xA9'
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
>>>

Emacs was running with -nw mode in rxvt-unicode terminal, system-wide LANG is set
to ru_RU.UTF-8. On step (1) sys.stdout.encoding is UTF-8, on step (2) it is None.

The same issue was found using python-mode.el (https://bugs.launchpad.net/python-mode/+bug/550661).
For more details see this thread: http://lists.gnu.org/archive/html/help-gnu-emacs/2010-03/msg00233.html


In GNU Emacs 23.1.1 (i686-pc-linux-gnu, GTK+ Version 2.14.7)
 of 2009-12-15 on bbone
configured using `configure  '--prefix=/usr' '--build=i686-pc-linux-gnu'
'--host=i686-pc-linux-gnu' '--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--program-suffix=-emacs-23' '--infodir=/usr/share/info/emacs-23'
'--with-sound' '--with-x' '--with-toolkit-scroll-bars' '--without-gif'
'--without-jpeg' '--without-png' '--without-rsvg' '--without-tiff'
'--without-xpm' '--with-xft' '--without-libotf' '--without-m17n-flt'
'--with-x-toolkit=gtk' '--without-hesiod' '--without-kerberos'
'--without-kerberos5' '--with-gpm' '--with-dbus' 'build_alias=i686-pc-linux-gnu'
'host_alias=i686-pc-linux-gnu' 'CFLAGS=-march=i686 -O2 -pipe' 'LDFLAGS=-Wl,-O1''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ru_RU.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  show-paren-mode: t
  iswitchb-mode: t
  global-hl-line-mode: t
  ergoemacs-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC a r e p o TAB r t - e TAB RET

Recent messages:
("emacs")
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...
Loading functions...done
Loading ergoemacs-unbind...done
Loading ergoemacs-layout-us...done
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...done
Ido mode enabled [3 times]
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Chong Yidong]

> When python shell invoked via C-c C-c, its stdout encoding is set to
> None and unicode characters can not be printed. Simple test case (0xA9
> should produce unicode symbol (c) ):
>
> 1. Invoke python shell manually:
>
> M-x python-shell
> >>> print u'\xA9'
> ©
> >>>
>
> 2. And this is how to trigger the error: close python shell opened at
> step (1), create new buffer containing the same print command, switch
> it to python-mode and press C-c C-c (python-send-buffer). Spawned
> python shell will show a traceback:

> >>> Traceback (most recent call last):
>   File "/tmp/py5308nvC", line 1, in <module>
>     print u'\xA9'
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> position 0: ordinal not in range(128)

This is due to the following line in `run-python', which I merged from
the upstream python.el on 2008-02-21:

       ;; Suppress use of pager for help output:
       (process-connection-type nil))

Dave, do you remember what the rationale for this is?  (I don't
understand the comment.)

I'm not sure why using a pipe rather than a pty makes a difference,
though.

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Max Arnold]

On Sat, Apr 10, 2010 at 02:21:40PM -0400, Chong Yidong wrote:
> > >>> Traceback (most recent call last):
> >   File "/tmp/py5308nvC", line 1, in <module>
> >     print u'\xA9'
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> > position 0: ordinal not in range(128)
> 
> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
> 
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
> 
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)
> 
> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

This is probably for python builtin help() function, which can use pager
to scroll documentation page by page. When help() detects that terminal is
a pipe, it disables interactive scrolling and just prints full output as is.
Try this for example:

import os
help(os)

Fortunately, python understands $PAGER environment variable and will use it
for scrolling, so PAGER="cat" will have the same effect as using pipe.

Hope this helps.

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Dave Love]

Chong Yidong <cyd@stupidchicken.com> writes:

> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
>
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
>
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)

It's the most likely way to suppress interactive i/o generally, which is
why it should normally be used for inferior interpreters and things like
VC, apart from the reasons in the manual to avoid PTYs.

> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

It's certainly odd behaviour, but as you're dropping python.el for
python-mode.el, you needn't worry about it.  Amongst other things,
python-mode.el doesn't have the features that require results from the
sub-process.

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Dave Love]

Max Arnold <lwarxx@gmail.com> writes:

> Fortunately, python understands $PAGER environment variable and will use it
> for scrolling, so PAGER="cat" will have the same effect as using pipe.

I assume it will lose on MS Windows, which is why I rejected it, not
that I have any interest in MS Windows.

For a working version see http://www.loveshack.ukfsn.org/emacs/.  As far
as I know it doesn't have any unfixed reported bugs, or any of the bugs
people keep reporting to me against the fork in Emacs.  You might want
it anyway when things like symbol help and completion are dropped from
Emacs, or for things like Python 3 support now.

bug#5794: 23.1; UnicodeEncodeError in python-shell

[Chong Yidong]

>>> When python shell invoked via C-c C-c, its stdout encoding is set to
>>> None and unicode characters can not be printed.
>
>> This is due to the following line in `run-python'
>>   (process-connection-type nil))
>> I'm not sure why using a pipe rather than a pty makes a difference,
>> though.
>
> This is probably for python builtin help() function, which can use
> pager to scroll documentation page by page. When help() detects that
> terminal is a pipe, it disables interactive scrolling and just prints
> full output as is.

I've changed python.el in the trunk to use a pty; this should remove the
problem.