unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#5794: 23.1; UnicodeEncodeError in python-shell
@ 2010-03-29  4:38 Max Arnold
  2010-04-10 18:21 ` Chong Yidong
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Max Arnold @ 2010-03-29  4:38 UTC (permalink / raw)
  To: 5794

When python shell invoked via C-c C-c, its stdout encoding is set to None and unicode
characters can not be printed. Simple test case (0xA9 should produce unicode symbol (c) ):

1. Invoke python shell manually:

M-x python-shell
>>> print u'\xA9'
©
>>>

2. And this is how to trigger the error: close python shell opened at step (1), create
new buffer containing the same print command, switch it to python-mode and press
C-c C-c (python-send-buffer). Spawned python shell will show a traceback:

>>> Traceback (most recent call last):
  File "/tmp/py5308nvC", line 1, in <module>
    print u'\xA9'
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
>>>

Emacs was running with -nw mode in rxvt-unicode terminal, system-wide LANG is set
to ru_RU.UTF-8. On step (1) sys.stdout.encoding is UTF-8, on step (2) it is None.

The same issue was found using python-mode.el (https://bugs.launchpad.net/python-mode/+bug/550661).
For more details see this thread: http://lists.gnu.org/archive/html/help-gnu-emacs/2010-03/msg00233.html


In GNU Emacs 23.1.1 (i686-pc-linux-gnu, GTK+ Version 2.14.7)
 of 2009-12-15 on bbone
configured using `configure  '--prefix=/usr' '--build=i686-pc-linux-gnu'
'--host=i686-pc-linux-gnu' '--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--program-suffix=-emacs-23' '--infodir=/usr/share/info/emacs-23'
'--with-sound' '--with-x' '--with-toolkit-scroll-bars' '--without-gif'
'--without-jpeg' '--without-png' '--without-rsvg' '--without-tiff'
'--without-xpm' '--with-xft' '--without-libotf' '--without-m17n-flt'
'--with-x-toolkit=gtk' '--without-hesiod' '--without-kerberos'
'--without-kerberos5' '--with-gpm' '--with-dbus' 'build_alias=i686-pc-linux-gnu'
'host_alias=i686-pc-linux-gnu' 'CFLAGS=-march=i686 -O2 -pipe' 'LDFLAGS=-Wl,-O1''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ru_RU.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  show-paren-mode: t
  iswitchb-mode: t
  global-hl-line-mode: t
  ergoemacs-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC a r e p o TAB r t - e TAB RET

Recent messages:
("emacs")
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...
Loading functions...done
Loading ergoemacs-unbind...done
Loading ergoemacs-layout-us...done
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...done
Ido mode enabled [3 times]
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...







^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#5794: 23.1; UnicodeEncodeError in python-shell
  2010-03-29  4:38 bug#5794: 23.1; UnicodeEncodeError in python-shell Max Arnold
@ 2010-04-10 18:21 ` Chong Yidong
  2010-04-11  2:47   ` Max Arnold
  2010-04-12 23:04 ` Dave Love
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 6+ messages in thread
From: Chong Yidong @ 2010-04-10 18:21 UTC (permalink / raw)
  To: Dave Love; +Cc: Max Arnold, 5794

> When python shell invoked via C-c C-c, its stdout encoding is set to
> None and unicode characters can not be printed. Simple test case (0xA9
> should produce unicode symbol (c) ):
>
> 1. Invoke python shell manually:
>
> M-x python-shell
> >>> print u'\xA9'
> ©
> >>>
>
> 2. And this is how to trigger the error: close python shell opened at
> step (1), create new buffer containing the same print command, switch
> it to python-mode and press C-c C-c (python-send-buffer). Spawned
> python shell will show a traceback:

> >>> Traceback (most recent call last):
>   File "/tmp/py5308nvC", line 1, in <module>
>     print u'\xA9'
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> position 0: ordinal not in range(128)

This is due to the following line in `run-python', which I merged from
the upstream python.el on 2008-02-21:

       ;; Suppress use of pager for help output:
       (process-connection-type nil))

Dave, do you remember what the rationale for this is?  (I don't
understand the comment.)

I'm not sure why using a pipe rather than a pty makes a difference,
though.






^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#5794: 23.1; UnicodeEncodeError in python-shell
  2010-04-10 18:21 ` Chong Yidong
@ 2010-04-11  2:47   ` Max Arnold
  0 siblings, 0 replies; 6+ messages in thread
From: Max Arnold @ 2010-04-11  2:47 UTC (permalink / raw)
  To: Chong Yidong; +Cc: 5794, Dave Love

On Sat, Apr 10, 2010 at 02:21:40PM -0400, Chong Yidong wrote:
> > >>> Traceback (most recent call last):
> >   File "/tmp/py5308nvC", line 1, in <module>
> >     print u'\xA9'
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> > position 0: ordinal not in range(128)
> 
> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
> 
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
> 
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)
> 
> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

This is probably for python builtin help() function, which can use pager
to scroll documentation page by page. When help() detects that terminal is
a pipe, it disables interactive scrolling and just prints full output as is.
Try this for example:

import os
help(os)

Fortunately, python understands $PAGER environment variable and will use it
for scrolling, so PAGER="cat" will have the same effect as using pipe.

Hope this helps.






^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#5794: 23.1; UnicodeEncodeError in python-shell
  2010-03-29  4:38 bug#5794: 23.1; UnicodeEncodeError in python-shell Max Arnold
  2010-04-10 18:21 ` Chong Yidong
@ 2010-04-12 23:04 ` Dave Love
  2010-04-12 23:08 ` Dave Love
  2010-08-24 20:22 ` Chong Yidong
  3 siblings, 0 replies; 6+ messages in thread
From: Dave Love @ 2010-04-12 23:04 UTC (permalink / raw)
  To: Chong Yidong; +Cc: Max Arnold, 5794@debbugs.gnu.org

Chong Yidong <cyd@stupidchicken.com> writes:

> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
>
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
>
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)

It's the most likely way to suppress interactive i/o generally, which is
why it should normally be used for inferior interpreters and things like
VC, apart from the reasons in the manual to avoid PTYs.

> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

It's certainly odd behaviour, but as you're dropping python.el for
python-mode.el, you needn't worry about it.  Amongst other things,
python-mode.el doesn't have the features that require results from the
sub-process.






^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#5794: 23.1; UnicodeEncodeError in python-shell
  2010-03-29  4:38 bug#5794: 23.1; UnicodeEncodeError in python-shell Max Arnold
  2010-04-10 18:21 ` Chong Yidong
  2010-04-12 23:04 ` Dave Love
@ 2010-04-12 23:08 ` Dave Love
  2010-08-24 20:22 ` Chong Yidong
  3 siblings, 0 replies; 6+ messages in thread
From: Dave Love @ 2010-04-12 23:08 UTC (permalink / raw)
  To: Max Arnold; +Cc: 5794@debbugs.gnu.org, Chong Yidong

Max Arnold <lwarxx@gmail.com> writes:

> Fortunately, python understands $PAGER environment variable and will use it
> for scrolling, so PAGER="cat" will have the same effect as using pipe.

I assume it will lose on MS Windows, which is why I rejected it, not
that I have any interest in MS Windows.

For a working version see http://www.loveshack.ukfsn.org/emacs/.  As far
as I know it doesn't have any unfixed reported bugs, or any of the bugs
people keep reporting to me against the fork in Emacs.  You might want
it anyway when things like symbol help and completion are dropped from
Emacs, or for things like Python 3 support now.






^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#5794: 23.1; UnicodeEncodeError in python-shell
  2010-03-29  4:38 bug#5794: 23.1; UnicodeEncodeError in python-shell Max Arnold
                   ` (2 preceding siblings ...)
  2010-04-12 23:08 ` Dave Love
@ 2010-08-24 20:22 ` Chong Yidong
  3 siblings, 0 replies; 6+ messages in thread
From: Chong Yidong @ 2010-08-24 20:22 UTC (permalink / raw)
  To: Max Arnold; +Cc: 5794

>>> When python shell invoked via C-c C-c, its stdout encoding is set to
>>> None and unicode characters can not be printed.
>
>> This is due to the following line in `run-python'
>>   (process-connection-type nil))
>> I'm not sure why using a pipe rather than a pty makes a difference,
>> though.
>
> This is probably for python builtin help() function, which can use
> pager to scroll documentation page by page. When help() detects that
> terminal is a pipe, it disables interactive scrolling and just prints
> full output as is.

I've changed python.el in the trunk to use a pty; this should remove the
problem.





^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2010-08-24 20:22 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-03-29  4:38 bug#5794: 23.1; UnicodeEncodeError in python-shell Max Arnold
2010-04-10 18:21 ` Chong Yidong
2010-04-11  2:47   ` Max Arnold
2010-04-12 23:04 ` Dave Love
2010-04-12 23:08 ` Dave Love
2010-08-24 20:22 ` Chong Yidong

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).