python mode shell and unicode

python mode shell and unicode

[Max Arnold]

Hi all!

My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits
UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and
os.environ.get('LANG') in python shell confirms this.

When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas?

Cheers, Max

Re: python mode shell and unicode

[Andreas Röhler]

Max Arnold wrote:
> Hi all!
> 
> My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits
> UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and
> os.environ.get('LANG') in python shell confirms this.
> 
> When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas?
> 
> Cheers, Max
> 
> 
> 


C-c C-c is bound here to

(python-send-buffer)

Send the current buffer to the inferior Python process.

What you get from

C-h v buffer-file-coding-system?


Andreas

--
https://code.launchpad.net/~a-roehler/python-mode
https://code.launchpad.net/s-x-emacs-werkstatt/

Re: python mode shell and unicode

[Max Arnold]

On Sat, Mar 27, 2010 at 06:48:00PM +0100, Andreas Röhler wrote:
> > My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits
> > UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and
> > os.environ.get('LANG') in python shell confirms this.
> >
> > When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas?
> >
> What you get from
> C-h v buffer-file-coding-system?

---
buffer-file-coding-system is a variable defined in `C source code'.
Its value is utf-8-unix
Local in buffer test1.py; global value is utf-8-unix
...
It does not apply to sending output to subprocesses, however.
---

Python module contains encoding specification # -*- coding: utf-8 -*- at the beginning.
Emacs version is 23.1.

Re: python mode shell and unicode

[Andreas Röhler]

Max Arnold wrote:
> On Sat, Mar 27, 2010 at 06:48:00PM +0100, Andreas Röhler wrote:
>>> My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits
>>> UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and
>>> os.environ.get('LANG') in python shell confirms this.
>>>
>>> When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas?
>>>
>> What you get from
>> C-h v buffer-file-coding-system?
> 
> ---
> buffer-file-coding-system is a variable defined in `C source code'.
> Its value is utf-8-unix
> Local in buffer test1.py; global value is utf-8-unix
> ...
> It does not apply to sending output to subprocesses, however.
> ---
> 
> Python module contains encoding specification # -*- coding: utf-8 -*- at the beginning.
> Emacs version is 23.1.
> 
> 
> 

Hmm, see inside `python-send-region'

  ;; Fixme: Write a `coding' header to the temp file if the region is
  ;; non-ASCII.

Maybe that indicates the cause?

You could try `py-execute-file' from python-mode.el

CC to mailing list there.


Andreas

--
https://code.launchpad.net/~a-roehler/python-mode
https://code.launchpad.net/s-x-emacs-werkstatt/



_______________________________________________
Python-mode mailing list
Python-mode@python.org
http://mail.python.org/mailman/listinfo/python-mode

Re: python mode shell and unicode

[Max Arnold]

On Sun, Mar 28, 2010 at 09:25:02AM +0200, Andreas Röhler wrote:
> Hmm, see inside `python-send-region'
> 
>   ;; Fixme: Write a `coding' header to the temp file if the region is
>   ;; non-ASCII.
> 
> Maybe that indicates the cause?

I think this is unrelated to the issue because my python code contains no
unicode characters; they are received from parsed web page and then printed
to stdout. The problem is with python process spawned by Emacs - its stdout
encoding is not specified (sys.stdout.encoding is None).

Maybe this is a python feature: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=415968

But then I do not understand why manually invoked pyton-shell handles unicode
characters just fine...

> You could try `py-execute-file' from python-mode.el

Sorry, didn't mentioned that I use python.el shipped with emacs. How to disable
built-in one and enable python-mode.el?  How to invoke py-execute-file from
M-x prompt?

BTW, print u'\xA9' is a nice test for this issue.

Cheers, Max

Re: python mode shell and unicode

[Andreas Röhler]

Max Arnold wrote:
> On Sun, Mar 28, 2010 at 09:25:02AM +0200, Andreas Röhler wrote:
>> Hmm, see inside `python-send-region'
>>
>>   ;; Fixme: Write a `coding' header to the temp file if the region is
>>   ;; non-ASCII.
>>
>> Maybe that indicates the cause?
> 
> I think this is unrelated to the issue because my python code contains no
> unicode characters; they are received from parsed web page and then printed
> to stdout. The problem is with python process spawned by Emacs - its stdout
> encoding is not specified (sys.stdout.encoding is None).
> 
> Maybe this is a python feature: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=415968
> 
> But then I do not understand why manually invoked pyton-shell handles unicode
> characters just fine...

Looks like your problem is rather on the python than the emacs side.

If I change the function in the bugreport given above into

def output(s):
    # print s.decode('utf-8')
    print s

redirecting into FILE with > works


> 
>> You could try `py-execute-file' from python-mode.el
> 
> Sorry, didn't mentioned that I use python.el shipped with emacs. How to disable
> built-in one and enable python-mode.el? 

Download it here

http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el

and get it loaded

 How to invoke py-execute-file from
> M-x prompt?

Just

M-x py-execute-file

should do it

Andreas

> 
> BTW, print u'\xA9' is a nice test for this issue.
> 
> Cheers, Max
> 
> 
> 

_______________________________________________
Python-mode mailing list
Python-mode@python.org
http://mail.python.org/mailman/listinfo/python-mode

Re: python mode shell and unicode

[Max Arnold]

On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote:
> Looks like your problem is rather on the python than the emacs side.

> Download it here
> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el
> 
> M-x py-execute-file

Ok, I placed (require 'python-mode) to init.el and it seems to be activated
(autoloading didn't worked for me). But M-x shows no available completions
for py-execute-file:

Possible completions are:
py-electric-backspace                  py-electric-colon
py-electric-delete                     py-end-of-def-or-class
py-execute-buffer                      py-execute-def-or-class
py-execute-import-or-reload            py-execute-region
py-execute-string

Although C-h f py-execute-file shows it and says it is defined in python-mode.el.


Next, quick test with print u'\xA9':

1. Invoke python shell manually:

M-x py-shell
>>> print u'\xA9'
©
>>>


2. Create new buffer containing the same print command, switch it to python-mode
and use py-execute-buffer:

>>> ## working on region in file /usr/tmp/python-9773IlV.py...
©
>>>


3. Close python shell (opened at step 1) and invoke py-execute-buffer again:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)


Is this really a python problem?  I think there is a difference in how Emacs spawns python
process in each case.

Re: python mode shell and unicode

[Andreas Röhler]

Max Arnold wrote:
> On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote:
>> Looks like your problem is rather on the python than the emacs side.
> 
>> Download it here
>> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el
>>
>> M-x py-execute-file
> 
> Ok, I placed (require 'python-mode) to init.el and it seems to be activated
> (autoloading didn't worked for me). But M-x shows no available completions
> for py-execute-file:
> 
> Possible completions are:
> py-electric-backspace                  py-electric-colon
> py-electric-delete                     py-end-of-def-or-class
> py-execute-buffer                      py-execute-def-or-class
> py-execute-import-or-reload            py-execute-region
> py-execute-string
> 
> Although C-h f py-execute-file shows it and says it is defined in python-mode.el.
> 
> 
> Next, quick test with print u'\xA9':
> 
> 1. Invoke python shell manually:
> 
> M-x py-shell
>>>> print u'\xA9'
> ©
> 
> 
> 2. Create new buffer containing the same print command, switch it to python-mode
> and use py-execute-buffer:
> 
>>>> ## working on region in file /usr/tmp/python-9773IlV.py...
> ©
> 

So that's what it should do(?)

> 
> 3. Close python shell (opened at step 1) and invoke py-execute-buffer again:
> 
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
> 
> 
> Is this really a python problem?  I think there is a difference in how Emacs spawns python
> process in each case.
> 
> 
> 

Hmm, yes, get the same error.
However, if I re-start ipython-shell parallel

it works again

In [11]: ## working on region in file /usr/tmp/python-3766xFD.py...
©


Maybe just start a python-shell to have a work-around?

Sorry, I'm not able to dive into now.

It may help, if you make a bug-report, having it noticed at least here:

https://bugs.launchpad.net/python-mode

Thanks

Andreas

Re: python mode shell and unicode

[Max Arnold]

On Sun, Mar 28, 2010 at 06:34:06PM +0200, Andreas Röhler wrote:
> > On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote:
> >> Looks like your problem is rather on the python than the emacs side.
> > 
> >> Download it here
> >> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el
> >>
> >> M-x py-execute-file
> > 
> > Ok, I placed (require 'python-mode) to init.el and it seems to be activated
> > (autoloading didn't worked for me). But M-x shows no available completions
> > for py-execute-file:
> > 
> > Possible completions are:
> > py-electric-backspace                  py-electric-colon
> > py-electric-delete                     py-end-of-def-or-class
> > py-execute-buffer                      py-execute-def-or-class
> > py-execute-import-or-reload            py-execute-region
> > py-execute-string
> > 
> > Although C-h f py-execute-file shows it and says it is defined in python-mode.el.
> > 
> > 
> > Next, quick test with print u'\xA9':
> > 
> > 1. Invoke python shell manually:
> > 
> > M-x py-shell
> >>>> print u'\xA9'
> > ©
> > 
> > 
> > 2. Create new buffer containing the same print command, switch it to python-mode
> > and use py-execute-buffer:
> > 
> >>>> ## working on region in file /usr/tmp/python-9773IlV.py...
> > ©
> > 
> 
> So that's what it should do(?)


Yes, (1) and (2) is expected behaviour, 0xA9 is the UTF-8 code for copyright symbol (C).

> > 
> > 3. Close python shell (opened at step 1) and invoke py-execute-buffer again:
> > 
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
> > 
> > 
> > Is this really a python problem?  I think there is a difference in how Emacs spawns python
> > process in each case.
> > 
> > 
> 
> Hmm, yes, get the same error.
> However, if I re-start ipython-shell parallel
> 
> it works again
> 
> In [11]: ## working on region in file /usr/tmp/python-3766xFD.py...
> ©
> 
> 
> Maybe just start a python-shell to have a work-around?

Initially I discovered this problem using python.el. It spawns new python
process upon C-c C-c even if there is existing one (looks like it launches
one process per buffer). So this workaround applicable only for python-mode.el


> Sorry, I'm not able to dive into now.
> 
> It may help, if you make a bug-report, having it noticed at least here:
> 
> https://bugs.launchpad.net/python-mode

Ok, I'll do this.  Should I do the same for python.el somewhere?

Re: python mode shell and unicode

[Andreas Röhler]

Max Arnold wrote:
> On Sun, Mar 28, 2010 at 06:34:06PM +0200, Andreas Röhler wrote:
>>> On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote:
>>>> Looks like your problem is rather on the python than the emacs side.
>>>> Download it here
>>>> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el
>>>>
>>>> M-x py-execute-file
>>> Ok, I placed (require 'python-mode) to init.el and it seems to be activated
>>> (autoloading didn't worked for me). But M-x shows no available completions
>>> for py-execute-file:
>>>
>>> Possible completions are:
>>> py-electric-backspace                  py-electric-colon
>>> py-electric-delete                     py-end-of-def-or-class
>>> py-execute-buffer                      py-execute-def-or-class
>>> py-execute-import-or-reload            py-execute-region
>>> py-execute-string
>>>
>>> Although C-h f py-execute-file shows it and says it is defined in python-mode.el.
>>>
>>>
>>> Next, quick test with print u'\xA9':
>>>
>>> 1. Invoke python shell manually:
>>>
>>> M-x py-shell
>>>>>> print u'\xA9'
>>> ©
>>>
>>>
>>> 2. Create new buffer containing the same print command, switch it to python-mode
>>> and use py-execute-buffer:
>>>
>>>>>> ## working on region in file /usr/tmp/python-9773IlV.py...
>>> ©
>>>
>> So that's what it should do(?)
> 
> 
> Yes, (1) and (2) is expected behaviour, 0xA9 is the UTF-8 code for copyright symbol (C).
> 
>>> 3. Close python shell (opened at step 1) and invoke py-execute-buffer again:
>>>
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
>>>
>>>
>>> Is this really a python problem?  I think there is a difference in how Emacs spawns python
>>> process in each case.
>>>
>>>
>> Hmm, yes, get the same error.
>> However, if I re-start ipython-shell parallel
>>
>> it works again
>>
>> In [11]: ## working on region in file /usr/tmp/python-3766xFD.py...
>> ©
>>
>>
>> Maybe just start a python-shell to have a work-around?
> 
> Initially I discovered this problem using python.el. It spawns new python
> process upon C-c C-c even if there is existing one (looks like it launches
> one process per buffer). So this workaround applicable only for python-mode.el
> 
> 
>> Sorry, I'm not able to dive into now.
>>
>> It may help, if you make a bug-report, having it noticed at least here:
>>
>> https://bugs.launchpad.net/python-mode
> 
> Ok, I'll do this.  Should I do the same for python.el somewhere?
> 
> 
> 

Yes, thanks

M-x report-emacs-bug

will provide the destination.

Andreas
_______________________________________________
Python-mode mailing list
Python-mode@python.org
http://mail.python.org/mailman/listinfo/python-mode