* python mode shell and unicode
@ 2010-03-26 15:35 Max Arnold
2010-03-27 17:48 ` Andreas Röhler
0 siblings, 1 reply; 10+ messages in thread
From: Max Arnold @ 2010-03-26 15:35 UTC (permalink / raw)
To: help-gnu-emacs
Hi all!
My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits
UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and
os.environ.get('LANG') in python shell confirms this.
When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas?
Cheers, Max
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: python mode shell and unicode 2010-03-26 15:35 python mode shell and unicode Max Arnold @ 2010-03-27 17:48 ` Andreas Röhler 2010-03-28 2:47 ` Max Arnold 0 siblings, 1 reply; 10+ messages in thread From: Andreas Röhler @ 2010-03-27 17:48 UTC (permalink / raw) To: help-gnu-emacs Max Arnold wrote: > Hi all! > > My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits > UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and > os.environ.get('LANG') in python shell confirms this. > > When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas? > > Cheers, Max > > > C-c C-c is bound here to (python-send-buffer) Send the current buffer to the inferior Python process. What you get from C-h v buffer-file-coding-system? Andreas -- https://code.launchpad.net/~a-roehler/python-mode https://code.launchpad.net/s-x-emacs-werkstatt/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-27 17:48 ` Andreas Röhler @ 2010-03-28 2:47 ` Max Arnold 2010-03-28 7:25 ` Andreas Röhler 0 siblings, 1 reply; 10+ messages in thread From: Max Arnold @ 2010-03-28 2:47 UTC (permalink / raw) To: help-gnu-emacs On Sat, Mar 27, 2010 at 06:48:00PM +0100, Andreas Röhler wrote: > > My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits > > UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and > > os.environ.get('LANG') in python shell confirms this. > > > > When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas? > > > What you get from > C-h v buffer-file-coding-system? --- buffer-file-coding-system is a variable defined in `C source code'. Its value is utf-8-unix Local in buffer test1.py; global value is utf-8-unix ... It does not apply to sending output to subprocesses, however. --- Python module contains encoding specification # -*- coding: utf-8 -*- at the beginning. Emacs version is 23.1. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 2:47 ` Max Arnold @ 2010-03-28 7:25 ` Andreas Röhler 2010-03-28 13:08 ` Max Arnold 0 siblings, 1 reply; 10+ messages in thread From: Andreas Röhler @ 2010-03-28 7:25 UTC (permalink / raw) To: help-gnu-emacs; +Cc: python-mode Max Arnold wrote: > On Sat, Mar 27, 2010 at 06:48:00PM +0100, Andreas Röhler wrote: >>> My python-shell invoked via C-c C-c from python buffer can not print unicode characters (emits >>> UnicodeEncodeError) and sys.stdout.encoding is empty. System wide LANG set to "ru_RU.UTF-8" and >>> os.environ.get('LANG') in python shell confirms this. >>> >>> When python-shell invoked manually (M-x python-shell) there is no encoding issues. Any ideas? >>> >> What you get from >> C-h v buffer-file-coding-system? > > --- > buffer-file-coding-system is a variable defined in `C source code'. > Its value is utf-8-unix > Local in buffer test1.py; global value is utf-8-unix > ... > It does not apply to sending output to subprocesses, however. > --- > > Python module contains encoding specification # -*- coding: utf-8 -*- at the beginning. > Emacs version is 23.1. > > > Hmm, see inside `python-send-region' ;; Fixme: Write a `coding' header to the temp file if the region is ;; non-ASCII. Maybe that indicates the cause? You could try `py-execute-file' from python-mode.el CC to mailing list there. Andreas -- https://code.launchpad.net/~a-roehler/python-mode https://code.launchpad.net/s-x-emacs-werkstatt/ _______________________________________________ Python-mode mailing list Python-mode@python.org http://mail.python.org/mailman/listinfo/python-mode ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 7:25 ` Andreas Röhler @ 2010-03-28 13:08 ` Max Arnold 2010-03-28 14:56 ` Andreas Röhler 0 siblings, 1 reply; 10+ messages in thread From: Max Arnold @ 2010-03-28 13:08 UTC (permalink / raw) To: help-gnu-emacs On Sun, Mar 28, 2010 at 09:25:02AM +0200, Andreas Röhler wrote: > Hmm, see inside `python-send-region' > > ;; Fixme: Write a `coding' header to the temp file if the region is > ;; non-ASCII. > > Maybe that indicates the cause? I think this is unrelated to the issue because my python code contains no unicode characters; they are received from parsed web page and then printed to stdout. The problem is with python process spawned by Emacs - its stdout encoding is not specified (sys.stdout.encoding is None). Maybe this is a python feature: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=415968 But then I do not understand why manually invoked pyton-shell handles unicode characters just fine... > You could try `py-execute-file' from python-mode.el Sorry, didn't mentioned that I use python.el shipped with emacs. How to disable built-in one and enable python-mode.el? How to invoke py-execute-file from M-x prompt? BTW, print u'\xA9' is a nice test for this issue. Cheers, Max ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 13:08 ` Max Arnold @ 2010-03-28 14:56 ` Andreas Röhler 2010-03-28 16:37 ` Max Arnold 0 siblings, 1 reply; 10+ messages in thread From: Andreas Röhler @ 2010-03-28 14:56 UTC (permalink / raw) To: help-gnu-emacs; +Cc: python-mode Max Arnold wrote: > On Sun, Mar 28, 2010 at 09:25:02AM +0200, Andreas Röhler wrote: >> Hmm, see inside `python-send-region' >> >> ;; Fixme: Write a `coding' header to the temp file if the region is >> ;; non-ASCII. >> >> Maybe that indicates the cause? > > I think this is unrelated to the issue because my python code contains no > unicode characters; they are received from parsed web page and then printed > to stdout. The problem is with python process spawned by Emacs - its stdout > encoding is not specified (sys.stdout.encoding is None). > > Maybe this is a python feature: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=415968 > > But then I do not understand why manually invoked pyton-shell handles unicode > characters just fine... Looks like your problem is rather on the python than the emacs side. If I change the function in the bugreport given above into def output(s): # print s.decode('utf-8') print s redirecting into FILE with > works > >> You could try `py-execute-file' from python-mode.el > > Sorry, didn't mentioned that I use python.el shipped with emacs. How to disable > built-in one and enable python-mode.el? Download it here http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el and get it loaded How to invoke py-execute-file from > M-x prompt? Just M-x py-execute-file should do it Andreas > > BTW, print u'\xA9' is a nice test for this issue. > > Cheers, Max > > > _______________________________________________ Python-mode mailing list Python-mode@python.org http://mail.python.org/mailman/listinfo/python-mode ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 14:56 ` Andreas Röhler @ 2010-03-28 16:37 ` Max Arnold 2010-03-28 16:34 ` Andreas Röhler 0 siblings, 1 reply; 10+ messages in thread From: Max Arnold @ 2010-03-28 16:37 UTC (permalink / raw) To: help-gnu-emacs On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote: > Looks like your problem is rather on the python than the emacs side. > Download it here > http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el > > M-x py-execute-file Ok, I placed (require 'python-mode) to init.el and it seems to be activated (autoloading didn't worked for me). But M-x shows no available completions for py-execute-file: Possible completions are: py-electric-backspace py-electric-colon py-electric-delete py-end-of-def-or-class py-execute-buffer py-execute-def-or-class py-execute-import-or-reload py-execute-region py-execute-string Although C-h f py-execute-file shows it and says it is defined in python-mode.el. Next, quick test with print u'\xA9': 1. Invoke python shell manually: M-x py-shell >>> print u'\xA9' © >>> 2. Create new buffer containing the same print command, switch it to python-mode and use py-execute-buffer: >>> ## working on region in file /usr/tmp/python-9773IlV.py... © >>> 3. Close python shell (opened at step 1) and invoke py-execute-buffer again: Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128) Is this really a python problem? I think there is a difference in how Emacs spawns python process in each case. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 16:37 ` Max Arnold @ 2010-03-28 16:34 ` Andreas Röhler 2010-03-28 17:58 ` Max Arnold 0 siblings, 1 reply; 10+ messages in thread From: Andreas Röhler @ 2010-03-28 16:34 UTC (permalink / raw) To: help-gnu-emacs; +Cc: python-mode Max Arnold wrote: > On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote: >> Looks like your problem is rather on the python than the emacs side. > >> Download it here >> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el >> >> M-x py-execute-file > > Ok, I placed (require 'python-mode) to init.el and it seems to be activated > (autoloading didn't worked for me). But M-x shows no available completions > for py-execute-file: > > Possible completions are: > py-electric-backspace py-electric-colon > py-electric-delete py-end-of-def-or-class > py-execute-buffer py-execute-def-or-class > py-execute-import-or-reload py-execute-region > py-execute-string > > Although C-h f py-execute-file shows it and says it is defined in python-mode.el. > > > Next, quick test with print u'\xA9': > > 1. Invoke python shell manually: > > M-x py-shell >>>> print u'\xA9' > © > > > 2. Create new buffer containing the same print command, switch it to python-mode > and use py-execute-buffer: > >>>> ## working on region in file /usr/tmp/python-9773IlV.py... > © > So that's what it should do(?) > > 3. Close python shell (opened at step 1) and invoke py-execute-buffer again: > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128) > > > Is this really a python problem? I think there is a difference in how Emacs spawns python > process in each case. > > > Hmm, yes, get the same error. However, if I re-start ipython-shell parallel it works again In [11]: ## working on region in file /usr/tmp/python-3766xFD.py... © Maybe just start a python-shell to have a work-around? Sorry, I'm not able to dive into now. It may help, if you make a bug-report, having it noticed at least here: https://bugs.launchpad.net/python-mode Thanks Andreas ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 16:34 ` Andreas Röhler @ 2010-03-28 17:58 ` Max Arnold 2010-03-28 17:28 ` Andreas Röhler 0 siblings, 1 reply; 10+ messages in thread From: Max Arnold @ 2010-03-28 17:58 UTC (permalink / raw) To: help-gnu-emacs On Sun, Mar 28, 2010 at 06:34:06PM +0200, Andreas Röhler wrote: > > On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote: > >> Looks like your problem is rather on the python than the emacs side. > > > >> Download it here > >> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el > >> > >> M-x py-execute-file > > > > Ok, I placed (require 'python-mode) to init.el and it seems to be activated > > (autoloading didn't worked for me). But M-x shows no available completions > > for py-execute-file: > > > > Possible completions are: > > py-electric-backspace py-electric-colon > > py-electric-delete py-end-of-def-or-class > > py-execute-buffer py-execute-def-or-class > > py-execute-import-or-reload py-execute-region > > py-execute-string > > > > Although C-h f py-execute-file shows it and says it is defined in python-mode.el. > > > > > > Next, quick test with print u'\xA9': > > > > 1. Invoke python shell manually: > > > > M-x py-shell > >>>> print u'\xA9' > > © > > > > > > 2. Create new buffer containing the same print command, switch it to python-mode > > and use py-execute-buffer: > > > >>>> ## working on region in file /usr/tmp/python-9773IlV.py... > > © > > > > So that's what it should do(?) Yes, (1) and (2) is expected behaviour, 0xA9 is the UTF-8 code for copyright symbol (C). > > > > 3. Close python shell (opened at step 1) and invoke py-execute-buffer again: > > > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128) > > > > > > Is this really a python problem? I think there is a difference in how Emacs spawns python > > process in each case. > > > > > > Hmm, yes, get the same error. > However, if I re-start ipython-shell parallel > > it works again > > In [11]: ## working on region in file /usr/tmp/python-3766xFD.py... > © > > > Maybe just start a python-shell to have a work-around? Initially I discovered this problem using python.el. It spawns new python process upon C-c C-c even if there is existing one (looks like it launches one process per buffer). So this workaround applicable only for python-mode.el > Sorry, I'm not able to dive into now. > > It may help, if you make a bug-report, having it noticed at least here: > > https://bugs.launchpad.net/python-mode Ok, I'll do this. Should I do the same for python.el somewhere? ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: python mode shell and unicode 2010-03-28 17:58 ` Max Arnold @ 2010-03-28 17:28 ` Andreas Röhler 0 siblings, 0 replies; 10+ messages in thread From: Andreas Röhler @ 2010-03-28 17:28 UTC (permalink / raw) To: help-gnu-emacs; +Cc: python-mode Max Arnold wrote: > On Sun, Mar 28, 2010 at 06:34:06PM +0200, Andreas Röhler wrote: >>> On Sun, Mar 28, 2010 at 04:56:49PM +0200, Andreas Röhler wrote: >>>> Looks like your problem is rather on the python than the emacs side. >>>> Download it here >>>> http://launchpad.net/python-mode/trunk/5.1.0/+download/python-mode.el >>>> >>>> M-x py-execute-file >>> Ok, I placed (require 'python-mode) to init.el and it seems to be activated >>> (autoloading didn't worked for me). But M-x shows no available completions >>> for py-execute-file: >>> >>> Possible completions are: >>> py-electric-backspace py-electric-colon >>> py-electric-delete py-end-of-def-or-class >>> py-execute-buffer py-execute-def-or-class >>> py-execute-import-or-reload py-execute-region >>> py-execute-string >>> >>> Although C-h f py-execute-file shows it and says it is defined in python-mode.el. >>> >>> >>> Next, quick test with print u'\xA9': >>> >>> 1. Invoke python shell manually: >>> >>> M-x py-shell >>>>>> print u'\xA9' >>> © >>> >>> >>> 2. Create new buffer containing the same print command, switch it to python-mode >>> and use py-execute-buffer: >>> >>>>>> ## working on region in file /usr/tmp/python-9773IlV.py... >>> © >>> >> So that's what it should do(?) > > > Yes, (1) and (2) is expected behaviour, 0xA9 is the UTF-8 code for copyright symbol (C). > >>> 3. Close python shell (opened at step 1) and invoke py-execute-buffer again: >>> >>> Traceback (most recent call last): >>> File "<stdin>", line 1, in <module> >>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128) >>> >>> >>> Is this really a python problem? I think there is a difference in how Emacs spawns python >>> process in each case. >>> >>> >> Hmm, yes, get the same error. >> However, if I re-start ipython-shell parallel >> >> it works again >> >> In [11]: ## working on region in file /usr/tmp/python-3766xFD.py... >> © >> >> >> Maybe just start a python-shell to have a work-around? > > Initially I discovered this problem using python.el. It spawns new python > process upon C-c C-c even if there is existing one (looks like it launches > one process per buffer). So this workaround applicable only for python-mode.el > > >> Sorry, I'm not able to dive into now. >> >> It may help, if you make a bug-report, having it noticed at least here: >> >> https://bugs.launchpad.net/python-mode > > Ok, I'll do this. Should I do the same for python.el somewhere? > > > Yes, thanks M-x report-emacs-bug will provide the destination. Andreas _______________________________________________ Python-mode mailing list Python-mode@python.org http://mail.python.org/mailman/listinfo/python-mode ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-03-28 17:58 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-03-26 15:35 python mode shell and unicode Max Arnold 2010-03-27 17:48 ` Andreas Röhler 2010-03-28 2:47 ` Max Arnold 2010-03-28 7:25 ` Andreas Röhler 2010-03-28 13:08 ` Max Arnold 2010-03-28 14:56 ` Andreas Röhler 2010-03-28 16:37 ` Max Arnold 2010-03-28 16:34 ` Andreas Röhler 2010-03-28 17:58 ` Max Arnold 2010-03-28 17:28 ` Andreas Röhler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).