bug#3616: 23.0.94; vc-bzr coding system bug

bug#3616: 23.0.94; vc-bzr coding system bug

[端瑞]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=GB2312, Size: 10449 bytes --]

In short, the bug is, when I use Chinese, it cannot commit successfully.
The version of my bazaar is 1.15.

Next I will show 3 bazaar sessions and their bzr log, 1 and 3 are with this bug.
Obviously, the command string passed from Emacs to bazaar was wrongly encoded.

1. When I run the command bzr commit in Emacs *shell* buffer, my
comment message is written in Chinese, the result was:
--------------------------------
d:\ehome\5-Dev\Mobile\Brew\Ver0.9\ehome>bzr commit -m
"ÐÞÕýÁËÁªÏµÈËÁбíµÄ½¹µãÒƶ¯ÎÊÌâºÍ°´#¼üʱµÄ²Ëµ¥¸üÐÂÎÊÌâ¡£"
bzr commit -m "ä¿  了联系人列表的焦点移动问题和æŒ?é” —¶çš„菜单更新问题ã ?
Traceback (most recent call last):
 File "bzr", line 130, in <module>
 File "bzrlib\commands.pyo", line 969, in main
bzrlib.errors.BzrError: Parameter
''\xe4\xbf\xae\xe6\xad\xa3\xe4\xba\x86\xe8\x81\x94\xe7\xb3\xbb\xe4\xba\xba\xe5\x88\x97\xe8\xa1\xa8\xe7\x9a\x84\xe7\x84\xa6\xe7\x82\xb9\xe7\xa7\xbb\xe5\x8a\xa8\xe9\x97\xae\xe9\xa2\x98\xe5\x92\x8c\xe6\x8c?\xe9\x94\xae\xe6\x97\xb6\xe7\x9a\x84\xe8\x8f\x9c\xe5\x8d\x95\xe6\x9b\xb4\xe6\x96\xb0\xe9\x97\xae\xe9\xa2\x98\xe3\x80?''
is unsupported by the current encoding.
--------------------------------
I run the above command twice.  Both got the same result.  There was
no bzr log  for these two operations in the file .bzr.log.

2. It could not commit until I changed the comment message to English:
--------------------------------
d:\ehome\5-Dev\Mobile\Brew\Ver0.9\ehome>bzr commit -m "Fixed the focus
move problems in the contact list and the update problem on the menu
bar when # key is clicked."
bzr commit -m "Fixed the focus move problems in the contact list and
the update problem on the menu bar when # key is clicked."
Committing to: D:/ehome/5-Dev/Mobile/Brew/Ver0.9/ehome/
modified .bzrignore
modified App/ChattingListBox.h
modified App/ChattingListBox.inl
modified App/EhomeAppLayer.h
modified App/EhomeAppLayer.inl
modified App/EhomeAppLayer_Session.inl
modified App/SipPriorityTable.h
modified App/StructInfo/Message.h
modified App/StructInfo/Message.inl
modified App/StructInfo/session.inl
modified AppUI/ChattingListBoxItem.inl
modified AppUI/ChattingTabPage.h
modified AppUI/ChattingTabPage.inl
modified AppUI/ContactsListTabPage.inl
modified AppUI/ContactsListTree.inl
modified AppUI/FormChat.inl
modified AppUI/MsgItemTrans.inl
modified AppUI/SendTakePic.inl
modified AppUI/SetInfoTakePic.inl
modified AppUI/TreeViewWithAds.h
modified AppUI/TreeViewWithAds.inl
modified AppUI/WithAdItemListbox.inl
modified AppUI/AdsUI/AdsAction.h
modified AppUI/AdsUI/AdsAction.inl
modified AppUI/AdsUI/AdsComm.inl
modified AppUI/AdsUI/AdsHttpRequest.h
modified AppUI/AdsUI/AdsHttpRequest.inl
modified AppUI/AdsUI/AdsItemPainter.h
modified AppUI/AdsUI/AdsItemPainter.inl
modified common/SaveRecord.inl
modified common/config.h
modified common/ehomesound.h
modified common/ehomesound.inl
modified common/logger.inl
modified common/sound.inl
modified common/util.h
modified common/util.inl
modified common/net/MySocket.inl
modified common/stl/astringbuilder.inl
modified common/stl/wstring.h
added doc/contacts_focus.txt
added doc/tilemgr_test_cases.html
Committed revision 56.
--------------------------------

The log in .bzr.log is:
--------------------------------
ÐÇÆÚÎå 2009-06-19 15:46:19 +0800
0.125  bzr arguments: [u'commit', u'-m', u'Fixed the focus move
problems in the contact list and the update problem on the menu bar
when # key is clicked.']
0.140  looking for plugins in C:/Documents and
Settings/Ryan/Application Data/bazaar/2.0/plugins
0.140  looking for plugins in C:/Program Files/Bazaar/plugins
0.312  encoding stdout as osutils.get_user_encoding() 'cp936'
0.375  opening working tree 'D:/ehome/5-Dev/Mobile/Brew/Ver0.9/ehome'
0.437  preparing to commit
[ 4452] 2009-06-19 15:46:19.703 INFO: Committing to:
D:/ehome/5-Dev/Mobile/Brew/Ver0.9/ehome/
0.453  Selecting files for commit with filter []
[ 4452] 2009-06-19 15:46:19.875 INFO: modified .bzrignore
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/ChattingListBox.h
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/ChattingListBox.inl
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/EhomeAppLayer.h
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/EhomeAppLayer.inl
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/EhomeAppLayer_Session.inl
[ 4452] 2009-06-19 15:46:19.875 INFO: modified App/SipPriorityTable.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified App/StructInfo/Message.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified App/StructInfo/Message.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified App/StructInfo/session.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/ChattingListBoxItem.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/ChattingTabPage.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/ChattingTabPage.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/ContactsListTabPage.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/ContactsListTree.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/FormChat.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/MsgItemTrans.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/SendTakePic.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/SetInfoTakePic.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/TreeViewWithAds.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/TreeViewWithAds.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/WithAdItemListbox.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsAction.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsAction.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsComm.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsHttpRequest.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsHttpRequest.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsItemPainter.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified AppUI/AdsUI/AdsItemPainter.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/SaveRecord.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/config.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/ehomesound.h
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/ehomesound.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/logger.inl
[ 4452] 2009-06-19 15:46:19.890 INFO: modified common/sound.inl
[ 4452] 2009-06-19 15:46:19.905 INFO: modified common/util.h
[ 4452] 2009-06-19 15:46:19.905 INFO: modified common/util.inl
[ 4452] 2009-06-19 15:46:19.905 INFO: modified common/net/MySocket.inl
[ 4452] 2009-06-19 15:46:19.905 INFO: modified common/stl/astringbuilder.inl
[ 4452] 2009-06-19 15:46:19.905 INFO: modified common/stl/wstring.h
[ 4452] 2009-06-19 15:46:19.905 INFO: added doc/contacts_focus.txt
[ 4452] 2009-06-19 15:46:19.905 INFO: added doc/tilemgr_test_cases.html
[ 4452] 2009-06-19 15:46:20.733 INFO: Committed revision 56.
1.531  return code 0
--------------------------------

3. When I committed a single file from the *vc-dir* buffer using the v
shortcut, and write the message in Chinese, it also failed and
complained as follows:
--------------------------------
Traceback (most recent call last):
 File "bzr", line 130, in <module>
 File "bzrlib\commands.pyo", line 969, in main
bzrlib.errors.BzrError: Parameter
''\xe8\xaf\x95\xe8\xaf\x95\xe7\x94\xa8\xe4\xb8\xad\xe6\x96\x87\xe3\x80?''
is unsupported by the current encoding.
--------------------------------
It didn't generate any bzr log, either.



In GNU Emacs 23.0.94.1 (i386-mingw-nt5.1.2600)
of 2009-05-24 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'

Important settings:
 value of $LC_ALL: nil
 value of $LC_COLLATE: nil
 value of $LC_CTYPE: nil
 value of $LC_MESSAGES: nil
 value of $LC_MONETARY: nil
 value of $LC_NUMERIC: nil
 value of $LC_TIME: nil
 value of $LANG: CHS
 value of $XMODIFIERS: nil
 locale-coding-system: cp936
 default-enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
 diff-auto-refine-mode: t
 shell-dirtrack-mode: t
 desktop-save-mode: t
 show-paren-mode: t
 tooltip-mode: t
 tool-bar-mode: t
 mouse-wheel-mode: t
 menu-bar-mode: t
 file-name-shadow-mode: t
 global-font-lock-mode: t
 font-lock-mode: t
 blink-cursor-mode: t
 global-auto-composition-mode: t
 auto-composition-mode: t
 auto-encryption-mode: t
 auto-compression-mode: t
 column-number-mode: t
 line-number-mode: t
 transient-mark-mode: t

Recent input:
p d a t e SPC p r <backspace> <backspace> p r o b l
e m SPC o n SPC t h e m <backspace> SPC m e n u SPC
b a r SPC w h e n SPC p r e s s i n g <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> # SPC k e y SPC i s SPC p r e s s e d .
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> c l i c k e d .
C-f <return> M-v M-v C-l C-h l C-x 1 C-h L <return>
C-x o v C-v M-v C-x k <return> C-x o C-x 1 C-h k C-x
<return> f C-x <return> C-h C-x <return> t C-g C-h
k C-x <return> t C-x o C-v M-v C-p C-p C-p C-p C-p
C-p C-p C-p C-p C-p C-p C-n C-n C-n C-p C-p C-p C-p
C-p C-p C-p C-n <tab> <return> C-v C-x b <return> C-h
v t e r m i n a l <tab> <tab> <backspace> <backspace>
<backspace> <backspace> <backspace> c o <tab> <backspace>
<backspace> <tab> C-g C-x b <return> C-x b <return>
C-h v d e f a u l t SPC t e r <tab> <return> C-x 1
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
M-x f i n d SPC l i b <tab> <return> b z r <return>
M-x M-p <return> v c - v b <backspace> <backspace>
b a <backspace> z <tab> <return> C-v C-n C-n C-n C-n
C-n C-n C-n C-n C-f C-f C-f C-f C-f C-f <C-f2> <return>
M-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-x k <return>
M-x r e p o r t SPC <tab> <return>

Recent messages:
History item: 2
Quit [3 times]
History item: 1 [2 times]
Type C-x 1 to delete the help window, C-M-v to scroll help.
Buffer is read-only: #<buffer *Help*>
Type C-x 1 to delete the help window, C-M-v to scroll help.
Quit
mouse-2, RET: find function's definition
Quit
find-library-name: Can't find library bzr
call-interactively: End of buffer

bug#3616: 23.0.94; vc-bzr coding system bug

[Eli Zaretskii]

> Date: Fri, 19 Jun 2009 16:24:37 +0800
> From: =?UTF-8?Q?=E7=AB=AF=E7=91=9E?= <duanpanda@gmail.com>
> Cc: 
> Reply-To: =?UTF-8?Q?=E7=AB=AF=E7=91=9E?= <duanpanda@gmail.com>,
> 	3616@emacsbugs.donarmstrong.com
> 
> In short, the bug is, when I use Chinese, it cannot commit successfully.
> The version of my bazaar is 1.15.

Does it work for you from the command line?  If it does, what encoding
of Chinese do you use in that case?

> Next I will show 3 bazaar sessions and their bzr log, 1 and 3 are with this bug.
> Obviously, the command string passed from Emacs to bazaar was wrongly encoded.
> 
> 1. When I run the command bzr commit in Emacs *shell* buffer, my
> comment message is written in Chinese, the result was:

What is the value of buffer-file-coding-system in the *shell* buffer?
Does it help to change it to cp936?

bug#3616: 23.0.94; vc-bzr coding system bug

[Ryan Duan]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=GB2312, Size: 4040 bytes --]

It works from the command line which is part of Windows XP and uses
Windows ANSI coding system.  Windows command line seems use cp936 as
the coding system.
The value of buffer-file-coding-system in the *shell* buffer is
chinese-gbk-dos, one of whose alias is cp936-dos.  It doesn't help to
change it to any of cp936 nor chinese-iso-8bit.

I observe that *shell* and *VC-log* buffers pass UTF-8 encoded string
(Is Emacs's internal buffer code UTF-8?) to Windows command line,
which might be the real cause of this bug and other related bugs.
Three examples following.

EXAMPLE 1
--------------------------------
In *shell*,
d:\code>bzr commit -m "µÚ¶þ"
bzr commit -m "ç¬ ºŒ"
Traceback (most recent call last):
 File "bzr", line 130, in <module>
 File "bzrlib\commands.pyo", line 969, in main
bzrlib.errors.BzrError: Parameter ''\xe7\xac\xac\xe4\xba\x8c'' is
unsupported by the current encoding.

Notice ''\xe7\xac\xac\xe4\xba\x8c'' which is the UTF-8 encoding of my
inputted Chinese characters.  It was these UTF-8 string causing the
above error.

Apply C-u C-x = on the Chinese character "µÚ":
       character: µÚ (31532, #o75454, #x7b2c)
preferred charset: chinese-gbk (GBK Chinese simplified.)
      code point: 0xB5DA
          syntax: w    which means: word
        category:
                  .:Base, C:2-byte han, c:Chinese, h:Korean,
j:Japanese, |:line breakable
     buffer code: #xE7 #xAC #xAC
       file code: #xB5 #xDA (encoded by coding system chinese-gbk-dos)
         display: by this font (glyph code)
   uniscribe:-outline-ÐÂËÎÌå-normal-normal-normal-mono-13-*-*-*-c-*-gb2312.1980-0
(#x3100)

Notice its buffer code is "\xe7\xac\xac" which is the first substring
of ''\xe7\xac\xac\xe4\xba\x8c''.  The file code "\xb5\xda" is
chinse-gbk encoded, and is what I expect to pass to the command line
system in Windows, which might work correctly.  But unfortunately,
instead of passing Chinese GBK encoded string to SHELL, Emacs passes
UTF-8 encoded string to SHELL.

EXAMPLE 2
--------------------------------
In *VC-log* buffer, I inputted two Chinese characters "µÚ¶þ" which was
the same as that in EXAMPLE 1.
After C-c C-c, the same error occurs: bzrlib.errors.BzrError:
Parameter ''\xe7\xac\xac\xe4\xba\x8c'' is unsupported by the current
encoding.
Apply C-u C-x = on "µÚ" returned the same information as that in EXAMPLE 1.

EXAMPLE 3 (Another related bug)
--------------------------------
In Windows, I created a directory (folder) named "µÚ¶þ".
In dired, it works all right.
But in *shell*,
d:\>cd µÚ¶þ
cd ç¬ ºŒ
ϵͳÕÒ²»µ½Ö¸¶¨µÄ·¾¶¡£

It complains that the system cannot find the specified path.  Because
the "\xb5\xda\xb6\xfe"(Chinese GBK) is converted to
''\xe7\xac\xac\xe4\xba\x8c''(UTF-8) to pass to the SHELL, but the
SHELL can only process Chinese GBK characters.

CONCLUSION
--------------------------------
When we use Emacs on Chinese Windows, Chinese GBK characters are
converted to UTF-8 characters to pass to Windows command line, but
Windows command line cannot process UTF-8 characters, which causes
this bug and other related bugs.

I feel that this is not a small problem.  Emacs should detect the OS's
locale, then use the correct encoding system to interact with the OS.
It seems to do well on Linux but badly on Windows.  Dired seems do
well on Windows but shell.el and vc-bzr.el do badly.  I didn't test
other vc-* modes.

I hope the information above will help solve this problem.  Thank you!
HAPPY HACKING!

2009/6/19 Eli Zaretskii <eliz@gnu.org>:
>> Date: Fri, 19 Jun 2009 16:24:37 +0800
>> From: =?UTF-8?Q?=E7=AB=AF=E7=91=9E?= <duanpanda@gmail.com>
>> Cc:
>> Reply-To: =?UTF-8?Q?=E7=AB=AF=E7=91=9E?= <duanpanda@gmail.com>,
>>       3616@emacsbugs.donarmstrong.com

> Does it work for you from the command line?  If it does, what encoding
> of Chinese do you use in that case?
>
> What is the value of buffer-file-coding-system in the *shell* buffer?
> Does it help to change it to cp936?

bug#3616: 23.0.94; vc-bzr coding system bug

[Andreas Schwab]

Ryan Duan <duanpanda@gmail.com> writes:

> EXAMPLE 3 (Another related bug)
> --------------------------------
> In Windows, I created a directory (folder) named "第二".
> In dired, it works all right.
> But in *shell*,
> d:\>cd 第二
> cd 绗 簩
> 系统找不到指定的路径。
>
> It complains that the system cannot find the specified path.  Because
> the "\xb5\xda\xb6\xfe"(Chinese GBK) is converted to
> ''\xe7\xac\xac\xe4\xba\x8c''(UTF-8) to pass to the SHELL, but the
> SHELL can only process Chinese GBK characters.

What does (process-coding-system (get-buffer-process "*shell*")) return?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#3616: 23.0.94; vc-bzr coding system bug

[Ryan Duan]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=GB2312, Size: 913 bytes --]

(chinese-gbk-dos . undecided-dos)

2009/6/23 Andreas Schwab <schwab@linux-m68k.org>:
> Ryan Duan <duanpanda@gmail.com> writes:
>
>> EXAMPLE 3 (Another related bug)
>> --------------------------------
>> In Windows, I created a directory (folder) named "µÚ¶þ".
>> In dired, it works all right.
>> But in *shell*,
>> d:\>cd µÚ¶þ
>> cd ç¬ ºŒ
>> ϵͳÕÒ²»µ½Ö¸¶¨µÄ·¾¶¡£
>>
>> It complains that the system cannot find the specified path.  Because
>> the "\xb5\xda\xb6\xfe"(Chinese GBK) is converted to
>> ''\xe7\xac\xac\xe4\xba\x8c''(UTF-8) to pass to the SHELL, but the
>> SHELL can only process Chinese GBK characters.
>
> What does (process-coding-system (get-buffer-process "*shell*")) return?
>
> Andreas.
>
> --
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>