all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: "Laimonas Vėbra" <laimonas.vebra@gmail.com>
To: 6705@debbugs.gnu.org
Subject: bug#6705: w32 cmdproxy.c pass args to cygwin; erroneous charset conversion (problem description, solution/suggestion)
Date: Thu, 22 Jul 2010 15:31:44 +0300	[thread overview]
Message-ID: <4C483A30.9010804@gmail.com> (raw)

Below is a comment that i wrote for myself in the cmpdproxy.c (it 
explains the problem). I have a half (i suppose -- portable enough) of 
working solution/fix for it using MultiByteToWideChar API function, but 
i won't send a (partly working) patch, unless someone from the 
developers who agree with the problem and intend to fix it will ask for it.

Besides, the patch itself is larger than 10 diff lines and it uses 
(duplicates by copying) some helper functions/declarations 
(open_input_file(), close_file_data(), rva_to_section(), 
w32_executable_type, RVA_TO_PTR) from unexw32.c, so it may need some 
code refactoring.

This problem certainly needs some discussion (how best to solve it) 
because it addresses unicode communication aspects/issues. If some won't 
bother reading all the description, then here is a simple question -- 
how do one can/should (clearly) pass utf-8 arguments to an external 
(cygwin) app on windows? I suppose, now it's not possible.

Thank you for your attention.


> /* When calling cygwin executable we need to explicitly convert utf-8
>    arguments (it's encoding yhat Emacs uses internally and passes args to
>    external commands, when coding-system-for-write is nil) to utf-16 and
>    call unicode (wide) API function CreateProcess(W).
>    That needs to be done, because of this transcoding chain which
>    migth (and it definitely WILL if args contains unicode, i.e. non
>    ascii/locale_charset character) result in corrupted args:
>
>    WINAPI/OS layer:
>    multibyte string args (utf-8) -> CreateProcessA():
>    locale_codepage -> unicode (utf-16)
>
>    ->
>
>    CYGWIN layer:
>    unicode (utf-16) <-> utf-8 ->
>    cygwin locale env (LC_XXX, LANG; default: C.UTF-8)
>
>
>    Example #1:
>    utf-8 string 'žą'; 'ž'(0xC5, 0xBE) 'ą'(0xC4, 0x85) transcoding
>    (to cygwin locale env charset) chain:
>
>    converting #1:
>    locale_codepage (lt, LCID: 1063, ansi/oem cp: cp1257/cp775) -> utf-16;
>
>    utf-8 string 'žą' in locale codepage (cp1257) represenation: 'žą'
>    'Å'(0xC5), '¾'(0xBE), 'Ä'(0xC4), '…'(0x85).
>
>    string converted to utf-16: 'žą'
>    U+00C5(Å), U+00BE(¾), U+00C4(Ä), U+2026(…).
>
>    utf-16: 'žą': 'Å'(U+00C5), '¾'(U+00BE), 'Ä'(U+00C4), '…'(U+2026).
>    <->
>    utf-8 : 'žą': 'Å'(0xC385), '¾'(0xC2BE), 'Ä'(0xC384), '…'(0xE280).
>
>    converting #2:
>    utf-16/utf-8 -> cygwin locale env (LANG = lt_LT.cp1257);
>
>    utf-8 string 'žą' (0xC3, 0x85, 0xC2, 0xBE, 0xC3, 0x84, 0xE2, 0x80)
>    converted to cp1257: 'žą' (0xC5, 0xBE, 0xC4, 0x85)
>
>    cp1257 string 'žą' in utf-8 representation: 'žą'; 'ž'(0xC5BE), 'ą'(0xC485)
>
>    Although string was (should be) converted to cp1257 (according to
>    cygwin locale env variables), its original value ('žą'), after transcoding
>    to cp1257 (in cp1257 representation as it should be), is corrupted and indeed
>    passed args are (were preserved) in utf-8 encoding.
>    It's important to note that such "original value preservation" happens
>    only because of successful circumstances, when we are converting to windows
>    locale codepage/charset and arg string (utf-8) in  windows locale
>    representation doesn't result in some unconvertible character/combination
>    (e.g. undefined characters) and it's possible to convert back (from utf-16/utf-8
>    to locale charset). Corruption _always_ occurs  if we ar converting to other 	
>    codepage/charset than the current windows locale codepage.
>
>    Consider unsuccessful/erroneous conversion example:
>    utf-8 string/character 'ĥ' (U+0125) passed to cygwin (utf-8):
>
>    utf-8 string 'ĥ'(0xC4A5) in locale codepage (cp1257) representation: 'Ä'
>    (0xA5('') is undefined in cp1257 and it doesn't map to unicode)
>
>    converting #1:
>    locale_codepage (lt, LCID: 1063, ansi/oem cp: cp1257/cp775) -> utf-16;
>
>    utf-8 string 'ĥ' in cp1257 representation: 'Ä'
>
>    string converted to utf-16: 'Ä' (0x00C4, 0xF8FD)
>    (http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit1257.txt)
>    0xA5 (cp1257) is mapped to 0xF8FD in Unicode (Private Use Area Range: E000–F8FF)
>
>    utf-16: 'Ä': 'Ä'(U+00C4), ''(U+F8FD)
>    <->
>    utf-8 : 'Ä': 'Ä'(0xC384), ''(0xEFA3BD)
>
>    converting #2:
>    utf-16/utf-8 -> cygwin locale env (LANG = C.UTF-8);
>
>
>    utf-16 string 'Ä': 'Ä'(U+00C4), ''(U+F8FD)
>    converted to utf-8: 'Ä': 'Ä'(0xC384), ''(0xEFA3BD)
>
>    So, original string value 'ĥ' is transcoded to an invalid 'Ä' although that
>    shouldn't happen (as no conversion is supposed; neither implicitly, nor
>    explicitly)
>
>
>    Concluding all: erroneous conversion _always_ occurs, when we are converting
>    to codepage/charset other than the current windows locale codepage, although
>    corruption might occur even if we are not supposed to convert at all
>    (just pass utf-8 encoded arguments).
>
>
> */





             reply	other threads:[~2010-07-22 12:31 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-07-22 12:31 Laimonas Vėbra [this message]
2010-07-22 14:33 ` bug#6705: w32 cmdproxy.c pass args to cygwin; erroneous charset conversion (problem description, solution/suggestion) Jason Rumney
2010-07-22 18:14   ` bug#6546: " Laimonas Vėbra
2010-07-22 19:50 ` Eli Zaretskii
2010-07-22 20:59   ` Laimonas Vėbra
2010-07-23 10:21     ` Eli Zaretskii
2010-07-23 12:57       ` Laimonas Vėbra
2010-07-23 14:35         ` Eli Zaretskii
2010-07-23 15:35           ` Laimonas Vėbra
2010-07-23 18:06             ` Eli Zaretskii
2010-07-23 18:53               ` Laimonas Vėbra
2010-07-23 21:25                 ` Eli Zaretskii
2010-07-23 21:53                   ` Laimonas Vėbra
2010-07-24 20:33                     ` Eli Zaretskii
2010-07-25 10:10                       ` Laimonas Vėbra
2010-07-23 19:07             ` Juanma Barranquero
2010-07-23 19:51               ` Laimonas Vėbra
2010-07-23 19:59                 ` Juanma Barranquero
2010-07-22 22:56   ` Juanma Barranquero
2010-07-23 10:26     ` Eli Zaretskii
2010-07-23 10:45       ` Juanma Barranquero
2010-07-23 14:16         ` Eli Zaretskii
2010-07-25 23:38           ` Stefan Monnier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4C483A30.9010804@gmail.com \
    --to=laimonas.vebra@gmail.com \
    --cc=6705@debbugs.gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.