From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Noam Postavsky Newsgroups: gmane.emacs.bugs Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space Date: Mon, 28 Aug 2017 10:42:14 -0400 Message-ID: References: <87d17oba8j.fsf@users.sourceforge.net> <83y3qbabzw.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="001a11c183aa03278b0557d14df9" X-Trace: blaine.gmane.org 1503931411 27989 195.159.176.226 (28 Aug 2017 14:43:31 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 28 Aug 2017 14:43:31 +0000 (UTC) Cc: 28180@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Aug 28 16:43:25 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dmLG8-0006yD-Vl for geb-bug-gnu-emacs@m.gmane.org; Mon, 28 Aug 2017 16:43:25 +0200 Original-Received: from localhost ([::1]:40132 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dmLGF-00022f-Ox for geb-bug-gnu-emacs@m.gmane.org; Mon, 28 Aug 2017 10:43:31 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:45429) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dmLFp-0001mf-B8 for bug-gnu-emacs@gnu.org; Mon, 28 Aug 2017 10:43:06 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dmLFm-0002HJ-5e for bug-gnu-emacs@gnu.org; Mon, 28 Aug 2017 10:43:05 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:50862) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dmLFm-0002HD-1T for bug-gnu-emacs@gnu.org; Mon, 28 Aug 2017 10:43:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dmLFl-0006NW-Rw for bug-gnu-emacs@gnu.org; Mon, 28 Aug 2017 10:43:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Noam Postavsky Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 28 Aug 2017 14:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 28180 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 28180-submit@debbugs.gnu.org id=B28180.150393134224474 (code B ref 28180); Mon, 28 Aug 2017 14:43:01 +0000 Original-Received: (at 28180) by debbugs.gnu.org; 28 Aug 2017 14:42:22 +0000 Original-Received: from localhost ([127.0.0.1]:59543 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dmLF8-0006Mg-3r for submit@debbugs.gnu.org; Mon, 28 Aug 2017 10:42:22 -0400 Original-Received: from mail-oi0-f47.google.com ([209.85.218.47]:35350) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dmLF7-0006MU-Am for 28180@debbugs.gnu.org; Mon, 28 Aug 2017 10:42:21 -0400 Original-Received: by mail-oi0-f47.google.com with SMTP id k77so5446303oib.2 for <28180@debbugs.gnu.org>; Mon, 28 Aug 2017 07:42:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=lecipUaW1VIs8rlAlFIqgVsJZGhd4R8AzPcX/MwKNOg=; b=d7UFUYV+Kath5heyXDw8Rd4y9iD+l6c0PbcSfYpLcPuxaHo5zKmsUOQCYB4q2wVzc7 ix8C47TAYK5wOHzXCNSMHWAbIBWNRP6MFj0idlchPk5GfKMIH3vxfqmbtaaAOzM0ih5t yyucdYZpimin0tGI9d3EmE0Wc+on4Vjt27rnWlcIfP+v7fGCGyDC2rfzvyit2EtDAb74 pXdvfVp2tHJKXvjTRv1rcKdNPQJnCH2tYX+Z5caHEmGH8aRbHHl7b4Zs1rsFwlxEHRfY ruU6SIPoaxTSDVxk+YMNIu0gXp3X5Pe8sqnQbe5g5LODBypC4jCXVWjQFOKpzH3fIQ3/ gVSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=lecipUaW1VIs8rlAlFIqgVsJZGhd4R8AzPcX/MwKNOg=; b=ay11a7lf/rBAgek75PgVR2eqbXO4KlsDZgbKc3DGQfrSipLVRh5QbFeIJaP6k7l7kZ KNZbr5smA7kFhQ6HQ069enmt4x4PcERwAD8LC+A+iSbF3rZBbDgis14cMYZ/Ql5KN9Pr PzLnCTBHH5oy1YAH24rXB7TGuucLUo406Sh5vd/1azxrrsq1QEhGjUaSV8+cK8MlgXbZ 4jAJOX28bM/nBIvSjotNf+JRdFfvZCbbMyYuFmhYzBEWq0kO58hOdNfrQSEa+NYEzZal brOUCa+JlxvYbtrAJ5IsCvHCf5TyzFXEMfHT6EaGdtxEtUzwX1mFzYW8UVEe2CpbjIP3 Y/Eg== X-Gm-Message-State: AHYfb5hTwgCqrnd3LU62DVaMpQKZD9mPtqhyL5A9ErWyvmVhsgzHdliN vKjJ5z4rtg56ietcGoF9nk9Omno4Ng== X-Received: by 10.202.78.85 with SMTP id c82mr669662oib.96.1503931335364; Mon, 28 Aug 2017 07:42:15 -0700 (PDT) Original-Received: by 10.74.17.199 with HTTP; Mon, 28 Aug 2017 07:42:14 -0700 (PDT) In-Reply-To: <83y3qbabzw.fsf@gnu.org> X-Google-Sender-Auth: UV20xlhB8iPjccvU4OLCrNrpU84 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:136295 Archived-At: --001a11c183aa03278b0557d14df9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Aug 22, 2017 at 10:54 AM, Eli Zaretskii wrote: > As to the "other places", the only problem I'm aware of is that the > encoding of the command-line arguments, when they arrive at w32proc.c, > is not known in advance, so this must be somehow fixed/changed, > otherwise we will be unable to re-encode them in UTF-16. I believe > the comment in w32.c does mention that. Just to understand the issue better, I applied the attached diff to use CreateProcessW. It seemed to work, but only when I start emacs from mingw's msys shell. When running from cmd.exe it still translates to space. Furthermore, when I run an unpatched Emacs from the msys shell, the output of the test I posted above is different: (call-process "git" nil '(t t) nil "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x" "x" "(=E5=A5=BD)") 0000000 28 c3 a5 c2 a5 c2 bd 29 0000010 Do you have any idea what setting could cause this? --001a11c183aa03278b0557d14df9 Content-Type: text/plain; charset="US-ASCII"; name="CreateProcessW.diff" Content-Disposition: attachment; filename="CreateProcessW.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_j6w9ncre0 ZGlmZiAtLWdpdCBjL3NyYy9wcm9jZXNzLmMgaS9zcmMvcHJvY2Vzcy5jCmluZGV4IGU3ZWU5OWEu LjJjMTdiNWEgMTAwNjQ0Ci0tLSBjL3NyYy9wcm9jZXNzLmMKKysrIGkvc3JjL3Byb2Nlc3MuYwpA QCAtMTg4MSw4ICsxODgxLDcgQEAgdXNhZ2U6IChtYWtlLXByb2Nlc3MgJnJlc3QgQVJHUykgICov KQogCSAgaWYgKFNUUklOR19NVUxUSUJZVEUgKGFyZykpCiAJICAgIHsKIAkgICAgICBpZiAoTklM UCAoYXJnX2VuY29kaW5nKSkKLQkJYXJnX2VuY29kaW5nID0gKGNvbXBsZW1lbnRfcHJvY2Vzc19l bmNvZGluZ19zeXN0ZW0KLQkJCQkoWFBST0NFU1MgKHByb2MpLT5lbmNvZGVfY29kaW5nX3N5c3Rl bSkpOworCQlhcmdfZW5jb2RpbmcgPSBRdXRmXzg7CiAJICAgICAgYXJnID0gY29kZV9jb252ZXJ0 X3N0cmluZ19ub3JlY29yZCAoYXJnLCBhcmdfZW5jb2RpbmcsIDEpOwogCSAgICB9CiAJICB0ZW0g PSBGY29ucyAoYXJnLCB0ZW0pOwpkaWZmIC0tZ2l0IGMvc3JjL3czMnByb2MuYyBpL3NyYy93MzJw cm9jLmMKaW5kZXggNzZhZjU1Zi4uODZhYWVlMiAxMDA2NDQKLS0tIGMvc3JjL3czMnByb2MuYwor KysgaS9zcmMvdzMycHJvYy5jCkBAIC0xMjA0LDE0ICsxMjA0LDEzIEBAIHN0YXRpYyBCT09MCiBj cmVhdGVfY2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNf Z3VpX2FwcCwKIAkgICAgICBwaWRfdCAqIHBQaWQsIGNoaWxkX3Byb2Nlc3MgKmNwKQogewotICBT VEFSVFVQSU5GTyBzdGFydDsKKyAgU1RBUlRVUElORk9XIHN0YXJ0OwogICBTRUNVUklUWV9BVFRS SUJVVEVTIHNlY19hdHRyczsKICNpZiAwCiAgIFNFQ1VSSVRZX0RFU0NSSVBUT1Igc2VjX2Rlc2M7 CiAjZW5kaWYKICAgRFdPUkQgZmxhZ3M7CiAgIGNoYXIgZGlyWyBNQVhfUEFUSCBdOwotICBjaGFy ICpwOwogICBjb25zdCBjaGFyICpleHQ7CiAKICAgaWYgKGNwID09IE5VTEwpIGVtYWNzX2Fib3J0 ICgpOwpAQCAtMTI0MiwxNCArMTI0MSw4IEBAIGNyZWF0ZV9jaGlsZCAoY2hhciAqZXhlLCBjaGFy ICpjbWRsaW5lLCBjaGFyICplbnYsIGludCBpc19ndWlfYXBwLAogICBzZWNfYXR0cnMubHBTZWN1 cml0eURlc2NyaXB0b3IgPSBOVUxMIC8qICZzZWNfZGVzYyAqLzsKICAgc2VjX2F0dHJzLmJJbmhl cml0SGFuZGxlID0gRkFMU0U7CiAKLSAgZmlsZW5hbWVfdG9fYW5zaSAocHJvY2Vzc19kaXIsIGRp cik7Ci0gIC8qIENhbid0IHVzZSB1bml4dG9kb3NfZmlsZW5hbWUgaGVyZSwgc2luY2UgdGhhdCBu ZWVkcyBpdHMgZmlsZSBuYW1lCi0gICAgIGFyZ3VtZW50IGVuY29kZWQgaW4gVVRGLTguICBPVE9I LCBwcm9jZXNzX2Rpciwgd2hpY2ggX2lzXyBpbgotICAgICBVVEYtOCwgcG9pbnRzLCB0byB0aGUg ZGlyZWN0b3J5IGNvbXB1dGVkIGJ5IG91ciBjYWxsZXIsIGFuZCB3ZQotICAgICBkb24ndCB3YW50 IHRvIG1vZGlmeSB0aGF0LCBlaXRoZXIuICAqLwotICBmb3IgKHAgPSBkaXI7ICpwOyBwID0gQ2hh ck5leHRBIChwKSkKLSAgICBpZiAoKnAgPT0gJy8nKQotICAgICAgKnAgPSAnXFwnOworICBzdHJj cHkgKGRpciwgcHJvY2Vzc19kaXIpOworICB1bml4dG9kb3NfZmlsZW5hbWUgKGRpcik7CiAKICAg LyogQ3JlYXRlUHJvY2VzcyBoYW5kbGVzIGJhdGNoIGZpbGVzIGFzIGV4ZSBzcGVjaWFsbHkuICBU aGlzIHNwZWNpYWwKICAgICAgaGFuZGxpbmcgZmFpbHMgd2hlbiBib3RoIHRoZSBiYXRjaCBmaWxl IGFuZCBhcmd1bWVudHMgYXJlIHF1b3RlZC4KQEAgLTEyNjUsOCArMTI1OCwxMyBAQCBjcmVhdGVf Y2hpbGQgKGNoYXIgKmV4ZSwgY2hhciAqY21kbGluZSwgY2hhciAqZW52LCBpbnQgaXNfZ3VpX2Fw cCwKIAkgICA6IENSRUFURV9ORVdfQ09OU09MRSk7CiAgIGlmIChOSUxQIChWdzMyX3N0YXJ0X3By b2Nlc3NfaW5oZXJpdF9lcnJvcl9tb2RlKSkKICAgICBmbGFncyB8PSBDUkVBVEVfREVGQVVMVF9F UlJPUl9NT0RFOwotICBpZiAoIUNyZWF0ZVByb2Nlc3NBIChleGUsIGNtZGxpbmUsICZzZWNfYXR0 cnMsIE5VTEwsIFRSVUUsCi0JCSAgICAgICBmbGFncywgZW52LCBkaXIsICZzdGFydCwgJmNwLT5w cm9jaW5mbykpCisKKyAgd2NoYXJfdCBleGVXW01BWF9QQVRIXSwgY21kbGluZVdbTUFYX1BBVEhd LCBkaXJXW01BWF9QQVRIXTsKKyAgZmlsZW5hbWVfdG9fdXRmMTYgKGV4ZSwgZXhlVyk7CisgIGZp bGVuYW1lX3RvX3V0ZjE2IChjbWRsaW5lLCBjbWRsaW5lVyk7CisgIGZpbGVuYW1lX3RvX3V0ZjE2 IChkaXIsIGRpclcpOworICBpZiAoIUNyZWF0ZVByb2Nlc3NXIChleGVXLCBjbWRsaW5lVywgJnNl Y19hdHRycywgTlVMTCwgVFJVRSwKKyAgICAgICAgICAgICAgICAgICAgICAgZmxhZ3MsIGVudiwg ZGlyVywgJnN0YXJ0LCAmY3AtPnByb2NpbmZvKSkKICAgICBnb3RvIEVIX0ZhaWw7CiAKICAgY3At PnBpZCA9IChpbnQpIGNwLT5wcm9jaW5mby5kd1Byb2Nlc3NJZDsK --001a11c183aa03278b0557d14df9--