From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: npostavs@users.sourceforge.net Newsgroups: gmane.emacs.bugs Subject: bug#28180: [w32] Unicode characters in subprocess (git) arguments changed to space Date: Mon, 21 Aug 2017 22:35:24 -0400 Message-ID: <87d17oba8j.fsf@users.sourceforge.net> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1503369322 9282 195.159.176.226 (22 Aug 2017 02:35:22 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 22 Aug 2017 02:35:22 +0000 (UTC) To: 28180@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Aug 22 04:35:16 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1djz27-0001fh-4U for geb-bug-gnu-emacs@m.gmane.org; Tue, 22 Aug 2017 04:35:11 +0200 Original-Received: from localhost ([::1]:54224 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1djz2A-0007uB-Ou for geb-bug-gnu-emacs@m.gmane.org; Mon, 21 Aug 2017 22:35:14 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:41920) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1djz22-0007s8-4l for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:35:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1djz1z-0003GM-1Q for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:35:06 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:39738) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1djz1y-0003GG-Si for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:35:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1djz1y-0003bl-KJ for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:35:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: npostavs@users.sourceforge.net Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 22 Aug 2017 02:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 28180 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.150336924213797 (code B ref -1); Tue, 22 Aug 2017 02:35:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 22 Aug 2017 02:34:02 +0000 Original-Received: from localhost ([127.0.0.1]:48419 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1djz10-0003aT-9l for submit@debbugs.gnu.org; Mon, 21 Aug 2017 22:34:02 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:50819) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1djz0y-0003a0-I3 for submit@debbugs.gnu.org; Mon, 21 Aug 2017 22:34:01 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1djz0s-0002rM-24 for submit@debbugs.gnu.org; Mon, 21 Aug 2017 22:33:55 -0400 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:40748) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1djz0r-0002rG-UN for submit@debbugs.gnu.org; Mon, 21 Aug 2017 22:33:53 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:41785) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1djz0q-0007S5-Fq for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:33:53 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1djz0l-0002pB-OX for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:33:52 -0400 Original-Received: from mail-it0-x22d.google.com ([2607:f8b0:4001:c0b::22d]:37703) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1djz0l-0002ow-Ho for bug-gnu-emacs@gnu.org; Mon, 21 Aug 2017 22:33:47 -0400 Original-Received: by mail-it0-x22d.google.com with SMTP id 76so40288085ith.0 for ; Mon, 21 Aug 2017 19:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:mime-version; bh=PeXHU65WctXgemL2+Un3L/NOUHEZKv4Ue6Q2teo/Noc=; b=YHxRF7+G3hQSfvVl67GqN6iuy4Ozl3NkVgBBCmMtqMIbeOtoXkWrwKvtE/YIh6c/p1 /FQy1SGGAFapmP87cKNxVyiCf0DE/7wORkCjCQMYsbIX+aOlKdkBd5kjb6yWXHlxsnHh aLmQXQ9byQa7AJrghNKPtzAZf0S1GVQOZ5QFQ3EMGgLQHdvqxaKTz4vAjnGQw1zDkS5n pmex6uHdhwINzCxygSJOXH3ju2alKBWok6pvynPSxxJYO6Hc6SH652vt34VOVpAwoo64 sVQfHnE6XCfIPKGOi7JY67TuYMOO+4RT01Gg11OwqTtnJOC7EKEGtAySP1Us0y5vd1sr aQzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :mime-version; bh=PeXHU65WctXgemL2+Un3L/NOUHEZKv4Ue6Q2teo/Noc=; b=ruMl5AXgXV4zykqqyMLTyF+TmFkCbiKiK9SzXe+gFwlbdki0F/x3ytfv6ahebL/zrv FkczT4kzB1On5Jb1CvKKL7OUzCCnPkZiEfrxQcTQlZuE76Rz2HKrHDoBGKF27ujgNOIE H1VAxeDD8XNLmqN+QsIxYr5YoNu89LpspK4W+39jfzmatf43oKijoSX8MMPyjpmOuAaQ tOlEcNM9iFuPuXoWFQ6Rt2gEokIAT03NdIdm7jC/CuZ94Nv5aFdPbRpVf3ENHQ/N8opB rZlD9DLiGHgWYG+KGxw4xur7PzMze0ZLXEDxYgkTPLoVgeoiZ1hOF3yDNkksTglEPIjj qmXQ== X-Gm-Message-State: AHYfb5hCcC/Nccy+JM31ZmHlIEkBwZH8fDidQHxPyJMdaVaGnd3Exuyt iQjNTP90jFDp8krl X-Received: by 10.36.40.147 with SMTP id h141mr8042989ith.85.1503369226713; Mon, 21 Aug 2017 19:33:46 -0700 (PDT) Original-Received: from zony ([45.2.119.49]) by smtp.googlemail.com with ESMTPSA id f11sm631037itf.1.2017.08.21.19.33.45 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 21 Aug 2017 19:33:45 -0700 (PDT) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:136034 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable In w32.c there is a comment saying . Running subprocesses in non-ASCII directories and with non-ASCII file arguments is limited to the current codepage [...] This should be fixed, but will also require changes in cmdproxy. The current limitation is not terribly bad anyway, since very few, if any, Windows console programs that are likely to be invoked by Emacs support UTF-16 encoded command lines. I believe we're running into this limitation with git: staging a file named =E5=A5=BD.txt fails from magit[1] (I tried also with vc, same problem= ). A quick way to see the problem is evaluating the call-process form below, the output shows that the Chinese character has been transformed into a space. This happens even if I execute 'chcp 65001' before starting Emacs (a possible workaround I saw suggested in a few places). The short file name trick doesn't help either. (call-process "git" nil '(t t) nil "-c" "alias.x=3D!x() { printf '%s' \"$1\" | od -tx1; }; x" "x= " "(=E5=A5=BD)") 0000000 28 20 29 0000003 As far as I can tell, git does support UTF-16 encoded command lines, as demonstrated by the attached git-args.c, which produces the utf8 encoding of the character (this is also what the call-process form produces when I run it on GNU/Linux): C:\Users\npostavs\src\win32-args>.\git-args.exe 0000000 28 e5 a5 bd 29 0000005 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=git-args.c Content-Description: test program, C source #include #include int main(int argc, char **argv) { STARTUPINFOW wsi; PROCESS_INFORMATION pi; WCHAR wcmdline[] = L"git -c \"alias.x=!x() { printf '%s' \\\"$1\\\" | od -tx1; }; x\" x (\x597d)"; ZeroMemory( &wsi, sizeof(wsi) ); wsi.cb = sizeof(wsi); ZeroMemory( &pi, sizeof(pi) ); if (!CreateProcessW(NULL, wcmdline, NULL, NULL, // proc, thread attrs FALSE, // handle inheritance 0, // creation flags NULL, // env NULL, // cur dir &wsi, &pi)) { printf( "CreateProcess failed (%d).\n", GetLastError() ); return 1; } // Wait until child process exits. WaitForSingleObject( pi.hProcess, INFINITE ); // Close process and thread handles. CloseHandle( pi.hProcess ); CloseHandle( pi.hThread ); return 0; } --=-=-= Content-Type: text/plain Am I correct that this problem is related the w32.c comment? It's not clear to me what changes are needed in cmdproxy (and other places?) to address it. [1]: https://github.com/magit/magit/issues/3111 --=-=-=--