From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#12055: 24.1.50; Characters "=?UTF-8?Q?=C3=A1?=" and "=?UTF-8?Q?=C3=A9?=" are not correctly displayed on a Windows terminal Date: Fri, 27 Jul 2012 21:03:43 +0300 Message-ID: <83wr1phyio.fsf@gnu.org> References: <83vchajyb1.fsf@gnu.org> <83txwujwyg.fsf@gnu.org> <83pq7ijv9m.fsf@gnu.org> <83k3xqjnns.fsf@gnu.org> <83hastk8i1.fsf@gnu.org> <83d33hk212.fsf@gnu.org> <83zk6li6gd.fsf@gnu.org> <87hastywxb.fsf@gnu.org> Reply-To: Eli Zaretskii NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Trace: dough.gmane.org 1343412251 29395 80.91.229.3 (27 Jul 2012 18:04:11 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Fri, 27 Jul 2012 18:04:11 +0000 (UTC) Cc: lekktu@gmail.com, 12055@debbugs.gnu.org To: Jason Rumney Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Jul 27 20:04:11 2012 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1SuotW-0003Cq-Ng for geb-bug-gnu-emacs@m.gmane.org; Fri, 27 Jul 2012 20:04:10 +0200 Original-Received: from localhost ([::1]:40140 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SuotV-0007MX-RG for geb-bug-gnu-emacs@m.gmane.org; Fri, 27 Jul 2012 14:04:09 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:55534) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SuotT-0007MN-2N for bug-gnu-emacs@gnu.org; Fri, 27 Jul 2012 14:04:08 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SuotR-0006Ua-L0 for bug-gnu-emacs@gnu.org; Fri, 27 Jul 2012 14:04:06 -0400 Original-Received: from debbugs.gnu.org ([140.186.70.43]:34901) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SuotR-0006UO-HH for bug-gnu-emacs@gnu.org; Fri, 27 Jul 2012 14:04:05 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.72) (envelope-from ) id 1Sup0A-0005Ms-Dn for bug-gnu-emacs@gnu.org; Fri, 27 Jul 2012 14:11:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 27 Jul 2012 18:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 12055 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 12055-submit@debbugs.gnu.org id=B12055.134341264020601 (code B ref 12055); Fri, 27 Jul 2012 18:11:02 +0000 Original-Received: (at 12055) by debbugs.gnu.org; 27 Jul 2012 18:10:40 +0000 Original-Received: from localhost ([127.0.0.1]:44447 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Suozn-0005ME-9r for submit@debbugs.gnu.org; Fri, 27 Jul 2012 14:10:39 -0400 Original-Received: from mtaout21.012.net.il ([80.179.55.169]:50313) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Suozj-0005M3-R9 for 12055@debbugs.gnu.org; Fri, 27 Jul 2012 14:10:37 -0400 Original-Received: from conversion-daemon.a-mtaout21.012.net.il by a-mtaout21.012.net.il (HyperSendmail v2007.08) id <0M7T00G00YUHO200@a-mtaout21.012.net.il> for 12055@debbugs.gnu.org; Fri, 27 Jul 2012 21:03:36 +0300 (IDT) Original-Received: from HOME-C4E4A596F7 ([87.69.210.75]) by a-mtaout21.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0M7T00GWSZHZKM50@a-mtaout21.012.net.il>; Fri, 27 Jul 2012 21:03:36 +0300 (IDT) In-reply-to: <87hastywxb.fsf@gnu.org> X-012-Sender: halo1@inter.net.il X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 140.186.70.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:62468 Archived-At: > From: Jason Rumney > Cc: lekktu@gmail.com, 12055@debbugs.gnu.org > Date: Sat, 28 Jul 2012 00:46:08 +0800 > > > Well, I see some strange stuff in the input processing. > > /* Get the codepage to interpret this key with. */ > GetLocaleInfo (GetThreadLocale (), > LOCALE_IDEFAULTANSICODEPAGE, cp, 20); > cpId = atoi (cp); > > is quite suspicious. It appears in two places - one is a fallback for > older versions of Windows that do not fully support Unicode, the other > is more interesting for this case, as it is in the dead key handling, > and from Juanma's description, a dead key is being used to input the > problem characters. > > The above lines should probably be replaced with > > cpId = GetConsoleCP (); Thanks. Yes, I wondered about that as well. However, this is not my problem right now. If we were decoding input with a wrong codepage, I should have at least seen correct Unicode character codes right at entry into key_event. But what I see on my machine (whose ANSI encoding is cp1255 and the corresponding OEM encoding is cp862) is something really weird. When I switch the keyboard to Hebrew and type ALEPH, BET, GIMEL, whose Unicode codepoints are, respectively, u+05D0, u+05D1, u+05D2, I see 0x0580, 0x0581, and 0x0582 instead. That makes no sense at all, and no amount of tinkering with input codepage can ever fix that. Besides, at least in my locale, the code that you mention is never executed at all. Instead, we return the original Unicode character codepoint via this fragment: else if (event->uChar.UnicodeChar > 0) { emacs_ev->kind = MULTIBYTE_CHAR_KEYSTROKE_EVENT; emacs_ev->code = event->uChar.UnicodeChar; } And since, at least in my locale, event->uChar.UnicodeChar is wrong, the rest is a logical consequence of this. So my current theory is that it is simply wrong to look at uChar.UnicodeChar unless we call ReadConsoleInputW, the wide-character version of the API. But I need data from other locales to make sure this theory is correct. The theory is based on the following vague portion of the ReadConsoleInput's documentation: This function uses either Unicode characters or 8-bit characters from the console's current code page. There isn't a word about when it does one or the other (AFAICS), which led me to the above hypothesis, since that's the only cause that doesn't need to be explicitly documented. Btw, the MSDN documentation about stuff this is not as helpful as it could have been (so what else is new?). This page http://msdn.microsoft.com/en-us/library/windows/desktop/ms684166%28v=vs.85%29.aspx says: uChar A union of the following members. UnicodeChar Translated Unicode character. AsciiChar Translated ASCII character. What the heck do they mean by "translated" here? "Translated" by whom and how?