bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

In MS Windows, users can easily create custom keyboard layouts using
Microsoft Keyboard Creator [1].

This can be used to for example to make typing certain Unicode symbols
easier (like Greek letters when typing mathematical formula), and is
very useful.

However, Emacs doesn't seem to handle the case when the keyboard
layout contains characters not available in the ANSI code page, and
just prints a question mark character instead. For certain characters,
a character that is visually similar to the actual character is
printed instead of a question mark. For example, if I use a layout
where AltGr+O produces U+2218 RING OPERATOR, Emacs prints U+00B0
DEGREE SYMBOL instead. The degree symbol is available in Windows 1252,
the default ANSI code page on my system, but the ring operator isn't.
However, if the layout maps AltGr+R to U+0220A SMALL ELEMENT OF, Emacs
just prints a question mark, presumably because Windows 1252 doesn't
contain a reasonable replacement for that character.

I can reproduce this on Emacs 24.0.92.1 (i386-mingw-nt5.1.2600) even
when started with -q to skip any customization.

I'd be happy to help debug this but I have no idea where to even
start. Is there an easy way to find out if it's the C code that
clobbers the character or if it happens in lisp for example?

[1]: http://msdn.microsoft.com/en-us/goglobal/bb964665

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Wed, 14 Dec 2011 21:39:28 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> 
> However, Emacs doesn't seem to handle the case when the keyboard
> layout contains characters not available in the ANSI code page, and
> just prints a question mark character instead.

Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
input.  Does it help to play with the value of keyboard-coding-system?

> For certain characters,
> a character that is visually similar to the actual character is
> printed instead of a question mark. For example, if I use a layout
> where AltGr+O produces U+2218 RING OPERATOR, Emacs prints U+00B0
> DEGREE SYMBOL instead. The degree symbol is available in Windows 1252,
> the default ANSI code page on my system, but the ring operator
> isn't.

I'm guessing that this is Windows trying to translate the characters
to the ANSI codepage behind the scenes.

> However, if the layout maps AltGr+R to U+0220A SMALL ELEMENT OF, Emacs
> just prints a question mark, presumably because Windows 1252 doesn't
> contain a reasonable replacement for that character.

Will inputting these characters with "C-x 8 RET 0220a RET" or "C-x 8
RET SMALL ELEMENT OF RET" be a good enough solution for you?  You can
input any Unicode character by its name or codepoint using "C-x 8 RET".

> I'd be happy to help debug this but I have no idea where to even
> start. Is there an easy way to find out if it's the C code that
> clobbers the character or if it happens in lisp for example?

I don't think there any "clobbering".  Emacs deliberately converts the
Unicode characters to the current locale's ANSI codepage.  I think
(but I'm not sure) the reason is that Emacs cannot use UTF-16 for
keyboard input.  Perhaps Jason and Handa-san could comment on this.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Kenichi Handa]

In article <E1Rb4hn-0002C6-5C@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> I don't think there any "clobbering".  Emacs deliberately converts the
> Unicode characters to the current locale's ANSI codepage.  I think
> (but I'm not sure) the reason is that Emacs cannot use UTF-16 for
> keyboard input.  Perhaps Jason and Handa-san could comment on this.

Currently only ASCII compatible coding system can be used
for keyboard-coding-system.  But, keyboard-coding-system is
for decoding tty input.  On graphic device, Emacs can decode
a system event directly.  For instance, handle_one_event (in
xterm.c) decodes the output of XIM (X Input Method)
directly.

---
Kenichi Handa
handa@m17n.org

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 15 December 2011 07:22, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 14 Dec 2011 21:39:28 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>
>> However, Emacs doesn't seem to handle the case when the keyboard
>> layout contains characters not available in the ANSI code page, and
>> just prints a question mark character instead.
>
> Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
> input.  Does it help to play with the value of keyboard-coding-system?

No, changing keyboard-coding-system doesn't help, and utf-16le-dos
isn't a valid setting for keyboard-coding-system anyway.

>> For certain characters,
>> a character that is visually similar to the actual character is
>> printed instead of a question mark. For example, if I use a layout
>> where AltGr+O produces U+2218 RING OPERATOR, Emacs prints U+00B0
>> DEGREE SYMBOL instead. The degree symbol is available in Windows 1252,
>> the default ANSI code page on my system, but the ring operator
>> isn't.
>
> I'm guessing that this is Windows trying to translate the characters
> to the ANSI codepage behind the scenes.
>
>> However, if the layout maps AltGr+R to U+0220A SMALL ELEMENT OF, Emacs
>> just prints a question mark, presumably because Windows 1252 doesn't
>> contain a reasonable replacement for that character.
>
> Will inputting these characters with "C-x 8 RET 0220a RET" or "C-x 8
> RET SMALL ELEMENT OF RET" be a good enough solution for you?  You can
> input any Unicode character by its name or codepoint using "C-x 8 RET".

Using C-x 8 is too cumbersome. I guess I could write my own custom
Emacs input method, but since Emacs now has good support for Unicode,
it would seem easier if it handled Unicode key events from the OS
correctly.

>> I'd be happy to help debug this but I have no idea where to even
>> start. Is there an easy way to find out if it's the C code that
>> clobbers the character or if it happens in lisp for example?
>
> I don't think there any "clobbering".  Emacs deliberately converts the
> Unicode characters to the current locale's ANSI codepage.  I think
> (but I'm not sure) the reason is that Emacs cannot use UTF-16 for
> keyboard input.  Perhaps Jason and Handa-san could comment on this.

I really don't know my way around the Emacs source, but a quick look
at w32_kbd_patch_key in w32inevt.c seems to indicate that Emacs really
is decoding the Unicode character event correctly, both
uChar.UnicodeChar and uChar.AsciiChar seem to be set correctly.

  /* On NT, call ToUnicode instead and then convert to the current
     locale's default codepage.  */
  if (os_subtype == OS_NT)
    {
      WCHAR buf[128];

      isdead = ToUnicode (event->wVirtualKeyCode, event->wVirtualScanCode,
			  keystate, buf, 128, 0);
      if (isdead > 0)
	{
	  char cp[20];
	  int cpId;

	  event->uChar.UnicodeChar = buf[isdead - 1];

	  GetLocaleInfo (GetThreadLocale (),
			 LOCALE_IDEFAULTANSICODEPAGE, cp, 20);
	  cpId = atoi (cp);
	  isdead = WideCharToMultiByte (cpId, 0, buf, isdead,
					ansi_code, 4, NULL, NULL);
	}
      else
	isdead = 0;
    }

However, this bit from w32_wnd_proc in w32fns.c looks suspicious to me:

		  else
		    {
		      /* Try to handle other keystrokes by determining the
			 base character (ie. translating the base key plus
			 shift modifier).  */
		      int add;
		      KEY_EVENT_RECORD key;

		      key.bKeyDown = TRUE;
		      key.wRepeatCount = 1;
		      key.wVirtualKeyCode = wParam;
		      key.wVirtualScanCode = (lParam & 0xFF0000) >> 16;
		      key.uChar.AsciiChar = 0;
		      key.dwControlKeyState = modifiers;

		      add = w32_kbd_patch_key (&key);
		      /* 0 means an unrecognized keycode, negative means
			 dead key.  Ignore both.  */
		      while (--add >= 0)
			{
			  /* Forward asciified character sequence.  */
			  post_character_message
			    (hwnd, WM_CHAR,
                             (unsigned char) key.uChar.AsciiChar, lParam,
			     w32_get_key_modifiers (wParam, lParam));
			  w32_kbd_patch_key (&key);
			}
		      return 0;
		    }

It looks like it's re-posting the event with just the Ascii key code,
clobbering the Unicode info that's originally in wParam. Or maybe the
idea is to translate characters that require multiple bytes into
multiple events?

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Thu, 15 Dec 2011 08:53:15 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: 10299@debbugs.gnu.org, handa@m17n.org, jasonr@gnu.org
> 
> I really don't know my way around the Emacs source, but a quick look
> at w32_kbd_patch_key in w32inevt.c seems to indicate that Emacs really
> is decoding the Unicode character event correctly, both
> uChar.UnicodeChar and uChar.AsciiChar seem to be set correctly.
> [...]
> However, this bit from w32_wnd_proc in w32fns.c looks suspicious to me:
> [...]
> It looks like it's re-posting the event with just the Ascii key code,
> clobbering the Unicode info that's originally in wParam.

I'd begin by establishing what kind of message Emacs gets when you
type a key created by MKLC.  It could be WM_KEYDOWN, but it could also
be something else, like WM_UNICHAR.

Can you find that out, either with a debugger or with some other tool?
If you can, please show the parameters of the message sent to Emacs.
Then we can continue tracing the code execution from there.

Note that Emacs has a separate thread that reads the Windows events,
and that thread then passes the events to the main thread, where the
event undergoes several levels of processing.  It's a complicated
issue, so we had better established from the get-go what kind of input
are we processing.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

Ok, I'll try to build Emacs on Windows and see if I can set a
breakpoint and look at the message Emacs is getting.

On 15 December 2011 11:52, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 15 Dec 2011 08:53:15 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> Cc: 10299@debbugs.gnu.org, handa@m17n.org, jasonr@gnu.org
>>
>> I really don't know my way around the Emacs source, but a quick look
>> at w32_kbd_patch_key in w32inevt.c seems to indicate that Emacs really
>> is decoding the Unicode character event correctly, both
>> uChar.UnicodeChar and uChar.AsciiChar seem to be set correctly.
>> [...]
>> However, this bit from w32_wnd_proc in w32fns.c looks suspicious to me:
>> [...]
>> It looks like it's re-posting the event with just the Ascii key code,
>> clobbering the Unicode info that's originally in wParam.
>
> I'd begin by establishing what kind of message Emacs gets when you
> type a key created by MKLC.  It could be WM_KEYDOWN, but it could also
> be something else, like WM_UNICHAR.
>
> Can you find that out, either with a debugger or with some other tool?
> If you can, please show the parameters of the message sent to Emacs.
> Then we can continue tracing the code execution from there.
>
> Note that Emacs has a separate thread that reads the Windows events,
> and that thread then passes the events to the main thread, where the
> event undergoes several levels of processing.  It's a complicated
> issue, so we had better established from the get-go what kind of input
> are we processing.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Thu, 15 Dec 2011 12:11:18 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: 10299@debbugs.gnu.org, handa@m17n.org, jasonr@gnu.org
> 
> Ok, I'll try to build Emacs on Windows and see if I can set a
> breakpoint and look at the message Emacs is getting.

Thanks.

For the record, my guess is that you will see Emacs getting a WM_CHAR
messages with character codes below 256, i.e. characters encoded in
the current ANSI codepage.  This is because in w32fns.c:w32_init_class
we call the ANSI variant of RegisterClass (since UNICODE is not
defined), which tells Windows that we want messages with text in the
ANSI character set.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Jason Rumney]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 14 Dec 2011 21:39:28 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> 
>> However, Emacs doesn't seem to handle the case when the keyboard
>> layout contains characters not available in the ANSI code page, and
>> just prints a question mark character instead.
>
> Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
> input.  Does it help to play with the value of keyboard-coding-system?

It tries to use Unicode if the Windows input method is sending it.  The
problem here is that there is no standard for Windows input methods, so
even the ones built into Windows act differently for different languages
and for the same language across different versions of Windows.

Some input methods will send WM_UNICHAR messages with the Unicode
character, which can be intercepted before they are turned into question
marks by Windows. These will just work with Emacs.

Others will send character codes corresponding to the codepoints in the
default Windows codepage for that language (regardless of your system
codepage). These can be handled by changing keyboard-coding-system
within Emacs.

Others will not send anything useful unless your system codepage is set
appropriately for the language being input.

It sounds like this one may be in the third category.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Lennart Borgman]

On Thu, Dec 15, 2011 at 15:40, Jason Rumney <jasonr@gnu.org> wrote:
>
> It tries to use Unicode if the Windows input method is sending it.  The
> problem here is that there is no standard for Windows input methods, so
> even the ones built into Windows act differently for different languages
> and for the same language across different versions of Windows.
>
> Some input methods will send WM_UNICHAR messages with the Unicode
> character, which can be intercepted before they are turned into question
> marks by Windows. These will just work with Emacs.
>
> Others will send character codes corresponding to the codepoints in the
> default Windows codepage for that language (regardless of your system
> codepage). These can be handled by changing keyboard-coding-system
> within Emacs.
>
> Others will not send anything useful unless your system codepage is set
> appropriately for the language being input.
>
> It sounds like this one may be in the third category.

Since Windows can handle this there must be a way to do it. But maybe
there are no docs from MS that explains how?

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 15 December 2011 15:40, Jason Rumney <jasonr@gnu.org> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Wed, 14 Dec 2011 21:39:28 +0100
>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>
>>> However, Emacs doesn't seem to handle the case when the keyboard
>>> layout contains characters not available in the ANSI code page, and
>>> just prints a question mark character instead.
>>
>> Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
>> input.  Does it help to play with the value of keyboard-coding-system?
>
> It tries to use Unicode if the Windows input method is sending it.  The
> problem here is that there is no standard for Windows input methods, so
> even the ones built into Windows act differently for different languages
> and for the same language across different versions of Windows.
>
> Some input methods will send WM_UNICHAR messages with the Unicode
> character, which can be intercepted before they are turned into question
> marks by Windows. These will just work with Emacs.
>
> Others will send character codes corresponding to the codepoints in the
> default Windows codepage for that language (regardless of your system
> codepage). These can be handled by changing keyboard-coding-system
> within Emacs.
>
> Others will not send anything useful unless your system codepage is set
> appropriately for the language being input.
>
> It sounds like this one may be in the third category.

Well, the custom keyboard layout works in all other programs I've
tried so Windows is certainly sending something useful to them.

However, I think IMEs use a slightly different, more complex method to
send input events. Emacs seems to handle IMEs better than custom
keyboard layouts, with an IME you can get ungarbled Unicode characters
into Emacs.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Cc: Joakim Hårsman <joakim.harsman@gmail.com>,
>   10299@debbugs.gnu.org,  handa@m17n.org
> Date: Thu, 15 Dec 2011 22:40:15 +0800
> 
> > Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
> > input.  Does it help to play with the value of keyboard-coding-system?
> 
> It tries to use Unicode if the Windows input method is sending it.

But am I right that, except for perhaps IME, we mostly get WM_CHAR
messages with ANSI character set, because we call the ANSI version of
RegisterClass?

If that is the reason, are there any complications to call
RegisterClassW instead (on NT and later systems; I think Windows 9X
are safer with ANSI)?

> Some input methods will send WM_UNICHAR messages with the Unicode
> character, which can be intercepted before they are turned into question
> marks by Windows. These will just work with Emacs.
> 
> Others will send character codes corresponding to the codepoints in the
> default Windows codepage for that language (regardless of your system
> codepage). These can be handled by changing keyboard-coding-system
> within Emacs.
> 
> Others will not send anything useful unless your system codepage is set
> appropriately for the language being input.
> 
> It sounds like this one may be in the third category.

I'm not sure MKLC works through IME.  Do you know that for a fact?

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 15 December 2011 18:34, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Jason Rumney <jasonr@gnu.org>
>> Cc: Joakim Hårsman <joakim.harsman@gmail.com>,
>>   10299@debbugs.gnu.org,  handa@m17n.org
>> Date: Thu, 15 Dec 2011 22:40:15 +0800
>>
>> > Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
>> > input.  Does it help to play with the value of keyboard-coding-system?
>>
>> It tries to use Unicode if the Windows input method is sending it.
>
> But am I right that, except for perhaps IME, we mostly get WM_CHAR
> messages with ANSI character set, because we call the ANSI version of
> RegisterClass?
>
> If that is the reason, are there any complications to call
> RegisterClassW instead (on NT and later systems; I think Windows 9X
> are safer with ANSI)?
>
>> Some input methods will send WM_UNICHAR messages with the Unicode
>> character, which can be intercepted before they are turned into question
>> marks by Windows. These will just work with Emacs.
>>
>> Others will send character codes corresponding to the codepoints in the
>> default Windows codepage for that language (regardless of your system
>> codepage). These can be handled by changing keyboard-coding-system
>> within Emacs.
>>
>> Others will not send anything useful unless your system codepage is set
>> appropriately for the language being input.
>>
>> It sounds like this one may be in the third category.
>
> I'm not sure MKLC works through IME.  Do you know that for a fact?

No, I'm just guessing since Emacs has worked with Unicode IMEs I've
tried in the past, but doesn't work with MKLC layouts. Instead of
debating, it seems easier to just get Emacs running with a debugger
and see what messages get delivered and what happens to them.

Anyway, I've gotten the latest Bazaar trunk but I haven't managed to
get it to build, compilation of make-docfile.c fails because it can't
find config.h:

gcc -I. -c -gdwarf-2 -g3  -mtune=pentium4 -O2      -Demacs=1 -DHAVE_CONFIG_H -I.
./lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5
000000 -o oo-spd/i386/make-docfile.o make-docfile.c
make-docfile.c:36:20: config.h: No such file or directory
make-docfile.c:79: error: syntax error before "NO_RETURN"
make-docfile.c:79: warning: data definition has no type or storage class
make[8]: *** [oo-spd/i386/make-docfile.o] Error 1
make[8]: Leaving directory `D:/Dev_projects/emacs/trunk/lib-src'
make[7]: *** [bootstrap-gmake] Error 2
make[7]: Leaving directory `D:/Dev_projects/emacs/trunk/nt'

I'm not really sure what to make of this, config.h is in trunk/src,
but gcc doesn't seem to be looking there from what I can see. Are
there more detailed build instructions available or is nt/INSTALL all
there is?

All I'm doing is cd:ing into the nt directory, running configure and
then make bootstrap.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 15 December 2011 21:50, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
> On 15 December 2011 18:34, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Jason Rumney <jasonr@gnu.org>
>>> Cc: Joakim Hårsman <joakim.harsman@gmail.com>,
>>>   10299@debbugs.gnu.org,  handa@m17n.org
>>> Date: Thu, 15 Dec 2011 22:40:15 +0800
>>>
>>> > Yes, Emacs on Windows uses the ANSI codepage to read the keyboard
>>> > input.  Does it help to play with the value of keyboard-coding-system?
>>>
>>> It tries to use Unicode if the Windows input method is sending it.
>>
>> But am I right that, except for perhaps IME, we mostly get WM_CHAR
>> messages with ANSI character set, because we call the ANSI version of
>> RegisterClass?
>>
>> If that is the reason, are there any complications to call
>> RegisterClassW instead (on NT and later systems; I think Windows 9X
>> are safer with ANSI)?
>>
>>> Some input methods will send WM_UNICHAR messages with the Unicode
>>> character, which can be intercepted before they are turned into question
>>> marks by Windows. These will just work with Emacs.
>>>
>>> Others will send character codes corresponding to the codepoints in the
>>> default Windows codepage for that language (regardless of your system
>>> codepage). These can be handled by changing keyboard-coding-system
>>> within Emacs.
>>>
>>> Others will not send anything useful unless your system codepage is set
>>> appropriately for the language being input.
>>>
>>> It sounds like this one may be in the third category.
>>
>> I'm not sure MKLC works through IME.  Do you know that for a fact?
>
> No, I'm just guessing since Emacs has worked with Unicode IMEs I've
> tried in the past, but doesn't work with MKLC layouts. Instead of
> debating, it seems easier to just get Emacs running with a debugger
> and see what messages get delivered and what happens to them.
>
> Anyway, I've gotten the latest Bazaar trunk but I haven't managed to
> get it to build, compilation of make-docfile.c fails because it can't
> find config.h:
>
> gcc -I. -c -gdwarf-2 -g3  -mtune=pentium4 -O2      -Demacs=1 -DHAVE_CONFIG_H -I.
> ./lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5
> 000000 -o oo-spd/i386/make-docfile.o make-docfile.c
> make-docfile.c:36:20: config.h: No such file or directory
> make-docfile.c:79: error: syntax error before "NO_RETURN"
> make-docfile.c:79: warning: data definition has no type or storage class
> make[8]: *** [oo-spd/i386/make-docfile.o] Error 1
> make[8]: Leaving directory `D:/Dev_projects/emacs/trunk/lib-src'
> make[7]: *** [bootstrap-gmake] Error 2
> make[7]: Leaving directory `D:/Dev_projects/emacs/trunk/nt'
>
> I'm not really sure what to make of this, config.h is in trunk/src,
> but gcc doesn't seem to be looking there from what I can see. Are
> there more detailed build instructions available or is nt/INSTALL all
> there is?
>
> All I'm doing is cd:ing into the nt directory, running configure and
> then make bootstrap.

Never mind, I got the build to work.

It turns out I only got the above error the second time I ran "make
bootstrap", presumably because some environment variable lingers from
the first invocation and messes things up.

To get the build to work I had to use a fresh shell and explicitly set
SHELL=cmd.exe when invoking make, then everything worked. I'll try to
attach a debugger tomorrow.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Thu, 15 Dec 2011 22:47:17 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: Jason Rumney <jasonr@gnu.org>, 10299@debbugs.gnu.org, handa@m17n.org
> 
> > gcc -I. -c -gdwarf-2 -g3  -mtune=pentium4 -O2      -Demacs=1 -DHAVE_CONFIG_H -I.
> > ./lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5
> > 000000 -o oo-spd/i386/make-docfile.o make-docfile.c
> > make-docfile.c:36:20: config.h: No such file or directory
> > make-docfile.c:79: error: syntax error before "NO_RETURN"
> > make-docfile.c:79: warning: data definition has no type or storage class
> > make[8]: *** [oo-spd/i386/make-docfile.o] Error 1
> > make[8]: Leaving directory `D:/Dev_projects/emacs/trunk/lib-src'
> > make[7]: *** [bootstrap-gmake] Error 2
> > make[7]: Leaving directory `D:/Dev_projects/emacs/trunk/nt'
> >
> > I'm not really sure what to make of this, config.h is in trunk/src,
> > but gcc doesn't seem to be looking there from what I can see. Are
> > there more detailed build instructions available or is nt/INSTALL all
> > there is?
> >
> > All I'm doing is cd:ing into the nt directory, running configure and
> > then make bootstrap.
> 
> Never mind, I got the build to work.
> 
> It turns out I only got the above error the second time I ran "make
> bootstrap", presumably because some environment variable lingers from
> the first invocation and messes things up.

That compilation failed because -I../src is missing from the command
line:

> gcc -I. -c -gdwarf-2 -g3 -mtune=pentium4 -O2 -Demacs=1 -DHAVE_CONFIG_H -I../lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5000000 -o oo-spd/i386/make-docfile.o make-docfile.c

I have no idea why that happened; lib-src/makefile definitely includes
"-I../src" through the LOCAL_FLAGS variable.  Could it be that you
somehow set LOCAL_FLAGS in the environment?

In general, the compilation should never leave anything in the
environment, so I don't think your explanation is quite right.

Also, there's no need to "make bootstrap" twice; once you ran
bootstrap once, the subsequent builds should be doe with just "make"
in the nt/ subdirectory.

> To get the build to work I had to use a fresh shell and explicitly set
> SHELL=cmd.exe when invoking make

Do you have a sh.exe somewhere on PATH?  Or do you have SHELL in your
environment set to something other than cmd.exe?  Otherwise, there
should be no need to set SHELL.

Not that all this matters, now that you've succeeded to build.

The only thing that does matter is that you've compiled with
optimizations (which is the default), which will make it hard to
examine the code under the debugger.  So I suggest to reconfigure and
rebuild without optimizations, like this:

  cd nt
  configure --no-opt --enable-checking
  make

You will then have an unoptimized emacs.exe in src/oo/i386/emacs.exe.
If you want it in bin/, say "make install" after the last "make".  The
old optimized build will still be available in src/oo-spd/i386.

Sorry, I should have mentioned this to begin with.

And thanks for working on this.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 16 December 2011 09:13, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 15 Dec 2011 22:47:17 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> Cc: Jason Rumney <jasonr@gnu.org>, 10299@debbugs.gnu.org, handa@m17n.org
>>
>> > gcc -I. -c -gdwarf-2 -g3  -mtune=pentium4 -O2      -Demacs=1 -DHAVE_CONFIG_H -I.
>> > ./lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5
>> > 000000 -o oo-spd/i386/make-docfile.o make-docfile.c
>> > make-docfile.c:36:20: config.h: No such file or directory
>> > make-docfile.c:79: error: syntax error before "NO_RETURN"
>> > make-docfile.c:79: warning: data definition has no type or storage class
>> > make[8]: *** [oo-spd/i386/make-docfile.o] Error 1
>> > make[8]: Leaving directory `D:/Dev_projects/emacs/trunk/lib-src'
>> > make[7]: *** [bootstrap-gmake] Error 2
>> > make[7]: Leaving directory `D:/Dev_projects/emacs/trunk/nt'
>> >
>> > I'm not really sure what to make of this, config.h is in trunk/src,
>> > but gcc doesn't seem to be looking there from what I can see. Are
>> > there more detailed build instructions available or is nt/INSTALL all
>> > there is?
>> >
>> > All I'm doing is cd:ing into the nt directory, running configure and
>> > then make bootstrap.
>>
>> Never mind, I got the build to work.
>>
>> It turns out I only got the above error the second time I ran "make
>> bootstrap", presumably because some environment variable lingers from
>> the first invocation and messes things up.
>
> That compilation failed because -I../src is missing from the command
> line:
>
>> gcc -I. -c -gdwarf-2 -g3 -mtune=pentium4 -O2 -Demacs=1 -DHAVE_CONFIG_H -I../lib -I../nt/inc -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000 -DPURESIZE=5000000 -o oo-spd/i386/make-docfile.o make-docfile.c
>
> I have no idea why that happened; lib-src/makefile definitely includes
> "-I../src" through the LOCAL_FLAGS variable.  Could it be that you
> somehow set LOCAL_FLAGS in the environment?
>
> In general, the compilation should never leave anything in the
> environment, so I don't think your explanation is quite right.
>
> Also, there's no need to "make bootstrap" twice; once you ran
> bootstrap once, the subsequent builds should be doe with just "make"
> in the nt/ subdirectory.
>
>> To get the build to work I had to use a fresh shell and explicitly set
>> SHELL=cmd.exe when invoking make
>
> Do you have a sh.exe somewhere on PATH?  Or do you have SHELL in your
> environment set to something other than cmd.exe?  Otherwise, there
> should be no need to set SHELL.
>
> Not that all this matters, now that you've succeeded to build.
>
> The only thing that does matter is that you've compiled with
> optimizations (which is the default), which will make it hard to
> examine the code under the debugger.  So I suggest to reconfigure and
> rebuild without optimizations, like this:
>
>  cd nt
>  configure --no-opt --enable-checking
>  make
>
> You will then have an unoptimized emacs.exe in src/oo/i386/emacs.exe.
> If you want it in bin/, say "make install" after the last "make".  The
> old optimized build will still be available in src/oo-spd/i386.
>
> Sorry, I should have mentioned this to begin with.
>
> And thanks for working on this.

I think I have a MSYS sh.exe somewhere in my path which caused the
build to fail if I didn't explicitly set SHELL.

Anyway, I managed to do some debugging and your theory seems to be
correct. The WM_KEYDOWN is passed to TranslateMessage which posts a
WM_CHAR with a question mark, so Emacs never sees the correct Unicode
character.

Also, the documentation for RegisterClass at
http://msdn.microsoft.com/en-us/library/windows/desktop/ms633586%28v=vs.85%29.aspx
says:

"If you register the window class by using RegisterClassA, the
application tells the system that the windows of the created class
expect messages with text or character parameters to use the ANSI
character set; if you register it by using RegisterClassW, the
application requests that the system pass text parameters of messages
as Unicode."

So I'll try to change to using RegisterClassW which should fix this
issue but might break other stuff since more messages will deliver
Unicode text then. An alternative might be to compare the result of
ToUnicode and ToAscii inside WM_KEYDOWN and skip calling
TranslateMessage when it will lose the correct character.

I'll get back to you with the results.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Dani Moncayo]

> I think I have a MSYS sh.exe somewhere in my path which caused the
> build to fail if I didn't explicitly set SHELL.

Maybe if you try "mingw32-make" instead of "make", you will be able to
build Emacs successfully (see
http://lists.gnu.org/archive/html/emacs-devel/2011-12/msg00025.html)

HTH.

-- 
Dani Moncayo

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Fri, 16 Dec 2011 12:01:02 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: jasonr@gnu.org, 10299@debbugs.gnu.org, handa@m17n.org
> 
> So I'll try to change to using RegisterClassW which should fix this
> issue but might break other stuff since more messages will deliver
> Unicode text then.

I think using RegisterClassW is the right way.  But please pay
attention to this part of w32term.c:w32_read_socket:

              if (msg.msg.message == WM_UNICHAR)
                {
                  inev.code = msg.msg.wParam;
                }
              else if (msg.msg.wParam < 256)
                {
                  ...
                }
              else
                {
                  /* Windows shouldn't generate WM_CHAR events above 0xFF
                     in non-Unicode message handlers.  */
                  DebPrint (("Non-byte WM_CHAR: %d\n", msg.msg.wParam));
                  inev.kind = NO_EVENT;
                  break;
                }
              inev.kind = inev.code < 128 ? ASCII_KEYSTROKE_EVENT
                                          : MULTIBYTE_CHAR_KEYSTROKE_EVENT;

If Windows continues sending WM_CHAR to Emacs even if you call
RegisterClassW, you will probably need to modify the "else" clause to
use the characters with codepoints beyond 256, instead of throwing
them away.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Fri, 16 Dec 2011 12:14:50 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 10299@debbugs.gnu.org
> 
> > I think I have a MSYS sh.exe somewhere in my path which caused the
> > build to fail if I didn't explicitly set SHELL.
> 
> Maybe if you try "mingw32-make" instead of "make", you will be able to
> build Emacs successfully (see
> http://lists.gnu.org/archive/html/emacs-devel/2011-12/msg00025.html)

If his make.exe reacts correctly to SHELL=cmd.exe, it is most probably
a MinGW Make, just renamed to make.exe.  (FWIW, my MinGW compiled Make
is also called make.exe.)

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 16 December 2011 12:26, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Fri, 16 Dec 2011 12:14:50 +0100
>> From: Dani Moncayo <dmoncayo@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, 10299@debbugs.gnu.org
>>
>> > I think I have a MSYS sh.exe somewhere in my path which caused the
>> > build to fail if I didn't explicitly set SHELL.
>>
>> Maybe if you try "mingw32-make" instead of "make", you will be able to
>> build Emacs successfully (see
>> http://lists.gnu.org/archive/html/emacs-devel/2011-12/msg00025.html)
>
> If his make.exe reacts correctly to SHELL=cmd.exe, it is most probably
> a MinGW Make, just renamed to make.exe.  (FWIW, my MinGW compiled Make
> is also called make.exe.)

Ok, I tried switching to RegisterClassW, but that didn't really help,
after calling TranslateMessage I still got WM_CHAR with a question
mark. So I tried switching to CreateWindowW and GetMessageW as well,
and that did the trick. Here's my changes:

D:\Dev_projects\emacs\trunk\nt>bzr diff
=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2011-12-17 11:39:10 +0000
@@ -1785,7 +1785,7 @@
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
+  WNDCLASSW wc;

   wc.style = CS_HREDRAW | CS_VREDRAW;
   wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
@@ -1796,9 +1796,9 @@
   wc.hCursor = w32_load_cursor (IDC_ARROW);
   wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
   wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
+  wc.lpszClassName = L"Emacs";

-  return (RegisterClass (&wc));
+  return (RegisterClassW (&wc));
 }

 static HWND
@@ -1853,7 +1853,7 @@
     }

   FRAME_W32_WINDOW (f) = hwnd
-    = CreateWindow (EMACS_CLASS,
+    = CreateWindowW (L"Emacs",
                    f->namebuf,
                    f->output_data.w32->dwStyle | WS_CLIPCHILDREN,
                    EQ (left, Qunbound) ? CW_USEDEFAULT : XINT (left),
@@ -2248,7 +2248,7 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (GetMessageW (&msg, NULL, 0, 0))
     {
       if (msg.hwnd == NULL)
        {
@@ -2507,7 +2507,7 @@
      message that has no particular effect. */
   {
     int c = wParam;
-    if (isalpha (c) && wmsg.dwModifiers == ctrl_modifier)
+    if (iswalpha (c) && wmsg.dwModifiers == ctrl_modifier)
       c = make_ctrl_char (c) & 0377;
     if (c == quit_char
        || (wmsg.dwModifiers == 0 &&


I had to switch to iswalpha because isalpha segfaulted when sent
values larger than 255. wParam isn't really the Unicode code point
here either, it's encoded as UTF-16. With this change, Emacs no longer
prints a question mark when I press the special keys, it doesn't print
anything at all.

I figured this is beacause wParam isn't a valid Unicode codepoint
here, like it is when you get a WM_UNICHAR, so I tried a quick hack
converting from UTF-16 and re-posting as a WM_UNICHAR even. I changed
the handling of WM_CHAR in w32_wnd_proc:

    case WM_CHAR:
      if (wParam > 255 )
        {
          /* Hacky conversion from UTF-16 to UCS-4.
             Doesn't handle surrogate pairs. */
          unsigned short lo = wParam & 0x0000FFFF;
          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
          wParam  = hi | lo;

          W32Msg wmsg;
          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
          signal_user_input ();
          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);

        }
      else
        post_character_message (hwnd, msg, wParam, lParam,
                                w32_get_key_modifiers (wParam, lParam));

This fixed it! I can now enter the characters with a custom keyboard
layout so I'm happy. There's loads of bugs still (e.g. the frame title
is "e" instead of "Emacs@host" because I just pass the frame's namebuf
to CreateWindowW without converting it to a wide string), but at least
the main issue is fixed.

I guess these changes need to only happen when running on NT for Emacs
to still work on Windows 95 or does Emacs use MSLU? Maybe they should
depend on some lisp variable being set?

Here's my changes in full so far:

=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2011-12-17 12:47:44 +0000
@@ -1785,7 +1785,7 @@
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
+  WNDCLASSW wc;

   wc.style = CS_HREDRAW | CS_VREDRAW;
   wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
@@ -1796,9 +1796,9 @@
   wc.hCursor = w32_load_cursor (IDC_ARROW);
   wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
   wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
+  wc.lpszClassName = L"Emacs";

-  return (RegisterClass (&wc));
+  return (RegisterClassW (&wc));
 }

 static HWND
@@ -1853,7 +1853,7 @@
     }

   FRAME_W32_WINDOW (f) = hwnd
-    = CreateWindow (EMACS_CLASS,
+    = CreateWindowW (L"Emacs",
                    f->namebuf,
                    f->output_data.w32->dwStyle | WS_CLIPCHILDREN,
                    EQ (left, Qunbound) ? CW_USEDEFAULT : XINT (left),
@@ -2248,7 +2248,7 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (GetMessageW (&msg, NULL, 0, 0))
     {
       if (msg.hwnd == NULL)
        {
@@ -2507,7 +2507,7 @@
      message that has no particular effect. */
   {
     int c = wParam;
-    if (isalpha (c) && wmsg.dwModifiers == ctrl_modifier)
+    if (iswalpha (c) && wmsg.dwModifiers == ctrl_modifier)
       c = make_ctrl_char (c) & 0377;
     if (c == quit_char
        || (wmsg.dwModifiers == 0 &&
@@ -2915,8 +2915,23 @@

     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-                             w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          /* Hacky conversion from UTF-16 to UCS-4.
+             Doesn't handle surrogate pairs. */
+          unsigned short lo = wParam & 0x0000FFFF;
+          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
+          wParam  = hi | lo;
+
+          W32Msg wmsg;
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;

     case WM_UNICHAR:

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Sat, 17 Dec 2011 13:52:29 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: 10299@debbugs.gnu.org
> 
> I had to switch to iswalpha because isalpha segfaulted when sent
> values larger than 255. wParam isn't really the Unicode code point
> here either, it's encoded as UTF-16. With this change, Emacs no longer
> prints a question mark when I press the special keys, it doesn't print
> anything at all.
> 
> I figured this is beacause wParam isn't a valid Unicode codepoint
> here, like it is when you get a WM_UNICHAR, so I tried a quick hack
> converting from UTF-16 and re-posting as a WM_UNICHAR even. I changed
> the handling of WM_CHAR in w32_wnd_proc:
> 
>     case WM_CHAR:
>       if (wParam > 255 )
>         {
>           /* Hacky conversion from UTF-16 to UCS-4.
>              Doesn't handle surrogate pairs. */
>           unsigned short lo = wParam & 0x0000FFFF;
>           unsigned short hi = (wParam & 0xFFFF0000) >> 8;
>           wParam  = hi | lo;
> 
>           W32Msg wmsg;
>           wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
>           signal_user_input ();
>           my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
> 
>         }
>       else
>         post_character_message (hwnd, msg, wParam, lParam,
>                                 w32_get_key_modifiers (wParam, lParam));
> 
> This fixed it! I can now enter the characters with a custom keyboard
> layout so I'm happy.

That's good news.  However, I'm puzzled: are you saying that the code
points passed by Windows to Emacs for the characters generated by MKLC
are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
points are they?

> There's loads of bugs still (e.g. the frame title
> is "e" instead of "Emacs@host" because I just pass the frame's namebuf
> to CreateWindowW without converting it to a wide string), but at least
> the main issue is fixed.

If at all possible, I'd like to keep the use of wide APIs to the
absolute minimum, because:

> I guess these changes need to only happen when running on NT for Emacs
> to still work on Windows 95 or does Emacs use MSLU?

We do use MSLU on Windows 9X (because the display routines need
that).  However, given that none of the developers has easy access to
a 9X machine, it is very easy to break Emacs on 9X by excess usage of
wide APIs.  In fact, all versions of Emacs 23 didn't work on Windows
9X and only recently we were lucky enough to find a 9X user who helped
us debug the problems and fix them.

So if possible, could you perhaps try finding out why you didn't get
UTF-16 codepoints without CreateWindowW and GetMessageW?  Maybe it's
TranslateMessage that causes that, and the original message Emacs gets
is okay?

> Maybe they should depend on some lisp variable being set?

Probably just on the value of os_subtype.

> Here's my changes in full so far:

Thanks.

This will probably need to wait until after v24.1 is released.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Sun, 18 Dec 2011 18:31:55 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> 
> > That's good news.  However, I'm puzzled: are you saying that the code
> > points passed by Windows to Emacs for the characters generated by MKLC
> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
> > points are they?
> 
> No, none of the characters I needed are outside the BMP.
> 
> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
> 0x220018.

??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
single 16-bit value that is numerically identical to the codepoint.
That is, you should have gotten 0x2218.  What am I missing?

> I experimented a bit, and CreateWindowW isn't needed after all. As
> long as I use RegisterClassW and GetMessageW, things work. I'm unsure
> if it's TranslateMessage that translates the key press to a question
> mark or if it's GetMessage that does it on receiving the message.

Question marks are a sign that Windows tried to convert the character
to its ANSI equivalent, and failed.  I.e., it means that Windows
thought the program asked for ANSI encoded characters.  So it's
probably TranslateMessage that did it.

> I'll try to get frame titles working again as well, then I can
> probably switch on os_subtype in two or three places and Windows 95
> won't be affected at all. Do you think that is a good plan?

Yes, thanks.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>
>> > That's good news.  However, I'm puzzled: are you saying that the code
>> > points passed by Windows to Emacs for the characters generated by MKLC
>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>> > points are they?
>>
>> No, none of the characters I needed are outside the BMP.
>>
>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>> 0x220018.
>
> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
> single 16-bit value that is numerically identical to the codepoint.
> That is, you should have gotten 0x2218.  What am I missing?

I just assumed Windows encoded the codepoints into a DWORD in some
funky way, but looking more closely at the documentattion it appears
like wParam should just be the codepoint. Even more strangely, some
places claim that if a keyboard produces a character outside the BMP,
you get two WM_CHAR events.

From what I can tell, Emacs itself never alters wParam, but I guess
Windows might do some funky multibyte encoding since Emacs isn't
completely Unicode?

>> I experimented a bit, and CreateWindowW isn't needed after all. As
>> long as I use RegisterClassW and GetMessageW, things work. I'm unsure
>> if it's TranslateMessage that translates the key press to a question
>> mark or if it's GetMessage that does it on receiving the message.
>
> Question marks are a sign that Windows tried to convert the character
> to its ANSI equivalent, and failed.  I.e., it means that Windows
> thought the program asked for ANSI encoded characters.  So it's
> probably TranslateMessage that did it.
>
>> I'll try to get frame titles working again as well, then I can
>> probably switch on os_subtype in two or three places and Windows 95
>> won't be affected at all. Do you think that is a good plan?
>
> Yes, thanks.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Lennart Borgman]

On Mon, Dec 19, 2011 at 11:44, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
> On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>
>>> > That's good news.  However, I'm puzzled: are you saying that the code
>>> > points passed by Windows to Emacs for the characters generated by MKLC
>>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>>> > points are they?
>>>
>>> No, none of the characters I needed are outside the BMP.
>>>
>>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>>> 0x220018.
>>
>> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
>> single 16-bit value that is numerically identical to the codepoint.
>> That is, you should have gotten 0x2218.  What am I missing?
>
> I just assumed Windows encoded the codepoints into a DWORD in some
> funky way, but looking more closely at the documentattion it appears
> like wParam should just be the codepoint. Even more strangely, some
> places claim that if a keyboard produces a character outside the BMP,
> you get two WM_CHAR events.
>
> From what I can tell, Emacs itself never alters wParam, but I guess
> Windows might do some funky multibyte encoding since Emacs isn't
> completely Unicode?

Maybe Emacs on windows still is using the ANSI version of DefWindowProc? See

http://blogs.msdn.com/b/michkap/archive/2007/03/25/1945659.aspx

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 19 December 2011 11:59, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> On Mon, Dec 19, 2011 at 11:44, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
>> On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>>
>>>> > That's good news.  However, I'm puzzled: are you saying that the code
>>>> > points passed by Windows to Emacs for the characters generated by MKLC
>>>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>>>> > points are they?
>>>>
>>>> No, none of the characters I needed are outside the BMP.
>>>>
>>>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>>>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>>>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>>>> 0x220018.
>>>
>>> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
>>> single 16-bit value that is numerically identical to the codepoint.
>>> That is, you should have gotten 0x2218.  What am I missing?
>>
>> I just assumed Windows encoded the codepoints into a DWORD in some
>> funky way, but looking more closely at the documentattion it appears
>> like wParam should just be the codepoint. Even more strangely, some
>> places claim that if a keyboard produces a character outside the BMP,
>> you get two WM_CHAR events.
>>
>> From what I can tell, Emacs itself never alters wParam, but I guess
>> Windows might do some funky multibyte encoding since Emacs isn't
>> completely Unicode?
>
> Maybe Emacs on windows still is using the ANSI version of DefWindowProc? See
>
> http://blogs.msdn.com/b/michkap/archive/2007/03/25/1945659.aspx

I looked at that page as well, but it says that the ANSI DefWindowProc
is supposed to post one or two ANSI characters, and it definitely
isn't doing that. I get teh correct Unciode character, just spread
over the low and high word of the wParam dword.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Lennart Borgman]

On Mon, Dec 19, 2011 at 12:04, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
> On 19 December 2011 11:59, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>> On Mon, Dec 19, 2011 at 11:44, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
>>> On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>>>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>>>
>>>>> > That's good news.  However, I'm puzzled: are you saying that the code
>>>>> > points passed by Windows to Emacs for the characters generated by MKLC
>>>>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>>>>> > points are they?
>>>>>
>>>>> No, none of the characters I needed are outside the BMP.
>>>>>
>>>>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>>>>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>>>>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>>>>> 0x220018.
>>>>
>>>> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
>>>> single 16-bit value that is numerically identical to the codepoint.
>>>> That is, you should have gotten 0x2218.  What am I missing?
>>>
>>> I just assumed Windows encoded the codepoints into a DWORD in some
>>> funky way, but looking more closely at the documentattion it appears
>>> like wParam should just be the codepoint. Even more strangely, some
>>> places claim that if a keyboard produces a character outside the BMP,
>>> you get two WM_CHAR events.
>>>
>>> From what I can tell, Emacs itself never alters wParam, but I guess
>>> Windows might do some funky multibyte encoding since Emacs isn't
>>> completely Unicode?
>>
>> Maybe Emacs on windows still is using the ANSI version of DefWindowProc? See
>>
>> http://blogs.msdn.com/b/michkap/archive/2007/03/25/1945659.aspx
>
> I looked at that page as well, but it says that the ANSI DefWindowProc
> is supposed to post one or two ANSI characters, and it definitely
> isn't doing that. I get teh correct Unciode character, just spread
> over the low and high word of the wParam dword.

Strange. What is the reason Emacs is still using the ANSI version?
Maybe a mix of ANSI and UNICODE versions gives strange results?

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

I'm not the right person to ask, but presumably to remain compatible with Windows 95. Not all wide/Unicode apis are available there. 


19 dec 2011 kl. 12:17 skrev Lennart Borgman <lennart.borgman@gmail.com>:

> On Mon, Dec 19, 2011 at 12:04, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
>> On 19 December 2011 11:59, Lennart Borgman <lennart.borgman@gmail.com> wrote:
>>> On Mon, Dec 19, 2011 at 11:44, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
>>>> On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>>>>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>>>> 
>>>>>>> That's good news.  However, I'm puzzled: are you saying that the code
>>>>>>> points passed by Windows to Emacs for the characters generated by MKLC
>>>>>>> are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>>>>>>> points are they?
>>>>>> 
>>>>>> No, none of the characters I needed are outside the BMP.
>>>>>> 
>>>>>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>>>>>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>>>>>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>>>>>> 0x220018.
>>>>> 
>>>>> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
>>>>> single 16-bit value that is numerically identical to the codepoint.
>>>>> That is, you should have gotten 0x2218.  What am I missing?
>>>> 
>>>> I just assumed Windows encoded the codepoints into a DWORD in some
>>>> funky way, but looking more closely at the documentattion it appears
>>>> like wParam should just be the codepoint. Even more strangely, some
>>>> places claim that if a keyboard produces a character outside the BMP,
>>>> you get two WM_CHAR events.
>>>> 
>>>> From what I can tell, Emacs itself never alters wParam, but I guess
>>>> Windows might do some funky multibyte encoding since Emacs isn't
>>>> completely Unicode?
>>> 
>>> Maybe Emacs on windows still is using the ANSI version of DefWindowProc? See
>>> 
>>> http://blogs.msdn.com/b/michkap/archive/2007/03/25/1945659.aspx
>> 
>> I looked at that page as well, but it says that the ANSI DefWindowProc
>> is supposed to post one or two ANSI characters, and it definitely
>> isn't doing that. I get teh correct Unciode character, just spread
>> over the low and high word of the wParam dword.
> 
> Strange. What is the reason Emacs is still using the ANSI version?
> Maybe a mix of ANSI and UNICODE versions gives strange results?

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Jason Rumney]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>

>> So if I press something which gives U+2218 RING OPERATOR, I get a
>> WM_CHAR event with a wParam of 2228248 or 0x220018.
>
> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
> single 16-bit value that is numerically identical to the codepoint.
> That is, you should have gotten 0x2218.  What am I missing?

It looks like there is an endianness mismatch somewhere.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>
>> > That's good news.  However, I'm puzzled: are you saying that the code
>> > points passed by Windows to Emacs for the characters generated by MKLC
>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>> > points are they?
>>
>> No, none of the characters I needed are outside the BMP.
>>
>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>> 0x220018.
>
> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
> single 16-bit value that is numerically identical to the codepoint.
> That is, you should have gotten 0x2218.  What am I missing?
>
>> I experimented a bit, and CreateWindowW isn't needed after all. As
>> long as I use RegisterClassW and GetMessageW, things work. I'm unsure
>> if it's TranslateMessage that translates the key press to a question
>> mark or if it's GetMessage that does it on receiving the message.
>
> Question marks are a sign that Windows tried to convert the character
> to its ANSI equivalent, and failed.  I.e., it means that Windows
> thought the program asked for ANSI encoded characters.  So it's
> probably TranslateMessage that did it.
>
>> I'll try to get frame titles working again as well, then I can
>> probably switch on os_subtype in two or three places and Windows 95
>> won't be affected at all. Do you think that is a good plan?
>
> Yes, thanks.

I've fixed the issues with the frame titles, and everything appears to
work, there are a number of issues I find very confusing however.

Here's the state of my changes as of now:

=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2011-12-20 20:46:40 +0000
@@ -1697,10 +1697,10 @@
   if (FRAME_W32_WINDOW (f))
     {
       if (STRING_MULTIBYTE (name))
-       name = ENCODE_SYSTEM (name);
+        name = ENCODE_SYSTEM (name);

       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1746,7 +1746,7 @@
        name = ENCODE_SYSTEM (name);

       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1785,7 +1785,7 @@
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
+  WNDCLASSW wc;

   wc.style = CS_HREDRAW | CS_VREDRAW;
   wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
@@ -1796,9 +1796,9 @@
   wc.hCursor = w32_load_cursor (IDC_ARROW);
   wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
   wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
+  wc.lpszClassName = L"Emacs";

-  return (RegisterClass (&wc));
+  return (RegisterClassW (&wc));
 }

 static HWND
@@ -2248,7 +2248,7 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (GetMessageW (&msg, NULL, 0, 0))
     {
       if (msg.hwnd == NULL)
        {
@@ -2915,8 +2915,21 @@

     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-                             w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          unsigned short lo = wParam & 0x0000FFFF;
+          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
+          wParam  = hi | lo;
+
+          W32Msg wmsg;
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;

     case WM_UNICHAR:

I should probably also only do this on NT (to avoid breaking stuff on
Windows 95), but that should be easy to fix.

There are a couple of very weird things going on however:

1. Why is wParam encoded in a weird format spread over the lo and hi
word of the wParam DWORD?

2. Why does sending 8-bit strings to SetWindowTextW work, but sending
8-bit strings to SetWindowTextA for a window with a "Unicode" window
class only use the first character?

My guess would be that the correct solution for 2 is to always encode
frame captions in utf-16le before sending them to SetWindowTextW,
however I'm not sure what the best way to do this is.

I figure I should use something like this:

Lisp_Object encoding = intern_c_string ("utf-16le-dos");
name = code_convert_string_norecord (name, encoding, 1);
SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));

Sadly that didn't work (I still get single char frame captions), and I
never managed to get gdb on Windows to print Lisp objects correctly,
so I had a hard time understanding why it didn't work. Looking at the
data that actually gets sent to SetWindowText might make things
clearer.

Anyway, the current patch works fine as far as I can tell, but it's a
bit disconcerting to not know *why* things work the way they do.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

Ok, here's the final version of the patch I'm currently using. It now
only switches to the new behavior when running on NT which should
maintain compatibility with Windows 95. This does make the code
somewhat uglier, but when Emacs stops supporting Windows 95 maybe the
source can be cleaned up and switched to always use the full Unicode
API on all Windows platforms.

I've been using an Emacs with this patch for a couple of weeks now and
haven't discovered any problems so far. I don't have a Windows 95
system to test on though.

=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2012-01-08 15:30:12 +0000
@@ -1697,10 +1697,13 @@
   if (FRAME_W32_WINDOW (f))
     {
       if (STRING_MULTIBYTE (name))
-       name = ENCODE_SYSTEM (name);
-
+        name = ENCODE_SYSTEM (name);
+
       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      if (os_subtype == OS_NT)
+        SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
+      else
+        SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1746,7 +1749,10 @@
        name = ENCODE_SYSTEM (name);

       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      if (os_subtype == OS_NT)
+        SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
+      else
+        SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1785,20 +1791,39 @@
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
-
-  wc.style = CS_HREDRAW | CS_VREDRAW;
-  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
-  wc.cbClsExtra = 0;
-  wc.cbWndExtra = WND_EXTRA_BYTES;
-  wc.hInstance = hinst;
-  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
-  wc.hCursor = w32_load_cursor (IDC_ARROW);
-  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
-  wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
-
-  return (RegisterClass (&wc));
+  WNDCLASSW uwc;
+  WNDCLASS  wc;
+
+  if (os_subtype == OS_NT)
+    {
+      uwc.style = CS_HREDRAW | CS_VREDRAW;
+      uwc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
+      uwc.cbClsExtra = 0;
+      uwc.cbWndExtra = WND_EXTRA_BYTES;
+      uwc.hInstance = hinst;
+      uwc.hIcon = LoadIcon (hinst, EMACS_CLASS);
+      uwc.hCursor = w32_load_cursor (IDC_ARROW);
+      uwc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
+      uwc.lpszMenuName = NULL;
+      uwc.lpszClassName = L"Emacs";
+
+      return (RegisterClassW (&uwc));
+    }
+  else
+    {
+      wc.style = CS_HREDRAW | CS_VREDRAW;
+      wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
+      wc.cbClsExtra = 0;
+      wc.cbWndExtra = WND_EXTRA_BYTES;
+      wc.hInstance = hinst;
+      wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
+      wc.hCursor = w32_load_cursor (IDC_ARROW);
+      wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
+      wc.lpszMenuName = NULL;
+      wc.lpszClassName = EMACS_CLASS;
+
+      return (RegisterClass (&wc));
+    }
 }

 static HWND
@@ -2248,8 +2273,16 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (1)
     {
+      if (os_subtype == OS_NT)
+        result = GetMessageW (&msg, NULL, 0, 0);
+      else
+        result = GetMessage (&msg, NULL, 0, 0);
+
+      if (!result)
+        break;
+
       if (msg.hwnd == NULL)
        {
          switch (msg.message)
@@ -2915,8 +2948,21 @@

     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-                             w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          unsigned short lo = wParam & 0x0000FFFF;
+          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
+          wParam  = hi | lo;
+
+          W32Msg wmsg;
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;

     case WM_UNICHAR:



On 20 December 2011 22:16, Joakim Hårsman <joakim.harsman@gmail.com> wrote:
> On 18 December 2011 19:13, Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Sun, 18 Dec 2011 18:31:55 +0100
>>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>>>
>>> > That's good news.  However, I'm puzzled: are you saying that the code
>>> > points passed by Windows to Emacs for the characters generated by MKLC
>>> > are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
>>> > points are they?
>>>
>>> No, none of the characters I needed are outside the BMP.
>>>
>>> WM_CHAR encodes the codepoint in UTF-16 inside wParam, while
>>> WM_UNICHAR uses UTF-32. So if I press something which gives U+2218
>>> RING OPERATOR, I get a WM_CHAR event with a wParam of 2228248 or
>>> 0x220018.
>>
>> ??? UTF-16 encodes the characters in the BMP as themselves, i.e. a
>> single 16-bit value that is numerically identical to the codepoint.
>> That is, you should have gotten 0x2218.  What am I missing?
>>
>>> I experimented a bit, and CreateWindowW isn't needed after all. As
>>> long as I use RegisterClassW and GetMessageW, things work. I'm unsure
>>> if it's TranslateMessage that translates the key press to a question
>>> mark or if it's GetMessage that does it on receiving the message.
>>
>> Question marks are a sign that Windows tried to convert the character
>> to its ANSI equivalent, and failed.  I.e., it means that Windows
>> thought the program asked for ANSI encoded characters.  So it's
>> probably TranslateMessage that did it.
>>
>>> I'll try to get frame titles working again as well, then I can
>>> probably switch on os_subtype in two or three places and Windows 95
>>> won't be affected at all. Do you think that is a good plan?
>>
>> Yes, thanks.
>
> I've fixed the issues with the frame titles, and everything appears to
> work, there are a number of issues I find very confusing however.
>
> Here's the state of my changes as of now:
>
> === modified file 'src/w32fns.c'
> --- src/w32fns.c        2011-12-04 08:02:42 +0000
> +++ src/w32fns.c        2011-12-20 20:46:40 +0000
> @@ -1697,10 +1697,10 @@
>   if (FRAME_W32_WINDOW (f))
>     {
>       if (STRING_MULTIBYTE (name))
> -       name = ENCODE_SYSTEM (name);
> +        name = ENCODE_SYSTEM (name);
>
>       BLOCK_INPUT;
> -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
> +      SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
>       UNBLOCK_INPUT;
>     }
>  }
> @@ -1746,7 +1746,7 @@
>        name = ENCODE_SYSTEM (name);
>
>       BLOCK_INPUT;
> -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
> +      SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
>       UNBLOCK_INPUT;
>     }
>  }
> @@ -1785,7 +1785,7 @@
>  static BOOL
>  w32_init_class (HINSTANCE hinst)
>  {
> -  WNDCLASS wc;
> +  WNDCLASSW wc;
>
>   wc.style = CS_HREDRAW | CS_VREDRAW;
>   wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
> @@ -1796,9 +1796,9 @@
>   wc.hCursor = w32_load_cursor (IDC_ARROW);
>   wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
>   wc.lpszMenuName = NULL;
> -  wc.lpszClassName = EMACS_CLASS;
> +  wc.lpszClassName = L"Emacs";
>
> -  return (RegisterClass (&wc));
> +  return (RegisterClassW (&wc));
>  }
>
>  static HWND
> @@ -2248,7 +2248,7 @@
>
>   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);
>
> -  while (GetMessage (&msg, NULL, 0, 0))
> +  while (GetMessageW (&msg, NULL, 0, 0))
>     {
>       if (msg.hwnd == NULL)
>        {
> @@ -2915,8 +2915,21 @@
>
>     case WM_SYSCHAR:
>     case WM_CHAR:
> -      post_character_message (hwnd, msg, wParam, lParam,
> -                             w32_get_key_modifiers (wParam, lParam));
> +      if (wParam > 255 )
> +        {
> +          unsigned short lo = wParam & 0x0000FFFF;
> +          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
> +          wParam  = hi | lo;
> +
> +          W32Msg wmsg;
> +          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
> +          signal_user_input ();
> +          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
> +
> +        }
> +      else
> +        post_character_message (hwnd, msg, wParam, lParam,
> +                                w32_get_key_modifiers (wParam, lParam));
>       break;
>
>     case WM_UNICHAR:
>
> I should probably also only do this on NT (to avoid breaking stuff on
> Windows 95), but that should be easy to fix.
>
> There are a couple of very weird things going on however:
>
> 1. Why is wParam encoded in a weird format spread over the lo and hi
> word of the wParam DWORD?
>
> 2. Why does sending 8-bit strings to SetWindowTextW work, but sending
> 8-bit strings to SetWindowTextA for a window with a "Unicode" window
> class only use the first character?
>
> My guess would be that the correct solution for 2 is to always encode
> frame captions in utf-16le before sending them to SetWindowTextW,
> however I'm not sure what the best way to do this is.
>
> I figure I should use something like this:
>
> Lisp_Object encoding = intern_c_string ("utf-16le-dos");
> name = code_convert_string_norecord (name, encoding, 1);
> SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
>
> Sadly that didn't work (I still get single char frame captions), and I
> never managed to get gdb on Windows to print Lisp objects correctly,
> so I had a hard time understanding why it didn't work. Looking at the
> data that actually gets sent to SetWindowText might make things
> clearer.
>
> Anyway, the current patch works fine as far as I can tell, but it's a
> bit disconcerting to not know *why* things work the way they do.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Stefan Monnier]

Note: I know very little about w32.

> -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
> +      if (os_subtype == OS_NT)
> +        SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
> +      else
> +        SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));

Can we move the test elsewhere by defining (or #defining) our own
SetWindowText which uses either of the two?

> @@ -1785,20 +1791,39 @@
>  static BOOL
>  w32_init_class (HINSTANCE hinst)
>  {
> -  WNDCLASS wc;
> -
> -  wc.style = CS_HREDRAW | CS_VREDRAW;
> -  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
> -  wc.cbClsExtra = 0;
> -  wc.cbWndExtra = WND_EXTRA_BYTES;
> -  wc.hInstance = hinst;
> -  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
> -  wc.hCursor = w32_load_cursor (IDC_ARROW);
> -  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
> -  wc.lpszMenuName = NULL;
> -  wc.lpszClassName = EMACS_CLASS;
> -
> -  return (RegisterClass (&wc));
> +  WNDCLASSW uwc;
> +  WNDCLASS  wc;
> +
> +  if (os_subtype == OS_NT)
> +    {
> +      uwc.style = CS_HREDRAW | CS_VREDRAW;
> +      uwc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
> +      uwc.cbClsExtra = 0;
> +      uwc.cbWndExtra = WND_EXTRA_BYTES;
> +      uwc.hInstance = hinst;
> +      uwc.hIcon = LoadIcon (hinst, EMACS_CLASS);
> +      uwc.hCursor = w32_load_cursor (IDC_ARROW);
> +      uwc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
> +      uwc.lpszMenuName = NULL;
> +      uwc.lpszClassName = L"Emacs";
> +
> +      return (RegisterClassW (&uwc));
> +    }
> +  else
> +    {
> +      wc.style = CS_HREDRAW | CS_VREDRAW;
> +      wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
> +      wc.cbClsExtra = 0;
> +      wc.cbWndExtra = WND_EXTRA_BYTES;
> +      wc.hInstance = hinst;
> +      wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
> +      wc.hCursor = w32_load_cursor (IDC_ARROW);
> +      wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
> +      wc.lpszMenuName = NULL;
> +      wc.lpszClassName = EMACS_CLASS;
> +
> +      return (RegisterClass (&wc));
> +    }
>  }

The two var declarations (WNDCLASSW uwc and WNDCLASS wc) should be moved
within their respective branch.  And it'd be better if we could share
some code between the two branches, e.g. using a macro.


        Stefan

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

I've amended the patch to take your comments into account.

Note that SetWindowText is already a macro that expands to
SetWindowTextA or SetWindowTextW depending on whether UNICODE is
defined. However, to retain compatibility with Windows 95, Emacs
doesn't define UNICODE and instead uses Unicode APIs directly where it
needs them. Still, the duplicated code was annoying, so I defined a
simple wrapper function which also gives a good place to document why
the code looks the way it does.

I also put the initialization of the window class structure into a
macro in w32.h, I wasn't entirely sure where to put it.

Here's the current version of the patch:


=== modified file 'src/w32.h'
--- src/w32.h   2011-10-28 09:54:02 +0000
+++ src/w32.h   2012-01-23 19:10:04 +0000
@@ -159,5 +159,18 @@
                                   const void* buf, size_t sz);
 #endif /* HAVE_GNUTLS */

+#define INIT_WINDOW_CLASS(wc)                   \
+    wc.style = CS_HREDRAW | CS_VREDRAW;         \
+    wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;    \
+    wc.cbClsExtra = 0;                          \
+    wc.cbWndExtra = WND_EXTRA_BYTES;            \
+    wc.hInstance = hinst;                       \
+    wc.hIcon = LoadIcon (hinst, EMACS_CLASS);   \
+    wc.hCursor = w32_load_cursor (IDC_ARROW);   \
+    wc.hbrBackground = NULL;                    \
+    wc.lpszMenuName = NULL;                     \
+
+
+
 #endif /* EMACS_W32_H */


=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2012-01-23 19:10:04 +0000
@@ -1641,6 +1641,20 @@

 }

+/* Set text of w32 frame with handle HWND to TEXT.
+
+   We explicitly switch between the Unicode and ANSI version of
+   SetWindowText because Emacs isn't compiled with UNICODE defined to
+   retain compatibility with Windows 95. */
+
+void
+w32_set_frame_text(HWND hwnd, LPCSTR text)
+{
+  if (os_subtype == OS_NT)
+    SetWindowTextW (hwnd, (LPCWSTR)text);
+  else
+    SetWindowTextA (hwnd, text);
+}

 /* Change the name of frame F to NAME.  If NAME is nil, set F's name to
        w32_id_name.
@@ -1697,10 +1711,10 @@
   if (FRAME_W32_WINDOW (f))
     {
       if (STRING_MULTIBYTE (name))
-       name = ENCODE_SYSTEM (name);
-
+        name = ENCODE_SYSTEM (name);
+
       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1746,7 +1760,7 @@
        name = ENCODE_SYSTEM (name);

       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1785,20 +1799,23 @@
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
-
-  wc.style = CS_HREDRAW | CS_VREDRAW;
-  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
-  wc.cbClsExtra = 0;
-  wc.cbWndExtra = WND_EXTRA_BYTES;
-  wc.hInstance = hinst;
-  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
-  wc.hCursor = w32_load_cursor (IDC_ARROW);
-  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
-  wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
-
-  return (RegisterClass (&wc));
+
+  if (os_subtype == OS_NT)
+    {
+      WNDCLASSW  uwc;
+      INIT_WINDOW_CLASS(uwc);
+      uwc.lpszClassName = L"Emacs";
+
+      return (RegisterClassW (&uwc));
+    }
+  else
+    {
+      WNDCLASS  wc;
+      INIT_WINDOW_CLASS(wc);
+      wc.lpszClassName = EMACS_CLASS;
+
+      return (RegisterClass (&wc));
+    }
 }

 static HWND
@@ -2248,8 +2265,16 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (1)
     {
+      if (os_subtype == OS_NT)
+        result = GetMessageW (&msg, NULL, 0, 0);
+      else
+        result = GetMessage (&msg, NULL, 0, 0);
+
+      if (!result)
+        break;
+
       if (msg.hwnd == NULL)
        {
          switch (msg.message)
@@ -2915,8 +2940,21 @@

     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-                             w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          unsigned short lo = wParam & 0x0000FFFF;
+          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
+          wParam  = hi | lo;
+
+          W32Msg wmsg;
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;

     case WM_UNICHAR:



On 16 January 2012 15:03, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> Note: I know very little about w32.
>
>> -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>> +      if (os_subtype == OS_NT)
>> +        SetWindowTextW (FRAME_W32_WINDOW (f), SDATA (name));
>> +      else
>> +        SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>
> Can we move the test elsewhere by defining (or #defining) our own
> SetWindowText which uses either of the two?
>
>> @@ -1785,20 +1791,39 @@
>>  static BOOL
>>  w32_init_class (HINSTANCE hinst)
>>  {
>> -  WNDCLASS wc;
>> -
>> -  wc.style = CS_HREDRAW | CS_VREDRAW;
>> -  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
>> -  wc.cbClsExtra = 0;
>> -  wc.cbWndExtra = WND_EXTRA_BYTES;
>> -  wc.hInstance = hinst;
>> -  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
>> -  wc.hCursor = w32_load_cursor (IDC_ARROW);
>> -  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
>> -  wc.lpszMenuName = NULL;
>> -  wc.lpszClassName = EMACS_CLASS;
>> -
>> -  return (RegisterClass (&wc));
>> +  WNDCLASSW uwc;
>> +  WNDCLASS  wc;
>> +
>> +  if (os_subtype == OS_NT)
>> +    {
>> +      uwc.style = CS_HREDRAW | CS_VREDRAW;
>> +      uwc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
>> +      uwc.cbClsExtra = 0;
>> +      uwc.cbWndExtra = WND_EXTRA_BYTES;
>> +      uwc.hInstance = hinst;
>> +      uwc.hIcon = LoadIcon (hinst, EMACS_CLASS);
>> +      uwc.hCursor = w32_load_cursor (IDC_ARROW);
>> +      uwc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
>> +      uwc.lpszMenuName = NULL;
>> +      uwc.lpszClassName = L"Emacs";
>> +
>> +      return (RegisterClassW (&uwc));
>> +    }
>> +  else
>> +    {
>> +      wc.style = CS_HREDRAW | CS_VREDRAW;
>> +      wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
>> +      wc.cbClsExtra = 0;
>> +      wc.cbWndExtra = WND_EXTRA_BYTES;
>> +      wc.hInstance = hinst;
>> +      wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
>> +      wc.hCursor = w32_load_cursor (IDC_ARROW);
>> +      wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
>> +      wc.lpszMenuName = NULL;
>> +      wc.lpszClassName = EMACS_CLASS;
>> +
>> +      return (RegisterClass (&wc));
>> +    }
>>  }
>
> The two var declarations (WNDCLASSW uwc and WNDCLASS wc) should be moved
> within their respective branch.  And it'd be better if we could share
> some code between the two branches, e.g. using a macro.
>
>
>        Stefan

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Stefan Monnier]

> I've amended the patch to take your comments into account.

Thank you.

> --- src/w32.h   2011-10-28 09:54:02 +0000
> +++ src/w32.h   2012-01-23 19:10:04 +0000
> @@ -159,5 +159,18 @@
>                                    const void* buf, size_t sz);
>  #endif /* HAVE_GNUTLS */

> +#define INIT_WINDOW_CLASS(wc)                   \
> +    wc.style = CS_HREDRAW | CS_VREDRAW;         \
> +    wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;    \
> +    wc.cbClsExtra = 0;                          \
> +    wc.cbWndExtra = WND_EXTRA_BYTES;            \
> +    wc.hInstance = hinst;                       \
> +    wc.hIcon = LoadIcon (hinst, EMACS_CLASS);   \
> +    wc.hCursor = w32_load_cursor (IDC_ARROW);   \
> +    wc.hbrBackground = NULL;                    \
> +    wc.lpszMenuName = NULL;                     \
> +
> +
> +
>  #endif /* EMACS_W32_H */

Please don't put this macro there: it's not something that's exported to
other files, so it is better to keep it close to where it's used.

> +/* Set text of w32 frame with handle HWND to TEXT.
> +
> +   We explicitly switch between the Unicode and ANSI version of
> +   SetWindowText because Emacs isn't compiled with UNICODE defined to
> +   retain compatibility with Windows 95. */
> +
> +void
> +w32_set_frame_text(HWND hwnd, LPCSTR text)

This should be `static', I think, and with a space before the open-paren.

> +      if (os_subtype == OS_NT)
> +        result = GetMessageW (&msg, NULL, 0, 0);
> +      else
> +        result = GetMessage (&msg, NULL, 0, 0);

I'd write it as follows:

  result = (os_subtype == OS_NT ? GetMessageW : GetMessage) (&msg, NULL, 0, 0);

other than that, the code looks fine for me (I'll let Windows users
assess whether it actually does what it's meant to do ;-),


        Stefan

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +      if (os_subtype == OS_NT)
>> +        result = GetMessageW (&msg, NULL, 0, 0);
>> +      else
>> +        result = GetMessage (&msg, NULL, 0, 0);
>
> I'd write it as follows:
>
>   result = (os_subtype == OS_NT ? GetMessageW : GetMessage) (&msg, NULL, 0, 0);

That can fail if either of the functions is defined only as a
function-like macro.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Juanma Barranquero]

On Tue, Jan 24, 2012 at 10:40, Andreas Schwab <schwab@linux-m68k.org> wrote:

>>   result = (os_subtype == OS_NT ? GetMessageW : GetMessage) (&msg, NULL, 0, 0);
>
> That can fail if either of the functions is defined only as a
> function-like macro.

OTOH, if the patch explicitly references W functions, it should be
using RegisterClassA and GetMessageA for the explicit A versions.

    Juanma

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

GetMessage is a macro, many Windows API functions are, and switches
between GetMessageA and GetMessageW based on whether UNICODE is
defined. I really should explicitly use GetMessageA, I just missed it.

I switched to Stefan's suggestion, but with explicit use of
GetMessageA. Note that this makes that line longer than 80 chars.

Here's an amended patch that also moves the definition of the
INIT_WINDOW_CLASS macro to just before use, and makes
w32_set_frame_text static.

=== modified file 'src/w32fns.c'
--- src/w32fns.c        2011-12-04 08:02:42 +0000
+++ src/w32fns.c        2012-01-24 20:22:52 +0000
@@ -1641,6 +1641,20 @@

 }

+/* Set text of w32 frame with handle HWND to TEXT.
+
+   We explicitly switch between the Unicode and ANSI version of
+   SetWindowText because Emacs isn't compiled with UNICODE defined to
+   retain compatibility with Windows 95. */
+
+static void
+w32_set_frame_text (HWND hwnd, LPCSTR text)
+{
+  if (os_subtype == OS_NT)
+    SetWindowTextW (hwnd, (LPCWSTR)text);
+  else
+    SetWindowTextA (hwnd, text);
+}

 /* Change the name of frame F to NAME.  If NAME is nil, set F's name to
        w32_id_name.
@@ -1697,10 +1711,10 @@
   if (FRAME_W32_WINDOW (f))
     {
       if (STRING_MULTIBYTE (name))
-       name = ENCODE_SYSTEM (name);
-
+        name = ENCODE_SYSTEM (name);
+
       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1746,7 +1760,7 @@
        name = ENCODE_SYSTEM (name);

       BLOCK_INPUT;
-      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
+      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
       UNBLOCK_INPUT;
     }
 }
@@ -1782,23 +1796,37 @@

 static LRESULT CALLBACK w32_wnd_proc (HWND, UINT, WPARAM, LPARAM);

+#define INIT_WINDOW_CLASS(wc)                   \
+  wc.style = CS_HREDRAW | CS_VREDRAW;           \
+  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;      \
+  wc.cbClsExtra = 0;                            \
+  wc.cbWndExtra = WND_EXTRA_BYTES;              \
+  wc.hInstance = hinst;                         \
+  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);     \
+  wc.hCursor = w32_load_cursor (IDC_ARROW);     \
+  wc.hbrBackground = NULL;                      \
+  wc.lpszMenuName = NULL;                       \
+
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
-
-  wc.style = CS_HREDRAW | CS_VREDRAW;
-  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
-  wc.cbClsExtra = 0;
-  wc.cbWndExtra = WND_EXTRA_BYTES;
-  wc.hInstance = hinst;
-  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
-  wc.hCursor = w32_load_cursor (IDC_ARROW);
-  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
-  wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
-
-  return (RegisterClass (&wc));
+
+  if (os_subtype == OS_NT)
+    {
+      WNDCLASSW  uwc;
+      INIT_WINDOW_CLASS(uwc);
+      uwc.lpszClassName = L"Emacs";
+
+      return (RegisterClassW (&uwc));
+    }
+  else
+    {
+      WNDCLASS  wc;
+      INIT_WINDOW_CLASS(wc);
+      wc.lpszClassName = EMACS_CLASS;
+
+      return (RegisterClassA (&wc));
+    }
 }

 static HWND
@@ -2248,8 +2276,13 @@

   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);

-  while (GetMessage (&msg, NULL, 0, 0))
+  while (1)
     {
+      result = (os_subtype == OS_NT ? GetMessageW : GetMessageA)
(&msg, NULL, 0, 0);
+
+      if (!result)
+        break;
+
       if (msg.hwnd == NULL)
        {
          switch (msg.message)
@@ -2915,8 +2948,21 @@

     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-                             w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          unsigned short lo = wParam & 0x0000FFFF;
+          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
+          wParam  = hi | lo;
+
+          W32Msg wmsg;
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;

     case WM_UNICHAR:



On 24 January 2012 13:03, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Tue, Jan 24, 2012 at 10:40, Andreas Schwab <schwab@linux-m68k.org> wrote:
>
>>>   result = (os_subtype == OS_NT ? GetMessageW : GetMessage) (&msg, NULL, 0, 0);
>>
>> That can fail if either of the functions is defined only as a
>> function-like macro.
>
> OTOH, if the patch explicitly references W functions, it should be
> using RegisterClassA and GetMessageA for the explicit A versions.
>
>     Juanma

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Tue, 24 Jan 2012 21:42:46 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, 10299@debbugs.gnu.org
> 
> GetMessage is a macro, many Windows API functions are, and switches
> between GetMessageA and GetMessageW based on whether UNICODE is
> defined. I really should explicitly use GetMessageA, I just missed it.
> 
> I switched to Stefan's suggestion, but with explicit use of
> GetMessageA. Note that this makes that line longer than 80 chars.
> 
> Here's an amended patch that also moves the definition of the
> INIT_WINDOW_CLASS macro to just before use, and makes
> w32_set_frame_text static.

Thanks.  It's been a while since you submitted this, so sorry for the
delay.  We are now past Emacs 24.1 release, so we are free to install
your changes.

However, when I applied the patch, I bumped into several problems.
First, I don't understand why we need this part:

  +/* Set text of w32 frame with handle HWND to TEXT.
  +
  +   We explicitly switch between the Unicode and ANSI version of
  +   SetWindowText because Emacs isn't compiled with UNICODE defined to
  +   retain compatibility with Windows 95. */
  +
  +static void
  +w32_set_frame_text (HWND hwnd, LPCSTR text)
  +{
  +  if (os_subtype == OS_NT)
  +    SetWindowTextW (hwnd, (LPCWSTR)text);
  +  else
  +    SetWindowTextA (hwnd, text);
  +}

   /* Change the name of frame F to NAME.  If NAME is nil, set F's name to
	  w32_id_name.
  @@ -1697,10 +1711,10 @@
     if (FRAME_W32_WINDOW (f))
       {
	 if (STRING_MULTIBYTE (name))
  -       name = ENCODE_SYSTEM (name);
  -
  +        name = ENCODE_SYSTEM (name);
  +
	 BLOCK_INPUT;
  -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
  +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
	 UNBLOCK_INPUT;
       }
   }
  @@ -1746,7 +1760,7 @@
	  name = ENCODE_SYSTEM (name);

	 BLOCK_INPUT;
  -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
  +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
	 UNBLOCK_INPUT;
       }
   }

Why do we need to use SetWindowTextW, when all we change is the way we
request messages for Emacs?  With the patch below, which doesn't use
this part, I see no problems at all with the titles of Emacs frames.
Can you tell me how to reproduce the problem with the title?

Moreover, passing 'SDATA (name)' to SetWindowTextW is wrong, since
that function expects a wchar_t *, not a char *, as pointer to the
text to display in the title.  (The compiler should warn about that,
didn't it?)  So if we do need this part, we will need to convert the
frame name into a wide character string.

Next, this part, whose need was not entirely understood even when you
submitted the patches:

  @@ -2915,8 +2948,21 @@

       case WM_SYSCHAR:
       case WM_CHAR:
  -      post_character_message (hwnd, msg, wParam, lParam,
  -                             w32_get_key_modifiers (wParam, lParam));
  +      if (wParam > 255 )
  +        {
  +          unsigned short lo = wParam & 0x0000FFFF;
  +          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
  +          wParam  = hi | lo;
  +
  +          W32Msg wmsg;
  +          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
  +          signal_user_input ();
  +          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
  +
  +        }
  +      else
  +        post_character_message (hwnd, msg, wParam, lParam,
  +                                w32_get_key_modifiers (wParam, lParam));
	 break;

This doesn't work in a locale where Unicode codepoints of the
characters generated by the localized keyboard are above 255.  (I live
in one such locale.)  In those locales, this snippet produces the
wrong characters.  The root cause of this is that WM_CHAR's wParam is
not really the Unicode codepoint of the character I type, it's some
garbled value.  After looking into this, my conclusion was that your
patches didn't do everything that's needed to get true Unicode
character input.  This page:

  http://social.msdn.microsoft.com/forums/en-US/windowssdk/thread/07afec87-68c1-4a56-bf46-a38a9c2232e9/

suggests what needs to be done.  So I went ahead and made the few
additional changes mentioned there, and the Unicode input started
working for me.  With these additional changes, the funky bit-tweaking
code is no longer needed, at least for the characters in the 0x05nn
range of Unicode that I tried.  I hope it will work with your custom
keyboard as well.

The patches I tried are below.  I'd appreciate if you could try them
and see if they work with your custom keyboard layouts.  If they do, I
will install them.

(Juanma, I'd appreciate if you could try this as well.)

Thanks.


=== modified file 'src/w32fns.c'
--- src/w32fns.c	2012-07-27 09:24:34 +0000
+++ src/w32fns.c	2012-07-28 14:28:29 +0000
@@ -1780,23 +1780,37 @@ w32_load_cursor (LPCTSTR name)
 
 static LRESULT CALLBACK w32_wnd_proc (HWND, UINT, WPARAM, LPARAM);
 
+#define INIT_WINDOW_CLASS(WC)			  \
+  (WC).style = CS_HREDRAW | CS_VREDRAW;		  \
+  (WC).lpfnWndProc = (WNDPROC) w32_wnd_proc;      \
+  (WC).cbClsExtra = 0;                            \
+  (WC).cbWndExtra = WND_EXTRA_BYTES;              \
+  (WC).hInstance = hinst;                         \
+  (WC).hIcon = LoadIcon (hinst, EMACS_CLASS);     \
+  (WC).hCursor = w32_load_cursor (IDC_ARROW);     \
+  (WC).hbrBackground = NULL;                      \
+  (WC).lpszMenuName = NULL;                       \
+
 static BOOL
 w32_init_class (HINSTANCE hinst)
 {
-  WNDCLASS wc;
 
-  wc.style = CS_HREDRAW | CS_VREDRAW;
-  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
-  wc.cbClsExtra = 0;
-  wc.cbWndExtra = WND_EXTRA_BYTES;
-  wc.hInstance = hinst;
-  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
-  wc.hCursor = w32_load_cursor (IDC_ARROW);
-  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
-  wc.lpszMenuName = NULL;
-  wc.lpszClassName = EMACS_CLASS;
+  if (os_subtype == OS_NT)
+    {
+      WNDCLASSW  uwc;
+      INIT_WINDOW_CLASS(uwc);
+      uwc.lpszClassName = L"Emacs";
+
+      return RegisterClassW (&uwc);
+    }
+  else
+    {
+      WNDCLASS  wc;
+      INIT_WINDOW_CLASS(wc);
+      wc.lpszClassName = EMACS_CLASS;
 
-  return (RegisterClass (&wc));
+      return RegisterClassA (&wc);
+    }
 }
 
 static HWND
@@ -2246,7 +2260,7 @@ w32_msg_pump (deferred_msg * msg_buf)
 
   msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);
 
-  while (GetMessage (&msg, NULL, 0, 0))
+  while ((os_subtype == OS_NT ? GetMessageW : GetMessageA) (&msg, NULL, 0, 0))
     {
       if (msg.hwnd == NULL)
 	{
@@ -2341,7 +2355,10 @@ w32_msg_pump (deferred_msg * msg_buf)
 	}
       else
 	{
-	  DispatchMessage (&msg);
+	  if (os_subtype == OS_NT)
+	    DispatchMessageW (&msg);
+	  else
+	    DispatchMessageA (&msg);
 	}
 
       /* Exit nested loop when our deferred message has completed.  */
@@ -2918,8 +2935,18 @@ w32_wnd_proc (HWND hwnd, UINT msg, WPARA
 
     case WM_SYSCHAR:
     case WM_CHAR:
-      post_character_message (hwnd, msg, wParam, lParam,
-			      w32_get_key_modifiers (wParam, lParam));
+      if (wParam > 255 )
+        {
+          W32Msg wmsg;
+
+          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
+          signal_user_input ();
+          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
+
+        }
+      else
+        post_character_message (hwnd, msg, wParam, lParam,
+                                w32_get_key_modifiers (wParam, lParam));
       break;
 
     case WM_UNICHAR:
@@ -3801,7 +3828,7 @@ w32_wnd_proc (HWND hwnd, UINT msg, WPARA
 	}
 
     dflt:
-      return DefWindowProc (hwnd, msg, wParam, lParam);
+      return (os_subtype == OS_NT ? DefWindowProcW :  DefWindowProcA) (hwnd, msg, wParam, lParam);
     }
 
   /* The most common default return code for handled messages is 0.  */

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

Sorry for the late reply, I've been away on vacation.

The change to SetWindowTextW for frame titles was needed because
otherwise the frame title was cut short after the first character
(presumably because the window class name is a UCS-2 wide string and
the second byte therefore is null). This is exaclty the case they talk
about at your link.

If I change w32_set frame text to always call SetWindowText I get a
default frame title of "e". Changing DispatchMessage to use the wide
version when appropriate doesn't help either.

I'm not surprised the WM_CHAR bit twiddling was wrong since the actual
format was so badly documented, your version looks much better.

I'll try your new patch tomorrow and get back with details on how it worked,

On 28 July 2012 16:50, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 24 Jan 2012 21:42:46 +0100
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> Cc: Andreas Schwab <schwab@linux-m68k.org>, 10299@debbugs.gnu.org
>>
>> GetMessage is a macro, many Windows API functions are, and switches
>> between GetMessageA and GetMessageW based on whether UNICODE is
>> defined. I really should explicitly use GetMessageA, I just missed it.
>>
>> I switched to Stefan's suggestion, but with explicit use of
>> GetMessageA. Note that this makes that line longer than 80 chars.
>>
>> Here's an amended patch that also moves the definition of the
>> INIT_WINDOW_CLASS macro to just before use, and makes
>> w32_set_frame_text static.
>
> Thanks.  It's been a while since you submitted this, so sorry for the
> delay.  We are now past Emacs 24.1 release, so we are free to install
> your changes.
>
> However, when I applied the patch, I bumped into several problems.
> First, I don't understand why we need this part:
>
>   +/* Set text of w32 frame with handle HWND to TEXT.
>   +
>   +   We explicitly switch between the Unicode and ANSI version of
>   +   SetWindowText because Emacs isn't compiled with UNICODE defined to
>   +   retain compatibility with Windows 95. */
>   +
>   +static void
>   +w32_set_frame_text (HWND hwnd, LPCSTR text)
>   +{
>   +  if (os_subtype == OS_NT)
>   +    SetWindowTextW (hwnd, (LPCWSTR)text);
>   +  else
>   +    SetWindowTextA (hwnd, text);
>   +}
>
>    /* Change the name of frame F to NAME.  If NAME is nil, set F's name to
>           w32_id_name.
>   @@ -1697,10 +1711,10 @@
>      if (FRAME_W32_WINDOW (f))
>        {
>          if (STRING_MULTIBYTE (name))
>   -       name = ENCODE_SYSTEM (name);
>   -
>   +        name = ENCODE_SYSTEM (name);
>   +
>          BLOCK_INPUT;
>   -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>   +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
>          UNBLOCK_INPUT;
>        }
>    }
>   @@ -1746,7 +1760,7 @@
>           name = ENCODE_SYSTEM (name);
>
>          BLOCK_INPUT;
>   -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>   +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
>          UNBLOCK_INPUT;
>        }
>    }
>
> Why do we need to use SetWindowTextW, when all we change is the way we
> request messages for Emacs?  With the patch below, which doesn't use
> this part, I see no problems at all with the titles of Emacs frames.
> Can you tell me how to reproduce the problem with the title?
>
> Moreover, passing 'SDATA (name)' to SetWindowTextW is wrong, since
> that function expects a wchar_t *, not a char *, as pointer to the
> text to display in the title.  (The compiler should warn about that,
> didn't it?)  So if we do need this part, we will need to convert the
> frame name into a wide character string.
>
> Next, this part, whose need was not entirely understood even when you
> submitted the patches:
>
>   @@ -2915,8 +2948,21 @@
>
>        case WM_SYSCHAR:
>        case WM_CHAR:
>   -      post_character_message (hwnd, msg, wParam, lParam,
>   -                             w32_get_key_modifiers (wParam, lParam));
>   +      if (wParam > 255 )
>   +        {
>   +          unsigned short lo = wParam & 0x0000FFFF;
>   +          unsigned short hi = (wParam & 0xFFFF0000) >> 8;
>   +          wParam  = hi | lo;
>   +
>   +          W32Msg wmsg;
>   +          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
>   +          signal_user_input ();
>   +          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
>   +
>   +        }
>   +      else
>   +        post_character_message (hwnd, msg, wParam, lParam,
>   +                                w32_get_key_modifiers (wParam, lParam));
>          break;
>
> This doesn't work in a locale where Unicode codepoints of the
> characters generated by the localized keyboard are above 255.  (I live
> in one such locale.)  In those locales, this snippet produces the
> wrong characters.  The root cause of this is that WM_CHAR's wParam is
> not really the Unicode codepoint of the character I type, it's some
> garbled value.  After looking into this, my conclusion was that your
> patches didn't do everything that's needed to get true Unicode
> character input.  This page:
>
>   http://social.msdn.microsoft.com/forums/en-US/windowssdk/thread/07afec87-68c1-4a56-bf46-a38a9c2232e9/
>
> suggests what needs to be done.  So I went ahead and made the few
> additional changes mentioned there, and the Unicode input started
> working for me.  With these additional changes, the funky bit-tweaking
> code is no longer needed, at least for the characters in the 0x05nn
> range of Unicode that I tried.  I hope it will work with your custom
> keyboard as well.
>
> The patches I tried are below.  I'd appreciate if you could try them
> and see if they work with your custom keyboard layouts.  If they do, I
> will install them.
>
> (Juanma, I'd appreciate if you could try this as well.)
>
> Thanks.
>
>
> === modified file 'src/w32fns.c'
> --- src/w32fns.c        2012-07-27 09:24:34 +0000
> +++ src/w32fns.c        2012-07-28 14:28:29 +0000
> @@ -1780,23 +1780,37 @@ w32_load_cursor (LPCTSTR name)
>
>  static LRESULT CALLBACK w32_wnd_proc (HWND, UINT, WPARAM, LPARAM);
>
> +#define INIT_WINDOW_CLASS(WC)                    \
> +  (WC).style = CS_HREDRAW | CS_VREDRAW;                  \
> +  (WC).lpfnWndProc = (WNDPROC) w32_wnd_proc;      \
> +  (WC).cbClsExtra = 0;                            \
> +  (WC).cbWndExtra = WND_EXTRA_BYTES;              \
> +  (WC).hInstance = hinst;                         \
> +  (WC).hIcon = LoadIcon (hinst, EMACS_CLASS);     \
> +  (WC).hCursor = w32_load_cursor (IDC_ARROW);     \
> +  (WC).hbrBackground = NULL;                      \
> +  (WC).lpszMenuName = NULL;                       \
> +
>  static BOOL
>  w32_init_class (HINSTANCE hinst)
>  {
> -  WNDCLASS wc;
>
> -  wc.style = CS_HREDRAW | CS_VREDRAW;
> -  wc.lpfnWndProc = (WNDPROC) w32_wnd_proc;
> -  wc.cbClsExtra = 0;
> -  wc.cbWndExtra = WND_EXTRA_BYTES;
> -  wc.hInstance = hinst;
> -  wc.hIcon = LoadIcon (hinst, EMACS_CLASS);
> -  wc.hCursor = w32_load_cursor (IDC_ARROW);
> -  wc.hbrBackground = NULL; /* GetStockObject (WHITE_BRUSH);  */
> -  wc.lpszMenuName = NULL;
> -  wc.lpszClassName = EMACS_CLASS;
> +  if (os_subtype == OS_NT)
> +    {
> +      WNDCLASSW  uwc;
> +      INIT_WINDOW_CLASS(uwc);
> +      uwc.lpszClassName = L"Emacs";
> +
> +      return RegisterClassW (&uwc);
> +    }
> +  else
> +    {
> +      WNDCLASS  wc;
> +      INIT_WINDOW_CLASS(wc);
> +      wc.lpszClassName = EMACS_CLASS;
>
> -  return (RegisterClass (&wc));
> +      return RegisterClassA (&wc);
> +    }
>  }
>
>  static HWND
> @@ -2246,7 +2260,7 @@ w32_msg_pump (deferred_msg * msg_buf)
>
>    msh_mousewheel = RegisterWindowMessage (MSH_MOUSEWHEEL);
>
> -  while (GetMessage (&msg, NULL, 0, 0))
> +  while ((os_subtype == OS_NT ? GetMessageW : GetMessageA) (&msg, NULL, 0, 0))
>      {
>        if (msg.hwnd == NULL)
>         {
> @@ -2341,7 +2355,10 @@ w32_msg_pump (deferred_msg * msg_buf)
>         }
>        else
>         {
> -         DispatchMessage (&msg);
> +         if (os_subtype == OS_NT)
> +           DispatchMessageW (&msg);
> +         else
> +           DispatchMessageA (&msg);
>         }
>
>        /* Exit nested loop when our deferred message has completed.  */
> @@ -2918,8 +2935,18 @@ w32_wnd_proc (HWND hwnd, UINT msg, WPARA
>
>      case WM_SYSCHAR:
>      case WM_CHAR:
> -      post_character_message (hwnd, msg, wParam, lParam,
> -                             w32_get_key_modifiers (wParam, lParam));
> +      if (wParam > 255 )
> +        {
> +          W32Msg wmsg;
> +
> +          wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
> +          signal_user_input ();
> +          my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
> +
> +        }
> +      else
> +        post_character_message (hwnd, msg, wParam, lParam,
> +                                w32_get_key_modifiers (wParam, lParam));
>        break;
>
>      case WM_UNICHAR:
> @@ -3801,7 +3828,7 @@ w32_wnd_proc (HWND hwnd, UINT msg, WPARA
>         }
>
>      dflt:
> -      return DefWindowProc (hwnd, msg, wParam, lParam);
> +      return (os_subtype == OS_NT ? DefWindowProcW :  DefWindowProcA) (hwnd, msg, wParam, lParam);
>      }
>
>    /* The most common default return code for handled messages is 0.  */
>

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Mon, 6 Aug 2012 22:20:01 +0200
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: lekktu@gmail.com, 10299@debbugs.gnu.org
> 
> Sorry for the late reply, I've been away on vacation.

No sweat.  Hope you had a good one.

> The change to SetWindowTextW for frame titles was needed because
> otherwise the frame title was cut short after the first character
> (presumably because the window class name is a UCS-2 wide string and
> the second byte therefore is null). This is exaclty the case they talk
> about at your link.
> 
> If I change w32_set frame text to always call SetWindowText I get a
> default frame title of "e". Changing DispatchMessage to use the wide
> version when appropriate doesn't help either.

I didn't see any problem with the frame title, with the patch I sent.
I'm curious to hear if you will still see that problem.

> I'll try your new patch tomorrow and get back with details on how it worked,

Thank you.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Jason Rumney]

Joakim Hårsman <joakim.harsman@gmail.com> writes:

>>
>>   +/* Set text of w32 frame with handle HWND to TEXT.
>>   +
>>   +   We explicitly switch between the Unicode and ANSI version of
>>   +   SetWindowText because Emacs isn't compiled with UNICODE defined to
>>   +   retain compatibility with Windows 95. */
>>   +
>>   +static void
>>   +w32_set_frame_text (HWND hwnd, LPCSTR text)
>>   +{
>>   +  if (os_subtype == OS_NT)
>>   +    SetWindowTextW (hwnd, (LPCWSTR)text);
>>   +  else
>>   +    SetWindowTextA (hwnd, text);
>>   +}

> The change to SetWindowTextW for frame titles was needed because
> otherwise the frame title was cut short after the first character
> (presumably because the window class name is a UCS-2 wide string and
> the second byte therefore is null). This is exaclty the case they talk
> about at your link.

>>          if (STRING_MULTIBYTE (name))
>>   -       name = ENCODE_SYSTEM (name);
>>   -
>>   +        name = ENCODE_SYSTEM (name);
>>   +
>>          BLOCK_INPUT;
>>   -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>>   +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));

But the above doesn't pass a UCS-2 wide string to SetWindowTextW.  It
passes an 8-bit string encoded in the system 8-bit encoding, then simply
casts it to a wchar_t!  You cannot possibly get the correct frame title using
this patch.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

Your patch works fine for me on Windows XP. Unicode input outside the
system code page works fine and so does non-ASCII input that is
present in the current code page.

Since this seems much cleaner than the previous patch, it is
definitely the way to go.

I think the critical change was switching to both DefWindowProcW and
DispatchMessageW (at least that's what
http://social.msdn.microsoft.com/forums/en-US/windowssdk/thread/07afec87-68c1-4a56-bf46-a38a9c2232e9/
leads you to believe).

I think I actually tried both those changes individually, but not both at once.

I'll test a patched build on Windows 7 tomorrow as well, sadly I don't
have access to anything running Windows 95.

Thanks for taking the time to look at this and test it.

On 7 August 2012 04:53, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 6 Aug 2012 22:20:01 +0200
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> Cc: lekktu@gmail.com, 10299@debbugs.gnu.org
>>
>> Sorry for the late reply, I've been away on vacation.
>
> No sweat.  Hope you had a good one.
>
>> The change to SetWindowTextW for frame titles was needed because
>> otherwise the frame title was cut short after the first character
>> (presumably because the window class name is a UCS-2 wide string and
>> the second byte therefore is null). This is exaclty the case they talk
>> about at your link.
>>
>> If I change w32_set frame text to always call SetWindowText I get a
>> default frame title of "e". Changing DispatchMessage to use the wide
>> version when appropriate doesn't help either.
>
> I didn't see any problem with the frame title, with the patch I sent.
> I'm curious to hear if you will still see that problem.
>
>> I'll try your new patch tomorrow and get back with details on how it worked,
>
> Thank you.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

I'm well aware of the insanity of this patch, but trust me, it does
actually work, I use a version built with this patch every day.

The reason for the weirdness is the partial use of Unicode APIs, this
lead to lots of weirdness with wide strings being interpreted as
single byte strings in various places.

But it doesn't matter anymore, Eli's modifications removes all of the
ugliness and work fine.

On 7 August 2012 14:15, Jason Rumney <jasonr@gnu.org> wrote:
> Joakim Hårsman <joakim.harsman@gmail.com> writes:
>
>>>
>>>   +/* Set text of w32 frame with handle HWND to TEXT.
>>>   +
>>>   +   We explicitly switch between the Unicode and ANSI version of
>>>   +   SetWindowText because Emacs isn't compiled with UNICODE defined to
>>>   +   retain compatibility with Windows 95. */
>>>   +
>>>   +static void
>>>   +w32_set_frame_text (HWND hwnd, LPCSTR text)
>>>   +{
>>>   +  if (os_subtype == OS_NT)
>>>   +    SetWindowTextW (hwnd, (LPCWSTR)text);
>>>   +  else
>>>   +    SetWindowTextA (hwnd, text);
>>>   +}
>
>> The change to SetWindowTextW for frame titles was needed because
>> otherwise the frame title was cut short after the first character
>> (presumably because the window class name is a UCS-2 wide string and
>> the second byte therefore is null). This is exaclty the case they talk
>> about at your link.
>
>>>          if (STRING_MULTIBYTE (name))
>>>   -       name = ENCODE_SYSTEM (name);
>>>   -
>>>   +        name = ENCODE_SYSTEM (name);
>>>   +
>>>          BLOCK_INPUT;
>>>   -      SetWindowText (FRAME_W32_WINDOW (f), SDATA (name));
>>>   +      w32_set_frame_text(FRAME_W32_WINDOW (f), SDATA (name));
>
> But the above doesn't pass a UCS-2 wide string to SetWindowTextW.  It
> passes an 8-bit string encoded in the system 8-bit encoding, then simply
> casts it to a wchar_t!  You cannot possibly get the correct frame title using
> this patch.
>
>

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Tue, 7 Aug 2012 21:47:38 +0200
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: lekktu@gmail.com, 10299@debbugs.gnu.org
> 
> Your patch works fine for me on Windows XP. Unicode input outside the
> system code page works fine and so does non-ASCII input that is
> present in the current code page.

Great.

> I'll test a patched build on Windows 7 tomorrow as well

Thanks.

> sadly I don't have access to anything running Windows 95.

Neither do I, but then the new code is not activated on Windows 9X, or
at least isn't supposed to be.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Joakim Hårsman]

Works fine on Windows 7 as well.

On 8 August 2012 04:48, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 7 Aug 2012 21:47:38 +0200
>> From: Joakim Hårsman <joakim.harsman@gmail.com>
>> Cc: lekktu@gmail.com, 10299@debbugs.gnu.org
>>
>> Your patch works fine for me on Windows XP. Unicode input outside the
>> system code page works fine and so does non-ASCII input that is
>> present in the current code page.
>
> Great.
>
>> I'll test a patched build on Windows 7 tomorrow as well
>
> Thanks.
>
>> sadly I don't have access to anything running Windows 95.
>
> Neither do I, but then the new code is not activated on Windows 9X, or
> at least isn't supposed to be.

bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii]

> Date: Wed, 8 Aug 2012 20:54:00 +0200
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: lekktu@gmail.com, 10299@debbugs.gnu.org
> 
> Works fine on Windows 7 as well.

Great, I committed the changes as trunk revision 109542.