bug#10299: Emacs doesn't handle Unicode characters in keyboard layout on MS Windows

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 17 Dec 2011 13:52:29 +0100
> From: Joakim Hårsman <joakim.harsman@gmail.com>
> Cc: 10299@debbugs.gnu.org
> 
> I had to switch to iswalpha because isalpha segfaulted when sent
> values larger than 255. wParam isn't really the Unicode code point
> here either, it's encoded as UTF-16. With this change, Emacs no longer
> prints a question mark when I press the special keys, it doesn't print
> anything at all.
> 
> I figured this is beacause wParam isn't a valid Unicode codepoint
> here, like it is when you get a WM_UNICHAR, so I tried a quick hack
> converting from UTF-16 and re-posting as a WM_UNICHAR even. I changed
> the handling of WM_CHAR in w32_wnd_proc:
> 
>     case WM_CHAR:
>       if (wParam > 255 )
>         {
>           /* Hacky conversion from UTF-16 to UCS-4.
>              Doesn't handle surrogate pairs. */
>           unsigned short lo = wParam & 0x0000FFFF;
>           unsigned short hi = (wParam & 0xFFFF0000) >> 8;
>           wParam  = hi | lo;
> 
>           W32Msg wmsg;
>           wmsg.dwModifiers = w32_get_key_modifiers (wParam, lParam);
>           signal_user_input ();
>           my_post_msg (&wmsg, hwnd, WM_UNICHAR, wParam, lParam);
> 
>         }
>       else
>         post_character_message (hwnd, msg, wParam, lParam,
>                                 w32_get_key_modifiers (wParam, lParam));
> 
> This fixed it! I can now enter the characters with a custom keyboard
> layout so I'm happy.

That's good news.  However, I'm puzzled: are you saying that the code
points passed by Windows to Emacs for the characters generated by MKLC
are outside the Unicode BMP, i.e. larger than 65535?  If so, what code
points are they?

> There's loads of bugs still (e.g. the frame title
> is "e" instead of "Emacs@host" because I just pass the frame's namebuf
> to CreateWindowW without converting it to a wide string), but at least
> the main issue is fixed.

If at all possible, I'd like to keep the use of wide APIs to the
absolute minimum, because:

> I guess these changes need to only happen when running on NT for Emacs
> to still work on Windows 95 or does Emacs use MSLU?

We do use MSLU on Windows 9X (because the display routines need
that).  However, given that none of the developers has easy access to
a 9X machine, it is very easy to break Emacs on 9X by excess usage of
wide APIs.  In fact, all versions of Emacs 23 didn't work on Windows
9X and only recently we were lucky enough to find a 9X user who helped
us debug the problems and fix them.

So if possible, could you perhaps try finding out why you didn't get
UTF-16 codepoints without CreateWindowW and GetMessageW?  Maybe it's
TranslateMessage that causes that, and the original message Emacs gets
is okay?

> Maybe they should depend on some lisp variable being set?

Probably just on the value of os_subtype.

> Here's my changes in full so far:

Thanks.

This will probably need to wait until after v24.1 is released.