Unicode input on Windows Emacs: any plans?

Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

Is there any plan to have Unicode input enabled on Windows?
Currently, this is what I’m forced to write about Emacs:

  NOTE: Among notable applications which fail spectacularly is
  Emacs. The developers forget that for a generation, it is already
  XXI century; so they use ToAscii() instead of ToUnicode()! (Even if
  ToUnicode() is available, its result is converted to the result of
  the corresponding ToAscii() code.)

  In addition to 8-bitness, Emacs also suffers from
  check-for-specials-first syndrome…

(from the discussion on how to implement a CORRECT Unicode input in an
Event-loop application on Windows; see:
  http://search.cpan.org/~ilyaz/UI-KeyboardLayout/lib/UI/KeyboardLayout.pm#Can_an_application_on_Windows_accept_keyboard_events?_Part_I:_insert_only
and 3 following parts).

Ilya

P.S.  I believe that until it was documented above, it was not known
      how to process Windows’ input correctly (and it is still not
      known how to do it CORRECTLY from console applications: I can
      find no way to find the input locale, so do not know how to
      distinguish a dead key from a possible part of “a command”).

      But now, when it IS documented, there IS a chance to make Emacs
      into a good citizen…

P.P.S.  The last time I inspected the source code of Emacs was several
	months ago; I could not post before, so maybe my info is
	obsolete now.

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Tue, 25 Mar 2014 15:40:29 +0000 (UTC)
> 
> Is there any plan to have Unicode input enabled on Windows?

That ship sailed a long time ago.  Emacs on Windows supports Unicode
keyboard input on Windows NT and later since Aug 2012, and Emacs 24.3
was released with that feature.  (This is limited to GUI sessions,
since Unicode support in Windows console applications sucks big time,
so much so that I consider it a waste of time and effort to support
that in text mode.)

> Currently, this is what I’m forced to write about Emacs:
> 
>   NOTE: Among notable applications which fail spectacularly is
>   Emacs. The developers forget that for a generation, it is already
>   XXI century; so they use ToAscii() instead of ToUnicode()! (Even if
>   ToUnicode() is available, its result is converted to the result of
>   the corresponding ToAscii() code.)

This is inaccurate for the current Emacs: ToAscii is only used on
Windows 9X, where Unicode keyboard input is not supported.

>   In addition to 8-bitness, Emacs also suffers from
>   check-for-specials-first syndrome…

Not sure what is this about; please elaborate.

> (from the discussion on how to implement a CORRECT Unicode input in an
> Event-loop application on Windows; see:
>   http://search.cpan.org/~ilyaz/UI-KeyboardLayout/lib/UI/KeyboardLayout.pm#Can_an_application_on_Windows_accept_keyboard_events?_Part_I:_insert_only
> and 3 following parts).

We do use ToUnicode (on NT and later).

> P.P.S.  The last time I inspected the source code of Emacs was several
> 	months ago; I could not post before, so maybe my info is
> 	obsolete now.

What version did you look in?

(This is not the best place to ask about Emacs development plans,
btw.)

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> Date: Tue, 25 Mar 2014 18:25:46 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Unicode support in Windows console applications sucks big time,
> so much so that I consider it a waste of time and effort to support
> that in text mode.

Actually, I see that even this is inaccurate: we do use
ReadConsoleInputW in text-mode session on Windows NT and later, so
Unicode keyboard input is supported even on a console.  (However,
console output still only supports the current OEM codepage.)

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-25, Eli Zaretskii <eliz@gnu.org> wrote:
>> Unicode support in Windows console applications sucks big time,
>> so much so that I consider it a waste of time and effort to support
>> that in text mode.

There is no difference in support of Event-Loop vs Console
applications on Windows, as far as the application has no need to
distinguish possible-accelerators from dead-keys.

> Actually, I see that even this is inaccurate: we do use
> ReadConsoleInputW in text-mode session on Windows NT and later, so
> Unicode keyboard input is supported even on a console.  (However,
> console output still only supports the current OEM codepage.)

Which means: “it does not work”.  Exactly the same as in the
Event-loop mode.

Ilya

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-28, Ilya Zakharevich <nospam-abuse@ilyaz.org> wrote:
> On 2014-03-25, Eli Zaretskii <eliz@gnu.org> wrote:
>> Actually, I see that even this is inaccurate: we do use
>> ReadConsoleInputW in text-mode session on Windows NT and later, so
>> Unicode keyboard input is supported even on a console.  (However,
>> console output still only supports the current OEM codepage.)
>
> Which means: “it does not work”.  Exactly the same as in the
> Event-loop mode.

Oups:
  • My first sentence is correct.
  • The second is not.

Sorry,
Ilya

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 28 Mar 2014 14:17:16 +0000 (UTC)
> Injection-Date: Fri, 28 Mar 2014 14:17:16 +0000 (UTC)
> 
> On 2014-03-28, Ilya Zakharevich <nospam-abuse@ilyaz.org> wrote:
> > On 2014-03-25, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Actually, I see that even this is inaccurate: we do use
> >> ReadConsoleInputW in text-mode session on Windows NT and later, so
> >> Unicode keyboard input is supported even on a console.  (However,
> >> console output still only supports the current OEM codepage.)
> >
> > Which means: “it does not work”.  Exactly the same as in the
> > Event-loop mode.
> 
> Oups:
>   • My first sentence is correct.
>   • The second is not.

What do you mean by "doesn't work"?  May I suggest reporting a bug
using "M-x report-emacs-bug RET", and including all the details of how
to reproduce the problems starting with "emacs -Q" or "emacs -Q -nw"?

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-28, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> we do use
>> >> ReadConsoleInputW in text-mode session on Windows NT and later, so
>> >> Unicode keyboard input is supported even on a console.  (However,
>> >> console output still only supports the current OEM codepage.)
>> >
>> > Which means: “it does not work”.

> What do you mean by "doesn't work"?

I meant only that “it only supports the current OEM codepage” is
synonymous with “it is not Unicode input”.

> May I suggest reporting a bug
> using "M-x report-emacs-bug RET", and including all the details of how
> to reproduce the problems starting with "emacs -Q" or "emacs -Q -nw"?

I did not do any experiments in console mode.  But looking at what
Emacs does in windowed mode, I do not see why to reproduce this
(extremely buggy) behaviour in console would represent any problem…

Ilya

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 28 Mar 2014 21:57:04 +0000 (UTC)
> 
> On 2014-03-28, Eli Zaretskii <eliz@gnu.org> wrote:
> >> >> we do use
> >> >> ReadConsoleInputW in text-mode session on Windows NT and later, so
> >> >> Unicode keyboard input is supported even on a console.  (However,
> >> >> console output still only supports the current OEM codepage.)
> >> >
> >> > Which means: “it does not work”.
> 
> > What do you mean by "doesn't work"?
> 
> I meant only that “it only supports the current OEM codepage” is
> synonymous with “it is not Unicode input”.

That is correct only for output.  On input, Emacs supports the same
Unicode characters it does in GUI sessions, i.e. not limited to the
OEM codepage.  Whatever characters the current keyboard layout can
send, Emacs will receive and correctly insert it in the buffer.  When
displaying that buffer, we are limited to the current console
codepage.

> > May I suggest reporting a bug
> > using "M-x report-emacs-bug RET", and including all the details of how
> > to reproduce the problems starting with "emacs -Q" or "emacs -Q -nw"?
> 
> I did not do any experiments in console mode.  But looking at what
> Emacs does in windowed mode, I do not see why to reproduce this
> (extremely buggy) behaviour in console would represent any problem…

I suggested reporting a bug because I think there might be a
misunderstanding here.  I think you are talking about a bug that
doesn't exist, or maybe I (still) don't understand what buggy behavior
you are describing.  A bug report is a good way of figuring out which
is which.

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-29, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> >> Unicode keyboard input is supported even on a console.  (However,
>> >> >> console output still only supports the current OEM codepage.)
                 ↑↑↑↑↑↑

>> >> > Which means: “it does not work”.

>> I meant only that “it only supports the current OEM codepage” is
>> synonymous with “it is not Unicode input”.

> That is correct only for output.

Oups, I missed the word marked above!

Sorry,
Ilya

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-25, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
>> Date: Tue, 25 Mar 2014 15:40:29 +0000 (UTC)
>> 
>> Is there any plan to have Unicode input enabled on Windows?
>
> That ship sailed a long time ago.  Emacs on Windows supports Unicode
> keyboard input on Windows NT and later since Aug 2012, and Emacs 24.3
> was released with that feature.

Hmm, I was reading the source code of 24.3 instead of running the
binary!  (My workhorse version is 23.2, and since I could not find
changes in the source code, I did not bother upgrading.)  Indeed, the
trivial stuff works fine in 24.3.  Sorry for misrepresentation!

  Now I need to find why I misinterpreted the source code.

> (This is limited to GUI sessions,
> since Unicode support in Windows console applications sucks big time,
> so much so that I consider it a waste of time and effort to support
> that in text mode.)

The only thing which sucks is impossibility to find whether a given
keypress is a part of a character input sequence (a deadkey).  I need
to read the source code more to find out whether this is actually
processed in Emacs.

>>   In addition to 8-bitness, Emacs also suffers from
>>   check-for-specials-first syndrome…

> Not sure what is this about; please elaborate.

The long explanation is in the Web document I quoted
(UI::KeyboardLayout below).  The short one: if I press
SuperDuper-RightArrow on the keyboard (here SuperDuper is a modifier
key), and the keyboard layout defines this combination to deliver a
character, this keypress should NOT be handled to the standard Emacs
keypress interpreter (which would most probably interpret it as just
RightArrow, ignoring the SuperDuper modifier).

  (I would just define a [WithCharacter] event (similar to [menubar])
   [which may “take parameters”], and would attach the keypress and
   the character it delivers as two parameters.  The standard binding
   for WithCharacter event should be to ignore the keypress, while
   inserting the character.  The users would be able to override
   this.)

Example: with izKeys keyboards, press AppMenu-RightArrow.  This should
insert →, but it acts the same as RightArrow.  (The layouts are on
  http://k.ilyaz.org/windows
)

>>   http://search.cpan.org/~ilyaz/UI-KeyboardLayout/lib/UI/KeyboardLayout.pm#Can_an_application_on_Windows_accept_keyboard_events?_Part_I:_insert_only
>> and 3 following parts).

>> P.P.S.  The last time I inspected the source code of Emacs was several
>> 	months ago; I could not post before, so maybe my info is
>> 	obsolete now.
>
> What version did you look in?

24.3 (but as I said, apparently, I misread the code).

Thanks and sorry,
Ilya

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-25, Eli Zaretskii <eliz@gnu.org> wrote:
>> P.P.S.  The last time I inspected the source code of Emacs was several
>> 	months ago; I could not post before, so maybe my info is
>> 	obsolete now.
>
> What version did you look in?

If I can navigate github (and googling for
    emacs 24.3 ToUnicode OR ToUnicodeEx WCHAR
) correctly, ToUnicode(Ex) is used only once,
in w32_kbd_patch_key(), and w32_kbd_patch_key() is used only once, in
w32inevt.c as
  w32_kbd_patch_key (event, -1);

The call with second argument -1 DOES NOT USE ToUnicode(): the
function is CALLED, but its result is immediately converted to the
corresponding result of ToAscii().

=======================================================

On the other hand, an application is not REQUIRED to call ToUnicode();
it may rely on delivery of WM_CHAR instead.  So it looks like what you
call “support of Unicode input” is that in some rare cases, Emacs
calls TranslateMessage() (which enables delivery of WM_CHAR).  (I
cannot still understand the logic of operation; to me, it looks like a
complete mess of special cases…)

This matches my experiments with 24.3: the simplest cases of Unicode
input work, the more complicated ones fail.¹⁾

It IS possible to handle keyboard input on Windows WITHOUT any special
cases.  (See the reference in the original message.)  You cannot
decide what to do with keypress without either calling ToUnicode(), or
inspecting the following WM_CHAR message (may be done by
peaking — after TranslateMessage()).

Ilya

  ¹⁾ See http://k.ilyaz.org/windows/izKeys-visual-maps.html#examples

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 28 Mar 2014 15:39:02 +0000 (UTC)
> 
> On the other hand, an application is not REQUIRED to call ToUnicode();
> it may rely on delivery of WM_CHAR instead.  So it looks like what you
> call “support of Unicode input” is that in some rare cases, Emacs
> calls TranslateMessage() (which enables delivery of WM_CHAR).  (I
> cannot still understand the logic of operation; to me, it looks like a
> complete mess of special cases…)

No, I meant that the input loop now gets Unicode codepoints on systems
that support that.  More often than not, ToUnicode is not needed.

> This matches my experiments with 24.3: the simplest cases of Unicode
> input work, the more complicated ones fail.¹⁾

Please report those cases as bugs, and please include all the
necessary details to reproduce the problems.  Thanks.

> It IS possible to handle keyboard input on Windows WITHOUT any special
> cases.  (See the reference in the original message.)  You cannot
> decide what to do with keypress without either calling ToUnicode(), or
> inspecting the following WM_CHAR message (may be done by
> peaking — after TranslateMessage()).

Patches to do that are welcome.  (I don't think we have anyone on
board right now who knows as much as you do about this.)

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-28, Eli Zaretskii <eliz@gnu.org> wrote:
> No, I meant that the input loop now gets Unicode codepoints on systems
> that support that.

Yes, this is a major simplification!

>  More often than not, ToUnicode is not needed.

AFAIU, getting any kind of close-to-correct behaviour requires either
calling TranslateMessage() on all Down/Up events, or calling
ToUnicode() explicitly on all events.

>> This matches my experiments with 24.3: the simplest cases of Unicode
>> input work, the more complicated ones fail.¹⁾
>
> Please report those cases as bugs, and please include all the
> necessary details to reproduce the problems.  Thanks.

Too much work…  But here is a copy of my notes of today (reading
  http://fossies.org/dox/emacs-24.3/w32fns_8c_source.html
).

Ilya

=============================================== lispy_function_keys[] (named non-main-island “unprintable” keys)
http://fossies.org/dox/emacs-24.3/keyboard_8c_source.html	(SPACE, ABNT_C1, ABNT_C2 considered “printable”; OK!)
	comment may be improved: KEYPAD stuff, actually, comes from the following:

=============================================== map_keypad_keys()
http://fossies.org/dox/emacs-24.3/w32fns_8c_source.html#l02209

=============================================== Logic of TranslateMessage() for non-special-cased keys:
			[Basically, all cases when TranslateMessage()
			 is not called are WRONG!]
  Convert numpad-vk to special values near 0x90;		(breaks translation, but translation is disabled anyway)
  If vk < 0x100 and has Emacs’ key-name: do not translate;	(SPACE has no name!)
  Otherwise:	LCONTROL & RMENU & !user_disabled_this: translate;
  Otherwise:	has recognized modifiers (except shift and CapsLock): manually translate:
  			if 'A' … 'Z': ————? WM_CHAR with 'A' … 'Z' or 'a' … 'z';
  			otherwise:	Using only SHIFT/CTRL/ALT/CAPSLOCK modifiers (chiral, and tuned for AltGr too!),
  					  call ToUnicode() (with correct limit on the length of the buffer);
  					  convert to ASCII (with wrong limit on the length of the buffer).
  						(Should return complete gibberish if output is of length>1:
  							See ansi_code[2] and no treatment for dead_key > 2)
  					  Post (Windows/Emacs???) messages for the ASCII chars.
  Otherwise:	translate

=============================================== WM_INPUTLANGCHANGE
Complete nonsense

=============================================== Treatment of WM_CHAR

BUG: The modifiers (Alt/Ctrl) are passed downstream.  Ctrl should not
be passed at all (unless the delivered char is in 0..0x20 or 0x7f);
Alt/Win/etc should be passed only if their presence is not changing
the result of ToUnicode() (this requires a non-destructive call of
ToUnicode()).

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 28 Mar 2014 22:08:14 +0000 (UTC)
> 
> >> This matches my experiments with 24.3: the simplest cases of Unicode
> >> input work, the more complicated ones fail.¹⁾
> >
> > Please report those cases as bugs, and please include all the
> > necessary details to reproduce the problems.  Thanks.
> 
> Too much work…  But here is a copy of my notes of today (reading
>   http://fossies.org/dox/emacs-24.3/w32fns_8c_source.html
> ).

Too bad, because I cannot make heads or tails of that.  I guess these
problems, whatever they are, will remain unsolved until someone does
consider them important enough to submit a bug report.

Re: Unicode input on Windows Emacs: any plans?

[Ilya Zakharevich]

On 2014-03-28, Eli Zaretskii <eliz@gnu.org> wrote:
>> It IS possible to handle keyboard input on Windows WITHOUT any special
>> cases.  (See the reference in the original message.)  You cannot
>> decide what to do with keypress without either calling ToUnicode(), or
>> inspecting the following WM_CHAR message (may be done by
>> peaking — after TranslateMessage()).
>
> Patches to do that are welcome.  (I don't think we have anyone on
> board right now who knows as much as you do about this.)

If I had a working build system for Emacs, I would love to do this.  I
have cygwin (full), Strawberry Perl (which should come with working
mingw, I presume), and MSKLC (which comes with some old MS C
compiler).

I will try to compile it, but a presence of somebody who would hold my
hands if/when something does not work would not hurt.  (I never
managed any C project on Windows.)

Ilya

Re: Unicode input on Windows Emacs: any plans?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 28 Mar 2014 22:12:16 +0000 (UTC)
> 
> On 2014-03-28, Eli Zaretskii <eliz@gnu.org> wrote:
> >> It IS possible to handle keyboard input on Windows WITHOUT any special
> >> cases.  (See the reference in the original message.)  You cannot
> >> decide what to do with keypress without either calling ToUnicode(), or
> >> inspecting the following WM_CHAR message (may be done by
> >> peaking — after TranslateMessage()).
> >
> > Patches to do that are welcome.  (I don't think we have anyone on
> > board right now who knows as much as you do about this.)
> 
> If I had a working build system for Emacs, I would love to do this.  I
> have cygwin (full), Strawberry Perl (which should come with working
> mingw, I presume), and MSKLC (which comes with some old MS C
> compiler).
> 
> I will try to compile it, but a presence of somebody who would hold my
> hands if/when something does not work would not hurt.  (I never
> managed any C project on Windows.)

You will find lots of support for that on emacs-devel@gnu.org.

I suggest to start with the current development trunk of Emacs.  When
you clone the trunk, read the file nt/INSTALL first, it includes
description of the tools (MSYS and MinGW) you need to install to have
an environment, and then you just do the usual Unix configure && make
thing.

Thanks.