bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[dmoncayo]

I see this problem in [2] but not in [1].

Recipe from "emacs -Q -nw":
  Type these characters: áéíóúñç

Note: With my spanish keyboard, I input the accented vowels by typing
the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
the "ç" characters, they have their own keys.

Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
in [2] I see these characters instead: ßÚݾ·±þ


--- Footnotes ---

[1] Build from the emacs-24 branch:

  In GNU Emacs 24.3.1 (i386-mingw-nt6.1.7601)
   of 2013-04-08 on LEG570
  Bzr revision: 111342 rgm@gnu.org-20130406230553-iw6czqc1chq2peld
  Windowing system distributor `Microsoft Corp.', version 6.1.7601
  Configured using:
   `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
   -IC:/emacs/libs/libXpm-3.5.10/include -IC:/emacs/libs/libXpm-3.5.10/src
   -IC:/emacs/libs/libpng-dev_1.4.3-1_win32/include
   -IC:/emacs/libs/zlib-dev_1.2.5-2_win32/include
   -IC:/emacs/libs/giflib-4.1.4-1-lib/include
   -IC:/emacs/libs/jpeg-6b-4-lib/include
   -IC:/emacs/libs/tiff-3.8.2-1-lib/include
   -IC:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
   -IC:/emacs/libs/gnutls-3.1.10-w32/include
   -IC:/emacs/libs/libiconv-1.14-2-mingw32-dev/include'

  Important settings:
    value of $LANG: ENU
    locale-coding-system: cp1252
    default enable-multibyte-characters: t


[2] Build from the trunk:

  In GNU Emacs 24.3.50.1 (i386-mingw-nt6.1.7601)
   of 2013-04-21 on LEG570
  Bzr revision: 112347 xfq.free@gmail.com-20130421120104-6nn62drdem8o2cud
  Windowing system distributor `Microsoft Corp.', version 6.1.7601
  Configured using:
   `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
   -IC:/emacs/libs/libXpm-3.5.10/include -IC:/emacs/libs/libXpm-3.5.10/src
   -IC:/emacs/libs/libpng-dev_1.4.3-1_win32/include
   -IC:/emacs/libs/zlib-dev_1.2.5-2_win32/include
   -IC:/emacs/libs/giflib-4.1.4-1-lib/include
   -IC:/emacs/libs/jpeg-6b-4-lib/include
   -IC:/emacs/libs/tiff-3.8.2-1-lib/include
   -IC:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
   -IC:/emacs/libs/gnutls-3.1.10-w32/include
   -IC:/emacs/libs/libiconv-1.14-2-mingw32-dev/include'
  
  Important settings:
    value of $LANG: ENU
    locale-coding-system: cp1252
    default enable-multibyte-characters: t

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: dmoncayo@gmail.com
> Date: Tue, 14 May 2013 21:11:22 +0200
> 
> 
> I see this problem in [2] but not in [1].
> 
> Recipe from "emacs -Q -nw":
>   Type these characters: áéíóúñç
> 
> Note: With my spanish keyboard, I input the accented vowels by typing
> the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
> the "ç" characters, they have their own keys.
> 
> Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
> in [2] I see these characters instead: ßÚݾ·±þ

This is an exact duplicate of 14368, discussed just yesterday.

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Stefan Monnier]

>> I see this problem in [2] but not in [1].
>> Recipe from "emacs -Q -nw":
>> Type these characters: áéíóúñç
>> Note: With my spanish keyboard, I input the accented vowels by typing
>> the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
>> the "ç" characters, they have their own keys.
>> Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
>> in [2] I see these characters instead: ßÚݾ·±þ

> This is an exact duplicate of 14368, discussed just yesterday.

Hmm, this looks different, since it doesn't seem to use quail, but
instead relies on the "native input method".  Is it the case that the
non-GUI code in Windows receives decoded chars in `read_char', contrary
to the posix code which receives encoded chars there?


        Stefan

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmoncayo@gmail.com,  14403@debbugs.gnu.org
> Date: Tue, 14 May 2013 18:28:35 -0400
> 
> > This is an exact duplicate of 14368, discussed just yesterday.
> 
> Hmm, this looks different, since it doesn't seem to use quail

Different, yes.  But caused by the same change in keyboard.c.

> Is it the case that the non-GUI code in Windows receives decoded
> chars in `read_char', contrary to the posix code which receives
> encoded chars there?

Both GUI and non-GUI keyboard input on Windows produce Unicode
codepoints of the characters the user types.  The produced input
events have the 'kind' of either ASCII_KEYSTROKE_EVENT or
MULTIBYTE_CHAR_KEYSTROKE_EVENT.  They are returned via this call
sequence:

  read_char -> kbd_buffer_get_event -> make_lispy_event

I guess the solution is to tell read_decoded_char more about the event
that produced the character?

Or maybe we should not set the keyboard encoding to the console
codepage on Windows (although I have no idea what kind of breakage
this could cause)?  What setting of keyboard-coding-system tells the
condition below that no decoding is needed?

Of course, we could always augment this condition:

      if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
	    && (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
		& CODING_REQUIRE_DECODING_MASK)))
	return nextevt;		/* No decoding needed.  */

to do something special for MS-Windows, but that sounds kludgey.

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Stefan Monnier]

> Of course, we could always augment this condition:

>       if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
> 	    && (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
> 		& CODING_REQUIRE_DECODING_MASK)))
> 	return nextevt;		/* No decoding needed.  */

Before my change, the test was just

          (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
 	   & CODING_REQUIRE_DECODING_MASK)

but was done inside tty_read_avail_input.  AFAICT, tty_read_avail_input
is only used under POSIX ttys and MSDOS, hence this bug: FRAME_TERMCAP_P
is true for w32 non-GUI frames even though they don't use
tty_read_avail_input (they use w32_console_read_socket instead).

So a crude fix would be to check
"terminal->read_socket_hook == &tty_read_avail_input".

What would be a good test to cleanly distinguish posix ttys from w32
"ttys"?


        Stefan

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Juanma Barranquero]

On Tue, May 14, 2013 at 9:48 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> This is an exact duplicate of 14368, discussed just yesterday.

Isn't this in fact a regression to bug#12055?  I can reproduce the bug
with console codepage 850, it works OK with 1252.

    J

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Wed, 22 May 2013 22:22:47 +0200
> Cc: Dani Moncayo Melgar <dmoncayo@gmail.com>, 14403@debbugs.gnu.org
> 
> On Tue, May 14, 2013 at 9:48 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > This is an exact duplicate of 14368, discussed just yesterday.
> 
> Isn't this in fact a regression to bug#12055?

No.

> I can reproduce the bug with console codepage 850, it works OK with
> 1252.

Revert the changes in keyboard.c I pointed to and try again.  Does the
bug still happen?

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmoncayo@gmail.com, 14403@debbugs.gnu.org
> Date: Wed, 22 May 2013 12:06:47 -0400
> 
> So a crude fix would be to check
> "terminal->read_socket_hook == &tty_read_avail_input".

Yes, crude.

> What would be a good test to cleanly distinguish posix ttys from w32
> "ttys"?

How are they different?

And what about the Leim use case, which trips there as well?

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Stefan Monnier]

>> So a crude fix would be to check
>> "terminal->read_socket_hook == &tty_read_avail_input".
> Yes, crude.
>> What would be a good test to cleanly distinguish posix ttys from w32
>> "ttys"?
> How are they different?

posix ttys only emit bytes (aka encoded characters) in the event queue,
whereas according to this bug, w32 ttys emit characters (aka decoded
characters).

> And what about the Leim use case, which trips there as well?

While the Leim use case comes from the same commit, its source is
slightly different, so the two fixes are unrelated.  I'm working on
a patch for that problem, but it doesn't fix this problem.


        Stefan

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dmoncayo@gmail.com, 14403@debbugs.gnu.org
> Date: Wed, 22 May 2013 17:22:15 -0400
> 
> >> So a crude fix would be to check
> >> "terminal->read_socket_hook == &tty_read_avail_input".
> > Yes, crude.
> >> What would be a good test to cleanly distinguish posix ttys from w32
> >> "ttys"?
> > How are they different?
> 
> posix ttys only emit bytes (aka encoded characters) in the event queue,
> whereas according to this bug, w32 ttys emit characters (aka decoded
> characters).

Agree, but then shouldn't we distinguish between them based on this
difference, rather than look for some external attribute?

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Stefan Monnier]

I installed the patch below which I hope fixes this problem.


        Stefan


=== modified file 'src/ChangeLog'
--- src/ChangeLog	2013-05-22 21:35:00 +0000
+++ src/ChangeLog	2013-05-23 13:22:33 +0000
@@ -1,3 +1,7 @@
+2013-05-23  Stefan Monnier  <monnier@iro.umontreal.ca>
+
+	* keyboard.c (read_decoded_char): Don't decode under w32 (bug#14403).
+
 2013-05-22  Barry OReilly  <gundaetiapo@gmail.com>  (tiny change)
 
 	* casetab.c (init_casetab_once): Fix last change (bug#14424).

=== modified file 'src/keyboard.c'
--- src/keyboard.c	2013-04-14 20:33:57 +0000
+++ src/keyboard.c	2013-05-23 13:21:53 +0000
@@ -6827,6 +6827,8 @@
   /* XXX I think the following code should be moved to separate hook
      functions in system-dependent files.  */
 #ifdef WINDOWSNT
+  /* FIXME: AFAIK, tty_read_avail_input is not used under w32 since the non-GUI
+     code sets read_socket_hook to w32_console_read_socket instead!  */
   return 0;
 #else /* not WINDOWSNT */
   if (! tty->term_initted)      /* In case we get called during bootstrap.  */
@@ -8700,6 +8702,10 @@
     {
       Lisp_Object nextevt
 	= read_char (commandflag, map, prev_event, used_mouse_menu, NULL);
+#ifdef WINDOWSNT
+      /* w32_console already returns decoded events.  */
+      return nextevt;
+#else
       struct frame *frame = XFRAME (selected_frame);
       struct terminal *terminal = frame->terminal;
       if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
@@ -8750,6 +8756,7 @@
 	      = Fcons (events[--n], Vunread_command_events);
 	  return events[0];
 	}
+#endif
     }
 }
 

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmoncayo@gmail.com,  14403-done@debbugs.gnu.org
> Date: Thu, 23 May 2013 10:02:14 -0400
> 
> I installed the patch below which I hope fixes this problem.

Thanks.  It seems to work for me, but let's wait for the Latin-1 guys
to see if it does for them.

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Dani Moncayo]

>> I installed the patch below which I hope fixes this problem.
>
> Thanks.  It seems to work for me, but let's wait for the Latin-1 guys
> to see if it does for them.

It works fine here too.

Thank you.

--
Dani Moncayo

bug#14403: 24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

[Juanma Barranquero]

On Thu, May 23, 2013 at 7:22 PM, Dani Moncayo <dmoncayo@gmail.com> wrote:

> It works fine here too.

Same here.

   J