bug#8522: Arrow key trouble when keyboard encoding is euc-japan

bug#8522: Arrow key trouble when keyboard encoding is euc-japan

[IIJIMA Hiromitsu]

Dear all,

I am using Emacs 23.2.1 on FreeBSD and RedHat (32-bit versions).

I have configured keyboard-encoding-system be "euc-japan" because
it is the terminal's default.

But when I upgraded Emacs to ver. 23, arrow keys began to cause
troubles when I run Emacs inside a terminal window (emacs -nw).

When I move cursor with an arrow key, the cursor movement is
reflected to the screen with one stroke delay.
C-l shows the cursor's "real" position.
It does not happen when I move cursor by C-f, C-n, etc.

HOW-TO-REPEAT: Open a file and type C-x RET k euc-japan RET.

According to the following site (in Japanese),
http://slashdot.jp/~doda/journal/516198
the cause of this trouble is that arrow keys are passed to Emacs as
ESC O {A,B,C,D} sequence and this "ESC O" is interpreted as ISO/IEC
2022's SS3 (single-shift 3) code.

This trouble occurs when the following conditions are all met:
- ISO/IEC 2022 compliant or their variants
- Using SS3
- A character set designated to G3 by default

At this moment, only Japanese EUC and its variants match the conditions.

There are some encodings that use single-shifts: iso-2022-jp-2,
iso-2022-cn, and iso-2022-cn-ext. But

   - iso-2022-jp-2 and iso-2022-cn use SS2 and do not use SS3,
   and
   - iso-2022-cn-ext uses SS3 but in this encoding G3 is empty
     at the boot time.

In addition, iso-2022-cn-ext is a 7-bit encoding and therefore you can
assume that in a 8-bit encoding, namely only Japanese EUC and its
variants, a SS3 is followed by GR byte sequence, and treat the sequence
"SS3 + GL-byte" as a void character.

The site above published the following patch.
Would you consider applying it? Thanks in advance.

--- src/coding.c.orig    2010-04-04 07:26:13.000000000 +0900
+++ src/coding.c    2010-09-24 16:42:33.000000000 +0900
@@ -3853,8 +3853,14 @@
           else
         charset = CHARSET_FROM_ID (charset_id_2);
           ONE_MORE_BYTE (c1);
-          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
-        goto invalid_code;
+          if (CODING_ISO_FLAGS (coding) & CODING_ISO_FLAG_SEVEN_BITS) {
+        if (c1 < 0x20 || c1 >= 0x80)
+          goto invalid_code;
+          }
+          else {
+        if (c1 < 0xA0)
+          goto invalid_code;
+          }
           break;

         case 'O':        /* invocation of single-shift-3 */
@@ -3867,8 +3873,14 @@
           else
         charset = CHARSET_FROM_ID (charset_id_3);
           ONE_MORE_BYTE (c1);
-          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
-        goto invalid_code;
+          if (CODING_ISO_FLAGS (coding) & CODING_ISO_FLAG_SEVEN_BITS) {
+        if (c1 < 0x20 || c1 >= 0x80)
+          goto invalid_code;
+          }
+          else {
+        if (c1 < 0xA0)
+          goto invalid_code;
+          }
           break;

         case '0': case '2':    case '3': case '4': /* start composition */

-- 
========================================================================
飯嶋 浩光 / でるもんた・いいじま      delmonta@dennougedougakkai-ndd.org
(Mr.) IIJIMA, Hiromitsu  http://www.dennougedougakkai-ndd.org/~delmonta/

bug#8522: Arrow key trouble when keyboard encoding is euc-japan

[Glenn Morris]

Please could you take a look at this report?
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8522

IIJIMA Hiromitsu wrote:

> I am using Emacs 23.2.1 on FreeBSD and RedHat (32-bit versions).
>
> I have configured keyboard-encoding-system be "euc-japan" because
> it is the terminal's default.
>
> But when I upgraded Emacs to ver. 23, arrow keys began to cause
> troubles when I run Emacs inside a terminal window (emacs -nw).
>
> When I move cursor with an arrow key, the cursor movement is
> reflected to the screen with one stroke delay.
> C-l shows the cursor's "real" position.
> It does not happen when I move cursor by C-f, C-n, etc.
>
> HOW-TO-REPEAT: Open a file and type C-x RET k euc-japan RET.
>
> According to the following site (in Japanese),
> http://slashdot.jp/~doda/journal/516198
> the cause of this trouble is that arrow keys are passed to Emacs as
> ESC O {A,B,C,D} sequence and this "ESC O" is interpreted as ISO/IEC
> 2022's SS3 (single-shift 3) code.
>
> This trouble occurs when the following conditions are all met:
> - ISO/IEC 2022 compliant or their variants
> - Using SS3
> - A character set designated to G3 by default
>
> At this moment, only Japanese EUC and its variants match the conditions.
>
> There are some encodings that use single-shifts: iso-2022-jp-2,
> iso-2022-cn, and iso-2022-cn-ext. But
>
>   - iso-2022-jp-2 and iso-2022-cn use SS2 and do not use SS3,
>   and
>   - iso-2022-cn-ext uses SS3 but in this encoding G3 is empty
>     at the boot time.
>
> In addition, iso-2022-cn-ext is a 7-bit encoding and therefore you can
> assume that in a 8-bit encoding, namely only Japanese EUC and its
> variants, a SS3 is followed by GR byte sequence, and treat the sequence
> "SS3 + GL-byte" as a void character.
>
> The site above published the following patch.
> Would you consider applying it? Thanks in advance.
>
> --- src/coding.c.orig    2010-04-04 07:26:13.000000000 +0900
> +++ src/coding.c    2010-09-24 16:42:33.000000000 +0900
> @@ -3853,8 +3853,14 @@
>           else
>         charset = CHARSET_FROM_ID (charset_id_2);
>           ONE_MORE_BYTE (c1);
> -          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
> -        goto invalid_code;
> +          if (CODING_ISO_FLAGS (coding) & CODING_ISO_FLAG_SEVEN_BITS) {
> +        if (c1 < 0x20 || c1 >= 0x80)
> +          goto invalid_code;
> +          }
> +          else {
> +        if (c1 < 0xA0)
> +          goto invalid_code;
> +          }
>           break;
>
>         case 'O':        /* invocation of single-shift-3 */
> @@ -3867,8 +3873,14 @@
>           else
>         charset = CHARSET_FROM_ID (charset_id_3);
>           ONE_MORE_BYTE (c1);
> -          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
> -        goto invalid_code;
> +          if (CODING_ISO_FLAGS (coding) & CODING_ISO_FLAG_SEVEN_BITS) {
> +        if (c1 < 0x20 || c1 >= 0x80)
> +          goto invalid_code;
> +          }
> +          else {
> +        if (c1 < 0xA0)
> +          goto invalid_code;
> +          }
>           break;
>
>         case '0': case '2':    case '3': case '4': /* start composition */

bug#8522: Arrow key trouble when keyboard encoding is euc-japan

[Handa Kenichi]

In article <70ppui8gr4.fsf@fencepost.gnu.org>, Glenn Morris
<rgm@gnu.org> writes:

> Please could you take a look at this report?
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8522

Ok.  Please wait for a while.

---
Kenichi Handa
handa@gnu.org

> IIJIMA Hiromitsu wrote:

> > I am using Emacs 23.2.1 on FreeBSD and RedHat (32-bit versions).
> >
> > I have configured keyboard-encoding-system be "euc-japan" because
> > it is the terminal's default.
> >
> > But when I upgraded Emacs to ver. 23, arrow keys began to cause
> > troubles when I run Emacs inside a terminal window (emacs -nw).
> >
> > When I move cursor with an arrow key, the cursor movement is
> > reflected to the screen with one stroke delay.
> > C-l shows the cursor's "real" position.
> > It does not happen when I move cursor by C-f, C-n, etc.
> >
> > HOW-TO-REPEAT: Open a file and type C-x RET k euc-japan RET.
> >
> > According to the following site (in Japanese),
> > http://slashdot.jp/~doda/journal/516198
> > the cause of this trouble is that arrow keys are passed to Emacs as
> > ESC O {A,B,C,D} sequence and this "ESC O" is interpreted as ISO/IEC
> > 2022's SS3 (single-shift 3) code.
> >
> > This trouble occurs when the following conditions are all met:
> > - ISO/IEC 2022 compliant or their variants
> > - Using SS3
> > - A character set designated to G3 by default
> >
> > At this moment, only Japanese EUC and its variants match the
> > conditions.
> >
> > There are some encodings that use single-shifts: iso-2022-jp-2,
> > iso-2022-cn, and iso-2022-cn-ext. But
> >
> >   - iso-2022-jp-2 and iso-2022-cn use SS2 and do not use SS3,
> >   and
> >   - iso-2022-cn-ext uses SS3 but in this encoding G3 is empty
> >     at the boot time.
> >
> > In addition, iso-2022-cn-ext is a 7-bit encoding and therefore you
> > can
> > assume that in a 8-bit encoding, namely only Japanese EUC and its
> > variants, a SS3 is followed by GR byte sequence, and treat the
> > sequence
> > "SS3 + GL-byte" as a void character.
> >
> > The site above published the following patch.
> > Would you consider applying it? Thanks in advance.
> >
> > --- src/coding.c.orig    2010-04-04 07:26:13.000000000 +0900
> > +++ src/coding.c    2010-09-24 16:42:33.000000000 +0900
> > @@ -3853,8 +3853,14 @@
> >           else
> >         charset = CHARSET_FROM_ID (charset_id_2);
> >           ONE_MORE_BYTE (c1);
> > -          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
> > -        goto invalid_code;
> > +          if (CODING_ISO_FLAGS (coding) &
> > CODING_ISO_FLAG_SEVEN_BITS) {
> > +        if (c1 < 0x20 || c1 >= 0x80)
> > +          goto invalid_code;
> > +          }
> > +          else {
> > +        if (c1 < 0xA0)
> > +          goto invalid_code;
> > +          }
> >           break;
> >
> >         case 'O':        /* invocation of single-shift-3 */
> > @@ -3867,8 +3873,14 @@
> >           else
> >         charset = CHARSET_FROM_ID (charset_id_3);
> >           ONE_MORE_BYTE (c1);
> > -          if (c1 < 0x20 || (c1 >= 0x80 && c1 < 0xA0))
> > -        goto invalid_code;
> > +          if (CODING_ISO_FLAGS (coding) &
> > CODING_ISO_FLAG_SEVEN_BITS) {
> > +        if (c1 < 0x20 || c1 >= 0x80)
> > +          goto invalid_code;
> > +          }
> > +          else {
> > +        if (c1 < 0xA0)
> > +          goto invalid_code;
> > +          }
> >           break;
> >
> >         case '0': case '2':    case '3': case '4': /* start
> >         composition */

bug#8522: Arrow key trouble when keyboard encoding is euc-japan

[Handa Kenichi]

In article <70ppui8gr4.fsf@fencepost.gnu.org>, Glenn Morris <rgm@gnu.org> writes:

> Please could you take a look at this report?
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8522

I've just installed a little bit different fix.

> IIJIMA Hiromitsu wrote:
[...]
> > According to the following site (in Japanese),
> > http://slashdot.jp/~doda/journal/516198
> > the cause of this trouble is that arrow keys are passed to Emacs as
> > ESC O {A,B,C,D} sequence and this "ESC O" is interpreted as ISO/IEC
> > 2022's SS3 (single-shift 3) code.
> >
> > This trouble occurs when the following conditions are all met:
> > - ISO/IEC 2022 compliant or their variants
> > - Using SS3
> > - A character set designated to G3 by default
> >
> > At this moment, only Japanese EUC and its variants match the
> > conditions.
> >
> > There are some encodings that use single-shifts: iso-2022-jp-2,
> > iso-2022-cn, and iso-2022-cn-ext. But
> >
> >   - iso-2022-jp-2 and iso-2022-cn use SS2 and do not use SS3,
> >   and
> >   - iso-2022-cn-ext uses SS3 but in this encoding G3 is empty
> >     at the boot time.
> >
> > In addition, iso-2022-cn-ext is a 7-bit encoding and therefore you
> > can
> > assume that in a 8-bit encoding, namely only Japanese EUC and its
> > variants, a SS3 is followed by GR byte sequence, and treat the
> > sequence
> > "SS3 + GL-byte" as a void character.

That analysis is correct.  But, as ISO 2022 has a concept of
"implementation level" which specifies which graphic plane
(GL or GR) is indentified as the single-shift area, I
introduced a new flag `8-bit-level-4' for ISO-2022 base
coding system.  By default, that flag is not set, thus the
implementation level of the 2022 decoder is "4A" which means
GR is identified as the single-shift area.  So, the cursor
key sequences ESC O A, etc are not recognized as a valid
single-shift sequence, and thus those sequences are given to
the normal key-sequence look-up mechanism.

---
Kenichi Handa
handa@gnu.org

bug#8522: Arrow key trouble when keyboard encoding is euc-japan

[Hiroki Sato]

[-- Attachment #1: Type: Text/Plain, Size: 313 bytes --]

Hi,

 I could reproduce the problem in #8522 and am using Emacs 24.3 with
 the patch in [*] (basically the same as one submitted in #8522) on
 FreeBSD.  It is working fine with no side effect.

 [*] http://svnweb.freebsd.org/ports/head/editors/emacs/files/patch-src_coding.c?view=markup&pathrev=281397

-- Hiroki

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]