bug#66970: 29.1; Overwrite-mode is not working with Korean characters

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[SUKBEOM KIM]

Hi,

I found a bug with typing Korean characters that "overwrite-mode" is not
actually working well. (For Japanese/Chinese characters, there is no
issue at all.)

aeoiu
=> あえu
=> 아에aeiou

In the thread, one of the Japanese users reported a bug but it is not
appropriate for Korean characters which requires a combination of
sub-characters to make a character unlike other Japanese or Chinese
characters.

- https://lists.nongnu.org/archive/html/emacs-devel/2015-05/msg00039.html

Important settings:
   value of $LC_ALL: en_US.UTF-8
   value of $LANG: C.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   overwrite-mode: overwrite-mode-textual

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cus-start
cus-load hangul hanja-util quail comp comp-cstr warnings icons subr-x rx
cl-seq cl-macs gv cl-extra help-mode cl-loaddefs cl-lib bytecomp
byte-compile rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 164922 8869)
  (symbols 48 8289 0)
  (strings 32 40752 3578)
  (string-bytes 1 1149899)
  (vectors 16 19161)
  (vector-slots 8 319338 11606)
  (floats 8 34 31)
  (intervals 56 269 0)
  (buffers 984 12))

bug#66970: (no subject)

[SUKBEOM KIM]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

I forgot to provide the test sequence to reproduce the issue:

 1.

    type 'aeiou' and <C-a>

 2.

    <M-x> set-input-method and set to korean-hangul (2-beoksik)

 3.

    <M-x> overwrite-mode.
      Replace 'ae' to '아' and 'oi' to '에'. You can type them with typing
    'dk', 'dp'.

 4.

    I expected '아에u', but the actual result is '아에aeiou'.
    Replacing Korean characters with alphabets is working but not vice
    versa.

[-- Attachment #2: Type: text/html, Size: 1391 bytes --]

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Eli Zaretskii]

tags 66970 moreinfo
thanks

> Date: Tue, 7 Nov 2023 01:07:00 +0900
> From: SUKBEOM KIM <sukbeom.kim@gmail.com>
> 
> I found a bug with typing Korean characters that "overwrite-mode" is not
> actually working well. (For Japanese/Chinese characters, there is no
> issue at all.)
> 
> aeoiu
> => あえu
> => 아에aeiou

Please provide a full recipe, starting from "emacs -Q", including from
what text to start and which characters to type.  I'm not sure I
understand the situation in enough detail to investigate.

> In the thread, one of the Japanese users reported a bug but it is not
> appropriate for Korean characters which requires a combination of
> sub-characters to make a character unlike other Japanese or Chinese
> characters.

To investigate, we need to see an example of characters that are
composed in the Korean case, and then are overwritten.  AFAIU, the
text you show above doesn't include character composition.

Emacs 29 also has the new variable composition-break-at-point, which
perhaps could be useful in at least some use cases of overwriting
Korean text (and any other text where characters are composed).

Thanks.

bug#66970: (no subject)

[Eli Zaretskii]

> Date: Tue, 7 Nov 2023 01:37:43 +0900
> From: SUKBEOM KIM <sukbeom.kim@gmail.com>
> 
> I forgot to provide the test sequence to reproduce the issue:
> 
> 1 type 'aeiou' and <C-a>
> 
> 2 <M-x> set-input-method and set to korean-hangul (2-beoksik)
> 
> 3 <M-x> overwrite-mode.
>   Replace 'ae' to '아' and 'oi' to '에'. You can type them with typing 'dk', 'dp'.
> 
> 4 I expected '아에u', but the actual result is '아에aeiou'.
>  Replacing Korean characters with alphabets is working but not vice versa.

What happens if you type the Korean characters using a Korean
keyboard, not via a Leim input method?  Does it also not work?  Or is
the problem only with typing via an input method?

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[SUKBEOM KIM]

Yes, it is the problem only with typing via an input method.
I checked that there is no issue if I type Korean characters with iBus 
(I guess this is the 'LEIM' what you mentioned).

>> Date: Tue, 7 Nov 2023 01:37:43 +0900
>> From: SUKBEOM KIM <sukbeom.kim@gmail.com>
>>
>> I forgot to provide the test sequence to reproduce the issue:
>>
>> 1 type 'aeiou' and <C-a>
>>
>> 2 <M-x> set-input-method and set to korean-hangul (2-beoksik)
>>
>> 3 <M-x> overwrite-mode.
>>    Replace 'ae' to '아' and 'oi' to '에'. You can type them with typing 'dk', 'dp'.
>>
>> 4 I expected '아에u', but the actual result is '아에aeiou'.
>>   Replacing Korean characters with alphabets is working but not vice versa.
> What happens if you type the Korean characters using a Korean
> keyboard, not via a Leim input method?  Does it also not work?  Or is
> the problem only with typing via an input method?

bug#66970: (no subject)

[Sukbeom Kim]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

The last comment was quite confusing. So I would like to re-summarize the
result.

   - Emacs's LEIM => overwrite-mode is not working.
   - Input Method (iBus) => ok


2023년 11월 7일 (화) 오전 2:04, Eli Zaretskii <eliz@gnu.org>님이 작성:

> > Date: Tue, 7 Nov 2023 01:37:43 +0900
> > From: SUKBEOM KIM <sukbeom.kim@gmail.com>
> >
> > I forgot to provide the test sequence to reproduce the issue:
> >
> > 1 type 'aeiou' and <C-a>
> >
> > 2 <M-x> set-input-method and set to korean-hangul (2-beoksik)
> >
> > 3 <M-x> overwrite-mode.
> >   Replace 'ae' to '아' and 'oi' to '에'. You can type them with typing
> 'dk', 'dp'.
> >
> > 4 I expected '아에u', but the actual result is '아에aeiou'.
> >  Replacing Korean characters with alphabets is working but not vice
> versa.
>
> What happens if you type the Korean characters using a Korean
> keyboard, not via a Leim input method?  Does it also not work?  Or is
> the problem only with typing via an input method?
>

[-- Attachment #2: Type: text/html, Size: 1546 bytes --]

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Eli Zaretskii]

> Date: Tue, 7 Nov 2023 08:19:53 +0900
> From: SUKBEOM KIM <sukbeom.kim@gmail.com>
> 
> Yes, it is the problem only with typing via an input method.
> I checked that there is no issue if I type Korean characters with iBus 
> (I guess this is the 'LEIM' what you mentioned).

It looks like the Korean input methods in Leim don't support Overwrite
mode: they use 'insert' (see 'hangul-insert-character'), whereas
Overwrite mode is supported only by self-insert-command, which
requires to push the characters on the unread-command-events instead
of explicitly inserting them.

Stefan, Kenichi: any ideas or suggestions for how to fix this?

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[João Távora]

Eli brought up this bug in emacs-devel and I thought
I would give it a shot.

I had never touched this code or anything like it, but I do
use input methods a bit for portuguese.  Let me know what
you think.

diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
index 46a2e5a6ba2..7e6c7310179 100644
--- a/lisp/leim/quail/hangul.el
+++ b/lisp/leim/quail/hangul.el
@@ -146,21 +146,34 @@ hangul-insert-character
       (progn
         (delete-region (region-beginning) (region-end))
         (deactivate-mark)))
-  (quail-delete-region)
-  (let ((first (car queues)))
-    (insert
-     (hangul-character
-      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
-      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
-      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
-  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
-  (dolist (queue (cdr queues))
-    (insert
-     (hangul-character
-      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
-      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
-      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
-    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
+  (let* ((chars-to-insert
+          (with-temp-buffer
+            (dolist (queue queues (mapcar #'identity (buffer-string)))
+              (insert
+               (hangul-character
+                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
(aref queue 1)))
+                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
(aref queue 3)))
+                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
(aref queue 5))))))))
+         (may-have-to-overwrite-p
+          (or
+           ;; If the overlay isn't showing (i.e. it has 0 length) then
+           ;; we may want to insert char overwriting (iff overwrite-mode is
+           ;; non-nil, of course)
+           (= (overlay-start quail-overlay) (overlay-end quail-overlay))
+           ;; Likewise we want to do it if there is more then one
+           ;; character that were combined.
+           (cdr chars-to-insert))))
+    (quail-delete-region) ; this kills the overlay
+    (dolist (c chars-to-insert)
+      (let ((last-command-event c)
+            (overwrite-mode (and overwrite-mode
+                                 may-have-to-overwrite-p
+                                 overwrite-mode)))
+        (self-insert-command 1)
+        ;; TODO: maybe use cl-loop for an easier-to-read loop?
+        (setq may-have-to-overwrite-p nil)))
+    ; this revives it (TODO: do we really always revive?)
+    (move-overlay quail-overlay (1- (point)) (point))))

 (defun hangul-djamo (jamo char1 char2)
   "Return the double Jamo index calculated from the arguments.



-- 
João Távora

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 16:37:17 +0000
> 
> Eli brought up this bug in emacs-devel and I thought
> I would give it a shot.
> 
> I had never touched this code or anything like it, but I do
> use input methods a bit for portuguese.  Let me know what
> you think.

Thanks.

Sukbeom, could you please try the patch and see if it gives good
results, both with and without Overwrite mode?

> diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
> index 46a2e5a6ba2..7e6c7310179 100644
> --- a/lisp/leim/quail/hangul.el
> +++ b/lisp/leim/quail/hangul.el
> @@ -146,21 +146,34 @@ hangul-insert-character
>        (progn
>          (delete-region (region-beginning) (region-end))
>          (deactivate-mark)))
> -  (quail-delete-region)
> -  (let ((first (car queues)))
> -    (insert
> -     (hangul-character
> -      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
> -      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
> -      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
> -  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
> -  (dolist (queue (cdr queues))
> -    (insert
> -     (hangul-character
> -      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
> -      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
> -      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
> -    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
> +  (let* ((chars-to-insert
> +          (with-temp-buffer
> +            (dolist (queue queues (mapcar #'identity (buffer-string)))
> +              (insert
> +               (hangul-character
> +                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
> (aref queue 1)))
> +                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
> (aref queue 3)))
> +                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
> (aref queue 5))))))))
> +         (may-have-to-overwrite-p
> +          (or
> +           ;; If the overlay isn't showing (i.e. it has 0 length) then
> +           ;; we may want to insert char overwriting (iff overwrite-mode is
> +           ;; non-nil, of course)
> +           (= (overlay-start quail-overlay) (overlay-end quail-overlay))
> +           ;; Likewise we want to do it if there is more then one
> +           ;; character that were combined.
> +           (cdr chars-to-insert))))
> +    (quail-delete-region) ; this kills the overlay
> +    (dolist (c chars-to-insert)
> +      (let ((last-command-event c)
> +            (overwrite-mode (and overwrite-mode
> +                                 may-have-to-overwrite-p
> +                                 overwrite-mode)))
> +        (self-insert-command 1)
> +        ;; TODO: maybe use cl-loop for an easier-to-read loop?
> +        (setq may-have-to-overwrite-p nil)))
> +    ; this revives it (TODO: do we really always revive?)
> +    (move-overlay quail-overlay (1- (point)) (point))))
> 
>  (defun hangul-djamo (jamo char1 char2)
>    "Return the double Jamo index calculated from the arguments.
> 
> 
> 
> -- 
> João Távora
> 

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Sukbeom Kim]

Thanks for the patch.

It seems that the issue within the overwrite-mode can be resolved with
the patch.

Anyway, I have one more question. I think there is a similar issue with
org-edit-special (table.el). I thought it would be resolved if the
overwrite-mode was fixed but it seems not.

Should I report it with a new bug ticket? Or can I describe it in this
ticket?

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Thu, 9 Nov 2023 16:37:17 +0000
>>
>> Eli brought up this bug in emacs-devel and I thought
>> I would give it a shot.
>>
>> I had never touched this code or anything like it, but I do
>> use input methods a bit for portuguese.  Let me know what
>> you think.
>
> Thanks.
>
> Sukbeom, could you please try the patch and see if it gives good
> results, both with and without Overwrite mode?
>
>> diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
>> index 46a2e5a6ba2..7e6c7310179 100644
>> --- a/lisp/leim/quail/hangul.el
>> +++ b/lisp/leim/quail/hangul.el
>> @@ -146,21 +146,34 @@ hangul-insert-character
>>        (progn
>>          (delete-region (region-beginning) (region-end))
>>          (deactivate-mark)))
>> -  (quail-delete-region)
>> -  (let ((first (car queues)))
>> -    (insert
>> -     (hangul-character
>> -      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
>> -      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
>> -      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
>> -  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
>> -  (dolist (queue (cdr queues))
>> -    (insert
>> -     (hangul-character
>> -      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
>> -      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
>> -      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
>> -    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
>> +  (let* ((chars-to-insert
>> +          (with-temp-buffer
>> +            (dolist (queue queues (mapcar #'identity (buffer-string)))
>> +              (insert
>> +               (hangul-character
>> +                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
>> (aref queue 1)))
>> +                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
>> (aref queue 3)))
>> +                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
>> (aref queue 5))))))))
>> +         (may-have-to-overwrite-p
>> +          (or
>> +           ;; If the overlay isn't showing (i.e. it has 0 length) then
>> +           ;; we may want to insert char overwriting (iff overwrite-mode is
>> +           ;; non-nil, of course)
>> +           (= (overlay-start quail-overlay) (overlay-end quail-overlay))
>> +           ;; Likewise we want to do it if there is more then one
>> +           ;; character that were combined.
>> +           (cdr chars-to-insert))))
>> +    (quail-delete-region) ; this kills the overlay
>> +    (dolist (c chars-to-insert)
>> +      (let ((last-command-event c)
>> +            (overwrite-mode (and overwrite-mode
>> +                                 may-have-to-overwrite-p
>> +                                 overwrite-mode)))
>> +        (self-insert-command 1)
>> +        ;; TODO: maybe use cl-loop for an easier-to-read loop?
>> +        (setq may-have-to-overwrite-p nil)))
>> +    ; this revives it (TODO: do we really always revive?)
>> +    (move-overlay quail-overlay (1- (point)) (point))))
>>
>>  (defun hangul-djamo (jamo char1 char2)
>>    "Return the double Jamo index calculated from the arguments.
>>
>>
>>
>> --
>> João Távora
>>

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Eli Zaretskii]

> From: Sukbeom Kim <sukbeom.kim@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,
>  66970@debbugs.gnu.org
> Date: Fri, 10 Nov 2023 09:34:11 +0900
> 
> Thanks for the patch.
> 
> It seems that the issue within the overwrite-mode can be resolved with
> the patch.

Thanks for testing.  João, please install this, and thanks.

> Anyway, I have one more question. I think there is a similar issue with
> org-edit-special (table.el). I thought it would be resolved if the
> overwrite-mode was fixed but it seems not.
> 
> Should I report it with a new bug ticket? Or can I describe it in this
> ticket?

Is this in Org?  I don't see any references to org-edit-special in
lisp/textmodes/table.el, if that is what you meant.

If this is about Org, I suggest to submit a bug report about this to
the Org developers, since they are those who develop and maintain
org-edit-special (AFAIU).

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Sukbeom Kim]

Okay. I will report about that to Org developers.

Thank you.

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sukbeom Kim <sukbeom.kim@gmail.com>
>> Cc: João Távora <joaotavora@gmail.com>,
>>  66970@debbugs.gnu.org
>> Date: Fri, 10 Nov 2023 09:34:11 +0900
>>
>> Thanks for the patch.
>>
>> It seems that the issue within the overwrite-mode can be resolved with
>> the patch.
>
> Thanks for testing.  João, please install this, and thanks.
>
>> Anyway, I have one more question. I think there is a similar issue with
>> org-edit-special (table.el). I thought it would be resolved if the
>> overwrite-mode was fixed but it seems not.
>>
>> Should I report it with a new bug ticket? Or can I describe it in this
>> ticket?
>
> Is this in Org?  I don't see any references to org-edit-special in
> lisp/textmodes/table.el, if that is what you meant.
>
> If this is about Org, I suggest to submit a bug report about this to
> the Org developers, since they are those who develop and maintain
> org-edit-special (AFAIU).

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[João Távora]

On Fri, Nov 10, 2023 at 7:13 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Sukbeom Kim <sukbeom.kim@gmail.com>
> > Cc: João Távora <joaotavora@gmail.com>,
> >  66970@debbugs.gnu.org
> > Date: Fri, 10 Nov 2023 09:34:11 +0900
> >
> > Thanks for the patch.
> >
> > It seems that the issue within the overwrite-mode can be resolved with
> > the patch.
>
> Thanks for testing.  João, please install this, and thanks.

Done in 8090ab05434.  Typo in the commit subject though, sorry about that.

João

bug#66970: 29.1; Overwrite-mode is not working with Korean characters

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 13 Nov 2023 12:41:42 +0000
> Cc: Sukbeom Kim <sukbeom.kim@gmail.com>, 66970@debbugs.gnu.org
> 
> > Thanks for testing.  João, please install this, and thanks.
> 
> Done in 8090ab05434.  Typo in the commit subject though, sorry about that.

Thanks, closing.