bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Keitaro Miyazaki]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

It is quite difficult to compose "input text" of language
such as Japanese by Input Methods like "Kotoeri" on MacOSX,
because emacs does not show clause boundary of the "input text".

I made an attached patch which improves text composition
by making a highlight on active clause of the "input text".

See also attached screen shots which shows effect of this patch:

  before.png: Before applying this patch,
              no clause boundary is shown.

  after.png:  After applying this patch,
              clause boundaries are indicated by
              a highlight on active clause.


In GNU Emacs 23.2.1 (x86_64-apple-darwin10.4.0, NS apple-appkit-1038.32)
 of 2010-10-26 on mac
Windowing system distributor `Apple', version 10.3.1038
configured using `configure  '--with-ns''

[-- Attachment #2: before.png --]
[-- Type: image/png, Size: 65339 bytes --]

[-- Attachment #3: after.png --]
[-- Type: image/png, Size: 65430 bytes --]

[-- Attachment #4: highlight-on-active-clause.diff --]
[-- Type: application/octet-stream, Size: 3773 bytes --]

diff -upr emacs-23.2.orig/lisp/term/ns-win.el emacs-23.2/lisp/term/ns-win.el
--- emacs-23.2.orig/lisp/term/ns-win.el	2010-04-04 07:26:08.000000000 +0900
+++ emacs-23.2/lisp/term/ns-win.el	2010-10-26 16:48:36.000000000 +0900
@@ -551,10 +551,19 @@ The properties returned may include `top
   "Face used to highlight working text during compose sequence insert."
   :group 'ns)
 
+(defface ns-active-clause-face
+  '((t :inherit ns-working-text-face :inverse-video t))
+  "Face used to highlight active clause of working text during compose sequence insert."
+  :group 'ns)
+
 (defvar ns-working-overlay nil
   "Overlay used to highlight working text during compose sequence insert.
 When text is in th echo area, this just stores the length of the working text.")
 
+(defvar ns-active-clause-overlay nil
+  "Overlay used to highlight active clause of working text during compose sequence insert.
+When text is in th echo area, this just stores the length of the working text.")
+
 (defvar ns-working-text)		; nsterm.m
 
 ;; Test if in echo area, based on mac-win.el 2007/08/26 unicode-2.
@@ -597,7 +606,12 @@ The overlay is assigned the face `ns-wor
     (insert ns-working-text)
     (overlay-put (setq ns-working-overlay (make-overlay start (point)
 							(current-buffer) nil t))
-		 'face 'ns-working-text-face)))
+		 'face 'ns-working-text-face)
+    (overlay-put (setq ns-active-clause-overlay
+                       (make-overlay (+ start (car ns-range-of-active-clause))
+                                     (+ start (cdr ns-range-of-active-clause))
+                                     nil t))
+                 'face 'ns-active-clause-face)))
 
 (defun ns-echo-working-text ()
   "Echo contents of `ns-working-text' in message display area.
@@ -622,7 +636,8 @@ See `ns-insert-working-text'."
     (with-current-buffer (overlay-buffer ns-working-overlay)
       (delete-region (overlay-start ns-working-overlay)
                      (overlay-end ns-working-overlay))
-      (delete-overlay ns-working-overlay)))
+      (delete-overlay ns-working-overlay)
+      (delete-overlay ns-active-clause-overlay)))
    ((integerp ns-working-overlay)
     (let ((msg (current-message))
           message-log-max)
diff -upr emacs-23.2.orig/src/nsterm.m emacs-23.2/src/nsterm.m
--- emacs-23.2.orig/src/nsterm.m	2010-04-04 07:26:04.000000000 +0900
+++ emacs-23.2/src/nsterm.m	2010-10-26 16:48:36.000000000 +0900
@@ -138,6 +138,7 @@ static unsigned convert_ns_to_X_keysym[]
 /* Lisp communications */
 Lisp_Object ns_input_file, ns_input_font, ns_input_fontsize, ns_input_line;
 Lisp_Object ns_input_color, ns_input_text, ns_working_text;
+Lisp_Object ns_range_of_active_clause;
 Lisp_Object ns_input_spi_name, ns_input_spi_arg;
 Lisp_Object Vx_toolkit_scroll_bars;
 static Lisp_Object Qmodifier_value;
@@ -4574,6 +4575,8 @@ ns_term_shutdown (int sig)
   processingCompose = YES;
   workingText = [str copy];
   ns_working_text = build_string ([workingText UTF8String]);
+  ns_range_of_active_clause = Fcons (make_number (selRange.location),
+                                     make_number (NSMaxRange (selRange)));
 
   emacs_event->kind = NS_TEXT_EVENT;
   emacs_event->code = KEY_NS_PUT_WORKING_TEXT;
@@ -6167,6 +6170,12 @@ syms_of_nsterm ()
               "String for visualizing working composition sequence.");
   ns_working_text =Qnil;
 
+  DEFVAR_LISP ("ns-range-of-active-clause", &ns_range_of_active_clause,
+              "Range of an active clause in composition sequence. \
+The value is a cons which holds start point of active clause in working \
+text as car, and end point as cdr.");
+  ns_range_of_active_clause =Qnil;
+
   DEFVAR_LISP ("ns-input-font", &ns_input_font,
               "The font specified in the last NS event.");
   ns_input_font =Qnil;

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 26 Oct 2010 21:20:13 +0900, Keitaro Miyazaki <keitaro.miyazaki@gmail.com> said:

> It is quite difficult to compose "input text" of language such as
> Japanese by Input Methods like "Kotoeri" on MacOSX, because emacs
> does not show clause boundary of the "input text".

> I made an attached patch which improves text composition by making a
> highlight on active clause of the "input text".

The members in `selRange' are in UTF-16 indices.  So they are not
necessarily coincide with character indices when the marked text
contains non-BMP characters.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[YAMAMOTO Mitsuharu]

>>>>> On Wed, 27 Oct 2010 14:46:56 +0900, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> said:

> The members in `selRange' are in UTF-16 indices.  So they are not

Oops, "they do not"...

> necessarily coincide with character indices when the marked text
> contains non-BMP characters.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Keitaro Miyazaki]

Yes, NSString counts length of string as number of Unicode characters,
and I suppose range is counted in a similar way.

     "... The length method returns the total number of Unicode
characters in the string ..."

       NSString Class Reference
       <http://developer.apple.com/library/mac/#documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/Reference/NSString.html>

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[YAMAMOTO Mitsuharu]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

>>>>> On Wed, 27 Oct 2010 23:56:47 +0900, Keitaro Miyazaki <keitaro.miyazaki@gmail.com> said:

> Yes, NSString counts length of string as number of Unicode characters,
> and I suppose range is counted in a similar way.

>      "... The length method returns the total number of Unicode
> characters in the string ..."

>        NSString Class Reference
>        <http://developer.apple.com/library/mac/#documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/Reference/NSString.html>

In this context, "number of Unicode characters" should be read as
"number of `unichar' values", where the type `unichar' is of 16-bit
width.  Actually, it is confusing.

Also, selected range handling is not enough for the following case:

  (activate Kotoeri) a i u e o left left left

The first screenshot is for the NS port with your patch, and the
second one is for my Mac port
(http://lists.gnu.org/archive/html/emacs-devel/2010-09/msg01439.html).

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

[-- Attachment #2: ns_port.png --]
[-- Type: image/png, Size: 2417 bytes --]

[-- Attachment #3: mac_port.png --]
[-- Type: image/png, Size: 2359 bytes --]

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Keitaro Miyazaki]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

Do you mean that there are some cases that the number of characters in
a string, in other words, the length of a string, is different from
the one counted by NSString to the one counted by Emacs?

If so, could you show me any example of these cases?


> Also, selected range handling is not enough for the following case:

Thank you for pointing that out.
I made an attached patch which may fix this problem
by moving cursor in composing text.

Apply this patch after applying my previous patch.

[-- Attachment #2: im-cursor-position-fix.diff --]
[-- Type: application/octet-stream, Size: 671 bytes --]

--- emacs-23.2/lisp/term/ns-win.el.orig	2010-10-28 14:57:50.000000000 +0900
+++ emacs-23.2/lisp/term/ns-win.el	2010-10-28 14:58:18.000000000 +0900
@@ -611,7 +611,8 @@ The overlay is assigned the face `ns-wor
                        (make-overlay (+ start (car ns-range-of-active-clause))
                                      (+ start (cdr ns-range-of-active-clause))
                                      nil t))
-                 'face 'ns-active-clause-face)))
+                 'face 'ns-active-clause-face)
+    (goto-char (+ start (cdr ns-range-of-active-clause)))))
 
 (defun ns-echo-working-text ()
   "Echo contents of `ns-working-text' in message display area.

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[YAMAMOTO Mitsuharu]

>>>>> On Thu, 28 Oct 2010 15:15:54 +0900, Keitaro Miyazaki <keitaro.miyazaki@gmail.com> said:

> Do you mean that there are some cases that the number of characters
> in a string, in other words, the length of a string, is different
> from the one counted by NSString to the one counted by Emacs?

> If so, could you show me any example of these cases?

I mentioned "non-BMP characters" in the previous reply.

(mapcar 'length (list (string #xffff) (string #x10000)))

=> (1 1)

#import <Cocoa/Cocoa.h>
#include <stdio.h>

main()
{
  NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
  const char str1[] = {0xef, 0xbf, 0xbf, '\0'}; // U+ffff in UTF-8
  const char str2[] = {0xf0, 0x90, 0x80, 0x80, '\0'}; // U+10000 in UTF-8
  NSString *string1 = [NSString stringWithUTF8String:str1];
  NSString *string2 = [NSString stringWithUTF8String:str2];

  printf ("%ld %ld\n", [string1 length], [string2 length]);
  [pool release];
}

=> 1 2

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Keitaro Miyazaki]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

> I mentioned "non-BMP characters" in the previous reply.

I understood that NSString counts numbers of UTF-16 units
instead of numbers of characters, so NSString counts length
of a non-BMP character as 2.

Thanks for your detailed description.


I fixed my patch to adjust the numbers in `selectedRange',
so that emacs can recognize the range properly.

Attached patch also includes the patch `im-cursor-position-fix.diff',
so you don't need to apply it now.


...BTW, I tried your Mac port today, and it seems to be working
fine for me. I hope it will be merged soon.

[-- Attachment #2: highlight-on-active-clause-v2.diff --]
[-- Type: application/octet-stream, Size: 4797 bytes --]

diff -Nrpu emacs-23.2.orig/lisp/term/ns-win.el emacs-23.2/lisp/term/ns-win.el
--- emacs-23.2.orig/lisp/term/ns-win.el	2010-04-04 07:26:08.000000000 +0900
+++ emacs-23.2/lisp/term/ns-win.el	2010-10-28 23:33:11.000000000 +0900
@@ -551,10 +551,19 @@ The properties returned may include `top
   "Face used to highlight working text during compose sequence insert."
   :group 'ns)
 
+(defface ns-active-clause-face
+  '((t :inherit ns-working-text-face :inverse-video t))
+  "Face used to highlight active clause of working text during compose sequence insert."
+  :group 'ns)
+
 (defvar ns-working-overlay nil
   "Overlay used to highlight working text during compose sequence insert.
 When text is in th echo area, this just stores the length of the working text.")
 
+(defvar ns-active-clause-overlay nil
+  "Overlay used to highlight active clause of working text during compose sequence insert.
+When text is in th echo area, this just stores the length of the working text.")
+
 (defvar ns-working-text)		; nsterm.m
 
 ;; Test if in echo area, based on mac-win.el 2007/08/26 unicode-2.
@@ -593,11 +602,20 @@ The overlay is assigned the face `ns-wor
   ;;  and if text is bound to a command, execute that instead (Bug#1453)
   (interactive)
   (ns-delete-working-text)
-  (let ((start (point)))
+  (let* ((start (point))
+         (ac-start (+ start (car ns-range-of-active-clause)))
+         (ac-end   (+ start (cdr ns-range-of-active-clause))))
     (insert ns-working-text)
     (overlay-put (setq ns-working-overlay (make-overlay start (point)
 							(current-buffer) nil t))
-		 'face 'ns-working-text-face)))
+		 'face 'ns-working-text-face)
+    (overlay-put (setq ns-active-clause-overlay
+                       (make-overlay ac-start
+                                     ac-end
+                                     nil t))
+                 'face 'ns-active-clause-face)
+    (when (= ac-start ac-end)
+      (goto-char ac-end))))
 
 (defun ns-echo-working-text ()
   "Echo contents of `ns-working-text' in message display area.
@@ -622,7 +640,8 @@ See `ns-insert-working-text'."
     (with-current-buffer (overlay-buffer ns-working-overlay)
       (delete-region (overlay-start ns-working-overlay)
                      (overlay-end ns-working-overlay))
-      (delete-overlay ns-working-overlay)))
+      (delete-overlay ns-working-overlay)
+      (delete-overlay ns-active-clause-overlay)))
    ((integerp ns-working-overlay)
     (let ((msg (current-message))
           message-log-max)
diff -Nrpu emacs-23.2.orig/src/nsterm.m emacs-23.2/src/nsterm.m
--- emacs-23.2.orig/src/nsterm.m	2010-04-04 07:26:04.000000000 +0900
+++ emacs-23.2/src/nsterm.m	2010-10-28 23:32:53.000000000 +0900
@@ -138,6 +138,7 @@ static unsigned convert_ns_to_X_keysym[]
 /* Lisp communications */
 Lisp_Object ns_input_file, ns_input_font, ns_input_fontsize, ns_input_line;
 Lisp_Object ns_input_color, ns_input_text, ns_working_text;
+Lisp_Object ns_range_of_active_clause;
 Lisp_Object ns_input_spi_name, ns_input_spi_arg;
 Lisp_Object Vx_toolkit_scroll_bars;
 static Lisp_Object Qmodifier_value;
@@ -4575,6 +4576,21 @@ ns_term_shutdown (int sig)
   workingText = [str copy];
   ns_working_text = build_string ([workingText UTF8String]);
 
+  /* NSString seems to be counting number of UTF-16 code units instead of
+     number of characters when calculating length of strings.
+     This causes NSString to count length of a non-BMP character as two,
+     not as one.
+     
+     Emacs expects that the length of a non-BMP character as one, so we
+     should adjust numbers in `selectedRange' before passing them to emacs,
+     in case of non-BMP characters are contained in the marked text. */
+  NSString *strToMin = [str substringWithRange: NSMakeRange (0, selRange.location)];
+  NSString *strToMax = [str substringWithRange: NSMakeRange (0, NSMaxRange (selRange))];
+  
+  /* Save the selected range, which was recalculated by emacs. */
+  ns_range_of_active_clause = Fcons (Flength (build_string ([strToMin UTF8String])),
+                                     Flength (build_string ([strToMax UTF8String])));
+
   emacs_event->kind = NS_TEXT_EVENT;
   emacs_event->code = KEY_NS_PUT_WORKING_TEXT;
   EV_TRAILER ((id)nil);
@@ -6167,6 +6183,12 @@ syms_of_nsterm ()
               "String for visualizing working composition sequence.");
   ns_working_text =Qnil;
 
+  DEFVAR_LISP ("ns-range-of-active-clause", &ns_range_of_active_clause,
+              "Range of an active clause in composition sequence. \
+The value is a cons which holds start point of active clause in working \
+text as car, and end point as cdr.");
+  ns_range_of_active_clause =Qnil;
+
   DEFVAR_LISP ("ns-input-font", &ns_input_font,
               "The font specified in the last NS event.");
   ns_input_font =Qnil;

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Alan Third]

Keitaro Miyazaki <keitaro.miyazaki@gmail.com> writes:
> It is quite difficult to compose "input text" of language
> such as Japanese by Input Methods like "Kotoeri" on MacOSX,
> because emacs does not show clause boundary of the "input text".

> I fixed my patch to adjust the numbers in `selectedRange',
> so that emacs can recognize the range properly.
>
> Attached patch also includes the patch `im-cursor-position-fix.diff',
> so you don't need to apply it now.

Can anyone confirm whether this is still an issue?

The patch no longer cleanly applies, but it looks like it might be
easily fixable.
-- 
Alan Third

bug#7282: 23.2; [PATCH] Improve text composition by Input Methods on MacOSX.

[Alan Third]

Alan Third <alan@idiocy.org> writes:

> Keitaro Miyazaki <keitaro.miyazaki@gmail.com> writes:
> Can anyone confirm whether this is still an issue?
>
> The patch no longer cleanly applies, but it looks like it might be
> easily fixable.

Looking at this again I just realised this was a bug for the Mac port.

Since it's no longer relevant to GNU Emacs, and nobody's responded in
over a year, I'm going to close it.
-- 
Alan Third