quail input methods and conversion functions

quail input methods and conversion functions

[Uwe Brauer]

Hello


In ancient pre Mule times, there was iso-unacentuate which for 
example would translatex ä to "a


Does something like this exist for all, or some quail input 
methods?

For example take greek-ibycus4 then
a) gives ἀ


So the function I am looking for would 

a) <--> ἀ

I think I know how to do that for this particular input method, 
but I don't want to reinvent the wheel. So any pointer would be 
wellcome.

Uwe Brauer 

Re: quail input methods and conversion functions

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

> In ancient pre Mule times, there was iso-unacentuate which for example
> would translatex ä to "a
>
>
> Does something like this exist for all, or some quail input methods?

It seems to exist.  The interactive version is `M-x quail-show-key', and
the programmatic version is `quail-find-key'.  For example, using the
latin-1-prefix input method, M-x quail-show-key on ä outputs:

  To input ‘ä’, type ""a"

and (quail-find-key ?ä) -> ("\"a")

Both of these depend on the current input method, of course.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: quail input methods and conversion functions

[Uwe Brauer]

>>> "Michael" == Michael Welsh Duggan <mwd@md5i.com> writes:

   > Uwe Brauer <oub@mat.ucm.es> writes:
   >> In ancient pre Mule times, there was iso-unacentuate which for example
   >> would translatex ä to "a
   >> 
   >> 
   >> Does something like this exist for all, or some quail input methods?

   > It seems to exist.  The interactive version is `M-x quail-show-key', and
   > the programmatic version is `quail-find-key'.  For example, using the
   > latin-1-prefix input method, M-x quail-show-key on ä outputs:

   >   To input ‘ä’, type ""a"

   > and (quail-find-key ?ä) -> ("\"a")

   > Both of these depend on the current input method, of course.

Thanks, but these functions do not convert globally, that is for a
region or file, between the two sets.

I mean suppose I have

 Über den Wolken, unter dem Öl,

Should give when converted

"Uber den Wolken, unter dem "Ol.

And vice versa, same way as iso-accentuate and iso-unaccentuate
worked/s.

Re: quail input methods and conversion functions

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

>>>> "Michael" == Michael Welsh Duggan <mwd@md5i.com> writes:
>
>    > Uwe Brauer <oub@mat.ucm.es> writes:
>    >> In ancient pre Mule times, there was iso-unacentuate which for example
>    >> would translatex ä to "a
>    >> 
>    >> 
>    >> Does something like this exist for all, or some quail input methods?
>
>    > It seems to exist.  The interactive version is `M-x quail-show-key', and
>    > the programmatic version is `quail-find-key'.  For example, using the
>    > latin-1-prefix input method, M-x quail-show-key on ä outputs:
>
>    >   To input ‘ä’, type ""a"
>
>    > and (quail-find-key ?ä) -> ("\"a")
>
>    > Both of these depend on the current input method, of course.
>
> Thanks, but these functions do not convert globally, that is for a
> region or file, between the two sets.
>
> I mean suppose I have
>
>  Über den Wolken, unter dem Öl,
>
> Should give when converted
>
> "Uber den Wolken, unter dem "Ol.
>
> And vice versa, same way as iso-accentuate and iso-unaccentuate
> worked/s.

In the unaccentuate case (which is what your original question), it
should be easy to write that functionality using `quail-find-key' as a
base.  You would have to decide how to handle failures (character can't
be typed using the current input method).

The accentuate case looks more difficult.  Quail allows more than one
translation candidate for a single key sequence, which leaves no good
way handle these characters.  A simple example from the ipa input
method: tsh can translate into ʧ, tʃ, or t⁀ʃ.  Something that works with
input methods that do not have this ambiguity is possible, though my
initial look through the code leads me to believe that this is still
going to be more difficult to write then the (fairly trivial)
unaccentuate case.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

[conversion fails] (was: quail input methods and conversion functions)

[Uwe Brauer]

   > Uwe Brauer <oub@mat.ucm.es> writes:

   > In the unaccentuate case (which is what your original question), it
   > should be easy to write that functionality using `quail-find-key'
   > as a base. You would have to decide how to handle failures
   > (character can't be typed using the current input method).

Actually taking iso-accentuate and un accuate as a reference it is the
accentuate part which I want to implement, for example
for the greek-ibycus

I want to replace    a)  by  ἀ


   > The accentuate case looks more difficult.  Quail allows more than one
   > translation candidate for a single key sequence, which leaves no good

I am puzzled now, I thought any input method should be an bijection. So
the old iso method was, wasn't it?


   > way handle these characters.  A simple example from the ipa input
   > method: tsh can translate into ʧ, tʃ, or t⁀ʃ.

I tried that out, but I obtain  ʧ none of the others.


   > Something that works with
   > input methods that do not have this ambiguity is possible, though my
   > initial look through the code leads me to believe that this is still
   > going to be more difficult to write then the (fairly trivial)
   > unaccentuate case.

I boldly took the code from iso accentuate/accentuate

And define 
(defun ibycus4-translate-conventions (from to trans-tab)
  "Translate between FROM and TO using the translation table TRANS-TAB.
  Based on iso-cvt"
  (save-excursion
    (save-restriction
      (narrow-to-region from to)
      (goto-char from)
      (let ((work-tab trans-tab)
	    (buffer-read-only nil)
	    (case-fold-search nil))
	(while work-tab
	  (save-excursion
	    (let ((trans-this (car work-tab)))
	      (while (re-search-forward (car trans-this) nil t)
		(replace-match (car (cdr trans-this)) t nil)))
	    (setq work-tab (cdr work-tab)))))
      (point-max))))

Then my table starts like this
(defvar ibycus4-ibycus4-to-utf8-trans-tab
  '(
   ("a)" "ἀ")
   ("a(" "ἁ")
   ("a)`" "ἂ")
   ("a(`" "ἃ")
   ("a)'" "ἄ")
   ("a('" "ἅ")))

But now surprise, surprise


   ("ἀ" "ἀ") correct
   ("ἁ" "ἁ") correct
   ("ἀ‘" "ἂ")wrong
   ("ἁ‘" "ἃ")wrong 
   ("ἀ’" "ἄ")wrong 
   ("ἁ’" "ἅ") wrong


So what is up here and what shall I do? Any help is now strongly
appreciated.

Re: [conversion fails]

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

>    > Uwe Brauer <oub@mat.ucm.es> writes:
>
>    > In the unaccentuate case (which is what your original question), it
>    > should be easy to write that functionality using `quail-find-key'
>    > as a base. You would have to decide how to handle failures
>    > (character can't be typed using the current input method).
>
> Actually taking iso-accentuate and un accuate as a reference it is the
> accentuate part which I want to implement, for example
> for the greek-ibycus
>
> I want to replace    a)  by  ἀ

Okay.

>    > The accentuate case looks more difficult.  Quail allows more than one
>    > translation candidate for a single key sequence, which leaves no good
>
> I am puzzled now, I thought any input method should be an bijection. So
> the old iso method was, wasn't it?

Not all input methods are bijections.  Especially not many of the Asian
input methods.

>    > way handle these characters.  A simple example from the ipa input
>    > method: tsh can translate into ʧ, tʃ, or t⁀ʃ.
>
> I tried that out, but I obtain  ʧ none of the others.

Really?  Try typing the following using the ipa input method:

  tsh1tsh2tsh3

The number selects the specific form.  If something else it typed rather
than the number, it will use the last remembered version, defaulting to
version 1.

I hacked up the following, which is minimally tested, but may work for
you.  I tried to use public functions from quail, but the (null (cdr
map)) is probably an implementation detail, and might break in a future
version.  There are almost certainly better ways to write this function.
If presented with an ambiguity (like above), it will choose the last
remembered variant (defaulting to 1).

(defun accentuate-region (start end)
  "Replace the region by using it as keys using the current input method"
  (interactive "r")
  (save-excursion 
    (goto-char start)
    (let* ((data (delete-and-extract-region start end))
           (size (length data))
           (idx 0) (cand ""))
      (while (< idx size)
        (let* ((next-cand (concat cand (list (aref data idx))))
               (map (quail-lookup-key next-cand))
               (def (and map (quail-map-definition map))))
          (if (null map)
              (progn
                (insert next-cand)
                (setq cand ""))
            (if (and def (null (cdr map)))
                (progn
                  (insert (quail-get-current-str (length next-cand) def))
                  (setq cand "")) 
              (setq cand next-cand)))
          (incf idx)))
      (insert cand))))

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: [conversion fails]

[Uwe Brauer]

   > Uwe Brauer <oub@mat.ucm.es> writes:

   > Okay.


   > Not all input methods are bijections.  Especially not many of the Asian
   > input methods.


   > Really?  Try typing the following using the ipa input method:

   >   tsh1tsh2tsh3

   > The number selects the specific form.  If something else it typed rather
   > than the number, it will use the last remembered version, defaulting to
   > version 1.

Ok I will 
   > I hacked up the following, which is minimally tested, but may work for
   > you.  I tried to use public functions from quail, but the (null (cdr
   > map)) is probably an implementation detail, and might break in a future
   > version.  There are almost certainly better ways to write this function.
   > If presented with an ambiguity (like above), it will choose the last
   > remembered variant (defaulting to 1).


Thanks very much, this looks very promising. I will start testing and
report back!


Uwe 

Re: [conversion fails]

[Uwe Brauer]

   > Uwe Brauer <oub@mat.ucm.es> writes:

   > Okay.



   > I hacked up the following, which is minimally tested, but may work for
   > you.  I tried to use public functions from quail, but the (null (cdr
   > map)) is probably an implementation detail, and might break in a future
   > version.  There are almost certainly better ways to write this function.
   > If presented with an ambiguity (like above), it will choose the last
   > remembered variant (defaulting to 1).

   > (defun accentuate-region (start end)
   >   "Replace the region by using it as keys using the current input method"
   >   (interactive "r")
   >   (save-excursion 
   >     (goto-char start)
   >     (let* ((data (delete-and-extract-region start end))
   >            (size (length data))
   >            (idx 0) (cand ""))
   >       (while (< idx size)
   >         (let* ((next-cand (concat cand (list (aref data idx))))
   >                (map (quail-lookup-key next-cand))
   >                (def (and map (quail-map-definition map))))
   >           (if (null map)
   >               (progn
   >                 (insert next-cand)
   >                 (setq cand ""))
   >             (if (and def (null (cdr map)))
   >                 (progn
   >                   (insert (quail-get-current-str (length next-cand) def))
   >                   (setq cand "")) 
   >               (setq cand next-cand)))
   >           (incf idx)))
   >       (insert cand))))

I tested it with latin-1-pre and it worked, but I tried also
greek-ibycus4

And it should translate

a)  to ἀ

But it does not. So I am puzzled.

Uwe 

Re: [conversion fails]

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

>    > I hacked up the following, which is minimally tested, but may work for
>    > you.  I tried to use public functions from quail, but the (null (cdr
>    > map)) is probably an implementation detail, and might break in a future
>    > version.  There are almost certainly better ways to write this function.
>    > If presented with an ambiguity (like above), it will choose the last
>    > remembered variant (defaulting to 1).

[...]

> I tested it with latin-1-pre and it worked, but I tried also
> greek-ibycus4
>
> And it should translate
>
> a)  to ἀ
>
> But it does not. So I am puzzled.

Right.  A corner case I missed.  Specifically, if the region ends on a
key sequence that encodes a valid character, but could be a prefix of
another character.

(defun accentuate-region (start end)
  "Replace the region by using it as keys using the current input method"
  (interactive "r")
  (save-excursion 
    (goto-char start)
    (let* ((data (delete-and-extract-region start end))
           (size (length data))
           (idx 0) (cand ""))
      (while (< idx size)
        (let* ((next-cand (concat cand (list (aref data idx))))
               (map (quail-lookup-key next-cand))
               (def (and map (quail-map-definition map))))
          (if (null map)
              (progn
                (insert next-cand)
                (setq cand ""))
            (if (and def (null (cdr map)))
                (progn
                  (insert (quail-get-current-str (length next-cand) def))
                  (setq cand "")) 
              (setq cand next-cand)))
          (incf idx)))
      (let* ((map (quail-lookup-key cand))
             (def (and map (quail-map-definition map))))
        (if def
            (insert (quail-get-current-str (length cand) def))
          (insert cand))))))

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: [conversion fails]

[Uwe Brauer]

    > Uwe Brauer <oub@mat.ucm.es> writes:

    > [...]


    > Right. A corner case I missed. Specifically, if the region ends on
    > a key sequence that encodes a valid character, but could be a
    > prefix of another character.

    > (defun accentuate-region (start end)
    >   "Replace the region by using it as keys using the current input method"
    >   (interactive "r")
    >   (save-excursion 
    >     (goto-char start)
    >     (let* ((data (delete-and-extract-region start end))
    >            (size (length data))
    >            (idx 0) (cand ""))
    >       (while (< idx size)
    >         (let* ((next-cand (concat cand (list (aref data idx))))
    >                (map (quail-lookup-key next-cand))
    >                (def (and map (quail-map-definition map))))
    >           (if (null map)
    >               (progn
    >                 (insert next-cand)
    >                 (setq cand ""))
    >             (if (and def (null (cdr map)))
    >                 (progn
    >                   (insert (quail-get-current-str (length next-cand) def))
    >                   (setq cand "")) 
    >               (setq cand next-cand)))
    >           (incf idx)))
    >       (let* ((map (quail-lookup-key cand))
    >              (def (and map (quail-map-definition map))))
    >         (if def
    >             (insert (quail-get-current-str (length cand) def))
    >           (insert cand))))))

Great, that looks very very promising. I will run various test cases and
come back when I finished. Maybe that code should make it, perhaps with a
different name, into quail.el?


I have to admit that this code is beyond me, while I more or less
understood the original iso-accentuate code, which worked with tables.

A last question then, for the sake of completion: you said the inverse
function would be trivial. In the case of the original iso-accentuate it
was. Would it be here as well? But as I said in my earlier mail I am
very happy with the accentuate version.

Thanks a zillon

Uwe 

Re: [conversion fails]

[Uwe Brauer]

   > Uwe Brauer <oub@mat.ucm.es> writes:

   > [...]


   > Right.  A corner case I missed.  Specifically, if the region ends on a
   > key sequence that encodes a valid character, but could be a prefix of
   > another character.

I cried victory to earlier, there seems more corner cases.

Take the following examples

1 )En a)rxh=| h)=n o( lo'gos, kai` o( lo'gos h)=n pro`s to`n qeo'n,
kai` qeo`s h)=n o( lo'gos.

Should be translated to 
1 Ἐν ἀρχῇ ἦν ὁ λόγος, καὶ ὁ λόγος ἦν πρὸς τὸν θεόν, καὶ θεὸς ἦν ὁ
λόγος.


But is translated to
1 Ἐν a)rχῇ h)=n o( λόγos, kaὶ o( λόγos h)=n πro`s τὸν θeo'n,
kaὶ θeo`s h)=n o( λόγos.


% 

%
2 ou(=tos h)=n e)n a)rxh=| pro`s to`n qeo'n.

Should be 
2 οὗτος ἦν ἐν ἀρχῇ πρὸς τὸν θεόν.

But gives
2 ou(=tos h)=n e)n a)rχῇ πro`s τὸν θeo'n.

That looks really tough. I tried to debug your function, but I don't
really understand the problem.


Thank you, really I appreciate your help


Uwe 

Re: [conversion fails]

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

>    > Uwe Brauer <oub@mat.ucm.es> writes:
>
>    > [...]
>
>
>    > Right.  A corner case I missed.  Specifically, if the region ends on a
>    > key sequence that encodes a valid character, but could be a prefix of
>    > another character.
>
> I cried victory to earlier, there seems more corner cases.
>
> Take the following examples
>
> 1 )En a)rxh=| h)=n o( lo'gos, kai` o( lo'gos h)=n pro`s to`n qeo'n,
> kai` qeo`s h)=n o( lo'gos.
>
> Should be translated to 
> 1 Ἐν ἀρχῇ ἦν ὁ λόγος, καὶ ὁ λόγος ἦν πρὸς τὸν θεόν, καὶ θεὸς ἦν ὁ
> λόγος.

Yup.  This version seems to do the trick, though:

(defun accentuate-region (start end)
  "Replace the region by using it as keys using the current input method"
  (interactive "r")
  (save-excursion
    (goto-char start)
    (let* ((data (delete-and-extract-region start end))
           (size (length data))
           (idx 0) (cand "") map def)
      (while (< idx size)
        (let* ((candx (concat cand (list (aref data idx))))
               (mapx (quail-lookup-key candx))
               (defx (and mapx (quail-map-definition mapx))))
          (if (null mapx)
              (if def
                  (progn
                    (insert (quail-get-current-str (length cand) def))
                    (setq cand "" map nil def nil)
                    (decf idx))
                (insert candx)
                (setq cand "" map nil def nil))
            (if (and defx (null (cdr mapx)))
                (progn
                  (insert (quail-get-current-str (length candx) defx))
                  (setq cand "" map nil def nil))
              (setq cand candx map mapx def defx)))
          (incf idx)))
      (if def
          (insert (quail-get-current-str (length cand) def))
        (insert cand)))))

I'll work on the inverse function in a bit.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: [conversion fails]

[Uwe Brauer]

 
    > Uwe Brauer <oub@mat.ucm.es> writes: 
 
    > Yup.  This version seems to do the trick, though: 

WOW, that works like CHARM,


Thanks a lot a lot. I run more test, most of the test files I have and
everything seems flawless. That is simply great. I am really grateful.
I could never ever have come up with something like this.

Honestly I think that should be part of quail.


    > I'll work on the inverse function in a bit.

That would also be great!

Uwe 

Re: [conversion fails]

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

>    > Uwe Brauer <oub@mat.ucm.es> writes: 
>
>    > Yup.  This version seems to do the trick, though: 
>
> WOW, that works like CHARM,
>
>
> Thanks a lot a lot. I run more test, most of the test files I have and
> everything seems flawless. That is simply great. I am really grateful.
> I could never ever have come up with something like this.
>
> Honestly I think that should be part of quail.
>
>
>    > I'll work on the inverse function in a bit.
>
> That would also be great!

Here you go.  I've included both, because this accentuate-region is an
improvement on the last one, especially if an error happens during
mid-translation.

(defun accentuate-region (start end)
  "Replace the region by using it as keys using the current input method"
  (interactive "r")
  (save-mark-and-excursion
    (when (< (mark) (point)) (exchange-point-and-mark))
    (let ((tend (1+ (point))) def)
      (while (<= tend (mark))
        (let* ((candx (buffer-substring (point) tend))
               (mapx (quail-lookup-key candx))
               (defx (and mapx (quail-map-definition mapx))))
          (if (null mapx)
              (progn
                (if def
                    (let* ((end (1- tend)) (len (- end (point))))
                      (delete-region (point) end)
                      (insert (quail-get-current-str len def))
                      (setq tend (point)))
                  (forward-char))
                (setq def nil))
            (if (and defx (null (cdr mapx)))
                (progn
                  (delete-region (point) tend)
                  (insert (quail-get-current-str (length candx) defx))
                  (setq def nil tend (point)))
              (setq def defx)))
          (incf tend)))
      (when def
        (let ((len (- (mark) (point))))
          (delete-region (point) (mark))
          (insert (quail-get-current-str (- (mark) (point)) def)))))))


(defun unaccentuate-region (start end)
  "Perform the reverse operation of typing in the current input method.
Replace the region with the keys used to type that text using the
current input method"
  (interactive "r")
  (save-mark-and-excursion
    (when (< (mark) (point)) (exchange-point-and-mark))
    (while (< (point) (mark))
      (condition-case nil
          (let ((keys (quail-find-key (following-char))))
            (if (consp keys)
                (progn (delete-char 1) (insert (car keys)))
              (forward-char)))))))

Sadly, I can't add these to quail, since it is too difficult to get a
copyright disclaimer from my place of work.  (It can be done, but has to
be done separately for every bit of code I contribute.)  Also, Handa-san
would almost certainly be able to write better versions than these that
can exploit the non-public interfaces in quail.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: [conversion fails]

[Uwe Brauer]

>>> "Michael" == Michael Welsh Duggan <mwd@md5i.com> writes: 
 
    > Uwe Brauer <oub@mat.ucm.es> writes: 
    >> > Uwe Brauer <oub@mat.ucm.es> writes:  
    >>  
    >> > Yup.  This version seems to do the trick, though:  
    >>  WOW, that works like CHARM,   Thanks a lot a lot. I run 
    >> more test, most of the test files I have and everything 
    >> seems flawless. That is simply great. I am really grateful. 
    >> I could never ever have come up with something like this. 
    >> Honestly I think that should be part of quail.   
    >> > I'll work on the inverse function in a bit. 
    >>  That would also be great! 
 
    > Here you go.  I've included both, because this 
    > accentuate-region is an improvement on the last one, 
    > especially if an error happens during mid-translation. 
    > Sadly, I can't add these to quail, since it is too difficult 
    > to get a copyright disclaimer from my place of work.  (It 
    > can be done, but has to be done separately for every bit of 
    > code I contribute.)  Also, Handa-san would almost certainly 
    > be able to write better versions than these that can exploit 
    > the non-public interfaces in quail. 

Thanks again, and it is a real pity that you have such 
difficulties in getting the papers. Maybe Handa-san, is reading 
this and will think about the issue.

Uwe

[MELPA] (was: [conversion fails])

[Uwe Brauer]

Possible doble posting my apologies.

    > Uwe Brauer <oub@mat.ucm.es> writes:


    > Sadly, I can't add these to quail, since it is too difficult to
    > get a copyright disclaimer from my place of work.

I just passed your code[1] to a colleague of mine and he was delighted
by the code. You saved his day. May I suggest that you put that small
pkg into MELPA? Or maybe the restrictions also apply to release your
code under the GPL?

Regards

Uwe 

Footnotes: 
[1]  I hope you don't mind. In any case I put your authorship in the
header of the file.

Re: [MELPA]

[Michael Welsh Duggan]

Uwe Brauer <oub@mat.ucm.es> writes:

> Possible doble posting my apologies.
>
>     > Uwe Brauer <oub@mat.ucm.es> writes:
>
>
>     > Sadly, I can't add these to quail, since it is too difficult to
>     > get a copyright disclaimer from my place of work.
>
> I just passed your code[1] to a colleague of mine and he was delighted
> by the code. You saved his day. May I suggest that you put that small
> pkg into MELPA? Or maybe the restrictions also apply to release your
> code under the GPL?

I actually have released code under the GPL as part of my job.

> Footnotes: 
> [1]  I hope you don't mind. In any case I put your authorship in the
> header of the file.

I do not mind.  If it's that useful, I'll contact Handa and see what he
thinks.  If he doesn't prefer to rewrite them himself, I'll see if I
can't get a disclaimer for this code in particular.

-- 
Michael Welsh Duggan
(md5i@md5i.com)