Using quail input methods to translate buffer text.

Using quail input methods to translate buffer text.

[Peter Heslin]

I have some ascii text that I want to convert to utf-8, and it is
encoded in a manner just like one of the quail input methods.  Can
anyone tell me if there is a function that will take text from a buffer
and treat it as if it were input as keystrokes to quail, and then output
the result in Unicode?

Thanks,

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Using quail input methods to translate buffer text.

[Evil Boris]

Peter Heslin <pj@heslin.eclipse.co.uk> writes:

> I have some ascii text that I want to convert to utf-8, and it is
> encoded in a manner just like one of the quail input methods.  Can
> anyone tell me if there is a function that will take text from a buffer
> and treat it as if it were input as keystrokes to quail, and then output
> the result in Unicode?

I have run into a similar problem.  I have not found a solution yet,
maybe because I did not look too hard.  :-)  

However, here's an exceedingly cheap trick that may work.  Run Emacs
in terminal mode (-nw) mode.  Open a new buffer in a new window and
set its input method to the one you want.  Now copy-paste the text you
want into the new buffer using the mouse and voila!

I just checked it here and it worked, sort of.  I have an Emacs
running in a Putty terminal window and I was able to turn text into
Cyrillic (Koi8-r, if you care) by just pasting from one window to
another.  (Accidentally picking up already encoded text and pasting it
produces spectacular gibberish.)

I would love to know of a nicer way of doing this.

Cheers,
        --Boris

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <87k66rlqys.fsf@heslin.eclipse.co.uk>, Peter Heslin <pj@heslin.eclipse.co.uk> writes:

> I have some ascii text that I want to convert to utf-8, and it is
> encoded in a manner just like one of the quail input methods.  Can
> anyone tell me if there is a function that will take text from a buffer
> and treat it as if it were input as keystrokes to quail, and then output
> the result in Unicode?

We don't have such a function, but this will work in many
cases.

(defun temp (input-method key-string)
  (let ((str ""))
    (activate-input-method input-method)
    (setq unread-command-events (string-to-list key-string))
    (while unread-command-events
      (let ((key (car unread-command-events))
	    events)
	(setq unread-command-events (cdr unread-command-events))
	(setq events (funcall input-method-function key))
	(dolist (elt events)
	  (if (integerp elt)
	      (setq str (concat str (string elt)))))))
    (activate-input-method nil)
    (decode-coding-string (encode-coding-string str 'utf-8) 'utf-8)))

For example:
(temp "cyrillic-yawerty" "abcd") => "абцд"

You don't need the last decode-coding-string if you use
emacs-unicode-2.

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

Kenichi Handa <handa@m17n.org> writes:

> We don't have such a function, but this will work in many
> cases.

Thank you very much!  This function will be extremely useful on a daily
basis (and I would guess also to others converting text from legacy
encodings to Unicode).

I append below a version of your function which suits my working habits.
The only problem I have noticed so far is this: if it encounters an
end-of-line at a point where it is still looking for possible keystrokes
to modify the current character, then the function stops there and does
not process the rest of the string.

I have worked around this for my purposes by appending a space to every
line before conversion, but I hope there is a cleaner solution.

(defun my-quail-conv (beg end)
  (interactive "r")
  (save-excursion
    (with-output-to-temp-buffer "*conversion output*"
      (princ (my-quail-conv-string
              current-input-method
              (replace-regexp-in-string "$" " "
                                        (buffer-substring-no-properties beg end)))))))

(defun my-quail-conv-string (input-method key-string)
  (let ((str "")
        (old-input-method current-input-method))
    (activate-input-method input-method)
    (setq unread-command-events (string-to-list key-string))
    (while unread-command-events
      (let ((key (car unread-command-events))
	    events)
	(setq unread-command-events (cdr unread-command-events))
	(setq events (funcall input-method-function key))
	(dolist (elt events)
	  (if (integerp elt)
	      (setq str (concat str (string elt)))))))
    (activate-input-method old-input-method)
    (decode-coding-string (encode-coding-string str 'utf-8) 'utf-8)))

Thanks again,

Peter

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <87y7v5fvp2.fsf@dur.ac.uk>, Peter Heslin <p.j.heslin@dur.ac.uk> writes:

> Thank you very much!  This function will be extremely useful on a daily
> basis (and I would guess also to others converting text from legacy
> encodings to Unicode).

> I append below a version of your function which suits my working habits.
> The only problem I have noticed so far is this: if it encounters an
> end-of-line at a point where it is still looking for possible keystrokes
> to modify the current character, then the function stops there and does
> not process the rest of the string.

> I have worked around this for my purposes by appending a space to every
> line before conversion, but I hope there is a cleaner solution.

Currently I think there's no clean solution.  Even appending
a space doesn't work for some input methods.  I have some
idea about making input methods work as batch, but that
requires big changes in the current code.  Perhaps for
emacs-unicode-2.

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

Kenichi Handa <handa@m17n.org> writes:

> Currently I think there's no clean solution.  Even appending
> a space doesn't work for some input methods.  

OK, so I tried a different approach.  I borrowed some lines of this code
from quail.el without fully understanding the data structures, so it may
be wrong, but it seems to work for my simple purposes.  Are there any
problems with this approach?


(defun my-quail-conv (beg end)
  (interactive "r")
  (save-excursion
    (let ((output))
      (goto-char beg)
      (while (< (point) end)
        (multiple-value-bind (trans map)
            (my-quail-conv-get-trans-and-map (quail-map) 0)
          (setf output (concat output
                               (or 
                                (if map
                                    (my-quail-conv-get-compound-char trans map)
                                  trans)
                                (char-to-string (char-after)))))
          (forward-char)))
      (with-output-to-temp-buffer "*conversion output*"
        (princ output)))))


(defun my-quail-conv-get-compound-char (trans map)
  (multiple-value-bind (next-trans next-map)
      (my-quail-conv-get-trans-and-map map 1)
    (cond
     (next-map
      (forward-char)
      (my-quail-conv-get-compound-char next-trans next-map))
     (next-trans
      (forward-char)
      next-trans)
     (t
      trans))))

(defun my-quail-conv-get-trans-and-map (alist offset)
  (let* ((ch (char-after (+ offset (point))))
         (map (cdr (assq ch alist)))
         (trans (and map (quail-get-translation
                          (car map) (char-to-string ch) 1))))
    (if (consp trans)
        (setq trans (aref (cdr trans) 0)))
    (setq trans
          (etypecase trans
            (string trans)
            (null trans)
            (integer (char-to-string trans))))
    (values trans (cdr map))))

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Using quail input methods to translate buffer text.

[Miles Bader]

Peter Heslin <pj@heslin.eclipse.co.uk> writes:
> OK, so I tried a different approach.  I borrowed some lines of this code
> from quail.el without fully understanding the data structures, so it may
> be wrong, but it seems to work for my simple purposes.  Are there any
> problems with this approach?

I tried both Kenichi's and your code using the `japanese' input method,
and yours seems to simply not work at all -- or at least I can't see
what it's doing:  it simply returns nil (and doesn't insert anything in
the buffer).  It does seem to work using the input method
`latin-1-postfix'.  In both cases, it leaves an empty scratch window
behind named "*conversion output*".

Kenichi's code does work properly for both cases, though a bit oddly for
Japanese -- because the japanese input method expects input to be
terminated with RET, you must append a "\r" to the string you pass to
his code to make sure it finishes correctly.

-Miles
-- 
Quidquid latine dictum sit, altum viditur.

Re: Using quail input methods to translate buffer text.

[Miles Bader]

I wrote:
> using the `japanese' input method, and yours seems to simply not work
> at all -- or at least I can't see what it's doing:  it simply returns
> nil (and doesn't insert anything in the buffer).

I was wrong here:  your code _does_ work properly for Japanese --
the problem was that it expects the BEG and END arguments to be in
proper order (and I did that for my latin-1-postfix test but not the
japanese test).

It also does have the advantage (over Kenichi's code) that it
doesn't require "\r" to be appended to the input string.

-Miles
-- 
/\ /\
(^.^)
(")")
*This is the cute kitty virus, please copy this into your sig so it can spread.

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

Miles Bader <miles@gnu.org> writes:

> your code _does_ work properly for Japanese --

Further testing revealed some odd corner cases where the code I posted
did not work correctly (for Greek).  I revised the code again, and I
think I fully understand what's going on now.  I cleaned it up and put
it in a file, which I'll attach.


[-- Attachment #2: quail-conv.el --]
[-- Type: application/emacs-lisp, Size: 4315 bytes --]

[-- Attachment #3: Type: text/plain, Size: 53 bytes --]



-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

In article <87ac7hupt2.fsf@heslin.eclipse.co.uk>, Peter Heslin <pj@heslin.eclipse.co.uk> writes:

> [1  <text/plain (7bit)>]
> Miles Bader <miles@gnu.org> writes:

>> your code _does_ work properly for Japanese --

> Further testing revealed some odd corner cases where the code I posted
> did not work correctly (for Greek).  I revised the code again, and I
> think I fully understand what's going on now.  I cleaned it up and put
> it in a file, which I'll attach.

It still doesn't work for chinese-py.  For instance, typing
"bei2" should input "北", but quail-conv doesn't handle the
last "2" correctly.

---
Kenichi Handa
handa@m17n.org



[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

Kenichi Handa <handa@m17n.org> writes:

> It still doesn't work for chinese-py.  For instance, typing
> "bei2" should input "北", but quail-conv doesn't handle the
> last "2" correctly.

Ah, thanks.  I think I've fixed this bug, but I'm not sure my fix is
correct.  I have assumed that each compound character sequence may be
followed by at most a single digit, and that this digit is used to
access the relevant vector like so: "1" gets the 0th element, "2" gets
the 1st ..., "9" gets the 8th, and "0" gets the 9th.  Does that sound
right?

PY.el says: "(each group contains at most 10 characters)", which I take
to mean that the index number following each group is never more than a
single digit.  Are there other input methods where this is not true?

Revised code attached.

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)


[-- Attachment #2: quail-conv.el --]
[-- Type: application/emacs-lisp, Size: 4781 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <87lkqob2fe.fsf@heslin.eclipse.co.uk>, Peter Heslin <pj@heslin.eclipse.co.uk> writes:

>> It still doesn't work for chinese-py.  For instance, typing
>> "bei2" should input "北", but quail-conv doesn't handle the
>> last "2" correctly.

> Ah, thanks.  I think I've fixed this bug, but I'm not sure my fix is
> correct.  I have assumed that each compound character sequence may be
> followed by at most a single digit, and that this digit is used to
> access the relevant vector like so: "1" gets the 0th element, "2" gets
> the 1st ..., "9" gets the 8th, and "0" gets the 9th.  Does that sound
> right?

> PY.el says: "(each group contains at most 10 characters)", which I take
> to mean that the index number following each group is never more than a
> single digit.  Are there other input methods where this is not true?

That's not accurate.  See the variable
quail-translation-keymap.  A user can select the next/prev
group by typing C-n/C-p.  In addition, several input methods
(including chinese-py) remembers the last selection.  So, to
get a constant result, we must reset the indices of last
selected translation, which requires modifying contents of
(quail-map).

I think it's not worth working on quail-conv to make it work
correctly for all cases.  How about just make it work well
for simple input methods (i.e. ones that (quail-simple)
return t) and reject complicated input methods.

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
> I think it's not worth working on quail-conv to make it work
> correctly for all cases.  How about just make it work well
> for simple input methods (i.e. ones that (quail-simple)
> return t) and reject complicated input methods.

That doesn't seem right -- (quail-simple) returns nil for Japanese, but
still, just the roman letters -> hiragana functionality of Peter's
function seems useful.

I'd say don't "reject" anything, simply put a note in the docstring that
the behavior is not necessarily useful for more complicated interactive
input methods.

-Miles
-- 
Everywhere is walking distance if you have the time.  -- Steven Wright

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <buoodvjd5s5.fsf@dhapc248.dev.necel.com>, Miles Bader <miles.bader@necel.com> writes:

> Kenichi Handa <handa@m17n.org> writes:
>> I think it's not worth working on quail-conv to make it work
>> correctly for all cases.  How about just make it work well
>> for simple input methods (i.e. ones that (quail-simple)
>> return t) and reject complicated input methods.

> That doesn't seem right -- (quail-simple) returns nil for Japanese, but
> still, just the roman letters -> hiragana functionality of Peter's
> function seems useful.

Even for japanese, it can't do correct converions of "kanji"
or "ippai" (I've not yet read quail-conv.el in detail why it
doesn't work for them).

> I'd say don't "reject" anything, simply put a note in the docstring that
> the behavior is not necessarily useful for more complicated interactive
> input methods.

Ummm, that may be ok.  But, if quail-conv gives up handling
of complicated one, I think the code can be far simpler
(like the original version).

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Richard Stallman]

    I think it's not worth working on quail-conv to make it work
    correctly for all cases.  How about just make it work well
    for simple input methods (i.e. ones that (quail-simple)
    return t) and reject complicated input methods.

That might be reasonable as a first step; but if quail-conv is useful,
we really should aim to make it work 100% as the ultimate goal.

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

Kenichi Handa <handa@m17n.org> writes:

>  A user can select the next/prev
> group by typing C-n/C-p.  In addition, several input methods
> (including chinese-py) remembers the last selection. 

OK, thanks for explaining this.

> I think it's not worth working on quail-conv to make it work
> correctly for all cases.  

I agree -- or rather I am now persuaded that there isn't a consistent
and useful automatic translation behavior when the input method requires
the user to choose between a list of possible translations.

> How about just make it work well for simple input methods (i.e. ones
> that (quail-simple) return t) and reject complicated input methods.

I have made and documented a small change to the code so that it throws
an error when it encounters a vector of multiple possible translations
rather than a single translation.  This way the simple roman -> hiragana
translation still works, but the chinese-py example will throw an error.

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <E1G3xkS-0007HB-AJ@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     I think it's not worth working on quail-conv to make it work
>     correctly for all cases.  How about just make it work well
>     for simple input methods (i.e. ones that (quail-simple)
>     return t) and reject complicated input methods.

> That might be reasonable as a first step; but if quail-conv is useful,
> we really should aim to make it work 100% as the ultimate goal.

As Peter wrote, there's no way to make it work 100% with the
current implementation of quail.  We have to change internal
structure/code of quail somehow.

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Richard Stallman]

    As Peter wrote, there's no way to make it work 100% with the
    current implementation of quail.  We have to change internal
    structure/code of quail somehow.

It is not out of the question to change the code of quail, or even its
internal structure.  Whether it is worth while to do so in this case
depends on details, such as how much benefit users would receive, and
on whether someone is willing to do the work.

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <E1G5546-0002Gq-T5@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     As Peter wrote, there's no way to make it work 100% with the
>     current implementation of quail.  We have to change internal
>     structure/code of quail somehow.

> It is not out of the question to change the code of quail, or even its
> internal structure.  Whether it is worth while to do so in this case
> depends on details, such as how much benefit users would receive, and
> on whether someone is willing to do the work.

Aren't we in feature-freeze state now?  The proposed one is
a new feature.  It may be ok to include it if the feature is
important enough to break `feature-freeze'.  But I think
that is only when the new feature is just addition and
doesn't require changing of existing code.

Or, are you talking about the work after the release?

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Richard Stallman]

    > It is not out of the question to change the code of quail, or even its
    > internal structure.  Whether it is worth while to do so in this case
    > depends on details, such as how much benefit users would receive, and
    > on whether someone is willing to do the work.

    Aren't we in feature-freeze state now?  The proposed one is
    a new feature.  It may be ok to include it if the feature is
    important enough to break `feature-freeze'.

I thought this was for after the release.
I don't think we should install this now.
So why hesitate to change Quail's structure?

Re: Using quail input methods to translate buffer text.

[Kenichi Handa]

In article <E1G5NfL-0005WY-Cn@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>> It is not out of the question to change the code of quail, or even its
>> internal structure.  Whether it is worth while to do so in this case
>> depends on details, such as how much benefit users would receive, and
>> on whether someone is willing to do the work.

>     Aren't we in feature-freeze state now?  The proposed one is
>     a new feature.  It may be ok to include it if the feature is
>     important enough to break `feature-freeze'.

> I thought this was for after the release.
> I don't think we should install this now.
> So why hesitate to change Quail's structure?

Ok, I see.  Then, I'll work on emacs-unicode-2 branch to
modify Quail so that such a feature can be more easily
implemented.

---
Kenichi Handa
handa@m17n.org

Re: Using quail input methods to translate buffer text.

[Peter Heslin]

Kenichi Handa <handa@m17n.org> writes:

> Ok, I see.  Then, I'll work on emacs-unicode-2 branch to
> modify Quail so that such a feature can be more easily
> implemented.

Just to make sure it is clear: my current quail-conv code does
everything I need it to do (which is translating text in polytonic Greek
LaTeX encodings which are not supported by any other tools such as GNU
recode).  I'm not sure whether it would be of real use to anyone to
extend this to the more complex interactive input methods.  I don't know
whether these complex input methods correspond exactly to text as it may
be stored in files, which is the case for many of the simple input
methods.  Before anyone goes to a great deal of trouble to implement
this, he or she should make sure that it will be genuinely useful
functionality.

-- 
Peter Heslin (http://www.dur.ac.uk/p.j.heslin)