bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

[Unknown]

in TeX input mode, I expect to be able to type ént by typing  "\'ent".
When I  set:

     (set-keyboard-coding-system 'utf-8)

typing those four characters gives me some Chinese character.

This is with the ubuntu Linux 8.04 emacs-snapshot package.

bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

[Stefan Monnier]

> in TeX input mode, I expect to be able to type ént by typing  "\'ent".
> When I  set:

>      (set-keyboard-coding-system 'utf-8)

> typing those four characters gives me some Chinese character.

IIUC the problem is that the input method processing is done before the
keyboard-coding-system processing.  So the sequence "\ ' e n t" is
turned by the input method into "é n t" (i.e. 233 110 116) which is then
interpreted as a utf-8 sequence which happens to give you
a chinese character.

In Emacs-22, bytes such as 233 were different from any other char, so
such confusion wasn't possible, which worked around the underlying
ordering problem.

A real fix would probably be to perform keyboard-coding-system decoding
earlier, but it's a non trivial change which may introduce its own set
of problems.

Maybe another way to fix the problem is to let quail emit not just é but
the sequence [im-char é] and then have input-decode-map strip the
`im-char' pseudo event; this would protect those im-generated chars from
being processed by the keyboard-coding-system processing.

Handa?


        Stefan

bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

[Kenichi Handa]

I've just noticed that this bug is not yet closed.

Laura Conrad <lconrad@laymusic.org> wrote:
> in TeX input mode, I expect to be able to type ént by typing  "\'ent".
> When I  set:
> 
>      (set-keyboard-coding-system 'utf-8)
> 
> typing those four characters gives me some Chinese character.
> 
> This is with the ubuntu Linux 8.04 emacs-snapshot package.

It seems that this bug is now fixed by Stefan's workaround
patch accompanied with this comment:

        ;; Invalid utf-8 sequence.  Might be because Quail got involved
        ;; in-between and the bytes we thought we were reading were actually
        ;; latin-1 chars.  Let's presume that `event' is the second "byte",
        ;; i.e. there weren't any "apprently correct" between `lead' and
        ;; `event': it's easy to recover in this case, and the more general
        ;; case seems pretty unlikely.
        ;; FIXME: We should really do encoded-kbd decoding before processing
        ;; input-methods.

The strange thing is that the date of the orignal bug report
is 2008-09-19 but the above patch was installed on
2008-06-15.  Laura, do you still see the problem?

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> A real fix would probably be to perform keyboard-coding-system decoding
> earlier, but it's a non trivial change which may introduce its own set
> of problems.

I vaguely remember that I sent a patch to do that long ago,
but it seems that the discussion fizzled out without
reaching any conclusion.

> Maybe another way to fix the problem is to let quail emit not just é but
> the sequence [im-char é] and then have input-decode-map strip the
> `im-char' pseudo event; this would protect those im-generated chars from
> being processed by the keyboard-coding-system processing.

That's one idea, but seems tricky.  I prefer the first
method.

---
Kenichi Handa
handa@m17n.org

bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

[Laura Conrad]

>>>>> "Kenichi" == Kenichi Handa <handa@m17n.org> writes:

    Kenichi> The strange thing is that the date of the orignal bug report
    Kenichi> is 2008-09-19 but the above patch was installed on
    Kenichi> 2008-06-15.  Laura, do you still see the problem?

No.  Not on ubuntu 9.04 with GNU Emacs 23.0.91.1.

The report was made when I upgraded to Ubuntu 8.04, with whatever
version their emacs-snapshot was.

Thanks for the fix.

-- 
Laura   (mailto:lconrad@laymusic.org http://www.laymusic.org/ )
(617) 661-8097	233 Broadway, Cambridge, MA 02139   

Jesus, I'm Thirsty.  (Danku)

'Who's getting the beer?'
Judas stepped up to the bar.
He'd had a windfall!

Danny Reynolds

bug#1003: 23.0.60; set-keyboard-coding-system incompatible with TeX input method

[Kenichi Handa]

In article <jwv1vqd47n7.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> A real fix would probably be to perform keyboard-coding-system decoding
>>> earlier, but it's a non trivial change which may introduce its own set
>>> of problems.

> > I vaguely remember that I sent a patch to do that long ago,
> > but it seems that the discussion fizzled out without
> > reaching any conclusion.

> I also had a vague feeling of seeing such a thing, but couldn't find it
> in the archives, so I figured I was confused.
> If not, please try and remember to install it for Emacs-23.2.

I've just installed the change.  As encoded-kb.el is now not
necessary, I deleted that file.

---
Kenichi Handa
handa@m17n.org