tmm.el: non-ASCII characters with locale-coding-system

tmm.el: non-ASCII characters with locale-coding-system

[Tatsuya Kinoshita]

Hi Emacs developers,

Could you please consider applying the following patch to handle
non-ASCII characters with locale-coding-system for tmm.el?

----
--- tmm.el.orig	2008-02-08 22:47:00.000000000 +0900
+++ tmm.el	2008-03-14 08:08:04.000000000 +0900
@@ -171,7 +171,7 @@
     ;; The order of elements in tmm-km-list is the order of the menu bar.
     (mapc (lambda (elt)
 	    (if (stringp elt)
-		(setq gl-str elt)
+		(setq gl-str (decode-coding-string elt locale-coding-system))
 	      (and (listp elt) (tmm-get-keymap elt not-menu))))
 	    menu)
     ;; Choose an element of tmm-km-list; put it in choice.
@@ -282,6 +282,7 @@
 
 (defsubst tmm-add-one-shortcut (elt)
 ;; uses the free vars tmm-next-shortcut-digit and tmm-short-cuts
+  (setcar elt (decode-coding-string (car elt) locale-coding-system))
   (cond
    ((eq (cddr elt) 'ignore)
     (cons (concat " " (make-string (length tmm-mid-prompt) ?\-)
----

The patch is created by Tetsuo Tsukamoto (czkmt at remus.dti.ne.jp)
at http://mail.ring.gr.jp/skk/200803/msg00009.html to prevent a
problem of displaying Japanese characters with tmm.el reported
by Noritada Kobayashi at Debian BTS http://bugs.debian.org/470646.

To reproduce the problem,

* Install Daredevil SKK 13.1 (http://openlab.jp/skk/) and Emacs
  23.0.60 or 22.2.

* Run Emacs with `LC_ALL=ja_JP.UTF-8 emacs -q'.

* Type `M-x skk-mode RET', `M-`' and `s' to select the menu `s==>SKK',
  and then garbled characters (\351\240\230...) are displayed.

The patch works nicely on my system for Debian's ddskk 13.1-1,
emacs22 22.1+1-3 and Romain's emacs-snapshot 1:20080406-2.

Thanks,
-- 
Tatsuya Kinoshita

Re: tmm.el: non-ASCII characters with locale-coding-system

[Stefan Monnier]

> Could you please consider applying the following patch to handle
> non-ASCII characters with locale-coding-system for tmm.el?

> -		(setq gl-str elt)
> +		(setq gl-str (decode-coding-string elt locale-coding-system))
[...]
> +  (setcar elt (decode-coding-string (car elt) locale-coding-system))

In both cases, you assume that the elt (or its `car') is a unibyte
string that needs to be decoded.  But those should have been decoded
already.  Please try and figure out why they're still unibyte strings
rather than pre-decoded multibyte strings.

> The patch is created by Tetsuo Tsukamoto (czkmt at remus.dti.ne.jp)
> at http://mail.ring.gr.jp/skk/200803/msg00009.html to prevent a
> problem of displaying Japanese characters with tmm.el reported
> by Noritada Kobayashi at Debian BTS http://bugs.debian.org/470646.

> To reproduce the problem,

> * Install Daredevil SKK 13.1 (http://openlab.jp/skk/) and Emacs
>   23.0.60 or 22.2.

> * Run Emacs with `LC_ALL=ja_JP.UTF-8 emacs -q'.

> * Type `M-x skk-mode RET', `M-`' and `s' to select the menu `s==>SKK',
>   and then garbled characters (\351\240\230...) are displayed.

My guess is that if you try the same thing with a different locale, it
will fail, because the skk-mode for some reasons places utf-8 encoded
unibyte strings in the menu, so the if any decoding is needed it should
be with utf-8 rther than with locale-coding-system.


        Stefan

Re: tmm.el: non-ASCII characters with locale-coding-system

[Tatsuya Kinoshita]

On April 16, 2008 at 11:38AM -0400,
monnier (at iro.umontreal.ca) wrote:

> > The patch is created by Tetsuo Tsukamoto (czkmt at remus.dti.ne.jp)
> > at http://mail.ring.gr.jp/skk/200803/msg00009.html to prevent a
> > problem of displaying Japanese characters with tmm.el reported
> > by Noritada Kobayashi at Debian BTS http://bugs.debian.org/470646.
> 
> > To reproduce the problem,
> 
> > * Install Daredevil SKK 13.1 (http://openlab.jp/skk/) and Emacs
> >   23.0.60 or 22.2.
> 
> > * Run Emacs with `LC_ALL=ja_JP.UTF-8 emacs -q'.
> 
> > * Type `M-x skk-mode RET', `M-`' and `s' to select the menu `s==>SKK',
> >   and then garbled characters (\351\240\230...) are displayed.
> 
> My guess is that if you try the same thing with a different locale, it
> will fail, because the skk-mode for some reasons places utf-8 encoded
> unibyte strings in the menu, so the if any decoding is needed it should
> be with utf-8 rther than with locale-coding-system.

Even if Emacs is invoked with LC_ALL=ja_JP.eucJP, the patch works
nicely.

Hmm, the encoding scheme is decided as follows, quoting from
skk-e21.el:

| (defvar skk-e21-coding-system (cond
|                                ((memq window-system '(w32 nil))
|                                 nil)
|                                ((and (boundp 'mac-carbon-version-string)
|                                      window-system
|                                      (find-coding-system 'utf-8))
|                                 'utf-8)
|                                (t
|                                 locale-coding-system)))

Should it be set to 'utf-8?  What's the correct way to handle a
non-ASCII character of a menu item?

Thanks,
-- 
Tatsuya Kinoshita

Re: tmm.el: non-ASCII characters with locale-coding-system

[Tatsuya Kinoshita]

On April 17, 2008 at 1:15AM +0900,
tats (at vega.ocn.ne.jp) wrote:

> > The patch is created by Tetsuo Tsukamoto (czkmt at remus.dti.ne.jp)
> > at http://mail.ring.gr.jp/skk/200803/msg00009.html to prevent a
> > problem of displaying Japanese characters with tmm.el reported
> > by Noritada Kobayashi at Debian BTS http://bugs.debian.org/470646.
[...]
> Hmm, the encoding scheme is decided as follows, quoting from
> skk-e21.el:
> 
> | (defvar skk-e21-coding-system (cond
> |                                ((memq window-system '(w32 nil))
> |                                 nil)
> |                                ((and (boundp 'mac-carbon-version-string)
> |                                      window-system
> |                                      (find-coding-system 'utf-8))
> |                                 'utf-8)
> |                                (t
> |                                 locale-coding-system)))
> 
> Should it be set to 'utf-8?  What's the correct way to handle a
> non-ASCII character of a menu item?

Wow, I set skk-e21-coding-system to nil, and then the problem has
gone away.  The SKK Japanese menu item can be displayed with tmm.el
and also with the menu bar.

Tetsuo, is locale-coding-system really needed?  If not, please set
skk-e21-coding-system to nil by default.

Thanks,
-- 
Tatsuya Kinoshita