Re: utf-8 cjk translation bug?

[Kenichi Handa <handa@m17n.org>]

In article <rzq65j9nn4j.fsf@albion.dl.ac.uk>, Dave Love <d.love@dl.ac.uk> writes:

>>  I think the reason why they are excluded from the
>>  translation is that they are representable by the charset
>>  mule-unicode-e000-ffff, thus there's no need of translation.

> That was part of the reason for it -- the hash-based translation code
> is only relevant because we more-or-less used up the code space for
> the BMP.  I also chose the boundaries to avoid breaking the region
> between the mule-unicode and CJK charsets.

Sorry, I don't understand the meaning of the last sentence.

>>  It seems to be a reasonable decision, but considering that
>>  most users don't have an ISO10646-1 font containing those
>>  glyphs,

> I thought they typically did if they had 10646 fonts at all.  Is the
> problem that in recent XFree86, for instance, the double-width
> characters are in different fonts which have `adstyl' `ja' or `ko'?

Ah, right, they have double-width glyphs for those chars.
But, I think there are still many those who are not using
the recent XFree86, or who have not installed those fonts.

> As far as I remember, the fontset code doesn't deal with that yet.
> (So many special cases, sigh.)

Right.  So, even for XFree86 users, to utilize those fonts,
we need extra work.

>>  and that those characters can also be regarded as
>>  CJK components (only CJK users uses them), I think we had
>>  better not exclude them from the translation.

> I'm not really convinced, but I don't feel strongly about it.  (If the
> extra charsets hadn't been added before mule-unicode, we'd just have
> covered the BMP with more mule-unicode ones.)

And if I knew it took that long time to release the code
that contains mule-unicode charsets, I'd implemented a
single 3-dimensional charset that covers almost all Unicode
characters (Charset-ID 159 is not yet used).

>>  So, I suggest changing the above line (and similar lines in
>>  the other subst-XXX.el) to:
>> 
>>       (if (>= unicode #x2e80)
>>  	 (puthash unicode  char ucs-unicode-to-mule-cjk))
>> 
>>  and modify ccl-decode-mule-utf-8 to check translation also
>>  for those characters.
>> 
>>  Dave, what do you think?  Does such a change leads to any
>>  problem?

> As far as I remember, it includes too much, and you end up displaying
> some characters double width that probably shouldn't be, but I don't
> remember which.  How about including the ranges of the double-width
> Western characters and the high CJK stuff explicitly?  I guess it
> doesn't expand the tables greatly.

Ok, I've just installed a code that include U+FF00..U+FFEF
in the decode tables.

Now, in utf-translate-cjk mode:

(decode-coding-string
 (encode-coding-string "ＮＥＣエレクトロニクス(株)" 'utf-8)
 'utf-8)
=> "ＮＥＣエレクトロニクス(株)"

---
Ken'ichi HANDA
handa@m17n.org