japanese vs. chinese fonts

japanese vs. chinese fonts

[Miles Bader]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=iso-2022-jp-2, Size: 1276 bytes --]

When I enter Japanese in emacs, it now uses a Chinese font to display
the characters!

Even though:

  (1) I used the "japanese" input method

  (2) My language-environment is "Japanese"

  (3) I have Japanese ttf fonts installed on my system, which fontconfig
      knows about, and which used to be used by emacs.

  (4) Emacs _used_ to do the right thing (displaying Japanese text using
      Japanese fonts).

What should I do to debug this?


Here's an example of a mis-displayed character (I don't know how this
will display on other people's systems, but note the font name).

        character: ^[$B2=^[(B (21270, #o51426, #x5316)
preferred charset: japanese-jisx0208
		   (JISX0208.1983/1990 Japanese Kanji: ISO-IR-87)
       code point: 0x323D
           syntax: w 	which means: word
         category: C:Chinese (Han) characters of 2-byte character sets
		   c:Chinese h:Korean j:Japanese
		   |:While filling, we can break a line at this character.
      buffer code: #xE5 #x8C #x96
        file code: #xE5 #x8C #x96 (encoded by coding system utf-8)
          display: by this font (glyph code)
    xft:-unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-12-*-*-*-*-0-iso10646-1 (#xDCA)


Thanks,

-Miles

-- 
Brain, n. An apparatus with which we think we think.

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <buohc7v1rq3.fsf@dhapc248.dev.necel.com>, Miles Bader <miles.bader@necel.com> writes:

> When I enter Japanese in emacs, it now uses a Chinese font to display
> the characters!

Please do this:

(1) Start Emacs with -Q
(2) ESC : (setq font-log nil)
(3) ESC : (insert #x5316)
(4) M-x font-show-log RET

and show me the result. 

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Miles Bader]

[-- Attachment #1: Type: text/plain, Size: 57 bytes --]

Hi, I've attached the font-log output.

Thanks,

-Miles


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: font-log --]
[-- Type: text/plain; charset=iso-2022-jp-2, Size: 2650 bytes --]

font for: (21270)
ftfont-list: -unknown-DejaVu Sans Mono-*-jisx0208*-*
list: -unknown-DejaVu Sans Mono-*-jisx0208*-*
ftfont-list: -*-DejaVu Sans Mono-*-jisx0208*-*
list: -*-DejaVu Sans Mono-*-jisx0208*-*
ftfont-list: -unknown-*-jisx0208*-*
  -unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIwL@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL ^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL P^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_L@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-KanjiStrokeOrders-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIw%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL ^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIwL@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_L@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIw%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
list: -unknown-*-jisx0208*-*
  -unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIwL@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL ^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL P^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_L@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-KanjiStrokeOrders-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIw%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-VL ^[$B%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIwL@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$B$5$6$J$_L@D+^[(B-normal-normal-normal-*-0-iso10646-1
  -unknown-^[$BElIw%4%7%C%/^[(B-normal-normal-normal-*-0-iso10646-1
sort-by: -unknown-DejaVu Sans Mono-normal-normal-normal-*-12-*-m-0-jisx0208*-*
  xft:-unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-0-iso10646-1
open: -unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-0-iso10646-1
  xft:-unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-12-*-0-iso10646-1
font for: (21270)
list: -unknown-DejaVu Sans Mono-normal-*-normal-*-m-0-iso10646-1
  -unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1
  -unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
sort-by: -unknown-DejaVu Sans Mono-normal-italic-normal-*-12-*-m-0-iso10646-1
  xft:-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
open: -unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
  xft:-unknown-DejaVu Sans Mono-normal-oblique-normal-*-12-*-m-0-iso10646-1

[-- Attachment #3: Type: text/plain, Size: 119 bytes --]



-- 
Pray, v. To ask that the laws of the universe be annulled in behalf of a
single petitioner confessedly unworthy.

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <buo3ajf1q6r.fsf@dhapc248.dev.necel.com>, Miles Bader <miles.bader@necel.com> writes:
[...]
> list: -*-DejaVu Sans Mono-*-jisx0208*-*
> ftfont-list: -unknown-*-jisx0208*-*
>   -unknown-文泉驛正黑-normal-normal-normal-*-0-iso10646-1
>   -unknown-東風明朝-normal-normal-normal-*-0-iso10646-1
[...]
> sort-by: -unknown-DejaVu Sans Mono-normal-normal-normal-*-12-*-m-0-jisx0208*-*
>   xft:-unknown-文泉驛正黑-normal-normal-normal-*-0-iso10646-1

Ummm, isn't that font listed by this command?
% fc-list :lang=ja

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Miles Bader]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=iso-2022-jp-2, Size: 1278 bytes --]

Kenichi Handa <handa@m17n.org> writes:
>>   -unknown-^[$BJ8@tqc@5^[$A:Z^[(B-normal-normal-normal-*-0-iso10646-1
>
> Ummm, isn't that font listed by this command?
> % fc-list :lang=ja

Hmm, you're right it does... but it also matches :lang=zh ... the debian
package (ttf-wqy-zenhei) description says:

   "WenQuanYi Zen Hei" A Hei-Ti Style (sans-serif) Chinese font

The reason I don't want it is because some of the glyphs look "weird",
e.g., ^[$B2=^[(B and ^[$B8@^[(B (not entirely unrecognizable, just ... "weird").
It doesn't seem like a good default for japanese text to me.

The debian description also says:

    This font provides full coverage of GBK (CP936) charset, CJK Unified
    Ideographs, as well as the code-points needed for zh_CN, zh_SG,
    zh_TW, zh_HK, zh_MO, ja (Japanese) and ko (Korean) locales for
    fontconfig.

I guess it's the coverage which is why it matches :lang=ja (it also
matches :lang=zh, whereas the various japanese fonts I have installed
don't).

What does :lang really mean, anyway?

I guess for now I'll just un-install the font package, but I wonder if
this is something that needs addressing somehow?

-Miles

-- 
Idiot, n. A member of a large and powerful tribe whose influence in human
affairs has always been dominant and controlling.

Re: japanese vs. chinese fonts

[Werner LEMBERG]

> The debian description also says:
> 
>     This font provides full coverage of GBK (CP936) charset, CJK
>     Unified Ideographs, as well as the code-points needed for zh_CN,
>     zh_SG, zh_TW, zh_HK, zh_MO, ja (Japanese) and ko (Korean)
>     locales for fontconfig.

Hmm.  Does this font also provides proper `localized' glyphs, this is,
you select an OpenType language tag so that you get proper glyph
shapes?  They can vary considerably (and this is what Miles
disturbes).  Without this, the font shouldn't announce itself as
covering more than a single CJK *language*.

Ken Lunde, the leading CJK expert of Adobe, wrote a few years ago an
article regarding CJK glyph shapes (for a Unicode conference).  He
estimates that about a third of all CJK glyphs (this means
approx. 20000 characters!) need localized forms.

An additional complication is ideographic variation (this is something
on the character level, thus relevant to editors like Emacs), cf.

  http://unicode.org/reports/tr37/


    Werner

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <20081002.113032.155350320.wl@gnu.org>, Werner LEMBERG <wl@gnu.org> writes:

> > The debian description also says:
> > 
> >     This font provides full coverage of GBK (CP936) charset, CJK
> >     Unified Ideographs, as well as the code-points needed for zh_CN,
> >     zh_SG, zh_TW, zh_HK, zh_MO, ja (Japanese) and ko (Korean)
> >     locales for fontconfig.

> Hmm.  Does this font also provides proper `localized' glyphs, this is,
> you select an OpenType language tag so that you get proper glyph
> shapes?  They can vary considerably (and this is what Miles
> disturbes).  Without this, the font shouldn't announce itself as
> covering more than a single CJK *language*.

I've just downloaded that font and checked the contents.
No, it doesn't have a LangSys for Japanese in GSUB table.
So, there's not that much Emacs can do.  If you want to
avoid using that font totally (i.e. don't use it even for
Chinese), you can use face-ignored-fonts.

> Ken Lunde, the leading CJK expert of Adobe, wrote a few years ago an
> article regarding CJK glyph shapes (for a Unicode conference).  He
> estimates that about a third of all CJK glyphs (this means
> approx. 20000 characters!) need localized forms.

> An additional complication is ideographic variation (this is something
> on the character level, thus relevant to editors like Emacs), cf.

>   http://unicode.org/reports/tr37/

Yes.  That's one of ugly Unicode features I hate.  I have
not yet got a good idea how to handle it.  I hope no one use
such a dirty feature.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Juanma Barranquero]

On Thu, Oct 2, 2008 at 13:54, Kenichi Handa <handa@m17n.org> wrote:

> If you want to
> avoid using that font totally (i.e. don't use it even for
> Chinese), you can use face-ignored-fonts.

Does it work? face-ignored-fonts doesn't seem to be used anywhere,
other than in startup.el to decide whether to clear the face cache.

  C:\...\trunk> grep -rE face[-_]ignored[-_]fonts src\*.c src\*.h lisp\*.el
  src\xfaces.c:   4. Setting face-ignored-fonts allows the user to
ignore specific
  src\xfaces.c:Lisp_Object Vface_ignored_fonts;
  src\xfaces.c:  DEFVAR_LISP ("face-ignored-fonts", &Vface_ignored_fonts,
  src\xfaces.c:  Vface_ignored_fonts = Qnil;
  lisp\startup.el:        (old-face-ignored-fonts face-ignored-fonts))
  lisp\startup.el:                 (eq face-ignored-fonts
old-face-ignored-fonts))

Also, I do

  emacs -Q --eval "(add-to-list 'face-ignored-fonts \"FreeMono\")"

and then C-u C-x = over a Braille char in etc/HELLO:

        character: ⠕ (10261, #o24025, #x2815)
preferred charset: mule-unicode-2500-33ff (Unicode characters of the
range U+2500..U+33FF.)
       code point: 0x2835
           syntax: w 	which means: word
      buffer code: #xE2 #xA0 #x95
        file code: ESC #x24 #x2C #x32 #x28 #x35 (encoded by coding
system iso-2022-7bit-dos)
          display: by this font (glyph code)
    uniscribe:-outline-FreeMono-normal-normal-normal-mono-13-*-*-*-p-*-iso10646-1
(#x8EF)

Character code properties: customize what to show
  name: BRAILLE PATTERN DOTS-135
  general-category: So (Symbol, Other)

There are text properties here:
  charset              mule-unicode-2500-33ff


             Juanma

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <f7ccd24b0810020724i2f9ffefahbdd1159e5228be99@mail.gmail.com>, "Juanma Barranquero" <lekktu@gmail.com> writes:

> On Thu, Oct 2, 2008 at 13:54, Kenichi Handa <handa@m17n.org> wrote:
> > If you want to
> > avoid using that font totally (i.e. don't use it even for
> > Chinese), you can use face-ignored-fonts.

> Does it work? face-ignored-fonts doesn't seem to be used anywhere,
> other than in startup.el to decide whether to clear the face cache.

Oops, I forgot to re-implement that feature for font
backend.  I'll implement it soon.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <buod4ijzcc1.fsf@dhapc248.dev.necel.com>, Miles Bader <miles.bader@necel.com> writes:

> What does :lang really mean, anyway?

Fontconfig has it's own database to decide which font is for
which language.  The database maps each language to a set of
characters that a font must support to claim that the font
is for that langauge.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Stefan Monnier]

>> What does :lang really mean, anyway?
> Fontconfig has it's own database to decide which font is for
> which language.  The database maps each language to a set of
> characters that a font must support to claim that the font
> is for that langauge.

So I think this problem should be pushed up to fontconfig's team.


        Stefan

Re: japanese vs. chinese fonts

[Kenichi Handa]

In article <jwv63obkwfs.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> What does :lang really mean, anyway?
> > Fontconfig has it's own database to decide which font is for
> > which language.  The database maps each language to a set of
> > characters that a font must support to claim that the font
> > is for that langauge.

> So I think this problem should be pushed up to fontconfig's team.

I'm not sure.  It's not wrong that fontconfig decides that
the font support Japanese.  Perhaps, the problem should be
reported to the font developper not to include ungly glyphs
in a font.  Fontconfig can't know how ugly glyphs are.

By the eay, I'm not familiar with fontconfig's configulation
file, but aren't their a way to customize fontconfig to
refuse that font as a Japanese font?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: japanese vs. chinese fonts

[Stefan Monnier]

>> So I think this problem should be pushed up to fontconfig's team.

> I'm not sure.  It's not wrong that fontconfig decides that
> the font support Japanese.  Perhaps, the problem should be
> reported to the font developper not to include ungly glyphs
> in a font.  Fontconfig can't know how ugly glyphs are.

But fontconfig needs to provide a way to select "font appropriate for
Japanese text".  Whether "lang=ja" is what it's for (in which case it
needs to be fixed/improved) or whether something else is needed
I don't know.  But it's fontconfig's job to provide the functionality,
because it's not specific to Emacs.

Let's let the fontconfig people decide whether it's a bug report,
feature request, ...


        Stefan