Re: bug#613: 23.0.60; Unicode display problems

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

Sorry for the late responce.

> Kenichi Handa writes:
> > Thank you.  But, unfortunately, the list was too long and
> > font-show-log truncated such long lists.  Please show me the
> > value of font-log itself instead:
> >
> > (1) Start Emacs as "% emacs -Q".
> > (2) ESC : (setq font-log nil)
> > (3) ESC : (insert #x201c)
> > (4) C-h v font-log RET

Thank you for the data.  I found very strange thing.  The
log has this entry.

>  (list "-unknown-DejaVu Sans Mono-*-iso10646-1"
>        [])

So, Emacs couldn't find a font matching with:
  -unknown-DejaVu Sans Mono-*-iso10646-1
But, the log also has this entry.

>  (list "-unknown-DejaVu Sans Mono-*-normal-normal-*-m-0-iso10646-1"
>        ["-unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1"])

So, Emacs found:
  -unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-iso10646-1
even if it should match with the former font pattern.

I tried various settings but I couldn't reproduce such a
thing.  Aren't there anyone else who has the same problem.

To find out what's going on in your environment, I need more
detailed logging info.  So I'm going to improve the font-log
facility.

By the way, what is the result of:

% fc-list 'Dejavu Sans Mono' file

---
Kenichi Handa
handa@ni.aist.go.jp

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <E1KTDsR-0007MI-4Y@etlken.m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> To find out what's going on in your environment, I need more
> detailed logging info.  So I'm going to improve the font-log
> facility.

Done.  Please try this:

(1) Start Emacs as "% emacs -Q".
(2) ESC : (setq font-log nil)
(3) ESC : (insert #x201c)
(4) C-u - M-x font-show-log RET

Note that the step (4) is slightly different from the
previous one.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: bug#613: 23.0.60; Unicode display problems

[Torsten Bronger]

Hallöchen!

Kenichi Handa writes:

> Kenichi Handa <handa@m17n.org> writes:
>
>> To find out what's going on in your environment, I need more
>> detailed logging info.  So I'm going to improve the font-log
>> facility.
>
> Done.  Please try this:
>
> (1) Start Emacs as "% emacs -Q".
> (2) ESC : (setq font-log nil)
> (3) ESC : (insert #x201c)
> (4) C-u - M-x font-show-log RET

It didn't work with a current CVS Emacs.  All it says is "Font
logging is currently supressed" in the minibuffer.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: torsten.bronger@jabber.rwth-aachen.de

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <873al2sd2h.fsf@physik.rwth-aachen.de>, Torsten Bronger <bronger@physik.rwth-aachen.de> writes:

> > (1) Start Emacs as "% emacs -Q".
> > (2) ESC : (setq font-log nil)
> > (3) ESC : (insert #x201c)
> > (4) C-u - M-x font-show-log RET

> It didn't work with a current CVS Emacs.  All it says is "Font
> logging is currently supressed" in the minibuffer.

??? Are you sure that you did: (setq font-log nil)

What is the value of font-log just after you did that?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: bug#613: 23.0.60; Unicode display problems

[Torsten Bronger]

Hallöchen!

Kenichi Handa writes:

> In article <873al2sd2h.fsf@physik.rwth-aachen.de>, Torsten Bronger
> <bronger@physik.rwth-aachen.de> writes:
>
>> > (1) Start Emacs as "% emacs -Q".
>> > (2) ESC : (setq font-log nil)
>> > (3) ESC : (insert #x201c)
>> > (4) C-u - M-x font-show-log RET
>
>> It didn't work with a current CVS Emacs.  All it says is "Font
>> logging is currently supressed" in the minibuffer.
>
> ??? Are you sure that you did: (setq font-log nil)

No sorry, after a third try, it worked.  Maybe I had entered
"font-lock".

finding a font for: (8220)
ftfont-list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
ftfont-list: -*-DejaVu Sans Mono-*-iso10646-1:script=symbol
list: -*-DejaVu Sans Mono-*-iso10646-1:script=symbol
ftfont-list: -unknown-*-iso10646-1:script=symbol
  -unknown-DejaVu Sans-normal-normal-semi-condensed-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-oblique-normal-*-0-iso10646-1
  -unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-oblique-semi-condensed-*-0-iso10646-1
  -unknown-UnBatang-bold-normal-normal-*-0-iso10646-1
  -unknown-FreeSerif-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-normal-normal-*-0-iso10646-1
  -unknown-UnDotum-bold-normal-normal-*-0-iso10646-1
  -unknown-Kochi Mincho-normal-normal-normal-*-0-iso10646-1
  -unknown-UnBatang-normal-normal-normal-*-0-iso10646-1
  -unknown-FreeSerif-bold-normal-normal-*-0-iso10646-1
  -unknown-FreeSans-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-normal-normal-*-0-iso10646-1
  -unknown-FreeMono-normal-normal-normal-*-m-0-iso10646-1
  -unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1
  -unknown-UnDotum-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-normal-semi-condensed-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-italic-semi-condensed-*-0-iso10646-1
  -unknown-OpenSymbol-normal-normal-normal-*-0-iso10646-1
  -unknown-Kochi Mincho-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-italic-normal-*-0-iso10646-1
list: -unknown-*-iso10646-1:script=symbol
  -unknown-DejaVu Sans-normal-normal-semi-condensed-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-oblique-normal-*-0-iso10646-1
  -unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-oblique-semi-condensed-*-0-iso10646-1
  -unknown-UnBatang-bold-normal-normal-*-0-iso10646-1
  -unknown-FreeSerif-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Sans-normal-normal-normal-*-0-iso10646-1
  -unknown-UnDotum-bold-normal-normal-*-0-iso10646-1
  -unknown-Kochi Mincho-normal-normal-normal-*-0-iso10646-1
  -unknown-UnBatang-normal-normal-normal-*-0-iso10646-1
  -unknown-FreeSerif-bold-normal-normal-*-0-iso10646-1
  -unknown-FreeSans-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-normal-normal-*-0-iso10646-1
  -unknown-FreeMono-normal-normal-normal-*-m-0-iso10646-1
  -unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1
  -unknown-UnDotum-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-normal-semi-condensed-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-italic-semi-condensed-*-0-iso10646-1
  -unknown-OpenSymbol-normal-normal-normal-*-0-iso10646-1
  -unknown-Kochi Mincho-normal-normal-normal-*-0-iso10646-1
  -unknown-DejaVu Serif-normal-italic-normal-*-0-iso10646-1
sort-by: -unknown-DejaVu Sans Mono-normal-normal-normal-*-12-*-m-0-iso10646-1
  xft:-unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1
open: -unknown-Kochi Gothic-normal-normal-normal-*-0-iso10646-1:script=symbol
  xft:-unknown-Kochi Gothic-normal-normal-normal-*-12-*-0-iso10646-1
finding a font for: (8220)
finding a font for: (8220)
finding a font for: (8220)
list: -unknown-DejaVu Sans Mono-normal-*-normal-*-m-0-iso10646-1
  -unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
  -unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1
sort-by: -unknown-DejaVu Sans Mono-normal-italic-normal-*-12-*-m-0-iso10646-1
  xft:-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
open: -unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1
  xft:-unknown-DejaVu Sans Mono-normal-oblique-normal-*-12-*-m-0-iso10646-1

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: torsten.bronger@jabber.rwth-aachen.de

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <87od3qqvlg.fsf@physik.rwth-aachen.de>, Torsten Bronger <bronger@physik.rwth-aachen.de> writes:

> finding a font for: (8220)
> ftfont-list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
> list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
[...]

Now I see what is wrong.  Dejavu Sans Mono lacks some of
charaters (#x201C #x2200 #x2500) to be used as `symbol'
font, so the other arbitrary font that contains all of those
characters are selected.  This setting was added by the
request in the attached mail, but it seems that the
restriction is too strong for symbol fonts.  So, I deleted
that restriction.  Please try again with the latest CVS code.

---
Kenichi Handa
handa@ni.aist.go.jp

In article <18477.2875.876651.446827@priss.frightenedpiglet.com>, sand@blarg.net writes:

> Below is the complete set of fontset changes that my system needed to
> display the HELLO file and Markus Kuhn's "UTF-8-demo.txt" file.  There
> are various problems with the fontset definitions that we can fix in
> Emacs Lisp.

>   1. The "symbol" and "phonetic" scripts don't have any representative
>   characters defined, and they don't have any fontset defined.

>   2. The Latin script doesn't have representative characters over
>   7-bits.  This means the Xft backend doesn't have any constraint that
>   would reject ISO8859-1 fonts.  Emacs appears to have special support
>   for real ISO8859-1 characters, so the new definition should not
>   break anything.

>   3. The Armenian script has representative characters, but no fontset
>   defined.

>   4. The Thai fontset definition assumes that you have OpenType set
>   up.  It needs a fallback in case you don't.

>   5. The Hangul fontset definition assumes that you have a language
>   definition in your Hangul font.  It needs a fallback in case you
>   don't.

> Note that for some people, the "symbol", "phonetic" and Latin scripts
> may display just fine.  The problem is that Emacs is not rejecting
> ISO8859-1 fonts---but if your installation happens to pick a
> ISO10646-1 font with coverage first then you won't see any weirdness.
> For example, my work machine had trouble displaying the scripts while
> my home machine did not.

> It may be that the Thai and Hangul definitions are the tip of the
> iceberg.  I suspect that *every* ISO10646-1 fontset definition that
> uses :otf or :language needs a fallback definition that uses :script.

> If your HELLO file shows weird fonts being picked (such as a
> double-width Chinese font for U+2018 LEFT SINGLE QUOTATION MARK) or
> "missing font" boxes, can you please try putting the below code into
> your .emacs file to see if it fixes the problem?  These were sufficent
> to get my work machine displaying every HELLO and "UTF-8-demo.txt"
> glyphs for every font that I had installed.

> This code can be applied to CVS HEAD before or after the font-backend
> merge.

> Derek

> --
> Derek Upham
> sand@blarg.net

> ------------------------------ cut here ------------------------------

> ;; The eval-after-load here is a hack for testing.  These should
> ;; really be changes in "fontset.el".
> ;;
> ;; Some of the fontset definitions have 'prepend for testing.  When
> ;; added to "fontset.el" they should go into some appropriate
> ;; location in the current lists.

> (eval-after-load "fontset"
>   '(progn

> ;; Representative characters for Latin need to include the
> ;; various extension blocks.  This should replace the existing
> ;; definition.
> (setq script-representative-chars
>       (cons '(latin ?A ?Z ?a ?z #x00C0 #x0100 #x0180 #x1e00)
>             script-representative-chars))

> ;; Representative characters for symbols were not defined.
> (setq script-representative-chars
>       (cons '(symbol #x201C #x2200 #x2500)
>             script-representative-chars))

> ;; Representative characters for phonetics were not defined.
> (setq script-representative-chars
>       (cons '(phonetic #x0250 #x0283)
>             script-representative-chars))

> ;; 'latin script needs a declaration that incorporates the Latin
> ;; representative characters.  None of the current ones do.
> (set-fontset-font "fontset-default" 'latin
>  (font-spec :registry "iso10646-1" :script 'latin) nil 'prepend)

> ;; No 'symbol script declaration exists.  Create one using the new
> ;; representative characters.
> (set-fontset-font "fontset-default" 'symbol
>  (font-spec :registry "iso10646-1" :script 'symbol))

> ;; No 'phonetic declaration exists.  Create one using the new
> ;; representative characters.  (Note that there is an 'ipa
> ;; declaration, but 'ipa doesn't exist as a real script.)
> (set-fontset-font "fontset-default" 'phonetic
>  (font-spec :registry "iso10646-1" :script 'phonetic))

> ;; No fontset was defined for Armenian.
> (set-fontset-font "fontset-default" 'armenian
>  (font-spec :registry "iso10646-1" :script 'armenian))

> ;; The Thai fontset definition assumes that you have OpenType
> ;; definitions set up properly.  We need an explicit :script
> ;; declaration as a backup.  I think this should go after the :otf
> ;; declaration, but I'm doing it as a 'prepend here.  (Perhaps
> ;; FreeSerif Thai fonts comply with OpenType, but Misc-Fixed
> ;; ones don't.)
> (set-fontset-font "fontset-default" 'thai
>  (font-spec :registry "iso10646-1" :script 'thai) nil 'prepend)

> ;; 'hangul script has a :language restriction for iso10646-1, but also
> ;; needs :script restriction as alternative.
> (set-fontset-font "fontset-default" 'hangul
>   (font-spec :registry "iso10646-1" :script 'hangul) nil 'prepend)

> ))

bug#613: 23.0.60; Unicode display problems

[sand]

Kenichi Handa writes:
> In article <87od3qqvlg.fsf@physik.rwth-aachen.de>, Torsten Bronger <bronger@physik.rwth-aachen.de> writes:
> 
> > finding a font for: (8220)
> > ftfont-list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
> > list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
> [...]
> 
> Now I see what is wrong.  Dejavu Sans Mono lacks some of
> charaters (#x201C #x2200 #x2500) to be used as `symbol'
> font, so the other arbitrary font that contains all of those
> characters are selected.  This setting was added by the
> request in the attached mail, but it seems that the
> restriction is too strong for symbol fonts.  So, I deleted
> that restriction.  Please try again with the latest CVS code.

Removing the symbol restrictions will allow Emacs to pick (incorrect)
ISO8859-1 fonts for these characters again.  Have you changed
something in the C code to prevent that?

Looking at the DejaVu Sans Mono glyphs in "gucharmap", it appears that
#x2200 is the problem.  The #x201C, #x2200, #x2500 combination was an
attempt to find "representative" symbol codepoints, but that
combination isn't all-or-nothing.  Can you first try leaving just
#x201C in `script-representative-chars'?  That's LEFT DOUBLE QUOTATION
MARK, so it should have wide coverage.

Derek

Re: bug#613: 23.0.60; Unicode display problems

[Torsten Bronger]

Hallöchen!

Kenichi Handa writes:

> In article <87od3qqvlg.fsf@physik.rwth-aachen.de>, Torsten Bronger
> <bronger@physik.rwth-aachen.de> writes:
>
>> finding a font for: (8220)
>> ftfont-list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
>> list: -unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol
>
> [...]
>
> Now I see what is wrong.  Dejavu Sans Mono lacks some of charaters
> (#x201C #x2200 #x2500) to be used as `symbol' font, so the other
> arbitrary font that contains all of those characters are selected.
> This setting was added by the request in the attached mail, but it
> seems that the restriction is too strong for symbol fonts.  So, I
> deleted that restriction.  Please try again with the latest CVS
> code.

It works now as for this change but apparently, it still is not
fully correct.  At least two characters that I use ("prime" #x2032
and "cubic root" #x221b) are still too wide, however, I don't know
whether they are available in the DejaVu font.

Do you think it's sensible to create a file will all codepoints and
look for problematic chars in a systematic way?

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: torsten.bronger@jabber.rwth-aachen.de

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <18604.9770.893099.425147@priss.frightenedpiglet.com>, sand@blarg.net writes:

> Kenichi Handa writes:
[...]
> > Now I see what is wrong.  Dejavu Sans Mono lacks some of
> > charaters (#x201C #x2200 #x2500) to be used as `symbol'
> > font, so the other arbitrary font that contains all of those
> > characters are selected.  This setting was added by the
> > request in the attached mail, but it seems that the
> > restriction is too strong for symbol fonts.  So, I deleted
> > that restriction.  Please try again with the latest CVS code.

> Removing the symbol restrictions will allow Emacs to pick (incorrect)
> ISO8859-1 fonts for these characters again.  Have you changed
> something in the C code to prevent that?

Now the entry for symbols is this:

     (symbol ,(font-spec :registry "iso10646-1"))

So, at least, Emacs tries to select an iso10646-1 font that
is closest (in family, style, size) to the default font.

> Looking at the DejaVu Sans Mono glyphs in "gucharmap", it appears that
> #x2200 is the problem.  The #x201C, #x2200, #x2500 combination was an
> attempt to find "representative" symbol codepoints, but that
> combination isn't all-or-nothing.  Can you first try leaving just
> #x201C in `script-representative-chars'?  That's LEFT DOUBLE QUOTATION
> MARK, so it should have wide coverage.

I think the typical usage of symbol characters is to use a
few of them in a normal text.  In that case, the most
suitable font for those symbols is the default font
(provided that it supports those symbols) rather than a font
that covers wider range of symbols.  So, I think there's no
merit in restricting fonts in the above way.

As a result, when a sequence of many symbols is displayed,
Emacs will not use the same font for them, but, I think it's
not a big problem.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <87k5ebbrq1.fsf@physik.rwth-aachen.de>, Torsten Bronger <bronger@physik.rwth-aachen.de> writes:

> It works now as for this change but apparently, it still is not
> fully correct.  At least two characters that I use ("prime" #x2032
> and "cubic root" #x221b) are still too wide, however, I don't know
> whether they are available in the DejaVu font.

Dejavu Sans Mono doesn't have those characters.  In such a
case, it's very difficult to decide which is the "correct"
font.

A while ago, stephen@xemacs.org suggested to utilize
fontconfig's language repertoires.  I'm going to implement
that somehow.  When it is done, for non-CJK locale users,
Emacs can select a single-width font.

> Do you think it's sensible to create a file will all codepoints and
> look for problematic chars in a systematic way?

No, it won't help.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: bug#613: 23.0.60; Unicode display problems

[Juanma Barranquero]

On Thu, Aug 21, 2008 at 03:01, Kenichi Handa <handa@m17n.org> wrote:

> Dejavu Sans Mono doesn't have those characters.

I think the newest DejaVu Sans Mono (2.26) does have U+201C, U+2200,
U+2500 and U+221B.

It still lacks U+2032, though.

  Juanma

Re: bug#613: 23.0.60; Unicode display problems

[sand]

Kenichi Handa writes:
> In article <18604.9770.893099.425147@priss.frightenedpiglet.com>, sand@blarg.net writes:
> 
> > Kenichi Handa writes:
> [...]
> > > Now I see what is wrong.  Dejavu Sans Mono lacks some of
> > > charaters (#x201C #x2200 #x2500) to be used as `symbol'
> > > font, so the other arbitrary font that contains all of those
> > > characters are selected.  This setting was added by the
> > > request in the attached mail, but it seems that the
> > > restriction is too strong for symbol fonts.  So, I deleted
> > > that restriction.  Please try again with the latest CVS code.
> 
> > Removing the symbol restrictions will allow Emacs to pick (incorrect)
> > ISO8859-1 fonts for these characters again.  Have you changed
> > something in the C code to prevent that?
> 
> Now the entry for symbols is this:
> 
>      (symbol ,(font-spec :registry "iso10646-1"))
> 
> So, at least, Emacs tries to select an iso10646-1 font that
> is closest (in family, style, size) to the default font.
> 

I'm seeing a complete regression.  I rebuilt from CVS HEAD, confirmed
that your change is in fontset.el, and Emacs picks AR PL ShanHeiSun
for open single quotes.  Putting the script definitions back in fixes
the problem.  I have pasted "good" and "bad" font log blocks to the
end of this message.  Those logs are particularly for Neep Alt, but
the Misc Fixed ISO10646 fonts have the same problem.

Derek

------------------------------ cut here ------------------------------

Successful load of LEFT SINGLE QUOTE with the old definition using
"script":

font for: (8216)
ftfont-list: -jmk-Neep Alt-*-iso10646-1:script=symbol
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
list: -jmk-Neep Alt-*-iso10646-1:script=symbol
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
sort-by: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
open: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1:script=symbol
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1


Unsuccessful load of LEFT SINGLE QUOTE with new definition:

font for: (8216)
ftfont-list: -jmk-Neep Alt-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-10-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
list: -jmk-Neep Alt-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-10-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
sort-by: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
open: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
ftfont-list: -jmk-Neep Alt-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-10-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
list: -jmk-Neep Alt-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-10-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
sort-by: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
open: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
list: -jmk-Neep Alt-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-normal-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-11-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-10-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-24-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-bold-normal-semi-condensed-*-13-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-15-*-c-*-iso10646-1
  -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
sort-by: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
  xft:-jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
ftfont-list: -gnu-unifont-*-iso10646-1
list: -gnu-unifont-*-iso10646-1
ftfont-list: -mutt-clearlyu-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
ftfont-list: -jmk-Neep Alt-*-gb2312.1980*-*
list: -jmk-Neep Alt-*-gb2312.1980*-*
ftfont-list: -*-Neep Alt-*-gb2312.1980*-*
list: -*-Neep Alt-*-gb2312.1980*-*
ftfont-list: -jmk-*-gb2312.1980*-*
list: -jmk-*-gb2312.1980*-*
ftfont-list: -*-gb2312.1980*-*
  -ISAS-Fangsong ti-normal-normal-normal-*-24-*-c-*-iso10646-1
  -unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
  -unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
  -unknown-AR PL ZenKai Uni-normal-normal-normal-*-0-iso10646-1
  -ISAS-Fangsong ti-normal-normal-normal-*-16-*-c-*-iso10646-1
list: -*-gb2312.1980*-*
  -ISAS-Fangsong ti-normal-normal-normal-*-24-*-c-*-iso10646-1
  -unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
  -unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
  -unknown-AR PL ZenKai Uni-normal-normal-normal-*-0-iso10646-1
  -ISAS-Fangsong ti-normal-normal-normal-*-16-*-c-*-iso10646-1
sort-by: -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-gb2312.1980*-*
  xft:-unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
open: -unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-0-iso10646-1
  xft:-unknown-AR PL ShanHeiSun Uni-normal-normal-normal-*-20-*-0-iso10646-1
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8216)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*
font for: (8217)
list: -gnu-unifont-*-iso10646-1
list: -mutt-clearlyu-*-iso10646-1
list: -jmk-Neep Alt-*-gb18030*-*
list: -*-Neep Alt-*-gb18030*-*
list: -jmk-*-gb18030*-*
list: -*-gb18030*-*

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <18613.24465.897881.636770@priss.frightenedpiglet.com>, sand@blarg.net writes:

> I'm seeing a complete regression.  I rebuilt from CVS HEAD, confirmed
> that your change is in fontset.el, and Emacs picks AR PL ShanHeiSun
> for open single quotes.  Putting the script definitions back in fixes
> the problem.  I have pasted "good" and "bad" font log blocks to the
> end of this message.  Those logs are particularly for Neep Alt, but
> the Misc Fixed ISO10646 fonts have the same problem.

[...]
> Unsuccessful load of LEFT SINGLE QUOTE with new definition:

> font for: (8216)
> ftfont-list: -jmk-Neep Alt-*-iso10646-1
[...]
>   -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
[...]
>   -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1

Don't you happen to have two sets of "Neep Alt" fonts in the
fontconfig's fontpath; one set supports LEFT SINGLE QUOTE,
and the other doesn't?

---
Kenichi Handa
handa@ni.aist.go.jp

bug#613: 23.0.60; Unicode display problems

[sand]

Kenichi Handa writes:
> In article <18613.24465.897881.636770@priss.frightenedpiglet.com>, sand@blarg.net writes:
> 
> > I'm seeing a complete regression.  I rebuilt from CVS HEAD, confirmed
> > that your change is in fontset.el, and Emacs picks AR PL ShanHeiSun
> > for open single quotes.  Putting the script definitions back in fixes
> > the problem.  I have pasted "good" and "bad" font log blocks to the
> > end of this message.  Those logs are particularly for Neep Alt, but
> > the Misc Fixed ISO10646 fonts have the same problem.
> 
> [...]
> > Unsuccessful load of LEFT SINGLE QUOTE with new definition:
> 
> > font for: (8216)
> > ftfont-list: -jmk-Neep Alt-*-iso10646-1
> [...]
> >   -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
> [...]
> >   -jmk-Neep Alt-normal-normal-normal-*-20-*-c-*-iso10646-1
> 
> Don't you happen to have two sets of "Neep Alt" fonts in the
> fontconfig's fontpath; one set supports LEFT SINGLE QUOTE,
> and the other doesn't?

Neep Alt was originally designed for ISO8859-1 encodings, and my
installation still has iso18859-1 fonts that definitely lack LEFT
SINGLE QUOTE.  However, I constructed additional fonts to support
ISO10646-1, by extracting the Misc Fixed glyphs outside of ISO8859-1,
and appending them to the Neep Alt font definitions.  Those fonts are
defined to have the iso10646-1 encoding.

When I first noticed the font display problem a couple of months ago,
I traced the font code in the debugger and found that Emacs considered
the ISO8859-1 fonts to be valid matches for LEFT SINGLE QUOTE and
similar codepoints.  It would:

  * load the ISO8859-1 font, rather than the ISO10646-1 font,
  * notice that LEFT SINGLE QUOTE was missing, and then
  * start looking in other font families for the codepoint.

Adding LEFT SINGLE QUOTE and a couple of others as "symbol"
representatives let Emacs filter out the ISO8859-1 fonts earlier in
the code path, leaving the ISO10646-1 fonts for Emacs to use.  The
Misc Fixed fonts had the same problem, as they also included ISO8859-1
and ISO10646-1 encodings, and this solution worked for them as well.

One approach that we might consider is to change the C code, so that
any ISO8859-1 font is immediately skipped when the codepoint is 256 or
greater.  Or we could change the font loading code to try multiple
encodings within a family before dropping back to other families.

Derek

Re: bug#613: 23.0.60; Unicode display problems

[Kenichi Handa]

In article <18614.49466.916241.572583@priss.frightenedpiglet.com>, sand@blarg.net writes:

> Neep Alt was originally designed for ISO8859-1 encodings,
> and my installation still has iso18859-1 fonts that
> definitely lack LEFT SINGLE QUOTE.  However, I constructed
> additional fonts to support ISO10646-1, by extracting the
> Misc Fixed glyphs outside of ISO8859-1, and appending them
> to the Neep Alt font definitions.  Those fonts are defined
> to have the iso10646-1 encoding.

Ah, I see.

> Adding LEFT SINGLE QUOTE and a couple of others as
> "symbol" representatives let Emacs filter out the
> ISO8859-1 fonts earlier in the code path, leaving the
> ISO10646-1 fonts for Emacs to use.  The Misc Fixed fonts
> had the same problem, as they also included ISO8859-1 and
> ISO10646-1 encodings, and this solution worked for them as
> well.

I re-added `symbol' entry to script-representative-chars but
with vector of characters as this:

	(symbol . [#x201C #x2200 #x2500])

and modified ftfont.c to handle that kind of format to
fileter out fonts that have none of characters in the
vector.

Please you try again with the latest code.

---
Kenichi Handa
handa@ni.aist.go.jp

bug#613: 23.0.60; Unicode display problems

[sand]

Kenichi Handa writes:
> I re-added `symbol' entry to script-representative-chars but
> with vector of characters as this:
> 
> 	(symbol . [#x201C #x2200 #x2500])
> 
> and modified ftfont.c to handle that kind of format to
> fileter out fonts that have none of characters in the
> vector.
> 
> Please you try again with the latest code.

I synced to CVS HEAD, and removed my local overrides.  The HELLO
document and UTF-8-demo document look fine now.  Thanks.

Derek