Display problems with non-ascii characters

Display problems with non-ascii characters

[Sascha Wilde]

Hi *,

I'm experiencing some strange display problems with non-ascii characters
for some time -- a saw various threads on the list which seemed to be
related, bus as they still persist I decided to write this report:

Emacs 23 CVS head, --with-x-toolkit=lucid

To reproduce:

; I use the font "terminus" http://www.is-vn.bg/hamster/ included in
; many GNU/Linux Distributions, for example Debian

1. emacs -Q

2. ; Type some German umlauts in *scratch*:
   ÄÖÜßäöü

3. ; evaluate:
   (set-face-font 'default "terminus 12")
   ; everything looks fine, but the umlauts disappeared and when you
   ; move the text cursor above them even the cursor disappears.

4. ; evaluate:
   (set-face-font 'default "terminus 11")
   ; the umlauts reappear.

Additional information:

- As 4. shows the problem seems to be related to some sizes (which used
  to work in Emacs 22).

- When you jump in an buffer in such an way, that the (invisible)
  umlauts would appear at the same position where some other character
  was drawn before, theses other characters stay on the screen.  So it
  seems that on the position of the umlauts no redraw takes place at
  all.

cheers
sascha
-- 
Sascha Wilde  :  "I heard that if you play the Windows CD backward, you
              :  get a satanic message. But that's nothing compared to
              :  when you play it forward: It installs Windows...." 
              :  -- G. R. Gaudreau

Re: Display problems with non-ascii characters

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 1015 bytes --]

In article <m2k5cldg0q.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

> Emacs 23 CVS head, --with-x-toolkit=lucid

> To reproduce:

> ; I use the font "terminus" http://www.is-vn.bg/hamster/ included in
> ; many GNU/Linux Distributions, for example Debian

I've just installed the debian package xfonts-terminus, and
followed what you did.

> 1. emacs -Q

> 2. ; Type some German umlauts in *scratch*:
>    ^[$(D*#*S*d)N+#+S+d^[(B

> 3. ; evaluate:
>    (set-face-font 'default "terminus 12")
>    ; everything looks fine, but the umlauts disappeared and when you
>    ; move the text cursor above them even the cursor disappears.

This bug doesn't happen for me.  The font used for those
German letters is:
    x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 (#xC4)

> 4. ; evaluate:
>    (set-face-font 'default "terminus 11")
>    ; the umlauts reappear.

This doesn't change the font for me.  Which font is used in
your case in 3 and 4?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Display problems with non-ascii characters

[Sascha Wilde]

Kenichi Handa <handa@m17n.org> wrote:
> In article <m2k5cldg0q.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:
>
>> Emacs 23 CVS head, --with-x-toolkit=lucid
>
>> To reproduce:
>
>> ; I use the font "terminus" http://www.is-vn.bg/hamster/ included in
>> ; many GNU/Linux Distributions, for example Debian
>
> I've just installed the debian package xfonts-terminus, and
> followed what you did.
>
>> 1. emacs -Q
>
>> 2. ; Type some German umlauts in *scratch*:
>>    ÄÖÜßäöü
>
>> 3. ; evaluate:
>>    (set-face-font 'default "terminus 12")
>>    ; everything looks fine, but the umlauts disappeared and when you
>>    ; move the text cursor above them even the cursor disappears.
>
> This bug doesn't happen for me. 

Stange enough: for me too (nightly cvs builds, so this is no more the
same CVS head), but in normal use the problem is still there...

Indeed this is a very annoying heisen-bug: I see the same symptoms under
varying conditions all the time, but its hard to get a reproduction
receipt.  A co-worker of mine has the same problems with a different
font -- both fonts have in common, that they are x11 bitmap fonts.

Maybe the used toolkit is a factor two?  This is with lucid (haven't
tested gtk2)...

> The font used for those German letters is:
>     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 (#xC4)

>> 4. ; evaluate:
>>    (set-face-font 'default "terminus 11")
>>    ; the umlauts reappear.
>
> This doesn't change the font for me.  Which font is used in
> your case in 3 and 4?

For 3 (Terminus 12):
  x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 
and for 4 (Terminus 11):
  x:-xos4-terminus-medium-r-normal--14-140-72-72-c-80-iso10646-1

cheers
sascha
-- 
Sascha Wilde
"Structure is _nothing_ if it is all you got.  Skeletons _spook_ people if
 thwy try to walk around on their own.  I really wonder why XML does
 not."            -- Erik Naggum <erik@naggum.net> in comp.lang.lisp

Re: Display problems with non-ascii characters

[Kenichi Handa]

In article <m2myhg2ug9.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

>>> 3. ; evaluate:
>>> (set-face-font 'default "terminus 12")
>>> ; everything looks fine, but the umlauts disappeared and when you
>>> ; move the text cursor above them even the cursor disappears.
> >
> > This bug doesn't happen for me. 

> Stange enough: for me too (nightly cvs builds, so this is no more the
> same CVS head), but in normal use the problem is still there...

Which font is used in that "normal use"?  C-u C-x = tells
the font name.

> Indeed this is a very annoying heisen-bug: I see the same symptoms under
> varying conditions all the time, but its hard to get a reproduction
> receipt.  A co-worker of mine has the same problems with a different
> font -- both fonts have in common, that they are x11 bitmap fonts.

> Maybe the used toolkit is a factor two?  This is with lucid (haven't
> tested gtk2)...

I don't think so because the display routine doesn't use
toolkit's renderer.

> > The font used for those German letters is:
> >     x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 (#xC4)

>>> 4. ; evaluate:
>>> (set-face-font 'default "terminus 11")
>>> ; the umlauts reappear.
> >
> > This doesn't change the font for me.  Which font is used in
> > your case in 3 and 4?

> For 3 (Terminus 12):
>   x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 
> and for 4 (Terminus 11):
>   x:-xos4-terminus-medium-r-normal--14-140-72-72-c-80-iso10646-1

It seems that your display has the different resolution than
mine (100dpi).  What resolution is yours (xdpyinfo command
tells it)?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Display problems with non-ascii characters

[Sascha Wilde]

Kenichi Handa <handa@m17n.org> wrote:
> In article <m2myhg2ug9.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:
>
>>>> 3. ; evaluate:
>>>> (set-face-font 'default "terminus 12")
>>>> ; everything looks fine, but the umlauts disappeared and when you
>>>> ; move the text cursor above them even the cursor disappears.
>> >
>> > This bug doesn't happen for me. 
>
>> Stange enough: for me too (nightly cvs builds, so this is no more the
>> same CVS head), but in normal use the problem is still there...
>
> Which font is used in that "normal use"?  C-u C-x = tells
> the font name.

x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso8859-1

[...]
>> > This doesn't change the font for me.  Which font is used in
>> > your case in 3 and 4?
>
>> For 3 (Terminus 12):
>>   x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 
>> and for 4 (Terminus 11):
>>   x:-xos4-terminus-medium-r-normal--14-140-72-72-c-80-iso10646-1
>
> It seems that your display has the different resolution than
> mine (100dpi).  What resolution is yours (xdpyinfo command
> tells it)?

resolution:    96x96 dots per inch

cheers
an many thanks for investigating this
sascha
-- 
Sascha Wilde   -.-. ..- .-. .. --- ... .. - -.-- 
               -.- .. .-.. .-.. . -.. 
               - .... .
               -.-. .- -

Re: Display problems with non-ascii characters

[Kenichi Handa]

In article <m24p3n34xc.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

>>>>> 3. ; evaluate:
>>>>> (set-face-font 'default "terminus 12")
>>>>> ; everything looks fine, but the umlauts disappeared and when you
>>>>> ; move the text cursor above them even the cursor disappears.
>>> >
>>> > This bug doesn't happen for me. 
> >
>>> Stange enough: for me too (nightly cvs builds, so this is no more the
>>> same CVS head), but in normal use the problem is still there...
> >
> > Which font is used in that "normal use"?  C-u C-x = tells
> > the font name.

> x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso8859-1

You at first wrote that in the case of "terminus 12", the
font used was:

x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1

It's different from the above only in CHARSET_REGISTRY
field, but it had the problem.  Now you wrote that the case
of "terminus 12" doesn't show a problem but "normal use"
shows the problem.  Does it mean that the font used for
"terminus 12" is different now?

>>> For 3 (Terminus 12):
>>> x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 
>>> and for 4 (Terminus 11):
>>> x:-xos4-terminus-medium-r-normal--14-140-72-72-c-80-iso10646-1
> >
> > It seems that your display has the different resolution than
> > mine (100dpi).  What resolution is yours (xdpyinfo command
> > tells it)?

> resolution:    96x96 dots per inch

Ah, that's why the 14 dots font is selected for terminus 11
point in your case.

Your 96 DPI case:
(* 96 (/ 11 72.27)) => 14.6  ==> 14-dot font matches best
(* 96 (/ 12 72.27)) => 15.9  ==> 16-dot font matches best

My 100 DPI case
(* 100 (/ 11 72.27)) => 15.2 ==> 16-dot font matches best
(* 100 (/ 12 72.27)) => 16.6 ==> 16-dot font matches best

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Display problems with non-ascii characters

[Sascha Wilde]

Kenichi Handa <handa@m17n.org> wrote:
> In article <m24p3n34xc.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:
>
>>>> > This bug doesn't happen for me. 
>> >
>>>> Stange enough: for me too (nightly cvs builds, so this is no more the
>>>> same CVS head), but in normal use the problem is still there...
>> >
>> > Which font is used in that "normal use"?  C-u C-x = tells
>> > the font name.
>
>> x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso8859-1
>
> You at first wrote that in the case of "terminus 12", the
> font used was:
>
> x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1
>
> It's different from the above only in CHARSET_REGISTRY
> field, but it had the problem.  Now you wrote that the case
> of "terminus 12" doesn't show a problem but "normal use"
> shows the problem.  Does it mean that the font used for
> "terminus 12" is different now?

Seems so.  Actually never qualified the CHARSET_REGISTRY part.  There
are various ways im using to set/change the default font:

1.  in my .emacs file:

   (custom-set-faces
     ;; custom-set-faces was added by Custom.
     ;; If you edit it by hand, you could mess it up, so be careful.
     ;; Your init file should contain only one such instance.
     ;; If there is more than one, they won't work right.
    '(default ((t (:inherit nil :stipple nil :background "Black"
      :foreground "Gray90" :inverse-video nil :box nil :strike-through nil
      :overline nil :underline nil :slant normal :weight normal :height 140
      :width normal :foundry "xos4" :family "terminus"))))
    ; ...
    )

2.  in some code I wrote for fast font switching which uses
    set-default-font and setting font in default-frame-alist

3.  in my (broken) test case: set-face-font for 'default

BUT in all cases I used "terminus 11", "terminus 12" and that like, no
further qualification of the charset...

So it seems that Emacs made different choices under certain conditions.

FWIW: for the first time in weeks I didn't experienced the bug today
(without changing any configuration!) -- maybe its shy and talking about
it makes it go away..?

cheers
sascha
-- 
Sascha Wilde

"Unix was the first OS where you could carry the media and system
documentation around in a briefcase. This was fixed in BSD4.2."

Re: Display problems with non-ascii characters

[Sascha Wilde]

Sascha Wilde <wilde@sha-bang.de> wrote:
> Kenichi Handa <handa@m17n.org> wrote:
>> In article <m24p3n34xc.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes
> FWIW: for the first time in weeks I didn't experienced the bug today
> (without changing any configuration!) -- maybe its shy and talking about
> it makes it go away..?

Well, no reason to believe in miracles:  Today I experienced the bug
again an so I came up with a new receipt to reproduce it.  Its simpler
than the first try and it showed for me the bug reproducible:

1. emacs -Q

2. C-h H    ; to bring up the Hello file
            ; watch out for the German line: "Guten Tag, Grüß Gott"

3. M-x set-default-font RET terminus RET
            ; now the bug shows: the umlauts are gone the line looks
            ; like: "Guten Tag, Gr   Gott"

C-u C-x =  on the vanished `ß' shows:

--8<---------------cut here---------------start------------->8---
        character: ß (223, #o337, #xdf)
preferred charset: latin-iso8859-1 (Right-Hand Part of ISO/IEC 8859/1
(Latin-1): ISO-IR-100)
       code point: 0x5F
           syntax: w    which means: word
         category: h:Korean j:Japanese l:Latin
      buffer code: #xC3 #x9F
        file code: ESC #x2C #x41 #x5F (encoded by coding system
        iso-2022-7bit-unix)
          display: by this font (glyph code)
    x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1
    (#xDF)

Character code properties: customize what to show
  name: LATIN SMALL LETTER SHARP S
  general-category: Ll (Letter, Lowercase)

There are text properties here:
  charset              latin-iso8859-1
--8<---------------cut here---------------end--------------->8---

(well of cause the `ß' in the output above is invisible, too).

Can you reproduce it this way?
Any new ideas?

cheers
sascha
-- 
Sascha Wilde  :  "The PROPER way to handle HTML postings is to cancel
the article, then hire a hitman to kill the poster, his wife and kids,
and fuck his dog and smash his computer into little bits. Anything
more is just extremism."  -- Paul Tomblin

Re: Display problems with non-ascii characters

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 935 bytes --]

In article <m2k5chg2qq.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

> Well, no reason to believe in miracles:  Today I experienced the bug
> again an so I came up with a new receipt to reproduce it.  Its simpler
> than the first try and it showed for me the bug reproducible:

> 1. emacs -Q

> 2. C-h H    ; to bring up the Hello file
>             ; watch out for the German line: "Guten Tag, Gr^[$(D+d)N^[(B Gott"

> 3. M-x set-default-font RET terminus RET
>             ; now the bug shows: the umlauts are gone the line looks
>             ; like: "Guten Tag, Gr   Gott"

> C-u C-x =  on the vanished `^[$(D)N^[(B' shows:

I still can't reproduce it.  C-u C-x = shows exactly the
same output as yours (except for that ^[$(D)N^[(B is shown
correctly).

By the way, in step 3, do you really see 3 spaces between
"Gr" and "Gott"?  Aren't there any other vanishing
characters?

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Display problems with non-ascii characters

[Sascha Wilde]

Kenichi Handa <handa@m17n.org> wrote:
> In article <m2k5chg2qq.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:
>> 1. emacs -Q
>
>> 2. C-h H    ; to bring up the Hello file
>>             ; watch out for the German line: "Guten Tag, Grüß Gott"
>
>> 3. M-x set-default-font RET terminus RET
>>             ; now the bug shows: the umlauts are gone the line looks
>>             ; like: "Guten Tag, Gr   Gott"
>
>> C-u C-x =  on the vanished `ß' shows:
>
> I still can't reproduce it.

I have some new findings on that, details below...

[...]
> By the way, in step 3, do you really see 3 spaces between
> "Gr" and "Gott"?  Aren't there any other vanishing
> characters?

Yes, 3: one for the missing ü one for the missing ß and the one which
belongs there (between the two words).

But please note that the two spaces which are really the vanished
Umlauts are more like "holes" in the X display: When I move the cursor
on them, the cursor vanishes, too -- and when I jump in the buffer, so
that the vanished chars would be displayed in a place where an other
character was displayed before, this character stays, no space, but the
wrong character, with fontification, color and all, just as if the
missing characters left a hole, do redraw takes place where they would
be, the space isn't even cleared...

That said, here are my latest findings:

Thirst thing I did this morning was to try my one receipt, with no
success(sic!) -- thirst thought: a change in the nightly build fixed or
at least changed it.

So I started my Emacs without `-Q' and started to work, running gnus I
immediately noticed a missing Umlaut in one of the mails Subjects...
:-( 

So I re tried my receipt with a fresh `emacs -Q' (running on a different
machine, but on the same X11 display) and what shall I say?  It "worked"
again, the bug was back...

So It seems the problem might really be a strange X11 bug, which is
triggered by the new Emacs display code?

Any Ideas how to debug this?
(FWIW this is xorg 7.1.0-19 from Debian Etch)

cheers
sascha
-- 
Sascha Wilde : VI is to EMACS as masturbation is to making love:
             : effective and always available but probably not your
             : first choice...

Re: Display problems with non-ascii characters

[Sascha Wilde]

Sascha Wilde <wilde@sha-bang.de> wrote:
> Kenichi Handa <handa@m17n.org> wrote:
>> In article <m2k5chg2qq.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

Thanks for you help and patience, Handa-san.

I'm not intending to be a pita, but this is still an issue and I think
it should be further investigated and resolved.

If I can help, by sending debugger info or the like please tell me what
to do.

On a second thought I think that my statement:
> So It seems the problem might really be a strange X11 bug, which is
> triggered by the new Emacs display code?

is somewhat to much of an excuse:  The problem really only shows with
recent Emacs, not with any other application where I'm using exactly the
same fonts, nor with older (22) Emacsn.

More additional information, I don't know if its useful but it might
contain some hints for those who know the dirty details of Emacs
display...  ;-)

I noticed the following phenomenon (which is really an extended version
of my observations from the last mail):

I have two systems, A and B, and two users foo and bar, they both have
access to the very same Emacs binary (over nfs) and they both can use
the same X display (via ssh -X).
My X-server runs an system A.

1. I start Emacs with -Q as user bar on system B and do my C-h H test.
   No problem when changing the default font to terminus.
   After that I quit Emacs.

2. I start Emacs as user foo on system A and do some work, until the
   "vanished special chars" bug shows up.  I keep that Emacs running and
   ...

3. ... start again Emacs -Q as bar on system B and do my C-h H test.
   After changing the default font to terminus the umlauts are vanished
   in this instance of Emacs, too!

   Please note that the only thing the two Emacsn have in common is the
   X Display, as they run on different hosts by different users.

4. Now: when I quit both Emacsn and restart at 1. the bug is gone again!

   So it affects all Emacsn on the same display once it appeared, but it
   "heals" again when all affected Emacsn are closed...

Any ideas?  Insights?  Suggestions on how to proceed debugging this?

cheers
sascha
-- 
Sascha Wilde
Well, *my* brain likes to think it's vastly more powerful than any
finite Turing machine but it hasn't proven that to me...
  -- Christopher Koppler in comp.lang.lisp

Re: Display problems with non-ascii characters

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 3097 bytes --]

In article <m2hc7fsfmf.fsf@kenny.sha-bang.de>, Sascha Wilde <wilde@sha-bang.de> writes:

> I'm not intending to be a pita, but this is still an issue and I think
> it should be further investigated and resolved.

Thank you for keeping on attacking this problem.  But...

> If I can help, by sending debugger info or the like please tell me what
> to do.

as the reported phenomenon is so mysterious that I can't
think of any good debugging strategy.  :-(

> On a second thought I think that my statement:
> > So It seems the problem might really be a strange X11 bug, which is
> > triggered by the new Emacs display code?

> is somewhat to much of an excuse:  The problem really only shows with
> recent Emacs, not with any other application where I'm using exactly the
> same fonts, nor with older (22) Emacsn.

> More additional information, I don't know if its useful but it might
> contain some hints for those who know the dirty details of Emacs
> display...  ;-)

> I noticed the following phenomenon (which is really an extended version
> of my observations from the last mail):

> I have two systems, A and B, and two users foo and bar, they both have
> access to the very same Emacs binary (over nfs) and they both can use
> the same X display (via ssh -X).
> My X-server runs an system A.

> 1. I start Emacs with -Q as user bar on system B and do my C-h H test.
>    No problem when changing the default font to terminus.
>    After that I quit Emacs.

> 2. I start Emacs as user foo on system A and do some work, until the
>    "vanished special chars" bug shows up.  I keep that Emacs running and
>    ...

> 3. ... start again Emacs -Q as bar on system B and do my C-h H test.
>    After changing the default font to terminus the umlauts are vanished
>    in this instance of Emacs, too!

>    Please note that the only thing the two Emacsn have in common is the
>    X Display, as they run on different hosts by different users.

> 4. Now: when I quit both Emacsn and restart at 1. the bug is gone again!

>    So it affects all Emacsn on the same display once it appeared, but it
>    "heals" again when all affected Emacsn are closed...

> Any ideas?  Insights?  Suggestions on how to proceed debugging this?

As far as I can't reproduce the problem, I think there's no
way other than you debug Emacs by gdb.

As you wrote:
----------------------------------------------------------------------
C-u C-x =  on the vanished `^[$(D)N^[(B' shows:
[...]
    x:-xos4-terminus-medium-r-normal--16-160-72-72-c-80-iso10646-1 (#xDF)
----------------------------------------------------------------------

Please find a way to reproduce the problem with a buffer
that contains only "^[$(D)N^[(B".  Then, run Emacs under gdb as this:

% cd .../src/emacs
% gdb emacs
(gdb) br xfont.c:833
(gdb) cond 3 len==1&& s->char2b.byte2==0xDF
(gdb) run

and repeat the above procedure.  Emacs should stop at line
841.  Then try this and see how screen changes:

(gdb) ff
(gdb) fin
(gdb) ff

See also emacs/etc/DEBUG.  It has many hints for debugging
Emacs with gdb.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Display problems with non-ascii characters

[Sascha Wilde]

Kenichi Handa <handa@m17n.org> wrote:

Thanks for your reply.

> As far as I can't reproduce the problem, I think there's no
> way other than you debug Emacs by gdb.

Thanks for the hints on that.  I'll give it a try asap.

cheers
sascha
-- 
Sascha Wilde

"Unix was the first OS where you could carry the media and system
documentation around in a briefcase. This was fixed in BSD4.2."