23.0.60; GNU Emacs does not handle composed characters

23.0.60; GNU Emacs does not handle composed characters

[Peter Dyballa]

Hello!

When pasting a copy of a filled in text from OmniWeb (a Web browser  
coming from the NeXT) into a file buffer in GNU Emacs (mode-line  
starts with -U:), which was launched with -Q, the German umlauts ä,  
ö, ü, Ä, Ö, Ü are stripped off their diaereses and mutate to a, o, u,  
A, O, U. Typing C-u C-x = on such a Marcel Duchamps character leads  
to an error message in echo area:

	Format specifier doesn't match argument type

In *Messages* buffer a ``describe-char: ´´is prepended the text.

In dired buffers the same umlauts in file names are stripped, while  
the date fields can display an abbreviated month's name as ``Mär´´.  
Typing on this ä C-c C-x = I get:

	        character: ä (228, #o344, #xe4)
	preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
	       code point: 0xE4
	           syntax: w 	which means: word
	         category: j:Japanese l:Latin
	      buffer code: #xC3 #xA4
	        file code: #xC3 #xA4 (encoded by coding system utf-8-unix)
	          display: by this font (glyph code)
	     -B&H-LucidaTypewriter-Medium-R-Normal-Sans-10-100-75-75-M-60- 
ISO8859-1 (#xE4)
	
	Character code properties are not shown: customize what to show
	
	There are text properties here:
	  auto-composed        t
	  fontified            t

instead of an error message. The error message is output when I type  
on as tripped umlaut in a file name!


Looking with GNU Emacs 23.0.50 into the file saved with its stripped  
umlauts I see instead of them ``{äöüÄÖÜ}<an open box>´. This box is  
described in GNU Emacs 23.0.50 as:

	        character:  (332488, #o1211310, #x512c8, U+0308)
	          charset: mule-unicode-0100-24ff
			   (Unicode characters of the range U+0100..U+24FF.)
	       code point: #x25 #x48
	           syntax: w 	which means: word
	         category: ^:Combining diacritic or mark
	      buffer code: #x9C #xF4 #xA5 #xC8
	        file code: #xCC #x88 (encoded by coding system mule-utf-8-unix)
	          display: by this font (glyph code)
	     -b&h-lucida sans typewriter-medium-r-normal--10-98-74-74-m-60- 
iso10646-1 (#x308)
	     Unicode data:
	             Name: COMBINING DIAERESIS
	         Category: non-spacing mark
	  Combining class: Above
	    Bidi category: Non-Spacing Mark
	         Old name: NON-SPACING DIAERESIS
	ISO 10646 comment: Dialytika


In GNU Emacs 23.0.60.1 (powerpc-apple-darwin8.11.0, GTK+ Version 2.6.10)
  of 2007-12-25 on Latsche.local
Windowing system distributor `The XFree86 Project, Inc', version  
11.0.40400000
configured using `configure  '--enable-font-backend' '--with-x- 
toolkit=gtk' '--with-dbus' '--without-sound' '--without-pop' '--with- 
xpm' '--with-jpeg' '--with-tiff' '--with-gif' '--with-png' '--enable- 
locallisppath=/Library/Application Support/Emacs/calendar22:/Library/ 
Application Support/Emacs/caml:/Library/Application Support/Emacs:/sw/ 
share/emacs21/site-lisp/elib' 'PKG_CONFIG_PATH=/sw/lib/freetype219/ 
lib/pkgconfig:/sw/lib/fontconfig2/lib/pkgconfig:/sw/lib/pkgconfig:/sw/ 
lib/system-openssl/lib/pkgconfig:/sw/share/pkgconfig:/usr/lib/ 
pkgconfig:/usr/local/lib/pkgconfig:/usr/local/clamXav/lib/pkgconfig:/ 
usr/local/lib/pkgconfig' 'CPPFLAGS=-no-cpp-precomp -D__BIND_NOSTATIC - 
I/usr/include/openssl -I/sw/include/pango-1.0 -I/sw/lib/fontconfig2/ 
include -I/sw/lib/freetype219/include -I/sw/lib/freetype219/include/ 
freetype2 -I/sw/include -I/usr/local/include -idirafter /usr/X11R6/ 
include' 'CXXFLAGS=-no-cpp-precomp -I/usr/include/openssl -I/sw/ 
include/pango-1.0 -I/sw/lib/fontconfig2/include -I/sw/lib/freetype219/ 
include -I/sw/lib/freetype219/include/freetype2 -I/sw/include -I/usr/ 
local/include' 'CFLAGS=-ggdb -g3 -gfull -H -bind_at_load -pipe -fPIC - 
mcpu=7450 -mtune=7450 -O0' 'LDFLAGS=-dead_strip -multiply_defined  
suppress -L/sw/lib/ncurses -L/sw/lib/fontconfig2/lib -L/sw/lib/ 
freetype219/lib -L/sw/lib -L/usr/local/lib -L/usr/X11R6/lib''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: de_DE.UTF-8
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: de_DE.UTF-8
   value of $XMODIFIERS: nil
   locale-coding-system: utf-8-unix
   default-enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
   tooltip-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   global-auto-composition-mode: t
   auto-composition-mode: t
   auto-compression-mode: t
   line-number-mode: t

Recent input:
<down-mouse-1> <mouse-1> C-x C-f D a t e i <return>
<down-mouse-2> <mouse-2> <down-mouse-1> <mouse-1> C-x


--
Greetings

   Pete

America believes in education: the average professor earns more money  
in a year than a professional athlete earns in a whole week.
				­ Evan Esar

Re: 23.0.60; GNU Emacs does not handle composed characters

[Andreas Schwab]

Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> When pasting a copy of a filled in text from OmniWeb (a Web browser coming
> from the NeXT) into a file buffer in GNU Emacs (mode-line  starts with
> -U:), which was launched with -Q, the German umlauts ä,  ö, ü, Ä, Ö, Ü are
> stripped off their diaereses and mutate to a, o, u,  A, O, U. Typing C-u
> C-x = on such a Marcel Duchamps character leads  to an error message in
> echo area:
>
> 	Format specifier doesn't match argument type

Did you configure with --enable-font-backend?  I see the same error with
tibetan characters in the HELLO file, and it goes away when I disable
the xft support.  The error manifests with internal-char-font returning
only a single element list for these characters.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: 23.0.60; GNU Emacs does not handle composed characters

[Peter Dyballa]

Am 26.12.2007 um 18:10 schrieb Andreas Schwab:

> Did you configure with --enable-font-backend?


Yes. GNU Emacs launches for me with enabled font backend when I have  
amongst my X resources

	Emacs.FontBackend:              x

instead of

	Emacs.FontBackend:              xft

(which makes it crash) so I see now the bug.

--
Mit friedvollen Grüßen

   Pete

Rain is saved up in cloud banks.

Re: 23.0.60; GNU Emacs does not handle composed characters

[Peter Dyballa]

Am 26.12.2007 um 18:10 schrieb Andreas Schwab:

> Peter Dyballa <Peter_Dyballa> writes:
>
>> When pasting a copy of a filled in text from OmniWeb (a Web  
>> browser coming
>> from the NeXT) into a file buffer in GNU Emacs (mode-line  starts  
>> with
>> -U:), which was launched with -Q, the German umlauts ä,  ö, ü, Ä,  
>> Ö, Ü are
>> stripped off their diaereses and mutate to a, o, u,  A, O, U.  
>> Typing C-u
>> C-x = on such a Marcel Duchamps character leads  to an error  
>> message in
>> echo area:
>>
>> 	Format specifier doesn't match argument type
>
> Did you configure with --enable-font-backend?  I see the same error  
> with
> tibetan characters in the HELLO file, and it goes away when I disable
> the xft support.  The error manifests with internal-char-font  
> returning
> only a single element list for these characters.
>

Hello!

Today I did an update from CVS and configured without font-backend.  
The problem stays the same.


--
Mit friedvollen Grüßen

   Pete

They that can give up essential liberty to obtain a little temporary  
safety deserve neither liberty nor safety.
		-Benjamin Franklin, Historical Review of Pennsylvania.

Re: 23.0.60; GNU Emacs does not handle composed characters

[James Cloos]

I just tried out some file (and file name in dired) where I expected to
find composed character sequences to compare against Peter’s findings.

I didn’t find the composed sequences, but did find that the character
U+1E08 LATIN CAPITAL LETTER C WITH CEDILLA AND ACUTE is being displayed
as the glyph for the character U+013D LATIN CAPITAL LETTER L WITH CARON.

U+013D itself also gets that glyph.

I have the following in my ~/.emacs:

,----[ excerpt from ~/.emacs ]
| (defun jhc-what-font () "" (interactive)
|   (message "%s" (aref (query-font (font-at (point))) 1)))
`----

and that returns my default font, DejaVu Sans Mono, for U+1E08.

The font does have the needed glyph, and it does display correctly
in other apps.

... Does more searching ...

I did find at least one file with a composed sequence.  It contains a
zero followed by U+0338 COMBINING LONG SOLIDUS OVERLAY.  That pair works
correctly for me in Emacs.  The following line also seems to work,
although I cannot say for sure; it does at least look better than it
does in my terminal app:

,----[ mhadmamrajamapt.utf8 ]
| မဟာဓမ္မရာဇာမ္‌ပတ
`----

This one is definitely ok:

,----[ macron.utf8.txt ]
| m̄ M̄ n̄ N̄ ē Ē ō Ō
| U+0304  ̄
`----

That is all I could find.

I run the unicode-2 branch on an x86 gentoo box using X11 frames.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: 23.0.60; GNU Emacs does not handle composed characters

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 1981 bytes --]

Sorry for the late response.

In article <ED11F1E7-697B-4B6C-8906-0223E4F4017D@Freenet.DE>, Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> When pasting a copy of a filled in text from OmniWeb (a Web browser  
> coming from the NeXT) into a file buffer in GNU Emacs (mode-line  
> starts with -U:), which was launched with -Q, the German umlauts ^[$(D+#^[(B,  
> ^[$(D+S^[(B, ^[$(D+d^[(B, ^[$(D*#^[(B, ^[$(D*S^[(B, ^[$(D*d^[(B are stripped off their diaereses and mutate to a, o, u,  
> A, O, U. Typing C-u C-x = on such a Marcel Duchamps character leads  
> to an error message in echo area:

> 	Format specifier doesn't match argument type

> In *Messages* buffer a ``describe-char: ^[$B!-!-^[(Bis prepended the text.

I can't reproduce that.  Please do M-x load-library RET
descr-text.el RET, M-x toggle-debug-on-error RET, and try C-u
C-x = again.  When you get *Backtrace* buffer, please show
me the contents.

> In dired buffers the same umlauts in file names are stripped, while  
> the date fields can display an abbreviated month's name as ``M^[$(D+#^[(Br^[$B!-!-^[(B.  
> Typing on this ^[$(D+#^[(B C-c C-x = I get:

> 	        character: ^[$(D+#^[(B (228, #o344, #xe4)
> 	preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
> 	       code point: 0xE4
> 	           syntax: w 	which means: word
> 	         category: j:Japanese l:Latin
> 	      buffer code: #xC3 #xA4
> 	        file code: #xC3 #xA4 (encoded by coding system utf-8-unix)
> 	          display: by this font (glyph code)
> 	     -B&H-LucidaTypewriter-Medium-R-Normal-Sans-10-100-75-75-M-60- 
> ISO8859-1 (#xE4)
	
> 	Character code properties are not shown: customize what to show
	
> 	There are text properties here:
> 	  auto-composed        t
> 	  fontified            t

> instead of an error message. The error message is output when I type  
> on as tripped umlaut in a file name!

That perhaps means that the umlaut is not stripped off but
just can't be displayed properly.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: 23.0.60; GNU Emacs does not handle composed characters

[Peter Dyballa]

Am 08.01.2008 um 07:29 schrieb Kenichi Handa:

>> 	Format specifier doesn't match argument type
>
>> In *Messages* buffer a ``describe-char: ´´is prepended the text.
>
> I can't reproduce that.  Please do M-x load-library RET
> descr-text.el RET, M-x toggle-debug-on-error RET, and try C-u
> C-x = again.  When you get *Backtrace* buffer, please show
> me the contents.
>


Debugger entered--Lisp error: (error "Format specifier doesn't match  
argument type")
   format("%s (#x%02X)" "-monotype-arial unicode ms-medium-r- 
normal--13-120-75-75-p-129-gb18030.2000-0" "(33072 . 48436)")
   (if (cdr elt) (format "%s (#x%02X)" (cadr elt) (cddr elt)) "-- no  
font --")
   (insert "\n " (car elt) 58 (propertize " " (quote display)  
(quote ...)) (if (cdr elt) (format "%s (#x%02X)" ... ...) "-- no font  
--"))
   (if (/= (car elt) 9) (insert "\n " (car elt) 58 (propertize "  
" ... ...) (if ... ... "-- no font --")))
   (while --dolist-tail-- (setq elt (car --dolist-tail--)) (if (/ 
= ... 9) (insert "\n " ... 58 ... ...)) (setq --dolist-tail-- (cdr -- 
dolist-tail--)))
   (let ((--dolist-tail-- component-chars) elt) (while --dolist- 
tail-- (setq elt ...) (if ... ...) (setq --dolist-tail-- ...)))
   (dolist (elt component-chars) (if (/= ... 9) (insert "\n " ...  
58 ... ...)))
   (progn (insert "these fonts (glyph codes):") (dolist (elt  
component-chars) (if ... ...)))
   (if (display-graphic-p (selected-frame)) (progn (insert "these  
fonts (glyph codes):") (dolist ... ...)) (insert "these terminal  
codes:") (dolist (elt component-chars) (insert "\n  " ... ":" ... ...)))
   (if (and (vectorp ...) (vectorp ...)) (progn (insert " using this  
font:\n  " ... "\nby these glyphs:\n") (mapc ... ...)) (insert " by  
the rule:\n	(" (mapconcat ... ... " ") ")") (insert "\nThe component  
character(s) are displayed by ") (if (display-graphic-p ...)  
(progn ... ...) (insert "these terminal codes:") (dolist ... ...))  
(insert "\nSee the variable `reference-point-alist' for " "the  
meaning of the rule.\n"))
   (progn (insert "\nComposed") (if (car composition)  
(if ... ... ...) (if ... ...)) (if (and ... ...) (progn ... ...)  
(insert " by the rule:\n	(" ... ")") (insert "\nThe component  
character(s) are displayed by ") (if ... ... ... ...) (insert "\nSee  
the variable `reference-point-alist' for " "the meaning of the rule. 
\n")))
   (if composition (progn (insert "\nComposed") (if ... ... ...)  
(if ... ... ... ... ... ...)))
   (when composition (insert "\nComposed") (if (car composition)  
(if ... ... ...) (if ... ...)) (if (and ... ...) (progn ... ...)  
(insert " by the rule:\n	(" ... ")") (insert "\nThe component  
character(s) are displayed by ") (if ... ... ... ...) (insert "\nSee  
the variable `reference-point-alist' for " "the meaning of the rule. 
\n")))
   (save-current-buffer (set-buffer standard-output) (set-buffer- 
multibyte multibyte-p) (let (...) (dolist ... ...)) (when overlays  
(save-excursion ... ... ...)) (when disp-vector (insert "\nThe  
display table entry is displayed by ") (if ... ... ... ...)) (when  
composition (insert "\nComposed") (if ... ... ...)  
(if ... ... ... ... ... ...)) (if (not describe-char-unidata-list)  
(insert "\nCharacter code properties are not shown: ") (insert  
"\nCharacter code properties: ")) (insert-text-button "customize what  
to show" (quote action) (lambda ... ...)) (insert "\n") (dolist (elt  
describe-char-unidata-list) (let ... ...)) (if text-props-desc  
(insert text-props-desc)) (setq help-xref-stack-item (list ... ...))  
(toggle-read-only 1))
   (with-current-buffer standard-output (set-buffer-multibyte  
multibyte-p) (let (...) (dolist ... ...)) (when overlays (save- 
excursion ... ... ...)) (when disp-vector (insert "\nThe display  
table entry is displayed by ") (if ... ... ... ...)) (when  
composition (insert "\nComposed") (if ... ... ...)  
(if ... ... ... ... ... ...)) (if (not describe-char-unidata-list)  
(insert "\nCharacter code properties are not shown: ") (insert  
"\nCharacter code properties: ")) (insert-text-button "customize what  
to show" (quote action) (lambda ... ...)) (insert "\n") (dolist (elt  
describe-char-unidata-list) (let ... ...)) (if text-props-desc  
(insert text-props-desc)) (setq help-xref-stack-item (list ... ...))  
(toggle-read-only 1))
   (progn (with-current-buffer standard-output (set-buffer-multibyte  
multibyte-p) (let ... ...) (when overlays ...) (when disp- 
vector ... ...) (when composition ... ... ...) (if ... ... ...)  
(insert-text-button "customize what to show" ... ...) (insert "\n")  
(dolist ... ...) (if text-props-desc ...) (setq help-xref-stack- 
item ...) (toggle-read-only 1)))
   (with-output-to-temp-buffer (help-buffer) (progn (with-current- 
buffer standard- 
output ... ... ... ... ... ... ... ... ... ... ... ...)))
   (let ((list-of-frames ...) (list-of-window-tuples ...)) (setq help- 
window t) (set-marker help-window-point-marker nil) (with-output-to- 
temp-buffer (help-buffer) (progn ...)) (when (windowp help-window)  
(help-window-setup list-of-frames list-of-window-tuples)) (setq help- 
window nil))
   (with-help-window (help-buffer) (with-current-buffer standard- 
output (set-buffer-multibyte multibyte-p) (let ... ...) (when  
overlays ...) (when disp-vector ... ...) (when  
composition ... ... ...) (if ... ... ...) (insert-text-button  
"customize what to show" ... ...) (insert "\n") (dolist ... ...) (if  
text-props-desc ...) (setq help-xref-stack-item ...) (toggle-read- 
only 1)))
   (let* ((char ...) (charset ...) (composition ...) (component-chars  
nil) (display-table ...) (disp-vector ...) (multibyte-p enable- 
multibyte-characters) (overlays ...) (char-description ...) (text- 
props-desc ...) item-list max-width code) (setq code (encode-char  
char charset)) (setq item-list (\` ...)) (setq max-width  
(apply ... ...)) (help-setup-xref nil (interactive-p)) (with-help- 
window (help-buffer) (with-current-buffer standard- 
output ... ... ... ... ... ... ... ... ... ... ... ...)))
   describe-char(17122)
   what-cursor-position((4))
   call-interactively(what-cursor-position nil nil)
   recursive-edit()
   byte-code("..." [unread-command-char debugger-args x debugger- 
buffer noninteractive debugger-batch-max-lines -1 debug backtrace- 
debug 4 t backtrace-frame lambda 5 pop-to-buffer debugger-mode  
debugger-setup-buffer count-lines 2 "...\n" message "%s" buffer- 
string kill-emacs "" nil recursive-edit middlestart buffer-read-only  
standard-output] 4)
   debug(error (error "Format specifier doesn't match argument type"))
   format("%s (#x%02X)" "-monotype-arial unicode ms-medium-r- 
normal--13-120-75-75-p-129-gb18030.2000-0" "(33072 . 48436)")
   describe-char(17122)
   what-cursor-position((4))
   call-interactively(what-cursor-position nil nil)
   recursive-edit()
   byte-code("..." [unread-command-char debugger-args x debugger- 
buffer noninteractive debugger-batch-max-lines -1 debug backtrace- 
debug 4 t backtrace-frame lambda 5 pop-to-buffer debugger-mode  
debugger-setup-buffer count-lines 2 "...\n" message "%s" buffer- 
string kill-emacs "" nil recursive-edit middlestart buffer-read-only  
standard-output] 4)
   debug(error (error "Format specifier doesn't match argument type"))
   format("%s (#x%02X)" "-monotype-arial unicode ms-medium-r- 
normal--13-120-75-75-p-129-gb18030.2000-0" "(33072 . 48693)")
   describe-char(29)
   what-cursor-position((4))
   call-interactively(what-cursor-position nil nil)




--
Greetings

   Pete

When in doubt, use brute force.
				– Ken Thompson

Re: 23.0.60; GNU Emacs does not handle composed characters

[Kenichi Handa]

In article <4B6DBC29-EF5C-4C8E-ABF0-726B61B61F6C@Freenet.DE>, Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> Am 08.01.2008 um 07:29 schrieb Kenichi Handa:

>>> Format specifier doesn't match argument type
> >
>>> In *Messages* buffer a ``describe-char: ´´is prepended the text.
> >
> > I can't reproduce that.  Please do M-x load-library RET
> > descr-text.el RET, M-x toggle-debug-on-error RET, and try C-u
> > C-x = again.  When you get *Backtrace* buffer, please show
> > me the contents.

> Debugger entered--Lisp error: (error "Format specifier doesn't match  
> argument type")
>    format("%s (#x%02X)" "-monotype-arial unicode ms-medium-r- 
> normal--13-120-75-75-p-129-gb18030.2000-0" "(33072 . 48436)")
>    (if (cdr elt) (format "%s (#x%02X)" (cadr elt) (cddr elt)) "-- no  
> font --")

Thank you for the info.  Now I see what's wrong.  I've just
installed a fix.

---
Kenichi Handa
handa@ni.aist.go.jp