bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 194 bytes --]

When using the font "Menlo" Emacs displays the cyrillic text in the
hello file incorrectly. "Monaco" works fine. This affects both Mac and
NS ports.

Emacs -Q -fn Monaco

C-h h C-s russian RET


[-- Attachment #2: Russian text in Monaco --]
[-- Type: image/png, Size: 9949 bytes --]

[-- Attachment #3: Type: text/plain, Size: 44 bytes --]


Emacs -Q -fn Menlo

C-h h C-s russian RET


[-- Attachment #4: menlo.png --]
[-- Type: Russian text in Menlo, Size: 9108 bytes --]

[-- Attachment #5: Type: text/plain, Size: 5669 bytes --]


It appears that two characters are being drawn in the same space.

In GNU Emacs 25.1.50.1 (x86_64-apple-darwin15.5.0, NS appkit-1404.47 Version 10.11.5 (Build 15F34))
 of 2016-07-10 built on breton.holly.idiocy.org
Repository revision: 344eb6ee66e515c812139492990001764724e506
Windowing system distributor 'Apple', version 10.3.1404
Recent messages:
Mark set
Mark saved where search started
Contacting host: debbugs.gnu.org:80
Opening nndoc server on /var/folders/rs/0j9q5wvj5qv1h4p08j_dnbvw0000gn/T/gnus-temp-group-51168zC2-ephemeral...done
Error during redisplay: (widget-mouse-help #<window 58 on *Article nndoc+ephemeral:bug#3994*> #<overlay in no buffer> 575) signaled (wrong-type-argument stringp nil) [2 times]
mwheel-scroll: End of buffer [3 times]
mwheel-scroll: Beginning of buffer [13 times]
Mark set
previous-line: Beginning of buffer [2 times]
Making completion list...

Configured features:
JPEG RSVG NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8

Major mode: Debbugs

Minor modes in effect:
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  global-flycheck-mode: t
  show-paren-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug gnuplot-gui gnuplot info-look rect debug epa-file
mailalias smtpmail sendmail nnir thai-util thai-word lao-util view
flow-fill tabify vc-git cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs ibuf-ext ibuffer
ibuffer-loaddefs sh-script smie executable ob-gnuplot ob-dot ob-js
org-habit org-agenda image-file org-element org-rmail org-mhe org-irc
org-info org-gnus org-docview doc-view jka-compr image-mode org-bibtex
bibtex org-bbdb org-w3m org org-macro org-footnote org-pcomplete
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys
ob-comint ob-core ob-eval org-compat org-macs org-loaddefs cal-menu
calendar cal-loaddefs linum magit-blame magit-stash magit-bisect
magit-remote magit-commit magit-sequence magit magit-apply magit-wip
magit-log magit-diff smerge-mode diff-mode magit-core magit-autorevert
autorevert filenotify magit-process magit-popup magit-mode magit-git
magit-section magit-utils git-commit help-fns radix-tree log-edit
pcvs-util with-editor async-bytecomp async tramp-sh tramp tramp-compat
tramp-loaddefs trampver shell pcomplete advice map misearch
multi-isearch shr-color color qp shr svg dom browse-url sort smiley
gnus-cite mail-extr gnus-async gnus-bcklg gnus-agent gnus-srvr
gnus-score score-mode nnvirtual nntp gnus-ml gnus-msg disp-table nndoc
gnus-cache gnus-dup gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range
gnus-win gnus nnheader wid-edit mm-archive message dired dired-loaddefs
format-spec rfc822 mml mml-sec epa epg gnus-util rmail rmail-loaddefs
mailabbrev gmm-utils mailheader mail-utils network-stream starttls
url-cache crm debbugs-gnu add-log debbugs soap-client mm-decode
mm-bodies mm-encode url-http tls gnutls url-auth mail-parse rfc2231
rfc2047 rfc2045 ietf-drums url-gw nsm puny warnings rng-xsd rng-dt
rng-util xsd-regexp xml term/xterm xterm mm-util mail-prsvr unichar
exec-path-from-shell powershell-mode speedbar sb-image ezimage dframe
post derived cl abc-mode paredit go-eldoc go-mode url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap find-file ffap thingatpt etags xref project compile
comint ansi-color ring geiser flycheck find-func rx subr-x dash windmove
server edmacro kmacro paren wombat-theme use-package diminish bind-key
easy-mmode finder-inf autoinsert go-mode-autoloads info package
epg-config url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cl-extra help-mode easymenu cconv cl-loaddefs pcase cl-lib
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/ns-win ns-win ucs-normalize term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote kqueue
cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 698126 81959)
 (symbols 48 56643 16)
 (miscs 40 5089 1029)
 (strings 32 139370 21892)
 (string-bytes 1 4623261)
 (vectors 16 89708)
 (vector-slots 8 2047057 36584)
 (floats 8 992 1205)
 (intervals 56 6204 1338)
 (buffers 976 62))

-- 
Alan Third

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Eli Zaretskii]

> From: Alan Third <alan@idiocy.org>
> Date: Sun, 10 Jul 2016 16:45:51 +0100
> 
> When using the font "Menlo" Emacs displays the cyrillic text in the
> hello file incorrectly. "Monaco" works fine. This affects both Mac and
> NS ports.

How do we know it isn't a problem with that font?

I think the problem is triggered by displaying the u+0301 acute accent
above the а (to show the vowel with the accent).  Does Menlo has a
glyph for that codepoint?

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 2179 bytes --]

On Sun, Jul 10, 2016 at 07:04:40PM +0300, Eli Zaretskii wrote:
> > From: Alan Third <alan@idiocy.org>
> > Date: Sun, 10 Jul 2016 16:45:51 +0100
> > 
> > When using the font "Menlo" Emacs displays the cyrillic text in the
> > hello file incorrectly. "Monaco" works fine. This affects both Mac and
> > NS ports.
> 
> How do we know it isn't a problem with that font?
> 
> I think the problem is triggered by displaying the u+0301 acute accent
> above the а (to show the vowel with the accent).  Does Menlo has a
> glyph for that codepoint?

Yes, Emacs seems to think that the composed character takes up no
space. This may be no help whatsoever, but I've attached another image
showing a cyrillic a with acute alongside the latin version. The latin
one is not, afaik, composed, because I don't know how to do that. Both
have spaces round them.

I don't know how to check whether the font supports it.
Here's what describe-char says:

             position: 148 of 234 (63%), column: 2
            character: а (displayed as а) (codepoint 1072, #o2060, #x430)
    preferred charset: cyrillic-iso8859-5 (Right-Hand Part of ISO/IEC 8859/5 (Latin/Cyrillic): ISO-IR-144)
code point in charset: 0x50
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 430" or "C-x 8 RET CYRILLIC SMALL LETTER A"
          buffer code: #xD0 #xB0
            file code: #xD0 #xB0 (encoded by coding system utf-8-unix)
              display: composed to form "а́" (see below)

Composed with the following character(s) "́" using this font:
  mac-ct:-*-Menlo-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1072 870 7 0 7 7 1 nil]
  [0 1 769 646 7 2 6 10 -7 [-8 -1 -8]]

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER A
  general-category: Ll (Letter, Lowercase)
  decomposition: (1072) ('а')

There are text properties here:
  charset              cyrillic-iso8859-5
  face                 font-lock-string-face
  fontified            t

[back]

-- 
Alan Third

[-- Attachment #2: a with acute --]
[-- Type: image/png, Size: 8181 bytes --]

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Eli Zaretskii]

> Date: Sun, 10 Jul 2016 17:28:02 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: 23934@debbugs.gnu.org
> 
> > How do we know it isn't a problem with that font?
> > 
> > I think the problem is triggered by displaying the u+0301 acute accent
> > above the а (to show the vowel with the accent).  Does Menlo has a
> > glyph for that codepoint?
> 
> Yes, Emacs seems to think that the composed character takes up no
> space. This may be no help whatsoever, but I've attached another image
> showing a cyrillic a with acute alongside the latin version. The latin
> one is not, afaik, composed, because I don't know how to do that.

The Latin character is composed, because Emacs knows how to do that.
You can verify with "C-u C-x =".

Can you tell what does window-text-pixel-size return for the а́ part of
the text, both with Menlo and with Monaco?

> I don't know how to check whether the font supports it.
> Here's what describe-char says:
> 
>              position: 148 of 234 (63%), column: 2
>             character: а (displayed as а) (codepoint 1072, #o2060, #x430)
>     preferred charset: cyrillic-iso8859-5 (Right-Hand Part of ISO/IEC 8859/5 (Latin/Cyrillic): ISO-IR-144)
> code point in charset: 0x50
>                script: cyrillic
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
>              to input: type "C-x 8 RET 430" or "C-x 8 RET CYRILLIC SMALL LETTER A"
>           buffer code: #xD0 #xB0
>             file code: #xD0 #xB0 (encoded by coding system utf-8-unix)
>               display: composed to form "а́" (see below)
> 
> Composed with the following character(s) "́" using this font:
>   mac-ct:-*-Menlo-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The above line says that Menlo does have the glyph for the accent.
Emacs will not compose characters from different fonts.

> by these glyphs:
>   [0 1 1072 870 7 0 7 7 1 nil]
>   [0 1 769 646 7 2 6 10 -7 [-8 -1 -8]]

Can you show the corresponding data when you use the Monaco font?

Thanks.

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Alan Third]

On Sun, Jul 10, 2016 at 07:52:57PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 10 Jul 2016 17:28:02 +0100
> > From: Alan Third <alan@idiocy.org>
> > Cc: 23934@debbugs.gnu.org
> > 
> > Yes, Emacs seems to think that the composed character takes up no
> > space. This may be no help whatsoever, but I've attached another image
> > showing a cyrillic a with acute alongside the latin version. The latin
> > one is not, afaik, composed, because I don't know how to do that.
> 
> The Latin character is composed, because Emacs knows how to do that.
> You can verify with "C-u C-x =".
> 
> Can you tell what does window-text-pixel-size return for the а́ part of
> the text, both with Menlo and with Monaco?

Using (window-text-pixel-size nil 3 5):

Menlo: (0 . 14)
Monaco: (7 . 16)

> > I don't know how to check whether the font supports it.
> > Here's what describe-char says:
> > 
> >              position: 148 of 234 (63%), column: 2
> >             character: а (displayed as а) (codepoint 1072, #o2060, #x430)
> >     preferred charset: cyrillic-iso8859-5 (Right-Hand Part of ISO/IEC 8859/5 (Latin/Cyrillic): ISO-IR-144)
> > code point in charset: 0x50
> >                script: cyrillic
> >                syntax: w 	which means: word
> >              category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
> >              to input: type "C-x 8 RET 430" or "C-x 8 RET CYRILLIC SMALL LETTER A"
> >           buffer code: #xD0 #xB0
> >             file code: #xD0 #xB0 (encoded by coding system utf-8-unix)
> >               display: composed to form "а́" (see below)
> > 
> > Composed with the following character(s) "́" using this font:
> >   mac-ct:-*-Menlo-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> The above line says that Menlo does have the glyph for the accent.
> Emacs will not compose characters from different fonts.
> 
> > by these glyphs:
> >   [0 1 1072 870 7 0 7 7 1 nil]
> >   [0 1 769 646 7 2 6 10 -7 [-8 -1 -8]]
> 
> Can you show the corresponding data when you use the Monaco font?

             position: 3 of 7 (29%), column: 2
            character: а (displayed as а) (codepoint 1072, #o2060, #x430)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x0430
               script: cyrillic
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET 430" or "C-x 8 RET CYRILLIC SMALL LETTER A"
          buffer code: #xD0 #xB0
            file code: #xD0 #xB0 (encoded by coding system utf-8-unix)
              display: composed to form "а́" (see below)

Composed with the following character(s) "́" using this font:
  mac-ct:-*-Monaco-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1072 449 7 0 7 7 1 nil]
  [0 1 769 375 0 -6 -1 10 -7 nil]

Character code properties: customize what to show
  name: CYRILLIC SMALL LETTER A
  general-category: Ll (Letter, Lowercase)
  decomposition: (1072) ('а')

-- 
Alan Third

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[YAMAMOTO Mitsuharu]

>>>>> On Sun, 10 Jul 2016 16:45:51 +0100, Alan Third <alan@idiocy.org> said:

> When using the font "Menlo" Emacs displays the cyrillic text in the
> hello file incorrectly. "Monaco" works fine. This affects both Mac and
> NS ports.

I've just pushed a fix to the trunk (commit
bfeda891a51178dd0032800ea4aef013c62bbe10).  Sorry, I forgot to add
Bug# in the commit log.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#23934: 25.1.50; Incorrect rendering of cyrillic text on OS X

[Eli Zaretskii]

> Date: Sun, 10 Jul 2016 18:09:02 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: 23934@debbugs.gnu.org
> 
> Using (window-text-pixel-size nil 3 5):
> 
> Menlo: (0 . 14)
> Monaco: (7 . 16)

It figures.

> > The above line says that Menlo does have the glyph for the accent.
> > Emacs will not compose characters from different fonts.
> > 
> > > by these glyphs:
> > >   [0 1 1072 870 7 0 7 7 1 nil]
> > >   [0 1 769 646 7 2 6 10 -7 [-8 -1 -8]]
                     ^
That "7" is IMO completely bogus.  All the fonts I tried, and also
Monaco you tried, give a zero there.