bug#20752: 24.3; Display of combining characters on OS X after 24.3

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Pavel Iosad]

Hello,

This appears to be a regression -- this works fine in 24.3 but not in
the latest versions. (I'm using the Emacs from emacsforosx.com, on
Mountain Lion, but compiling from source doesn't help). The display of
glyphs with combining diacritics where the precomposed glyph is
unavailable appears to be faulty. Specifically, when the cursor is on
the character, the diacritic is combined with the space to the
right. When the cursor is elsewhere, most frequently nothing at all is
shown (i.e. neither the main character nor the diacritic) or only the
main character remains shown. To reproduce:

Open a buffer (scratch will do)

C-x 8 " a => displays ä
C-\ ipa-x-sampa RET
a _ " => displays ä
b _ " => shows incorrect behaviour

A screenshot is here:
https://www.dropbox.com/s/cx8bg2qljwltbz1/Screenshot%202015-06-05%2011.12.04.png?dl=0

This does not appear to be a font problem, since I am getting this with
multiple fonts and in any case it works fine with 24.3

Thanks!

Pavel
-- 
Pavel Iosad
Roinn a' Chànanachais agus Cànain na Beurla
Oilthigh Dhùn Èideann
Togalach Dhùghaill Stiùbhairt
3 Sràid Theàrlaich
Dùn Èideann EH8 9AD
Alba

http://www.ppls.ed.ac.uk/people/pavel-iosad

Is e buidheann carthannais a tha ann an Oilthigh Dhùn Èideann,
clàraichte ann an Albainn, le àireamh clàraidh SC005336.


In GNU Emacs 24.3.1 (x86_64-apple-darwin12.5.0, NS apple-appkit-1187.40)
 of 2014-11-28 on oilthigh
Windowing system distributor `Apple', version 10.3.1187
Configured using:
 `configure '--with-ns' 'CC=cc -mmacosx-version-min=10.7 ''

Important settings:
  locale-coding-system: nil
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  global-flycheck-mode: t
  flycheck-mode: t
  TeX-PDF-mode: t
  magit-auto-revert-mode: t
  diff-auto-refine-mode: t
  recentf-mode: t
  async-bytecomp-package-mode: t
  helm-mode: t
  display-time-mode: t
  golden-ratio-mode: t
  firestarter-mode: t
  shell-dirtrack-mode: t
  pyvenv-mode: t
  show-paren-mode: t
  global-centered-cursor-mode: t
  centered-cursor-mode: t
  global-hl-line-mode: t
  override-global-mode: t
  pallet-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
SPC n o <return> M-3 + E N D _ S R C C-p C-e <return> 
C-c ' ( u s e - p a c k a g e SPC c a l d w <backspace> 
<backspace> f w - o r g C-h k C-c a c C-x o C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n M-< C-n C-n 
C-n C-n C-n C-n q <backspace> <return> <tab> : b i 
n d SPC C-x o C-s : b i n d C-s <return> C-x o ( " 
C - v <backspace> x SPC C-x o C-h k C-x c c C-x c c 
C-h k C-x c o C-x o C-x o C-x o q C-x o c SPC o " SPC 
M-x c f w SPC o C-g . SPC c f w M-x c f w C-g : o p 
e n - o r g - c a l e n d a r ) ) C-x C-e C-x c o q 
C-x C-b C-c ' C-x C-s C-n C-p C-p C-p C-n C-n C-x b 
r n w <return> C-n C-n C-p C-p C-n C-p C-x o C-x o 
q C-x C-g <backspace> C-x 1 C-p C-p C-p C-p C-p C-p 
C-n M-x g l y c h <backspace> <backspace> <backspace> 
<backspace> <backspace> f l y c h <return> C-n C-n 
C-n C-n M-x p a n d o c C-o <return> C-x b <return> 
C-x k <return> C-x b m e s s <return> M-> M-v C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-p C-p C-SPC C-n C-n 
C-n C-n C-n C-n M-w M-x r e p o r t SPC <return>

Recent messages:

Mark set [2 times]

Load-path shadows:
/Users/piosad/.emacs.d/.cask/24.3.1/elpa/ess-20150601.657/.dir-locals hides /Applications/Emacs.app/Contents/Resources/lisp/gnus/.dir-locals

Features:
(calfw-org cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew
org-capture org-mks org-element calfw holidays hol-loaddefs
calfw-autoloads google-maps-autoloads org-location-google-maps
google-maps google-maps-static google-maps-geocode google-maps-base ert
ewoc flycheck sh-script smie let-alist helm-imenu semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw mode-local
cedet pcase tar-mode autoload org-colview cal-iso icalendar diary-lib
diary-loaddefs org-id org-icalendar cal-move parse-time rfc2104
reftex-sel reftex-ref helm-ring texmathp preview prv-emacs tex-buf
tex-fold imenu autopair font-latex latex tex-style tex crm reftex-cite
hi-lock tabify dabbrev debug helm-org org-indent gnutls network-stream
starttls url-http tls url-gw url-cache url-auth reftex-parse
magit-key-mode magit view diff-mode autorevert git-rebase-mode
git-commit-mode server log-edit pcvs-util add-log term ehelp electric
sane-term hydra lv org-wl org-w3m org-vm org-rmail org-mhe org-mew
org-irc org-jsinfo org-infojs org-html org-exp ob-exp org-exp-blocks
org-agenda org-info org-gnus org-docview org-bibtex bibtex org-bbdb
org-table reftex-dcr reftex-auc reftex reftex-vars misearch
multi-isearch web-mode disp-table image-file w3m-form w3m-filter
w3m-cookie w3m-bookmark w3m-tabmenu w3m-session mule-util gnus-dired
mu4e mu4e-speedbar speedbar sb-image ezimage dframe mu4e-main mu4e-view
epa derived epg epg-config mu4e-headers mu4e-compose mu4e-draft
mu4e-actions smtpmail mu4e-mark mu4e-message html2text mu4e-proc
mu4e-utils mu4e-lists mu4e-about mu4e-vars mu4e-meta recentf tree-widget
helm-misc rfc2368 w3m doc-view jka-compr image-mode timezone w3m-hist
w3m-fb bookmark-w3m w3m-ems w3m-ccl ccl w3m-favicon w3m-image w3m-proc
w3m-util mailalias mailclient qp vc-git quail iso-transl shadow sort
mail-extr emacsbug message rfc822 mml mml-sec mailabbrev gmm-utils
mailheader sendmail mail-utils winner helm-command helm-elisp helm-eval
edebug helm-config async-bytecomp async helm-aliases helm-mode
helm-files image-dired tramp tramp-compat tramp-loaddefs dired-x
dired-aux ffap helm-buffers helm-elscreen helm-tags helm-bookmark
helm-adaptive helm-info bookmark pp helm-locate helm-help
helm-match-plugin helm-grep helm-regexp helm-plugin helm-external
helm-net xml url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util url-parse auth-source
gnus-util password-cache url-vars mailcap helm-utils dired helm
helm-source rainbow-delimiters time golden-ratio ack-and-a-half
firestarter org ob-ref ob-lob ob-table org-footnote org-pcomplete
org-list org-faces org-entities org-version ob-emacs-lisp org-loaddefs
find-func cal-menu calendar cal-loaddefs poly-R poly-markdown
markdown-mode noutline outline poly-noweb polymode poly-base
polymode-weave polymode-export polymode-methods polymode-classes
polymode-common eieio-custom eieio-base eieio ess-rutils ess-R-data-view
popup ctable ess-smart-underscore ess-toolbar ess-mouse mouseme
thingatpt browse-url ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode
ess-bugs-l essd-els ess-sas-d ess-sas-l ess-sas-a shell pcomplete
ess-sta-d ess-sta-l make-regexp ess-sp6-d ess-sp3-d ess-julia ess-r-d
ess-r-completion ess-tracebug format-spec ess-roxy hideshow ess-help
ess-developer ess-s-l ess ess-inf ess-mode ess-noweb-mode ess-utils
ess-custom executable ess-compat ess-site stan-snippets stan-mode
stan-keywords-lists cc-langs cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs yasnippet help-mode cl
highlight-indentation flymake eldoc company elpy pyvenv elpy-refactor
python rx ido grep compile files-x etags cus-edit cus-start cus-load
wid-edit flyspell ispell paren powerline powerline-separators color
powerline-themes centered-cursor-mode hl-line zenburn-theme edmacro
kmacro exec-path-from-shell byte-opt warnings use-package diminish
bytecomp byte-compile cconv bind-key easy-mmode ob-tangle org-src
ob-comint comint ansi-color ring ob-keys ob org-compat org-macs ob-eval
pallet advice advice-preload ack-and-a-half-autoloads autopair-autoloads
bibretrieve-autoloads centered-cursor-mode-autoloads
company-auctex-autoloads auctex-autoloads tex-site
crontab-mode-autoloads css-eldoc-autoloads dired-toggle-sudo-autoloads
elpy-autoloads company-autoloads ess-R-data-view-autoloads
ctable-autoloads ess-smart-underscore-autoloads
exec-path-from-shell-autoloads find-file-in-project-autoloads
firestarter-autoloads flycheck-autoloads golden-ratio-autoloads
gscholar-bibtex-autoloads helm-backup-autoloads helm-autoloads
async-autoloads highlight-indentation-autoloads let-alist-autoloads
magit-autoloads git-rebase-mode-autoloads git-commit-mode-autoloads
markdown-mode-autoloads pallet-autoloads cask-autoloads
pandoc-mode-autoloads hydra-autoloads pkg-info-autoloads
polymode-autoloads popup-autoloads powerline-autoloads
py-autopep8-autoloads pyvenv-autoloads rainbow-delimiters-autoloads
rainbow-mode-autoloads sane-term-autoloads slime-autoloads info easymenu
stan-snippets-autoloads stan-mode-autoloads twittering-mode-autoloads
use-package-autoloads diminish-autoloads bind-key-autoloads
w3m-autoloads web-mode-autoloads yasnippet-autoloads
zenburn-theme-autoloads cask help-fns cl-macs gv cask-bootstrap
package-build mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 ietf-drums mm-util mail-prsvr json lisp-mnt shut-up epl git
commander cl-lib f dash s ucs-normalize commander-autoloads
epl-autoloads git-autoloads f-autoloads dash-autoloads
package-build-autoloads s-autoloads shut-up-autoloads finder-inf package
time-date tooltip ediff-hook vc-hooks lisp-float-type mwheel ns-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer loaddefs button faces cus-face macroexp files text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process ns multi-tty
emacs)

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Eli Zaretskii]

> From: Pavel Iosad <pavel.iosad@ed.ac.uk>
> Date: Sat, 6 Jun 2015 13:59:10 +0100
> 
> This appears to be a regression -- this works fine in 24.3 but not in
> the latest versions. (I'm using the Emacs from emacsforosx.com, on
> Mountain Lion, but compiling from source doesn't help). The display of
> glyphs with combining diacritics where the precomposed glyph is
> unavailable appears to be faulty. Specifically, when the cursor is on
> the character, the diacritic is combined with the space to the
> right. When the cursor is elsewhere, most frequently nothing at all is
> shown (i.e. neither the main character nor the diacritic) or only the
> main character remains shown. To reproduce:
> 
> Open a buffer (scratch will do)
> 
> C-x 8 " a => displays ä
> C-\ ipa-x-sampa RET
> a _ " => displays ä
> b _ " => shows incorrect behaviour
> 
> A screenshot is here:
> https://www.dropbox.com/s/cx8bg2qljwltbz1/Screenshot%202015-06-05%2011.12.04.png?dl=0
> 
> This does not appear to be a font problem, since I am getting this with
> multiple fonts and in any case it works fine with 24.3

Does "C-u C-x =" say that the base character and the diacritical are
composed?  Do you see any difference in what "C-u C-x =" show in 24.3
and the "latest versions" (which means what, exactly, btw?).

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Pavel Iosad]

Hello,

> Does "C-u C-x =" say that the base character and the diacritical are
> composed? 

Yes.

> Do you see any difference in what "C-u C-x =" show in 24.3
> and the "latest versions" (which means what, exactly, btw?).

Sorry, this is 24.5.1. I've tried several versions post 24.3 but to no
avail. 

Here's the output of C-u C-x = under 24.3.1 (using b̈ as an example)

             position: 192 of 193 (99%), column: 0
            character: b (displayed as b) (codepoint 98, #o142, #x62)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x62
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #x62
            file code: #x62 (encoded by coding system utf-8-unix)
              display: composed to form "b̈" (see below)

Composed with the following character(s) "̈" using this font:
  nil:-apple-Menlo-medium-normal-normal-*-12-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 98 69 7 0 7 9 0 nil]
  [0 1 776 653 7 0 5 1 0 [-6 0 0]]

And in 24.5.1 (note that when yanked it still says "composed to form b̈"
correctly, i.e. the string is intact, but it doesn't show the diacritic)

             position: 192 of 193 (99%), column: 0
            character: b (displayed as b) (codepoint 98, #o142, #x62)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x62
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #x62
            file code: #x62 (encoded by coding system utf-8-unix)
              display: composed to form "b̈" (see below)

Composed with the following character(s) "̈" using this font:
  mac-ct:-*-Menlo-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 98 69 7 0 7 10 1 [0 0 0]]
  [0 1 776 653 7 1 6 10 -7 [0 -2 7]]


Pavel
-- 
Pavel Iosad
Linguistics and English Language
The University of Edinburgh
Dugald Stewart Building
3 Charles Street
Edinburgh EH8 9AD
Scotland

http://www.ppls.ed.ac.uk/people/pavel-iosad

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Eli Zaretskii]

> From: Pavel Iosad <pavel.iosad@ed.ac.uk>
> Date: Sat, 6 Jun 2015 20:19:36 +0100
> Cc: 20752@debbugs.gnu.org
> 
> Hello,
> 
> > Does "C-u C-x =" say that the base character and the diacritical are
> > composed? 
> 
> Yes.

But the composition data is different.  Isn't that strange?  Can you
try figuring out why?

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Pavel Iosad]

Hello,

>> > Does "C-u C-x =" say that the base character and the diacritical are
>> > composed? 
>> 
>> Yes.
>
> But the composition data is different.  Isn't that strange?  Can you
> try figuring out why?

I could try, but I'm not at all familiar with how this bit of Emacs is
supposed to work (sorry!), so I wouldn't know where to start -- is there
a particular thing I should try to look at? (It doesn't seem to be an
issue with showing the combining character as such -- turning off
auto-composition-mode does what you expect, showing the combining
character fine.)

Thanks!

Pavel
-- 
Pavel Iosad
Roinn a' Chànanachais agus Cànain na Beurla
Oilthigh Dhùn Èideann
Togalach Dhùghaill Stiùbhairt
3 Sràid Theàrlaich
Dùn Èideann EH8 9AD
Alba

http://www.ppls.ed.ac.uk/people/pavel-iosad

Is e buidheann carthannais a tha ann an Oilthigh Dhùn Èideann,
clàraichte ann an Albainn, le àireamh clàraidh SC005336.

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Eli Zaretskii]

> From: Pavel Iosad <pavel.iosad@ed.ac.uk>
> Date: Sat, 6 Jun 2015 21:11:32 +0100
> 
> >> > Does "C-u C-x =" say that the base character and the diacritical are
> >> > composed? 
> >> 
> >> Yes.
> >
> > But the composition data is different.  Isn't that strange?  Can you
> > try figuring out why?
> 
> I could try, but I'm not at all familiar with how this bit of Emacs is
> supposed to work (sorry!), so I wouldn't know where to start -- is there
> a particular thing I should try to look at?

I meant mainly the way both versions were built -- perhaps different
configuration options or different versions of some support libraries?

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[YAMAMOTO Mitsuharu]

>>>>> On Sat, 6 Jun 2015 20:19:36 +0100, Pavel Iosad <pavel.iosad@ed.ac.uk> said:

> Here's the output of C-u C-x = under 24.3.1 (using b̈ as an example)

> Composed with the following character(s) "̈" using this font:
>   nil:-apple-Menlo-medium-normal-normal-*-12-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 98 69 7 0 7 9 0 nil]
>   [0 1 776 653 7 0 5 1 0 [-6 0 0]]


> And in 24.5.1 (note that when yanked it still says "composed to form b̈"
> correctly, i.e. the string is intact, but it doesn't show the diacritic)

> Composed with the following character(s) "̈" using this font:
>   mac-ct:-*-Menlo-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 98 69 7 0 7 10 1 [0 0 0]]
>   [0 1 776 653 7 1 6 10 -7 [0 -2 7]]

Notice the difference of font backend drivers: nil for 24.3 and mac-ct
for 24.5.  The latter is originally developed for the Mac port and it
supports text shaping via Core Text layout engine in OS X.  The mac-ct
backend driver was incorporated for the NS port in Emacs 24.4.  Below
is the corresponding entry in etc/NEWS:

  * Changes in Emacs 24.4 on Non-Free Operating Systems

  ** New Core Text based font backend for Mac OS X 10.5 and newer.
  To use the old font backend, use the following on the command line:
    % defaults write org.gnu.Emacs FontBackend ns
  GNUstep and Mac OS X 10.4 use the old font backend.

There was a bug in the composition string display code of the NS port
(not in the font backend driver itself), and it is supposed to be
fixed in the trunk:

  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20537

Actually, I couldn't reproduce the bug you described with the trunk.
So, I think this is a duplicate of Bug#20537.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[Pavel Iosad]

Hello,

>   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20537
>
> Actually, I couldn't reproduce the bug you described with the trunk.
> So, I think this is a duplicate of Bug#20537.

Thanks! This seems about right. The bug goes away (checked with c941d47)
with most other fonts. (This solves my immediate problem, of course.)

It is still there with DejaVu Sans Mono. I've tried some other fonts I
have installed and most of them seem to work fine, so I presume it *is*
a font issue after all. Thanks for your help!

Best,
Pavel
-- 
Pavel Iosad
Adran Ieithyddiaeth ac Iaith Saesneg
Prifysgol Caeredin
Adeilad Dugald Stewart
3 Stryd Siarl
Caeredin EH8 9AD
Yr Alban

http://www.ppls.ed.ac.uk/people/pavel-iosad

Mae Prifysgol Caeredin yn elusen gofrestredig yn yr Alban,
gyda rhif cofrestru SC005336.

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

bug#20752: 24.3; Display of combining characters on OS X after 24.3

[YAMAMOTO Mitsuharu]

>>>>> On Mon, 8 Jun 2015 11:27:41 +0100, Pavel Iosad <pavel.iosad@ed.ac.uk> said:

>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20537
>> 
>> Actually, I couldn't reproduce the bug you described with the
>> trunk.  So, I think this is a duplicate of Bug#20537.

> Thanks! This seems about right. The bug goes away (checked with
> c941d47) with most other fonts. (This solves my immediate problem,
> of course.)

> It is still there with DejaVu Sans Mono. I've tried some other fonts
> I have installed and most of them seem to work fine, so I presume it
> *is* a font issue after all. Thanks for your help!

Actually, "b̈b̈b̈b̈b̈" in DejaVu Sans Mono is displayed strangely
(just like overstriking) even with TextEdit.app on OS X 10.10.
I don't know which is wrong, the font data or the layout engine in the
Core Text framework.

Anyway, I close the bug.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp