bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Stephen Eglen]

In brief: x̅ does not appear as x with an overline in the JuliaMono font.

emacs -Q

(set-face-attribute 'default nil :font "JuliaMono")

(insert 611 773 32 120 773)

ɣ̅ x̅

should generate a gamma and x, both with overline.

gamma is overlined, but not x.

Checking with JuliaMono author, this does not seem to be a font issue:

https://github.com/cormullion/juliamono/issues/87


This has been confirmed by a harfbuzz maintainer, where you can see
detailed discussion.  It was suggested there to report this as an Emacs
bug, which I am now doing.

https://github.com/harfbuzz/harfbuzz/discussions/2790

Thanks, Stephen

----------------------------------------------------------------------
In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.17.3)
 of 2020-08-28 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Manjaro Linux


Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-wide-int
 --with-modules --with-cairo --with-harfbuzz 'CFLAGS=-march=x86-64
 -mtune=generic -O2 -pipe -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

Important settings:
  value of $LC_MONETARY: en_GB.UTF-8
  value of $LC_NUMERIC: en_GB.UTF-8
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Image[png]

Minor modes in effect:
  pdf-occur-global-minor-mode: t
  global-magit-file-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  global-edit-server-edit-mode: t
  csv-field-index-mode: t
  shell-dirtrack-mode: t
  recentf-mode: t
  show-paren-mode: t
  TeX-PDF-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/stephen/.emacs.d/straight/build/csv-mode/csv-mode hides ~/langs/emacs/elisp-ds/csv-mode
~/langs/emacs/elisp-ds/emaxima/maxima-font-lock hides ~/langs/emacs/maxima-font-lock
~/langs/emacs/elisp-ds/emaxima/maxima hides ~/langs/emacs/maxima
~/langs/emacs/mspools hides /usr/share/emacs/27.1/lisp/mail/mspools
/home/stephen/.emacs.d/straight/build/let-alist/let-alist hides /usr/share/emacs/27.1/lisp/emacs-lisp/let-alist

Features:
(shell-toggle man locate shadow emacsbug descr-text cursor-sensor vc-mtn
vc-hg vc-git vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs macros misearch
multi-isearch dired-aux gnus-dired vc vc-dispatcher shr-color timezone
iso-transl gnus-cite smiley qp mm-archive mail-extr view mu4e desktop
frameset mu4e-org mu4e-main mu4e-view mu4e-headers mu4e-mark mule-util
hl-line windswap windmove windswap-autoloads list-unicode-display
list-unicode-display-autoloads shackle trace shackle-autoloads
visual-fill-column visual-fill-column-autoloads lua-mode
lua-mode-autoloads hugo-autoloads org-recoll yesterbox-autoloads
package-lint finder lisp-mnt package-lint-autoloads
docker-tramp-autoloads docker-tramp tramp-cache dockerfile-mode s
sh-script smie executable dockerfile-mode-autoloads yaml-mode
yaml-mode-autoloads vterm-toggle tramp-sh tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat ls-lisp vterm-toggle-autoloads
vterm face-remap term disp-table ehelp vterm-module vterm-autoloads
keychain-environment keychain-environment-autoloads sje-exwm exwm-randr
xcb-randr exwm-config exwm-systemtray xcb-systemtray xcb-xembed exwm
exwm-input xcb-keysyms xcb-xkb exwm-manage exwm-floating xcb-cursor
xcb-render exwm-layout exwm-workspace exwm-core xcb-ewmh xcb-icccm xcb
xcb-xproto xcb-types xcb-debug exwm-autoloads xelb-autoloads zygospore
zygospore-autoloads moody moody-autoloads solarized-light-theme
solarized-theme solarized solarized-faces solarized-theme-autoloads
pdf-occur ibuf-ext ibuffer ibuffer-loaddefs tablist tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local cedet pdf-isearch let-alist pdf-misc pdf-tools
cus-edit cus-start cus-load pdf-view magit-bookmark bookmark pp
pdf-cache pdf-info tq pdf-util pdf-tools-autoloads let-alist-autoloads
tablist-autoloads magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit package url-handlers magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff
diff-mode magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode git-commit transient magit-git
magit-section magit-utils log-edit pcvs-util add-log with-editor
async-bytecomp async dash magit-autoloads git-commit-autoloads
with-editor-autoloads transient-autoloads dash-autoloads async-autoloads
edit-server edit-server-autoloads poly-R poly-noweb ess-r-mode
ess-r-flymake flymake-proc flymake warnings ess-r-xref xref ess-trns
ess-r-package ess-r-completion ess-roxy ess-r-syntax ess-rd ess-s-lang
ess-help ess-mode ess-inf project ess-tracebug compile poly-R-autoloads
poly-noweb-autoloads poly-markdown markdown-mode rx polymode poly-lock
polymode-base polymode-weave polymode-export polymode-compat
polymode-methods polymode-core polymode-classes eieio-custom eieio-base
color poly-markdown-autoloads polymode-autoloads ess-knitr
markdown-mode-autoloads org-mu4e mu4e-compose mu4e-context mu4e-draft
mu4e-actions mu4e-message flow-fill mu4e-proc mu4e-utils doc-view
jka-compr image-mode exif mu4e-lists mu4e-vars mu4e-meta rfc2368
smtpmail sendmail midnight org-agenda mu4e-icalendar gnus-icalendar
org-capture gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-sum
url url-proxy url-privacy url-expand url-methods url-history mailcap shr
url-cookie url-domsuf url-util svg dom browse-url gnus-group gnus-undo
gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo
parse-time iso8601 gnus-spec gnus-int gnus-range message rmc puny
dired-x dired dired-loaddefs rfc822 mml mml-sec epa derived epg
epg-config mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs text-property-search mail-utils mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr gmm-utils icalendar diary-lib diary-loaddefs csv-mode sort
csv-mode-autoloads ob-latex ob-ditaa ob-dot ob-shell shell ob-R recentf
tree-widget wid-edit ox-beamer ox-odt rng-loc rng-uri rng-parse
rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok
nxml-util ox-latex ox-icalendar ox-html table ox-ascii ox-publish ox
org-element org ob ob-tangle ob-ref ob-lob ob-table org-macro
org-footnote org-src ob-comint org-pcomplete pcomplete org-list
org-faces org-entities time-date org-version ob-emacs-lisp org-table
org-keys org-loaddefs find-func avl-tree generator ol ob-exp ob-core
org-compat ob-eval org-macs format-spec server hideshow-org hideshow
cc-vars cc-defs ess ess-utils ess-custom ess-autoloads
julia-repl-autoloads s-autoloads julia-mode comint ansi-color ring
julia-mode-latexsubs julia-mode-autoloads fm emacs-keys paren ido move
ff-paths cl ffap thingatpt url-parse auth-source eieio eieio-core
eieio-loaddefs password-cache json map url-vars my-tex tex dbus xml crm
auctex-autoloads tex-site finder-inf my-elisp time-stamp my-c
tempo-examples tempo others-elisp cal-menu calendar cal-loaddefs
mailabbrev benchmark-init advice benchmark-init-autoloads
use-package-ensure cl-seq use-package-core use-package-autoloads
bind-key-autoloads straight-autoloads info cl-extra help-mode easymenu
seq byte-opt straight subr-x cl-macs gv bytecomp byte-compile cconv
outlines edmacro kmacro cl-loaddefs cl-lib noutline outline easy-mmode
time tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 760591 140616)
 (symbols 48 84280 1)
 (strings 32 297932 8875)
 (string-bytes 1 9266510)
 (vectors 16 106532)
 (vector-slots 8 2940584 207654)
 (floats 8 1223 536)
 (intervals 56 22449 4880)
 (buffers 1000 60))

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Stephen Eglen <sje30@cam.ac.uk> writes:

> In brief: x̅ does not appear as x with an overline in the JuliaMono font.
>
> emacs -Q
>
> (set-face-attribute 'default nil :font "JuliaMono")
>
> (insert 611 773 32 120 773)

I'm unable to reproduce this in Emacs 27.1 on Debian bullseye:


[-- Attachment #2: Type: image/png, Size: 27849 bytes --]

[-- Attachment #3: Type: text/plain, Size: 944 bytes --]


This is with:

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.23)
 of 2020-12-14 built on xo
Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
Repository branch: HEAD
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Debian GNU/Linux bullseye/sid

Which is quite similar to your build:

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.17.3)
 of 2020-08-28 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Manjaro Linux

Let's see...  the main difference seems to be that my 27.1 build is
without cairo?

Nope, even when building with cairo, I'm not able to reproduce this.  So
my guess would be that a font library in your distribution has a bug in
this area?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

On December 31, 2020 7:24:30 AM GMT+02:00, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Stephen Eglen <sje30@cam.ac.uk> writes:
> 
> > In brief: x̅ does not appear as x with an overline in the JuliaMono
> font.
> >
> > emacs -Q
> >
> > (set-face-attribute 'default nil :font "JuliaMono")
> >
> > (insert 611 773 32 120 773)
> 
> I'm unable to reproduce this in Emacs 27.1 on Debian bullseye:

And neither can I on MS-Windows.

Could be an issue with the version of HarfBuzz, perhaps?

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Stephen Eglen]

Thanks both for testing.

>> I'm unable to reproduce this in Emacs 27.1 on Debian bullseye:
>
> And neither can I on MS-Windows.
>
> Could be an issue with the version of HarfBuzz, perhaps?

$ pacman -Q harfbuzz

harfbuzz 2.7.2-1

I've tried with compiling emacs from master yesterday, and I get the
same error.

https://packages.debian.org/search?keywords=harfbuzz shows bulleyes is
on 2.6.7-1 (I think).

I'll check with a colleague as to what system he is on, as I think he
also gets the error.

Stephen

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> From: Stephen Eglen <sje30@cam.ac.uk>
> Date: Wed, 30 Dec 2020 13:42:40 +0000
> 
> emacs -Q
> 
> (set-face-attribute 'default nil :font "JuliaMono")
> 
> (insert 611 773 32 120 773)
> 
> ɣ̅ x̅
> 
> should generate a gamma and x, both with overline.
> 
> gamma is overlined, but not x.

If you move the cursor to x̅, does Emacs display a single cursor block
that includes both x and the overline, or does it behave as if those
were 2 separate glyphs?  If the latter, what does Emacs show in the
*Help* buffer if you go to the ̅ glyph and type "C-u C-x ="?

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Stephen Eglen]

> If you move the cursor to x̅, does Emacs display a single cursor block
> that includes both x and the overline, or does it behave as if those
> were 2 separate glyphs?  If the latter, what does Emacs show in the
> *Help* buffer if you go to the ̅ glyph and type "C-u C-x ="?

It appears as two separate glyphs.  The first is char "x" (decimal 120)
and then the second is given as below

position: 86 of 87 (98%), column: 3
            character: ̅ (displayed as ̅) (codepoint 773, #o1405, #x305)
              charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
code point in charset: 0x0305
               script: latin
               syntax: w 	which means: word
             category: ^:Combining
             to input: type "C-x 8 RET 305" or "C-x 8 RET COMBINING OVERLINE"
          buffer code: #xCC #x85
            file code: #xCC #x85 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1 (#x1F1D)

Character code properties: customize what to show
  name: COMBINING OVERLINE
  old-name: NON-SPACING OVERSCORE
  general-category: Mn (Mark, Nonspacing)
  decomposition: (773) ('̅')

There are text properties here:
  fontified            t

[back]

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> From: Stephen Eglen <sje30@cam.ac.uk>
> Cc: bug-gnu-emacs@gnu.org, Lars Ingebrigtsen <larsi@gnus.org>, Stephen Eglen
>  <sje30@cam.ac.uk>, 45557@debbugs.gnu.org
> Date: Thu, 31 Dec 2020 09:06:07 +0000
> 
> > Could be an issue with the version of HarfBuzz, perhaps?
> 
> $ pacman -Q harfbuzz
> 
> harfbuzz 2.7.2-1

Then I guess you should also look into Lars's guess: some of your
other font-related libraries, such as Freetype or Fontconfig.

What version of the offending font are you using?  I used the latest
released one available from their site.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> From: Stephen Eglen <sje30@cam.ac.uk>
> Cc: Stephen Eglen <sje30@cam.ac.uk>, 45557@debbugs.gnu.org
> Date: Thu, 31 Dec 2020 14:12:25 +0000
> 
> > If you move the cursor to x̅, does Emacs display a single cursor block
> > that includes both x and the overline, or does it behave as if those
> > were 2 separate glyphs?  If the latter, what does Emacs show in the
> > *Help* buffer if you go to the ̅ glyph and type "C-u C-x ="?
> 
> It appears as two separate glyphs.  The first is char "x" (decimal 120)
> and then the second is given as below
> 
> position: 86 of 87 (98%), column: 3
>             character: ̅ (displayed as ̅) (codepoint 773, #o1405, #x305)
>               charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
> code point in charset: 0x0305
>                script: latin
>                syntax: w 	which means: word
>              category: ^:Combining
>              to input: type "C-x 8 RET 305" or "C-x 8 RET COMBINING OVERLINE"
>           buffer code: #xCC #x85
>             file code: #xCC #x85 (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1 (#x1F1D)

And the same info about "x" also shows the same font, i.e.

  ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1

Anyway, the above means Emacs didn't compose these COMBINING OVERLINE
woth "x", for some reason.  The question is why.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Stephen Eglen]

> And the same info about "x" also shows the same font, i.e.
>
>   ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1
>
> Anyway, the above means Emacs didn't compose these COMBINING OVERLINE
> woth "x", for some reason.  The question is why.

yes; this is the output for "x":

             position: 85 of 87 (97%), column: 2
            character: x (displayed as x) (codepoint 120, #o170, #x78)
              charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x78
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET 78" or "C-x 8 RET LATIN SMALL LETTER X"
          buffer code: #x78
            file code: #x78 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1 (#xA7)

Character code properties: customize what to show
  name: LATIN SMALL LETTER X
  general-category: Ll (Letter, Lowercase)
  decomposition: (120) ('x')

There are text properties here:
  fontified            t

[back]

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Stephen Eglen]

On Thu, Dec 31 2020, Eli Zaretskii wrote:

>> From: Stephen Eglen <sje30@cam.ac.uk>
>> Cc: Stephen Eglen <sje30@cam.ac.uk>, 45557@debbugs.gnu.org
>> Date: Thu, 31 Dec 2020 14:12:25 +0000
>> 
>> > If you move the cursor to x̅, does Emacs display a single cursor block
>> > that includes both x and the overline, or does it behave as if those
>> > were 2 separate glyphs?  If the latter, what does Emacs show in the
>> > *Help* buffer if you go to the ̅ glyph and type "C-u C-x ="?
>> 
>> It appears as two separate glyphs.  The first is char "x" (decimal 120)
>> and then the second is given as below
>> 
>> position: 86 of 87 (98%), column: 3
>>             character: ̅ (displayed as ̅) (codepoint 773, #o1405, #x305)
>>               charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>> code point in charset: 0x0305
>>                script: latin
>>                syntax: w 	which means: word
>>              category: ^:Combining
>>              to input: type "C-x 8 RET 305" or "C-x 8 RET COMBINING OVERLINE"
>>           buffer code: #xCC #x85
>>             file code: #xCC #x85 (encoded by coding system utf-8-unix)
>>               display: by this font (glyph code)
>>     ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1 (#x1F1D)
>
> And the same info about "x" also shows the same font, i.e.
>
>   ftcrhb:-UKWN-JuliaMono-normal-normal-normal-*-19-*-*-*-m-0-iso10646-1
>
> Anyway, the above means Emacs didn't compose these COMBINING OVERLINE
> woth "x", for some reason.  The question is why.

I think this can be closed now!

After much debugging and reinstalls of Emacs under different
configurations, I found the problem is not with Emacs, but the way in
which I installed JuliaMono.

If I install the fonts using

yay -S ttf-JuliaMono

i.e. using the arch package manager, I get the problem.  If I uninstall
that package, and install the fonts manually, by putting them in my
~.fonts/ folder and running "fc-cache -fv", the combining characters
work fine.

I could not find out why this made any difference.  I can see one small
issue, i.e. that when installing fonts using the package manager, it
does a little bit of extra work, by running some hooks.  These hooks
effectively run "mkfontscale" and "mkfontdir" in the system directory,
and update the files /usr/share/fonts/TTF/fonts.dir,fonts.scale .
However, if I temporarily removed those hooks, I still get the problem.

Either way, this is much an arch problem not an Emacs problem I feel.

Thanks everyone for your help.

Stephen

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Lars Ingebrigtsen]

Stephen Eglen <sje30@cam.ac.uk> writes:

> Either way, this is much an arch problem not an Emacs problem I feel.

OK; closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[James Cloos]

that implies that your dist (arch, yes?)  does something to the font
when packaging it.

last i looked itjuliamono was ofl (good) but no src available (not as
good), so i can't guess what arch does when packaging it....

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Madhu]

[-- Attachment #1: Type: Text/Plain, Size: 3210 bytes --]


I'm not asking to reopen this bug but I'm seeing repeatable weirdness
over the state of auto-composition-mode.  Pardon the complicated
clunky test case. The important thing is that Emacs should start off
with a font which is unable to compose the combining character
correctly.

The attached file test.txt has two lines - the first line is from the
test case upthread.  LATIN SMALL LETTER X + COMBINING OVERLINE.

The second line has tentative alternative Devanagari spellings for
Emacs (all wrong).  The interesting composition is in the last
consonant K+S of the word Emacs.  Without composition it should read

DEVANAGARI LETTER KA + DEVANAGARI SIGN VIRAMA + DEVANAGARI LETTER SA +
DEVANAGARI SIGN VIRAMA

and with composition it should read

DEVANAGARI LETTER KA Composed with DEVANAGARI SIGN VIRAMA +
DEVANAGARI LETTER SA Composed with DEVANAGARI SIGN VIRAMA

Assuming you have JuliaMono (or some font which does compose x with
overbar), Monaco (or BitstreamVeraSansMono or some font which does not
compose x with overbar), and NotoSans (which handles composed
Devanagari combining characters)

1. emacs -Q -fn Monaco ~/test.txt
---------------------
M-: auto-composition-mode ; => t.

This always gets the first line "wrong":

LATIN SMALL LETTER X
              display: by this font (glyph code)
    ftcrhb:-APPL-Monaco-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1 (#x5B)
+
COMBINING OVERLINE
              display: by this font (glyph code)
    ftcrhb:-SIL -Charis SIL-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1 (#x8C

Those are *not* composeḍ.
But The second line is "right".  The K+S consonants show up as

DEVANAGARI LETTER KA
 Composed with the following character(s) "्" using this font:
  ftcrhb:-GOOG-Noto Sans Devanagari UI-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 2325 180 13 0 14 14 0 [0 0 12]]
with these character(s):
  ् (#x94d) DEVANAGARI SIGN VIRAMA

DEVANAGARI LETTER SA"
Composed with the following character(s) "्" using this font:
  ftcrhb:-GOOG-Noto Sans Devanagari UI-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1
by these glyphs:
  [2 3 2360 60 15 0 16 14 2 nil]
  [2 3 2381 80 0 -4 4 0 6 nil]
with these character(s):
  ् (#x94d) DEVANAGARI SIGN VIRAMA


2. M-: (set-frame-font "JuliaMonoLatin-14:hintstyle=none" nil nil)
----------------

- Everything should look the same except the first line is rendered in
Julia mono. The x and the overbar are still not composed.


3.  M-x auto-composition-mode  ; to toggle
----------------
;; Auto-Composition mode disabled in current buffer
M-: auto-composition-mode ; => nil

and voilà! Now the first line is rendered "correctly" with x and
overbar composed and the second line is now incorrect: the k + s
appear decomposed.

Toggling auto-composition-mode again reverses this.

Creating a different frame with a a problematic font and then toggling
auto-composition-mode also exposes this behaviour, and it is confusing
when the same buffer is displayed in two frames

If emacs starts off with the correct font:

4. emacs -Q -fn JuliaMono test.txt

Then it all works as I think it was intended.


[-- Attachment #2: test.txt --]
[-- Type: Text/Plain, Size: 104 bytes --]

x̅
एमैक्स् व ईमेक्स् व एमक्स् व इमक्स्

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> Date: Wed, 06 Jan 2021 08:08:03 +0530 (IST)
> From: Madhu <enometh@meer.net>
> 
> 1. emacs -Q -fn Monaco ~/test.txt
> ---------------------
> M-: auto-composition-mode ; => t.
> 
> This always gets the first line "wrong":
> 
> LATIN SMALL LETTER X
>               display: by this font (glyph code)
>     ftcrhb:-APPL-Monaco-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1 (#x5B)
> +
> COMBINING OVERLINE
>               display: by this font (glyph code)
>     ftcrhb:-SIL -Charis SIL-normal-normal-normal-*-22-*-*-*-*-0-iso10646-1 (#x8C
> 
> Those are *not* composeḍ.

They are not composed because Emacs didn't find a glyph in the Monaco
font for the COMBINING OVERLINE character.  Emacs only composes
characters if they have glyphs in the same font.

> 2. M-: (set-frame-font "JuliaMonoLatin-14:hintstyle=none" nil nil)
> ----------------
> 
> - Everything should look the same except the first line is rendered in
> Julia mono. The x and the overbar are still not composed.

For the same reason.

(On my system, with a different font, they _are_ composed.)

> 3.  M-x auto-composition-mode  ; to toggle
> ----------------
> ;; Auto-Composition mode disabled in current buffer
> M-: auto-composition-mode ; => nil
> 
> and voilà! Now the first line is rendered "correctly" with x and
> overbar composed and the second line is now incorrect: the k + s
> appear decomposed.

This can only happen if some other software involved in the display
(Cairo?) composes them.  In any case, this is not an Emacs issue.

> 4. emacs -Q -fn JuliaMono test.txt
> 
> Then it all works as I think it was intended.

Which might mean the hintstyle=none may be a factor here?

Anyway, I see no Emacs problems in your description, only font
problems.  The text you sent is displayed correctly on my system, both
of its lines.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Madhu]

*  Eli Zaretskii <eliz@gnu.org> <83k0sq2j9m.fsf@gnu.org>
Wrote on Wed, 06 Jan 2021 17:10:45 +0200

>> 1. emacs -Q -fn Monaco ~/test.txt
>> ---------------------
>> M-: auto-composition-mode ; => t.
>> Those are *not* composeḍ.
>
> They are not composed because Emacs didn't find a glyph in the Monaco
> font for the COMBINING OVERLINE character.  Emacs only composes
> characters if they have glyphs in the same font.
>
>> 2. M-: (set-frame-font "JuliaMonoLatin-14:hintstyle=none" nil nil)
>> ----------------
>> - Everything should look the same except the first line is rendered in
>> Julia mono. The x and the overbar are still not composed.
>
> For the same reason.
>
> (On my system, with a different font, they _are_ composed.)

Right. JuliaMono will compose them, JuliaMonoLatin won't.

>> 3.  M-x auto-composition-mode  ; to toggle
>> ----------------
>> ;; Auto-Composition mode disabled in current buffer
>> M-: auto-composition-mode ; => nil
>>
>> and voilà! Now the first line is rendered "correctly" with x and
>> overbar composed and the second line is now incorrect: the k + s
>> appear decomposed.
>
> This can only happen if some other software involved in the display
> (Cairo?) composes them.  In any case, this is not an Emacs issue.

I have a build compiled with --without-cairo --without-harfbuzz
--with-xft where it happens.  But xft pulls in both harfbuzz and cairo
anyway for its implementation.

Since emacs now hard-depends on harfbuzz, harfbuzz issues become emacs
issues!

>> 4. emacs -Q -fn JuliaMono test.txt
>>
>> Then it all works as I think it was intended.
>
> Which might mean the hintstyle=none may be a factor here?

just the difference between JuliaMono and JuliaMonoLatin I think,
which I was blind to. Thanks for checking

> Anyway, I see no Emacs problems in your description, only font
> problems.  The text you sent is displayed correctly on my system, both
> of its lines.

I assume the mswindows and applemac systems dont pull in harfbuzz?

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> Date: Wed, 06 Jan 2021 21:30:51 +0530 (IST)
> Cc: 45557@debbugs.gnu.org
> From: Madhu <enometh@meer.net>
> 
> > Anyway, I see no Emacs problems in your description, only font
> > problems.  The text you sent is displayed correctly on my system, both
> > of its lines.
> 
> I assume the mswindows and applemac systems dont pull in harfbuzz?

No, the Windows build does use HarfBuzz.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Madhu]

*  Eli Zaretskii <eliz@gnu.org> <83turu11xm.fsf@gnu.org>
Wrote on Wed, 06 Jan 2021 18:10:29 +0200
>> From: Madhu <enometh@meer.net>
>> > Anyway, I see no Emacs problems in your description, only font
>> > problems.  The text you sent is displayed correctly on my system, both
>> > of its lines.
Can you try using a font with emacs which does not compose the x and
overbar?

>> I assume the mswindows and applemac systems dont pull in harfbuzz?
> No, the Windows build does use HarfBuzz.
Whatever is doing the visual composition of x + overbar (when emacs is
explicitly asked not to do it by turning auto-composition-mode off) -
it isn't harfbuzz.  It happens on an emacs (--without-all) which links
to freetype without harfbuzz/cairo (and even when emacs isn't using m17n-lib)
Very puzzling.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Eli Zaretskii]

> Date: Thu, 07 Jan 2021 11:40:31 +0530 (IST)
> Cc: 45557@debbugs.gnu.org
> From: Madhu <enometh@meer.net>
> 
> *  Eli Zaretskii <eliz@gnu.org> <83turu11xm.fsf@gnu.org>
> Wrote on Wed, 06 Jan 2021 18:10:29 +0200
> >> From: Madhu <enometh@meer.net>
> >> > Anyway, I see no Emacs problems in your description, only font
> >> > problems.  The text you sent is displayed correctly on my system, both
> >> > of its lines.
> Can you try using a font with emacs which does not compose the x and
> overbar?

I already did.  The results are as I'd expect: the characters are
rendered separately.

> >> I assume the mswindows and applemac systems dont pull in harfbuzz?
> > No, the Windows build does use HarfBuzz.
> Whatever is doing the visual composition of x + overbar (when emacs is
> explicitly asked not to do it by turning auto-composition-mode off) -
> it isn't harfbuzz.  It happens on an emacs (--without-all) which links
> to freetype without harfbuzz/cairo (and even when emacs isn't using m17n-lib)
> Very puzzling.

Indeed.

bug#45557: 27.1; Incorrect rendering of COMBINING OVERLINE

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 07 Jan 2021 11:40:31 +0530 (IST)
>> Cc: 45557@debbugs.gnu.org
>> From: Madhu <enometh@meer.net>
>> 
>> *  Eli Zaretskii <eliz@gnu.org> <83turu11xm.fsf@gnu.org>
>> Wrote on Wed, 06 Jan 2021 18:10:29 +0200
>> >> From: Madhu <enometh@meer.net>
>> >> > Anyway, I see no Emacs problems in your description, only font
>> >> > problems.  The text you sent is displayed correctly on my system, both
>> >> > of its lines.
>> Can you try using a font with emacs which does not compose the x and
>> overbar?
>
> I already did.  The results are as I'd expect: the characters are
> rendered separately.
>
>> >> I assume the mswindows and applemac systems dont pull in harfbuzz?
>> > No, the Windows build does use HarfBuzz.
>> Whatever is doing the visual composition of x + overbar (when emacs is
>> explicitly asked not to do it by turning auto-composition-mode off) -
>> it isn't harfbuzz.  It happens on an emacs (--without-all) which links
>> to freetype without harfbuzz/cairo (and even when emacs isn't using m17n-lib)
>> Very puzzling.
>
> Indeed.

I can reproduce this behaviour using Bitstream Vera Mono and Julia
Mono. The display may show the two characters as composed, but they
actually aren't: after the 'x' thereʼs what is visually a space, but
'C-x =' says itʼs a COMBINING OVERLINE. If you then toggle
auto-composition again that visual space gets redrawn as an
OVERLINE. (and if I play with it some more, sometimes the OVERLINE
over the 'x' does not get cleared).

I can reproduce with a pgtk build as well.

Robert