bug#50951: 28.0.50; Urdu text is not displayed correctly

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1.1: Type: text/plain, Size: 30865 bytes --]

Starting with 'emacs -Q' and using the menu bar to paste urdu text

حرف نہیں جاں بخشی میں اس کی خوبی اپنی قسمت کی
ہم سے جو پہلے کہہ بھیجا سو مرنے کا پیغام کیا

the text is not rendered properly, some characters which should be
joined together are instead rendered individually.

It look like this
[image: 2021-10-01T21:43:10,566697499+02:00.png]
[image: 2021-10-01T21:49:10,611532571+02:00.png]
Changing the font to a Nastaliq font more common for Urdu via

(set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu"))

the text is again rendered improperly but in a different way with chracters
overlapping each other when they shouldn't, others have gaps where they
should be joined. It looks like this

[image: 2021-10-01T22:02:00,166113989+02:00.png]
while it should look like this

[image: 2021-10-01T22:05:28,910692941+02:00.png]

In GNU Emacs 28.0.50 (build 1, x86_64-suse-linux-gnu, GTK+ Version 3.24.30,
cairo version 1.16.0)
 of 2021-09-29 built on goat11
Windowing system distributor 'System Description: openSUSE Tumbleweed

Configured using:
 'configure --host=x86_64-suse-linux-gnu --build=x86_64-suse-linux-gnu
 --program-prefix= --disable-dependency-tracking --prefix=/usr
 --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
 --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
 --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --disable-dependency-tracking --with-pgtk
 --with-native-compilation --with-cairo --with-libotf --with-jpeg
 --with-tiff --with-gif --with-png --with-rsvg --with-xft --with-dbus
 --with-sound --with-xwidgets
 --enable-locallisppath=/usr/share/emacs/28.0.50/site-lisp:/usr/share/emacs/site-lisp
 'CFLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong
 -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
 -Werror=return-type -flto=auto -g' LDFLAGS=-Wl,-O2'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS XIM
XWIDGETS GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: DOOM v3.0.0-alpha

Minor modes in effect:
  diff-hl-margin-mode: t
  delete-selection-mode: t
  TeX-PDF-mode: t
  TeX-source-correlate-mode: t
  global-evil-surround-mode: t
  evil-surround-mode: t
  eros-mode: t
  projectile-mode: t
  solaire-global-mode: t
  solaire-mode: t
  global-git-commit-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  which-key-mode: t
  savehist-mode: t
  better-jumper-mode: t
  better-jumper-local-mode: t
  global-company-mode: t
  company-mode: t
  vertico-mode: t
  marginalia-mode: t
  evil-goggles-mode: t
  evil-escape-mode: t
  evil-snipe-override-mode: t
  evil-snipe-mode: t
  evil-snipe-override-local-mode: t
  evil-snipe-local-mode: t
  recentf-mode: t
  save-place-mode: t
  global-so-long-mode: t
  gcmh-mode: t
  global-hl-line-mode: t
  hl-line-mode: t
  winner-mode: t
  show-paren-mode: t
  smartparens-global-mode: t
  ws-butler-global-mode: t
  global-undo-fu-session-mode: t
  undo-fu-mode: t
  global-flycheck-mode: t
  persp-mode: t
  shell-dirtrack-mode: t
  evil-mode: t
  evil-local-mode: t
  windmove-mode: t
  +popup-mode: t
  +modeline-global-mode: t
  +modeline-mode: t
  general-override-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  window-divider-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  size-indication-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org-contrib/ox-koma-letter
hides /home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-koma-letter
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package
hides /home/azeem/.emacs.d/.local/straight/repos/use-package/use-package
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-lint
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-lint
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-jump
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-jump
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-ensure
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-ensure
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-diminish
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-diminish
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-delight
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-delight
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-core
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-core
/home/azeem/.emacs.d/.local/straight/build-28.0.50/use-package/use-package-bind-key
hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/use-package-bind-key
/home/azeem/.emacs.d/.local/straight/build-28.0.50/bind-key/bind-key hides
/home/azeem/.emacs.d/.local/straight/repos/use-package/bind-key
/home/azeem/.emacs.d/.local/straight/build-28.0.50/straight/straight hides
/home/azeem/.emacs.d/.local/straight/repos/straight.el/straight
/home/azeem/.emacs.d/.local/straight/build-28.0.50/straight/straight-x
hides /home/azeem/.emacs.d/.local/straight/repos/straight.el/straight-x
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-site hides
/usr/share/emacs/site-lisp/tex-site
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/toolbar-x hides
/usr/share/emacs/site-lisp/auctex/toolbar-x
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/texmathp hides
/usr/share/emacs/site-lisp/auctex/texmathp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex hides
/usr/share/emacs/site-lisp/auctex/tex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-style hides
/usr/share/emacs/site-lisp/auctex/tex-style
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-mik hides
/usr/share/emacs/site-lisp/auctex/tex-mik
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-jp hides
/usr/share/emacs/site-lisp/auctex/tex-jp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-ispell hides
/usr/share/emacs/site-lisp/auctex/tex-ispell
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-info hides
/usr/share/emacs/site-lisp/auctex/tex-info
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-font hides
/usr/share/emacs/site-lisp/auctex/tex-font
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-fold hides
/usr/share/emacs/site-lisp/auctex/tex-fold
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-buf hides
/usr/share/emacs/site-lisp/auctex/tex-buf
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/tex-bar hides
/usr/share/emacs/site-lisp/auctex/tex-bar
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/preview hides
/usr/share/emacs/site-lisp/auctex/preview
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/plain-tex hides
/usr/share/emacs/site-lisp/auctex/plain-tex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/multi-prompt
hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/latex hides
/usr/share/emacs/site-lisp/auctex/latex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/latex-flymake
hides /usr/share/emacs/site-lisp/auctex/latex-flymake
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/font-latex hides
/usr/share/emacs/site-lisp/auctex/font-latex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/context hides
/usr/share/emacs/site-lisp/auctex/context
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/context-nl hides
/usr/share/emacs/site-lisp/auctex/context-nl
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/context-en hides
/usr/share/emacs/site-lisp/auctex/context-en
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/bib-cite hides
/usr/share/emacs/site-lisp/auctex/bib-cite
/home/azeem/.emacs.d/.local/straight/build-28.0.50/auctex/auctex hides
/usr/share/emacs/site-lisp/site-start.d/auctex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/transient/transient
hides /usr/share/emacs/28.0.50/lisp/transient
/home/azeem/.emacs.d/.local/straight/repos/straight.el/indent hides
/usr/share/emacs/28.0.50/lisp/indent
/home/azeem/.emacs.d/.local/straight/build-28.0.50/xref/xref hides
/usr/share/emacs/28.0.50/lisp/progmodes/xref
/home/azeem/.emacs.d/.local/straight/build-28.0.50/project/project hides
/usr/share/emacs/28.0.50/lisp/progmodes/project
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox hides
/usr/share/emacs/28.0.50/lisp/org/ox
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-texinfo hides
/usr/share/emacs/28.0.50/lisp/org/ox-texinfo
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-publish hides
/usr/share/emacs/28.0.50/lisp/org/ox-publish
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-org hides
/usr/share/emacs/28.0.50/lisp/org/ox-org
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-odt hides
/usr/share/emacs/28.0.50/lisp/org/ox-odt
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-md hides
/usr/share/emacs/28.0.50/lisp/org/ox-md
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-man hides
/usr/share/emacs/28.0.50/lisp/org/ox-man
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-latex hides
/usr/share/emacs/28.0.50/lisp/org/ox-latex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-icalendar hides
/usr/share/emacs/28.0.50/lisp/org/ox-icalendar
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-html hides
/usr/share/emacs/28.0.50/lisp/org/ox-html
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-beamer hides
/usr/share/emacs/28.0.50/lisp/org/ox-beamer
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ox-ascii hides
/usr/share/emacs/28.0.50/lisp/org/ox-ascii
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org hides
/usr/share/emacs/28.0.50/lisp/org/org
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-version hides
/usr/share/emacs/28.0.50/lisp/org/org-version
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-timer hides
/usr/share/emacs/28.0.50/lisp/org/org-timer
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-tempo hides
/usr/share/emacs/28.0.50/lisp/org/org-tempo
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-table hides
/usr/share/emacs/28.0.50/lisp/org/org-table
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-src hides
/usr/share/emacs/28.0.50/lisp/org/org-src
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-refile hides
/usr/share/emacs/28.0.50/lisp/org/org-refile
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-protocol hides
/usr/share/emacs/28.0.50/lisp/org/org-protocol
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-plot hides
/usr/share/emacs/28.0.50/lisp/org/org-plot
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-pcomplete hides
/usr/share/emacs/28.0.50/lisp/org/org-pcomplete
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-num hides
/usr/share/emacs/28.0.50/lisp/org/org-num
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-mouse hides
/usr/share/emacs/28.0.50/lisp/org/org-mouse
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-mobile hides
/usr/share/emacs/28.0.50/lisp/org/org-mobile
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-macs hides
/usr/share/emacs/28.0.50/lisp/org/org-macs
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-macro hides
/usr/share/emacs/28.0.50/lisp/org/org-macro
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-loaddefs hides
/usr/share/emacs/28.0.50/lisp/org/org-loaddefs
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-list hides
/usr/share/emacs/28.0.50/lisp/org/org-list
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-lint hides
/usr/share/emacs/28.0.50/lisp/org/org-lint
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-keys hides
/usr/share/emacs/28.0.50/lisp/org/org-keys
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-install hides
/usr/share/emacs/28.0.50/lisp/org/org-install
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-inlinetask hides
/usr/share/emacs/28.0.50/lisp/org/org-inlinetask
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-indent hides
/usr/share/emacs/28.0.50/lisp/org/org-indent
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-id hides
/usr/share/emacs/28.0.50/lisp/org/org-id
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-habit hides
/usr/share/emacs/28.0.50/lisp/org/org-habit
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-goto hides
/usr/share/emacs/28.0.50/lisp/org/org-goto
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-footnote hides
/usr/share/emacs/28.0.50/lisp/org/org-footnote
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-feed hides
/usr/share/emacs/28.0.50/lisp/org/org-feed
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-faces hides
/usr/share/emacs/28.0.50/lisp/org/org-faces
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-entities hides
/usr/share/emacs/28.0.50/lisp/org/org-entities
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-element hides
/usr/share/emacs/28.0.50/lisp/org/org-element
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-duration hides
/usr/share/emacs/28.0.50/lisp/org/org-duration
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-datetree hides
/usr/share/emacs/28.0.50/lisp/org/org-datetree
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-ctags hides
/usr/share/emacs/28.0.50/lisp/org/org-ctags
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-crypt hides
/usr/share/emacs/28.0.50/lisp/org/org-crypt
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-compat hides
/usr/share/emacs/28.0.50/lisp/org/org-compat
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-colview hides
/usr/share/emacs/28.0.50/lisp/org/org-colview
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-clock hides
/usr/share/emacs/28.0.50/lisp/org/org-clock
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-capture hides
/usr/share/emacs/28.0.50/lisp/org/org-capture
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-attach hides
/usr/share/emacs/28.0.50/lisp/org/org-attach
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-attach-git hides
/usr/share/emacs/28.0.50/lisp/org/org-attach-git
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-archive hides
/usr/share/emacs/28.0.50/lisp/org/org-archive
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/org-agenda hides
/usr/share/emacs/28.0.50/lisp/org/org-agenda
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol hides
/usr/share/emacs/28.0.50/lisp/org/ol
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-w3m hides
/usr/share/emacs/28.0.50/lisp/org/ol-w3m
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-rmail hides
/usr/share/emacs/28.0.50/lisp/org/ol-rmail
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-mhe hides
/usr/share/emacs/28.0.50/lisp/org/ol-mhe
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-irc hides
/usr/share/emacs/28.0.50/lisp/org/ol-irc
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-info hides
/usr/share/emacs/28.0.50/lisp/org/ol-info
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-gnus hides
/usr/share/emacs/28.0.50/lisp/org/ol-gnus
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-eww hides
/usr/share/emacs/28.0.50/lisp/org/ol-eww
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-eshell hides
/usr/share/emacs/28.0.50/lisp/org/ol-eshell
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-docview hides
/usr/share/emacs/28.0.50/lisp/org/ol-docview
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-bibtex hides
/usr/share/emacs/28.0.50/lisp/org/ol-bibtex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ol-bbdb hides
/usr/share/emacs/28.0.50/lisp/org/ol-bbdb
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob hides
/usr/share/emacs/28.0.50/lisp/org/ob
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-vala hides
/usr/share/emacs/28.0.50/lisp/org/ob-vala
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-tangle hides
/usr/share/emacs/28.0.50/lisp/org/ob-tangle
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-table hides
/usr/share/emacs/28.0.50/lisp/org/ob-table
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-stan hides
/usr/share/emacs/28.0.50/lisp/org/ob-stan
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-sqlite hides
/usr/share/emacs/28.0.50/lisp/org/ob-sqlite
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-sql hides
/usr/share/emacs/28.0.50/lisp/org/ob-sql
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-shen hides
/usr/share/emacs/28.0.50/lisp/org/ob-shen
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-shell hides
/usr/share/emacs/28.0.50/lisp/org/ob-shell
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-sed hides
/usr/share/emacs/28.0.50/lisp/org/ob-sed
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-screen hides
/usr/share/emacs/28.0.50/lisp/org/ob-screen
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-scheme hides
/usr/share/emacs/28.0.50/lisp/org/ob-scheme
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-sass hides
/usr/share/emacs/28.0.50/lisp/org/ob-sass
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ruby hides
/usr/share/emacs/28.0.50/lisp/org/ob-ruby
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ref hides
/usr/share/emacs/28.0.50/lisp/org/ob-ref
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-python hides
/usr/share/emacs/28.0.50/lisp/org/ob-python
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-processing hides
/usr/share/emacs/28.0.50/lisp/org/ob-processing
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-plantuml hides
/usr/share/emacs/28.0.50/lisp/org/ob-plantuml
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-picolisp hides
/usr/share/emacs/28.0.50/lisp/org/ob-picolisp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-perl hides
/usr/share/emacs/28.0.50/lisp/org/ob-perl
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-org hides
/usr/share/emacs/28.0.50/lisp/org/ob-org
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-octave hides
/usr/share/emacs/28.0.50/lisp/org/ob-octave
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ocaml hides
/usr/share/emacs/28.0.50/lisp/org/ob-ocaml
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-mscgen hides
/usr/share/emacs/28.0.50/lisp/org/ob-mscgen
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-maxima hides
/usr/share/emacs/28.0.50/lisp/org/ob-maxima
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-matlab hides
/usr/share/emacs/28.0.50/lisp/org/ob-matlab
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-makefile hides
/usr/share/emacs/28.0.50/lisp/org/ob-makefile
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-lua hides
/usr/share/emacs/28.0.50/lisp/org/ob-lua
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-lob hides
/usr/share/emacs/28.0.50/lisp/org/ob-lob
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-lisp hides
/usr/share/emacs/28.0.50/lisp/org/ob-lisp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-lilypond hides
/usr/share/emacs/28.0.50/lisp/org/ob-lilypond
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ledger hides
/usr/share/emacs/28.0.50/lisp/org/ob-ledger
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-latex hides
/usr/share/emacs/28.0.50/lisp/org/ob-latex
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-js hides
/usr/share/emacs/28.0.50/lisp/org/ob-js
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-java hides
/usr/share/emacs/28.0.50/lisp/org/ob-java
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-io hides
/usr/share/emacs/28.0.50/lisp/org/ob-io
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-hledger hides
/usr/share/emacs/28.0.50/lisp/org/ob-hledger
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-haskell hides
/usr/share/emacs/28.0.50/lisp/org/ob-haskell
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-groovy hides
/usr/share/emacs/28.0.50/lisp/org/ob-groovy
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-gnuplot hides
/usr/share/emacs/28.0.50/lisp/org/ob-gnuplot
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-fortran hides
/usr/share/emacs/28.0.50/lisp/org/ob-fortran
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-forth hides
/usr/share/emacs/28.0.50/lisp/org/ob-forth
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-exp hides
/usr/share/emacs/28.0.50/lisp/org/ob-exp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-eval hides
/usr/share/emacs/28.0.50/lisp/org/ob-eval
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-eshell hides
/usr/share/emacs/28.0.50/lisp/org/ob-eshell
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-emacs-lisp hides
/usr/share/emacs/28.0.50/lisp/org/ob-emacs-lisp
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ebnf hides
/usr/share/emacs/28.0.50/lisp/org/ob-ebnf
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-dot hides
/usr/share/emacs/28.0.50/lisp/org/ob-dot
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-ditaa hides
/usr/share/emacs/28.0.50/lisp/org/ob-ditaa
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-css hides
/usr/share/emacs/28.0.50/lisp/org/ob-css
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-core hides
/usr/share/emacs/28.0.50/lisp/org/ob-core
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-coq hides
/usr/share/emacs/28.0.50/lisp/org/ob-coq
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-comint hides
/usr/share/emacs/28.0.50/lisp/org/ob-comint
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-clojure hides
/usr/share/emacs/28.0.50/lisp/org/ob-clojure
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-calc hides
/usr/share/emacs/28.0.50/lisp/org/ob-calc
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-awk hides
/usr/share/emacs/28.0.50/lisp/org/ob-awk
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-asymptote hides
/usr/share/emacs/28.0.50/lisp/org/ob-asymptote
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-abc hides
/usr/share/emacs/28.0.50/lisp/org/ob-abc
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-R hides
/usr/share/emacs/28.0.50/lisp/org/ob-R
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-J hides
/usr/share/emacs/28.0.50/lisp/org/ob-J
/home/azeem/.emacs.d/.local/straight/build-28.0.50/org/ob-C hides
/usr/share/emacs/28.0.50/lisp/org/ob-C

Features:
(shadow sort mail-extr emacsbug sendmail elisp-mode org-indent ol-bibtex
magit-patch magit-subtree magit-gitignore magit-ediff
evil-collection-ediff ediff ediff-merg ediff-mult ediff-wind ediff-diff
ediff-help ediff-init ediff-util diff-hl-margin diff-hl-dired diff-hl
evil-collection-log-view log-view evil-collection-vc-dir vc-dir ewoc
magit-extras goto-addr vc-mtn vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs
vc-rcs vc bug-reference magit-gitflow evil-collection-vc-git vc-git
vc-dispatcher company-yasnippet delsel restart-emacs desktop frameset
evil-collection-help vertico-directory tabify elisp-demos
evil-collection-indent evil-collection-helpful helpful trace
evil-collection-edebug edebug info-look evil-collection-info info
evil-collection-elisp-refs elisp-refs jka-compr hide-mode-line help-fns
radix-tree cl-print evil-collection-debug debug backtrace mule-util
vertico-repeat cursor-sensor consult-flycheck evil-collection-consult
consult-org consult-imenu consult-vertico consult magit-bookmark
evil-collection-bookmark bookmark preview cdlatex adaptive-wrap evil-tex
reftex-dcr reftex-auc laas aas tex-fold spell-fu font-latex
latex-mode-expansions bibtex-actions-filenotify bibtex-actions-latex
evil-collection-reftex reftex-toc reftex-cite reftex-ref reftex-parse
reftex reftex-loaddefs reftex-vars bibtex-actions bibtex-completion
biblio biblio-download biblio-dissemin biblio-ieee biblio-hal
biblio-dblp biblio-crossref biblio-arxiv timezone biblio-doi biblio-core
url-queue ido parsebib bibtex-actions-file org-id bibtex iso8601
auctex-latexmk tex-buf latex latex-flymake evil-collection-flymake
flymake-proc flymake project tex-ispell tex-style tex dbus xml texmathp
smartparens-latex tex-mode latexenc auto-minor-mode disp-table
whitespace flycheck-popup-tip evil-collection-popup popup flycheck-cask
evil-embrace evil-surround embrace expand-region text-mode-expansions
the-org-mode-expansions er-basic-expansions expand-region-core
expand-region-custom eros highlight-quoted rainbow-delimiters
highlight-numbers parent-mode display-line-numbers projectile ibuffer-vc
ibuf-ext evil-collection-ibuffer ibuffer ibuffer-loaddefs
doom-themes-ext-org solaire-mode face-remap doom-tomorrow-day-theme
doom-themes doom-themes-base org-capture org-agenda org-refile
evil-collection-magit-todos magit-todos pcre2el rxt re-builder hl-todo
async evil-collection-grep grep evil-collection-compile compile orgit
smartparens-org org-yt org-element avl-tree generator org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table ol org-keys oc
org-compat org-macs org-loaddefs evil-collection-calendar cal-menu
calendar cal-loaddefs github-review ghub-graphql treepy gsexp ghub
url-http url-gw nsm url-auth gnutls deferred a evil-collection-magit
magit-autoloads magit-submodule magit-obsolete magit-popup magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log which-func magit-diff smerge-mode diff
evil-collection-diff-mode diff-mode magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
magit-mode git-commit transient format-spec evil-collection-log-edit
log-edit message rmc puny evil-collection-dired dired dired-loaddefs
rfc822 mml mml-sec evil-collection-epa epa epg rfc6068 epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date mm-decode
mm-bodies mm-encode mailabbrev mail-utils gmm-utils mailheader pcvs-util
add-log magit-git magit-section magit-utils crm with-editor f s
doom-snippets doom-snippets-lib yasnippet evil-collection-elisp-mode
server dtrt-indent evil-collection-which-key which-key savehist
better-jumper company-capf company vertico orderless marginalia
evil-goggles pulse color evil-easymotion evil-escape evil-snipe recentf
tree-widget saveplace so-long gcmh hl-line winner paren
smartparens-config smartparens-text smartparens ws-butler
undo-fu-session undo-fu flycheck-package package-lint
evil-collection-imenu imenu evil-collection-finder finder finder-inf
lisp-mnt mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr evil-collection-package-menu core-packages package browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers url-parse auth-source eieio
eieio-core eieio-loaddefs password-cache url-vars
evil-collection-flycheck evil-collection-custom cus-edit cus-start
cus-load wid-edit evil-collection-comint evil-collection annalist
flycheck json map find-func dash persp-mode ibuf-macs evil
evil-integration evil-maps evil-commands reveal flyspell evil-jumps
evil-command-window evil-types evil-search shell pcomplete comint
ansi-color evil-macros evil-repeat evil-states evil-core advice
evil-common windmove calc calc-loaddefs calc-macs thingatpt rect
evil-digraphs evil-vars ring all-the-icons all-the-icons-faces
data-material data-weathericons data-octicons data-fileicons
data-faicons data-alltheicons let-alist derived edmacro kmacro
core-editor core-projects core-ui easy-mmode comp comp-cstr warnings rx
core-keybinds pp general cl-extra help-mode seq byte-opt cl-seq
use-package-core bytecomp byte-compile cconv core-modules tex-site core
core-lib pcase cl-macs gv cl-loaddefs cl-lib subr-x iso-transl tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/pgtk-win pgtk-win term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit pgtk multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 1057984 116256)
 (symbols 48 69380 3)
 (strings 32 354565 43630)
 (string-bytes 1 8886860)
 (vectors 16 122135)
 (vector-slots 8 3279970 69608)
 (floats 8 1209 337)
 (intervals 56 15396 641)
 (buffers 992 31))

[-- Attachment #1.2: Type: text/html, Size: 32061 bytes --]

[-- Attachment #2: 2021-10-01T21:43:10,566697499+02:00.png --]
[-- Type: image/png, Size: 22221 bytes --]

[-- Attachment #3: 2021-10-01T21:49:10,611532571+02:00.png --]
[-- Type: image/png, Size: 8086 bytes --]

[-- Attachment #4: 2021-10-01T22:02:00,166113989+02:00.png --]
[-- Type: image/png, Size: 30011 bytes --]

[-- Attachment #5: 2021-10-01T22:05:28,910692941+02:00.png --]
[-- Type: image/png, Size: 11781 bytes --]

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Fri, 1 Oct 2021 22:11:32 +0200
> 
> Starting with 'emacs -Q' and using the menu bar to paste urdu text

> حرف نہیں جاں بخشی میں اس کی خوبی اپنی قسمت کی
> ہم سے جو پہلے کہہ بھیجا سو مرنے کا پیغام کیا

> the text is not rendered properly, some characters which should be
> joined together are instead rendered individually.  
>
> It look like this
> 2021-10-01T21:43:10,566697499+02:00.png
> 2021-10-01T21:49:10,611532571+02:00.png

Can you give a few specific examples of characters that should be
joined, but aren't?  Please name the characters and also give they
positions relative to the beginning of this text, as I don't read
Urdu, so the images are useless for me without some additional data
and explanations.

> Changing the font to a Nastaliq font more common for Urdu via
> 
> (set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu"))
> 
> the text is again rendered improperly but in a different way with chracters overlapping each other when they
> shouldn't, others have gaps where they should be joined. It looks like this
> 
> 2021-10-01T22:02:00,166113989+02:00.png
> while it should look like this 
> 
> 2021-10-01T22:05:28,910692941+02:00.png

So isn't this a matter of finding a proper font, in particularly given
the "Nastaliq vs Naskh" issues?  NotoNastaliqUrdu is not the only font
supporting Nastaliq, so perhaps other fonts fare better?

Since Urdu uses the Arabic characters, Emacs uses character
composition rules for Arabic when displaying this text.  Do you know
if the composition rules for Urdu are different?

Also, which version of HarfBuzz do you have installed?

Thanks.

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1: Type: text/plain, Size: 3483 bytes --]

I forgot to reply all for my reply and it didn't go to the mailing list.
Sorry about that and I am forwarding it
to the mailing list now.

---------- Forwarded message ---------
From: Rah Guzar <aikrahguzar@gmail.com>
Date: Sat, Oct 2, 2021 at 1:40 PM
Subject: Re: bug#50951: 28.0.50; Urdu text is not displayed correctly
To: Eli Zaretskii <eliz@gnu.org>


Hi,
  Thanks a lot for the reply.

On Sat, Oct 2, 2021 at 8:07 AM Eli Zaretskii <eliz@gnu.org> wrote:

> -10-01T21:49:10,611532571+02:00.png
>
> Can you give a few specific examples of characters that should be
> joined, but aren't?  Please name the characters and also give they
> positions relative to the beginning of this text, as I don't read
> Urdu, so the images are useless for me without some additional data
> and explanations.
>

Let us consider the word نہیں

It is composed of four letters. I will use character field from
`describe-char` for each of them below
1) ن‎ (displayed as ن‎) (codepoint 1606, #o3106, #x646)
2)  ہ‎ (displayed as ہ‎) (codepoint 1729, #o3301, #x6c1)
3)  ی‎ (displayed as ی‎) (codepoint 1740, #o3314, #x6cc)
4) ں‎ (displayed as ں‎) (codepoint 1722, #o3272, #x6ba)

It should be displayed with all 4 characters joined together, instead they
are all displayed individually.
If I change to `NotoNastaliqUrdu` this word is displayed correctly. But
there is problem with   حرف

It consist of three letters,
1) ح‎ (displayed as ح‎) (codepoint 1581, #o3055, #x62d)
2) ر‎ (displayed as ر‎) (codepoint 1585, #o3061, #x631)
3) ف‎ (displayed as ف‎) (codepoint 1601, #o3101, #x641)

The first two characters should be joined and the last one should be on its
own. This seems to be the case.
But the two groups are rendered on top of each other making it illegible.

So isn't this a matter of finding a proper font, in particularly given
> the "Nastaliq vs Naskh" issues?  NotoNastaliqUrdu is not the only font
> supporting Nastaliq, so perhaps other fonts fare better?
>

My knowledge here is very deficient but my impression is Nastaliq and Naskh
are styles and shouldn't affect composition.
NotoNastaliqUrdu was the only Urdu font available from my distro.
Libreoffice which also uses harfbuzz renders it
correctly so I didn't try another font at first. Like emacs libreoffice
also uses a Naskh font by default but all the characters
are joined properly.

I did try some fonts from https://urdufonts.net/ after your suggestions and
they render correctly. Specifically the font I tried
were:
Jameel Noori Nastaleeq Regular
Alvi Nastaleeq
Zohra Unicode
Manzor Unicode

I didn't notice a problem with any of them except a very minor one for the
last two which have visible boundaries where glyphs
are joined.

Since Urdu uses the Arabic characters, Emacs uses character
> composition rules for Arabic when displaying this text.  Do you know
> if the composition rules for Urdu are different?
>

I think using Arabic composition rules might be part of the problem. Urdu
alphabet is a superset of Arabic alphabet and if I
don't set a font specifically designed for Urdu, the words where some
characters should be joined but aren't always seem to
include a character like ہ which is in Urdu alphabet but not in Arabic.

Also, which version of HarfBuzz do you have installed?
>
It is 2.9.1

Please let me know if you need any more information.

Thanks a lot again.

[-- Attachment #2: Type: text/html, Size: 5132 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 13:43:47 +0200
> 
> Let us consider the word نہیں
> 
> It is composed of four letters. I will use character field from `describe-char` for each of them below 
> 1) ن‎ (displayed as ن‎) (codepoint 1606, #o3106, #x646)
> 2)  ہ‎ (displayed as ہ‎) (codepoint 1729, #o3301, #x6c1)
> 3)  ی‎ (displayed as ی‎) (codepoint 1740, #o3314, #x6cc)
> 4) ں‎ (displayed as ں‎) (codepoint 1722, #o3272, #x6ba)
> 
> It should be displayed with all 4 characters joined together, instead they are all displayed individually.

What font displays them individually?  You should be able to tell that
if you type "C-u C-x =" on one of these characters.

For me, they display joined together.

> If I change to `NotoNastaliqUrdu` this word is displayed correctly. But there is problem with   حرف
> 
> It consist of three letters,
> 1) ح‎ (displayed as ح‎) (codepoint 1581, #o3055, #x62d)
> 2) ر‎ (displayed as ر‎) (codepoint 1585, #o3061, #x631)
> 3) ف‎ (displayed as ف‎) (codepoint 1601, #o3101, #x641)
> 
> The first two characters should be joined and the last one should be on its own. This seems to be the case.
> But the two groups are rendered on top of each other making it illegible.
> 
>  So isn't this a matter of finding a proper font, in particularly given
>  the "Nastaliq vs Naskh" issues?  NotoNastaliqUrdu is not the only font
>  supporting Nastaliq, so perhaps other fonts fare better?
>  
> My knowledge here is very deficient but my impression is Nastaliq and Naskh are styles and shouldn't affect
> composition.
> NotoNastaliqUrdu was the only Urdu font available from my distro.  Libreoffice which also uses harfbuzz
> renders it
> correctly so I didn't try another font at first. Like emacs libreoffice also uses a Naskh font by default but all the
> characters are joined properly.
> 
> I did try some fonts from https://urdufonts.net/ after your suggestions and they render correctly. Specifically
> the font I tried
> were: 
> Jameel Noori Nastaleeq Regular
> Alvi Nastaleeq 
> Zohra Unicode
> Manzor Unicode
> 
> I didn't notice a problem with any of them except a very minor one for the last two which have visible
> boundaries where glyphs
> are joined.  

So would it be correct to say that using a proper font solves the
problem?

>  Since Urdu uses the Arabic characters, Emacs uses character
>  composition rules for Arabic when displaying this text.  Do you know
>  if the composition rules for Urdu are different?
> 
> I think using Arabic composition rules might be part of the problem. Urdu alphabet is a superset of Arabic
> alphabet and if I
> don't set a font specifically designed for Urdu, the words where some characters should be joined but aren't
> always seem to
> include a character like ہ which is in Urdu alphabet but not in Arabic. 

I don't think the problem is with compositions, because in the 2
examples you described above, there are no character compositions.

Moreover, our pattern for asking HarfBuzz to shape Arabic text is
this:

   "[\u0600-\u074F\u200C\u200D]+"

which includes all of the characters, including U+06C1 which you say
causes problems.

You could try setting current-iso639-language to the symbol 'ur'
(without the quotes), that should tell HarfBuzz to shape the text as
appropriate for Urdu.  But I think the real problem is with the font,
not with shaping.

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

On Sat, Oct 2, 2021 at 2:18 PM Eli Zaretskii <eliz@gnu.org> wrote:

> What font displays them individually?  You should be able to tell that
> if you type "C-u C-x =" on one of these characters.
>
This is the default font that `emacs -Q` chooses. Using "C-u C-x =" I see,

ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
(#x56F)


> So would it be correct to say that using a proper font solves the
> problem?
>

This is almost correct except that the font (NotoNastaliqUrdu) that causes
problems with emacs
works fine in libreoffice and both use harfbuzz.


> I don't think the problem is with compositions, because in the 2
> examples you described above, there are no character compositions.
>
> Moreover, our pattern for asking HarfBuzz to shape Arabic text is
> this:
>
>    "[\u0600-\u074F\u200C\u200D]+"
>
> which includes all of the characters, including U+06C1 which you say
> causes problems.
>
> You could try setting current-iso639-language to the symbol 'ur'
> (without the quotes), that should tell HarfBuzz to shape the text as
> appropriate for Urdu.  But I think the real problem is with the font,
> not with shaping.
>

Sorry, my lack of understanding of terminology got in the way here. I
thought composition referred to
joining characters together.

I did try setting `current-iso639-language` to the symbol `ur`. As you
expected it didn't make a difference.

[-- Attachment #2: Type: text/html, Size: 2251 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 14:47:37 +0200
> Cc: 50951@debbugs.gnu.org
> 
> On Sat, Oct 2, 2021 at 2:18 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  What font displays them individually?  You should be able to tell that
>  if you type "C-u C-x =" on one of these characters.
> 
> This is the default font that `emacs -Q` chooses. Using "C-u C-x =" I see,
> 
> ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1 (#x56F)

I guess that font is problematic (in more than one way).

>  So would it be correct to say that using a proper font solves the
>  problem?
> 
> This is almost correct except that the font (NotoNastaliqUrdu) that causes problems with emacs
> works fine in libreoffice and both use harfbuzz.

The way to investigate such problems is to see what does hb-view, a
program that is part of the HarfBuzz installation, produce for the
same text with the same font.  If hb-view produces correct display,
but Emacs doesn't, then the problem is indeed in Emacs; otherwise the
problem is probably with the font, and in any case should be taken up
with the HarfBuzz developers.

(Are you sure that LibreOffice uses NotoNastaliqUrdu for the text you
type there?  They could use a different font under the hood.)

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Andreas Schwab]

On Okt 02 2021, Rah Guzar wrote:

> On Sat, Oct 2, 2021 at 2:18 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> What font displays them individually?  You should be able to tell that
>> if you type "C-u C-x =" on one of these characters.
>>
> This is the default font that `emacs -Q` chooses. Using "C-u C-x =" I see,
>
> ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
> (#x56F)

DejaVu Sans misses a glyph for ARABIC LETTER HEH GOAL, so Emacs has to
select a different font for it.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1.1: Type: text/plain, Size: 4656 bytes --]

On Sat, Oct 2, 2021 at 3:09 PM Eli Zaretskii <eliz@gnu.org> wrote:

> The way to investigate such problems is to see what does hb-view, a
> program that is part of the HarfBuzz installation, produce for the
> same text with the same font.  If hb-view produces correct display,
> but Emacs doesn't, then the problem is indeed in Emacs; otherwise the
> problem is probably with the font, and in any case should be taken up
> with the HarfBuzz developers.
>

I tried hb-view with NotoNastaliqUrdu and the text:
 خوبی اپنی قسمت کی
This is what I get
[image: urduhbtestnoto.png]
While in emacs I discovered how it is displayed depends a lot on the font
size.

For the same text at size 16, I get

[image: emacsq16.png]

At size 24 it looks almost correct
[image: emacsq24.png]
At size 32 it is really bad again
[image: emacsq32.png]
And the issue seem to be glyph placement rather than shaping.

NotoNastaliqUrdu seems to be the only font with this issue. I am not sure
if the problem is due to Nastaliq.
The other two Nastaliq fonts seem to handle joining characters through
composition. If I change font using

(set-fontset-font t 'arabic (font-spec :family "Jameel Noori Nastaleeq"
:size 32))

and move cursor to the word "قسمت" which has 4 characters, the cursor
encompasses all of them and "C-u C-x u"
gives

-----------------------------------------------------------------------------------------------------------------------
             position: 157 of 283 (55%), column: 11
            character: ق‎ (displayed as ق‎) (codepoint 1602, #o3102, #x642)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0642
               script: arabic
               syntax: w which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "C-x 8 RET 642" or "C-x 8 RET ARABIC LETTER QAF"
          buffer code: #xD9 #x82
            file code: #xD9 #x82 (encoded by coding system utf-8-unix)
              display: composed to form "قسمت" (see below)

Composed with the following character(s) "سمت" using this font:
  ftcrhb:-pdms-Jameel Noori
Nastaleeq-normal-normal-normal-*-32-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 3 1578 11352 50 1 51 30 1 nil]
with these character(s):
  س (#x633) ARABIC LETTER SEEN
  م (#x645) ARABIC LETTER MEEM
  ت (#x62a) ARABIC LETTER TEH

Character code properties: customize what to show
  name: ARABIC LETTER QAF
  general-category: Lo (Letter, Other)
  decomposition: (1602) ('ق')

There are text properties here:
  fontified            nil
-----------------------------------------------------------------------------------------------------------------------------

Changing to NotoNastaliqUrdu using

(set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu" :size 32))

the cursor moves through one character at a time and moving the cursor to
the beginning of the same word
"C-u C-x =" gives

-----------------------------------------------------------------------------------------------------------------------------------------
             position: 157 of 282 (55%), column: 11
            character: ق‎ (displayed as ق‎) (codepoint 1602, #o3102, #x642)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0642
               script: arabic
               syntax: w which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "C-x 8 RET 642" or "C-x 8 RET ARABIC LETTER QAF"
          buffer code: #xD9 #x82
            file code: #xD9 #x82 (encoded by coding system utf-8-unix)
              display: composed to form "ق" (see below)

Composed using this font:
  ftcrhb:-GOOG-Noto Nastaliq
Urdu-normal-normal-normal-*-32-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 0 1602 16 0 -6 6 35 -26 [3 -16 0]]
  [0 0 1602 983 0 0 0 0 0 nil]
  [0 0 1602 284 8 -1 8 24 6 [0 -23 8]]

Character code properties: customize what to show
  name: ARABIC LETTER QAF
  general-category: Lo (Letter, Other)
  decomposition: (1602) ('ق')

There are text properties here:
  fontified            t
-----------------------------------------------------------------------------------------------------------------------------------------

(Are you sure that LibreOffice uses NotoNastaliqUrdu for the text you
> type there?  They could use a different font under the hood.)
>

LibreOffice uses something else by default and when I changed to
NotoNastaliqUrdu the appearance changes
and is the same as what I get with hb-view.

[-- Attachment #1.2: Type: text/html, Size: 6234 bytes --]

[-- Attachment #2: urduhbtestnoto.png --]
[-- Type: image/png, Size: 28533 bytes --]

[-- Attachment #3: emacsq16.png --]
[-- Type: image/png, Size: 1642 bytes --]

[-- Attachment #4: emacsq24.png --]
[-- Type: image/png, Size: 3916 bytes --]

[-- Attachment #5: emacsq32.png --]
[-- Type: image/png, Size: 4810 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  50951@debbugs.gnu.org
> Date: Sat, 02 Oct 2021 16:18:35 +0200
> 
> On Okt 02 2021, Rah Guzar wrote:
> 
> > On Sat, Oct 2, 2021 at 2:18 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >> What font displays them individually?  You should be able to tell that
> >> if you type "C-u C-x =" on one of these characters.
> >>
> > This is the default font that `emacs -Q` chooses. Using "C-u C-x =" I see,
> >
> > ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
> > (#x56F)
> 
> DejaVu Sans misses a glyph for ARABIC LETTER HEH GOAL, so Emacs has to
> select a different font for it.

Thanks, should be fixed now.

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 16:19:01 +0200
> Cc: 50951@debbugs.gnu.org
> 
> I tried hb-view with NotoNastaliqUrdu and the text:
>  خوبی اپنی قسمت کی
> This is what I get
> urduhbtestnoto.png
> While in emacs I discovered how it is displayed depends a lot on the font size.
> 
> For the same text at size 16, I get 
> 
> emacsq16.png
>   
> At size 24 it looks almost correct 
> emacsq24.png
> At size 32 it is really bad again
> emacsq32.png
> And the issue seem to be glyph placement rather than shaping.
>  
> NotoNastaliqUrdu seems to be the only font with this issue. I am not sure if the problem is due to Nastaliq.

If you set bidi-display-reordering to a nil value, does the display
become better or worse?

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

On Sat, Oct 2, 2021 at 4:18 PM Andreas Schwab <schwab@linux-m68k.org> wrote:


> DejaVu Sans misses a glyph for ARABIC LETTER HEH GOAL, so Emacs has to
> select a different font for it.
>

This also true for ARABIC LETTER YEH BARREE which causes also causes the
same problem.

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 17:05:24 +0200
> 
>  If you set bidi-display-reordering to a nil value, does the display
>  become better or worse?
> 
> Setting it to nil completely garbles the text. This I what I see afterwards 
> reordering-nil.png

OK, that's what I expected.  It was a stab in the dark.

> By the way, the problem of size dependent misplacement happens with  another Urdu font
> Urdu_Emad_Nastaliq that doesn't rely on extensive composition which I tried. 

I'm afraid this is where I must stop trying to help investigating this
issue, as I'm long out of my depth in this area.  It sounds like some
tricky issue with the fonts, but maybe something else is at work here,
e.g., with Cairo display or something.  But I'm just guessing at this
point, sorry.

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 17:07:57 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, 50951@debbugs.gnu.org
> 
> On Sat, Oct 2, 2021 at 4:18 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
>  
>  DejaVu Sans misses a glyph for ARABIC LETTER HEH GOAL, so Emacs has to
>  select a different font for it.
> 
> This also true for ARABIC LETTER YEH BARREE which causes also causes the same problem. 

As I already added U+06C1 ARABIC LETTER HEH GOAL to the list of
required characters for displaying Arabic, the question is now: are
there fonts inappropriate for Arabic that support U+06C1, but do NOT
support U+06D2 ARABIC LETTER YEH BARREE?  If there are such fonts,
then we should add U+06D2 to the list of required characters as well.

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

On Sat, Oct 2, 2021 at 5:10 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I'm afraid this is where I must stop trying to help investigating this
> issue, as I'm long out of my depth in this area.  It sounds like some
> tricky issue with the fonts, but maybe something else is at work here,
> e.g., with Cairo display or something.  But I'm just guessing at this
> point, sorry.
>

Thanks a lot for the help. It would be nice if there is a proper solution to
this problem but there is at least an easy workaround of using large Urdu
fonts like "Jameel Noori Nastaleeq" or "Alvi Nastaleeq". It seems like only
small fonts like "NotoNastaliqUrdu" have this problem.

[-- Attachment #2: Type: text/html, Size: 1091 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 17:27:33 +0200
> 
>  As I already added U+06C1 ARABIC LETTER HEH GOAL to the list of
>  required characters for displaying Arabic, the question is now: are
>  there fonts inappropriate for Arabic that support U+06C1, but do NOT
>  support U+06D2 ARABIC LETTER YEH BARREE?  If there are such fonts,
>  then we should add U+06D2 to the list of required characters as well.
> 
> I haven't seen a font like this and I will reach out if I do. But I think this solution
> might cause issues with fonts for just Arabic which don't support Urdu (if there
> are such).

What kind of issues did you have in mind?

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

> What kind of issues did you have in mind?
>

This might be me not understanding how required characters work but if a
user sets a font
for only Arabic (not Urdu) and the font doesn't have these characters, will
it cause emacs
to use a fallback for Arabic text? Or is the restriction only relevant for
the selection of implicit
fallback?

[-- Attachment #2: Type: text/html, Size: 612 bytes --]

bug#50951: Fwd: bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <aikrahguzar@gmail.com>
> Date: Sat, 2 Oct 2021 18:06:09 +0200
> Cc: 50951@debbugs.gnu.org
> 
>  What kind of issues did you have in mind?
> 
> This might be me not understanding how required characters work but if a user sets a font
> for only Arabic (not Urdu) and the font doesn't have these characters, will it cause emacs
> to use a fallback for Arabic text? Or is the restriction only relevant for the selection of implicit
> fallback?

No, if the user customizes the fontset, the font specified there will
be used, no questions asked.  The representative characters are only
used for selecting the fonts by default.  AFAIK, at least.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Lars Ingebrigtsen]

I've only lightly skimmed this thread, but it seems like at least some
of the problems were solved.  Is there anything more to do here on the
Emacs side?

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rah Guzar <aikrahguzar@gmail.com>,  50951@debbugs.gnu.org
> Date: Sun, 04 Sep 2022 23:07:30 +0200
> 
> I've only lightly skimmed this thread, but it seems like at least some
> of the problems were solved.  Is there anything more to do here on the
> Emacs side?

Not from my POV, no.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've only lightly skimmed this thread, but it seems like at least some
> of the problems were solved.  Is there anything more to do here on the
> Emacs side?

Hi Lars and Eli,
   I am on emacs 28 so this might not be accurate for the development
   version of emacs. Right now urdu text renders fine for me using the
   default font which for me is,

   -AXIS-Zohra Unicode-normal-normal-normal-*-32-*-*-*-*-0-iso10646-1

   So that aspect has improved.

   However if I change my font to Noto Nastaliq by evaluating,

   (set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu"))

   rendering is still bad. There are nastaliq fonts
   (very big ones) that render fine. My experience is that if font size
   is around 10 megabytes it renders fine, if the size is less than that
   it has problems (on my system Noto Nastaliq is 1.1M).

   As in the thread this is specific to emacs and not a problem with
   harfbuzz generally.

   So situation is acceptable but needs workarounds to get a good
   experience.

Rahguzar

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <rahguzar@zohomail.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>, 50951@debbugs.gnu.org
> Date: Mon, 05 Sep 2022 13:57:24 +0200
> 
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > I've only lightly skimmed this thread, but it seems like at least some
> > of the problems were solved.  Is there anything more to do here on the
> > Emacs side?
> 
> Hi Lars and Eli,
>    I am on emacs 28 so this might not be accurate for the development
>    version of emacs. Right now urdu text renders fine for me using the
>    default font which for me is,
> 
>    -AXIS-Zohra Unicode-normal-normal-normal-*-32-*-*-*-*-0-iso10646-1
> 
>    So that aspect has improved.
> 
>    However if I change my font to Noto Nastaliq by evaluating,
> 
>    (set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu"))
> 
>    rendering is still bad. There are nastaliq fonts
>    (very big ones) that render fine. My experience is that if font size
>    is around 10 megabytes it renders fine, if the size is less than that
>    it has problems (on my system Noto Nastaliq is 1.1M).
> 
>    As in the thread this is specific to emacs and not a problem with
>    harfbuzz generally.

It is unclear to me why this conclusion.  Emacs uses HarfBuzz, and the
only factor that could affect that, apart from selecting the font, is
the setting of the current-iso639-language variable, which AFAIR
Rahguzar tried setting with no success.

>    So situation is acceptable but needs workarounds to get a good
>    experience.

My conclusion from this is that Noto Nastaliq is not a good font for
Urdu.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Eli,
Eli Zaretskii <eliz@gnu.org> writes:

> It is unclear to me why this conclusion.  Emacs uses HarfBuzz, and the
> only factor that could affect that, apart from selecting the font, is
> the setting of the current-iso639-language variable, which AFAIR
> Rahguzar tried setting with no success.

At your suggestion, I tried hb-view and it renders Noto Nastaliq fine.
Similarly Libre Office which also uses harfbuzz as far as I understand,
also renders it fine. Which is why I said that the problem is limited to
emacs. My understanding of font rendering is non-existent but visually
what seems to happen is that emacs displays all the individual atoms
(glyphs?) but it doesn't know how to position them relative to each
other so they overlap and obscure each other. This positioning is
especially tricky in Nastaliq fonts since it can require moving all of
up, down, left, right. The big fonts that emacs render correctly, take
care of this by prepackaging all these combinations of characters that
require anything other than right to left movement as separate shapes.

Sorry, if this reads as confused.

> My conclusion from this is that Noto Nastaliq is not a good font for
> Urdu.

On an aesthetic level, I tend to agree. But linux distributions package
very few Urdu fonts and it tends to be one of the few.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <rahguzar@zohomail.eu>
> Cc: larsi@gnus.org, 50951@debbugs.gnu.org
> Date: Mon, 05 Sep 2022 15:03:47 +0200
> 
> > My conclusion from this is that Noto Nastaliq is not a good font for
> > Urdu.
> 
> On an aesthetic level, I tend to agree. But linux distributions package
> very few Urdu fonts and it tends to be one of the few.

FWIW, on my system I cannot convince Emacs to use Noto Nastaliq Urdu
font for Urdu.  I have no idea what's going on here, sorry.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <rahguzar@zohomail.eu>
> Cc: Rah Guzar <rahguzar@zohomail.eu>, larsi@gnus.org, 50951@debbugs.gnu.org
> Date: Mon, 05 Sep 2022 16:09:31 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > FWIW, on my system I cannot convince Emacs to use Noto Nastaliq Urdu
> > font for Urdu.  I have no idea what's going on here, sorry.
> 
> I do it by evaluating,
> 
> (set-fontset-font t 'arabic (font-spec :family "NotoNastaliqUrdu")
> 
> and `describe-char` confirms that it is being used. I have attached a screenshot.

That's what I tried as well, among many other things.  Nothing worked.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Visuwesh]

[திங்கள் செப்டம்பர் 05, 2022] Rah Guzar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> Hi Eli,
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> It is unclear to me why this conclusion.  Emacs uses HarfBuzz, and the
>> only factor that could affect that, apart from selecting the font, is
>> the setting of the current-iso639-language variable, which AFAIR
>> Rahguzar tried setting with no success.
>
> At your suggestion, I tried hb-view and it renders Noto Nastaliq fine.
> Similarly Libre Office which also uses harfbuzz as far as I understand,
> also renders it fine. Which is why I said that the problem is limited to
> emacs. My understanding of font rendering is non-existent but visually
> what seems to happen is that emacs displays all the individual atoms
> (glyphs?) but it doesn't know how to position them relative to each
> other so they overlap and obscure each other. This positioning is
> especially tricky in Nastaliq fonts since it can require moving all of
> up, down, left, right. The big fonts that emacs render correctly, take
> care of this by prepackaging all these combinations of characters that
> require anything other than right to left movement as separate shapes.

To my ears, this sounds an awful lot like bug#54646 where I faced
similar font clipping issues with Noto Serif Tamil (and other Tamil
fonts).  The issues with clipping and other glyph placement issues went
away when I used the Xft+Harfbuzz backend, perhaps that might fix your
issue as well?  But you might trade crisp font rendering for a slightly
blurry one though; in my case it was a trade-off I had to make.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rah Guzar <rahguzar@zohomail.eu>,
>   larsi@gnus.org,  50951@debbugs.gnu.org
> Date: Tue, 06 Sep 2022 09:56:09 +0530
> 
> The issues with clipping and other glyph placement issues went
> away when I used the Xft+Harfbuzz backend

As opposed to what? Cairo+HarfBuzz? or something else?

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Visuwesh]

[செவ்வாய் செப்டம்பர் 06, 2022] Eli Zaretskii wrote:

>> From: Visuwesh <visuweshm@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Rah Guzar <rahguzar@zohomail.eu>,
>>   larsi@gnus.org,  50951@debbugs.gnu.org
>> Date: Tue, 06 Sep 2022 09:56:09 +0530
>> 
>> The issues with clipping and other glyph placement issues went
>> away when I used the Xft+Harfbuzz backend
>
> As opposed to what? Cairo+HarfBuzz? or something else?

Cairo+Harfbuzz.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

On Mon, 05 Sep 2022 22:03:47 +0900,
Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors wrote:
> 
> At your suggestion, I tried hb-view and it renders Noto Nastaliq fine.
> Similarly Libre Office which also uses harfbuzz as far as I understand,
> also renders it fine. Which is why I said that the problem is limited to
> emacs.

Probably the following change will fix the problem:

diff --git a/src/hbfont.c b/src/hbfont.c
index 2721a66120..01db522909 100644
--- a/src/hbfont.c
+++ b/src/hbfont.c
@@ -598,6 +598,8 @@ hbfont_shape (Lisp_Object lgstring, Lisp_Object direction)
 					      make_fixnum (xoff),
 					      make_fixnum (yoff),
 					      make_fixnum (wadjust)));
+      else
+	LGLYPH_SET_ADJUSTMENT (lglyph, Qnil);
     }
 
   return make_fixnum (glyph_len);


The other shapers except uniscribe_shape need to be changed, too.

Alternatively, one could clear garbage entries at the caller side:

diff --git a/src/composite.c b/src/composite.c
index 22422cca09..249d7587f6 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -876,7 +876,8 @@ fill_gstring_body (Lisp_Object gstring)
 	}
       LGLYPH_SET_ADJUSTMENT (g, Qnil);
     }
-  if (i < LGSTRING_GLYPH_LEN (gstring))
+  len = LGSTRING_GLYPH_LEN (gstring);
+  for (; i < len; i++)
     LGSTRING_SET_GLYPH (gstring, i, Qnil);
 }

 
				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> Date: Wed, 07 Sep 2022 15:18:21 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	larsi@gnus.org,
> 	50951@debbugs.gnu.org
> 
> Probably the following change will fix the problem:

Thanks.  Rah Guzar, can you try one of these changes and see if they
fix the problem?

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Visuwesh]

[புதன் செப்டம்பர் 07, 2022] Eli Zaretskii wrote:

>> Date: Wed, 07 Sep 2022 15:18:21 +0900
>> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> 	larsi@gnus.org,
>> 	50951@debbugs.gnu.org
>> 
>> Probably the following change will fix the problem:
>
> Thanks.  Rah Guzar, can you try one of these changes and see if they
> fix the problem?

Not sure about Rah Guzar, but it squashes bug#54646 for me.  Thank you
so very much, Yamamoto Mitsuharu!

With regards to the performance, my eyes say that patch no. 1 (i.e., the
one that patches src/hbfont.c) is slightly faster than patch no. 2 when
I continually zoomed to reproduce bug#54646.  The test material was this
random webpage https://www.dinamalar.com/news_detail.asp?id=3117719,
and M-x report-emacs-bug report on the Emacs with the patch is at the
end.
The test that I used is the same as I outlined in bug#54646, namely,
visit that webpage in eww, zoom in and out with C-scroll-wheel.  The
glyphs _do not_ bump into each other or get weirdly placed whereas
without the patch, glyphs can be oddly spaced.

In GNU Emacs 29.0.50 (build 6, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw scroll bars) of 2022-09-08 built on astatine
Repository revision: 9219e83b3c0ef53df02caf4c8ba38f482937ab50
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --with-sound=alsa --with-x-toolkit=lucid --with-json
 --without-xaw3d --without-gconf --without-libsystemd'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LC_MONETARY: ta_IN.UTF-8
  value of $LC_NUMERIC: ta_IN.UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Unfortunately I have very limited experience with building software and
none at all with applying patches so testing this will be a non-trivial
undertaking. I will try to do this next weekend.

But what Visuwesh describes seem exactly the symptoms I was experiencing
so I am pretty positive that it will work for me too. So thank you very
much indeed!

Rah Guzar

Visuwesh <visuweshm@gmail.com> writes:

> [புதன் செப்டம்பர் 07, 2022] Eli Zaretskii wrote:
>
>>> Date: Wed, 07 Sep 2022 15:18:21 +0900
>>> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
>>> Cc: Eli Zaretskii <eliz@gnu.org>,
>>> 	larsi@gnus.org,
>>> 	50951@debbugs.gnu.org
>>>
>>> Probably the following change will fix the problem:
>>
>> Thanks.  Rah Guzar, can you try one of these changes and see if they
>> fix the problem?
>
> Not sure about Rah Guzar, but it squashes bug#54646 for me.  Thank you
> so very much, Yamamoto Mitsuharu!
>
> With regards to the performance, my eyes say that patch no. 1 (i.e., the
> one that patches src/hbfont.c) is slightly faster than patch no. 2 when
> I continually zoomed to reproduce bug#54646.  The test material was this
> random webpage https://www.dinamalar.com/news_detail.asp?id=3117719,
> and M-x report-emacs-bug report on the Emacs with the patch is at the
> end.
> The test that I used is the same as I outlined in bug#54646, namely,
> visit that webpage in eww, zoom in and out with C-scroll-wheel.  The
> glyphs _do not_ bump into each other or get weirdly placed whereas
> without the patch, glyphs can be oddly spaced.
>
> In GNU Emacs 29.0.50 (build 6, x86_64-pc-linux-gnu, X toolkit, cairo
>  version 1.16.0, Xaw scroll bars) of 2022-09-08 built on astatine
> Repository revision: 9219e83b3c0ef53df02caf4c8ba38f482937ab50
> Repository branch: master
> Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
> System Description: Debian GNU/Linux bookworm/sid
>
> Configured using:
>  'configure --with-sound=alsa --with-x-toolkit=lucid --with-json
>  --without-xaw3d --without-gconf --without-libsystemd'
>
> Configured features:
> ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
> JSON LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
> SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM
> XINPUT2 XPM LUCID ZLIB
>
> Important settings:
>   value of $LC_MONETARY: ta_IN.UTF-8
>   value of $LC_NUMERIC: ta_IN.UTF-8
>   value of $LANG: en_GB.UTF-8
>   locale-coding-system: utf-8-unix

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Visuwesh]

[வியாழன் செப்டம்பர் 08, 2022] Visuwesh wrote:

> [புதன் செப்டம்பர் 07, 2022] Eli Zaretskii wrote:
>
>>> Date: Wed, 07 Sep 2022 15:18:21 +0900
>>> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
>>> Cc: Eli Zaretskii <eliz@gnu.org>,
>>> 	larsi@gnus.org,
>>> 	50951@debbugs.gnu.org
>>> 
>>> Probably the following change will fix the problem:
>>
>> Thanks.  Rah Guzar, can you try one of these changes and see if they
>> fix the problem?
>
> Not sure about Rah Guzar, but it squashes bug#54646 for me.  Thank you
> so very much, Yamamoto Mitsuharu!

Unfortunately, this turned out to be a false alarm.  Before I could
easily reproduce the issue at will but now it is not so easy—the fact
that it took me so long to notice that the bug is still present speaks
so.  I was using patch no. 1 for the past days, I will try patch no. 2
and see if it improves the situation.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Visuwesh]

[ஞாயிறு செப்டம்பர் 11, 2022] Visuwesh wrote:

>> Not sure about Rah Guzar, but it squashes bug#54646 for me.  Thank you
>> so very much, Yamamoto Mitsuharu!
>
> Unfortunately, this turned out to be a false alarm.  Before I could
> easily reproduce the issue at will but now it is not so easy—the fact
> that it took me so long to notice that the bug is still present speaks
> so.  I was using patch no. 1 for the past days, I will try patch no. 2
> and see if it improves the situation.

Patch no. 2 also exhibits the bug.  I tried Rah Guzar's original test
case given in [1] and [2] with the first and the second patch, and
without any patches and I saw no mistakes in glyph placements.  But of
course, I cannot read Urdu so take this with a pound of salt...

1. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50951#11
2. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50951#26

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

I finally tested the patches and both of them improve the situation by a
lot but problems still remain. One word that is not rendered by
accurately by them is

ہنگام

Where is problem is fourth character which is
        character: گ‎ (displayed as گ‎) (codepoint 1711, #o3257, #x6af)
        charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
        code point in charset: 0x06AF
        script: arabic

This character should be rendered as a circle and two slanted lines
which seem to get clipped.

Both the patches seem to have the same effect, except that with the
first one above a threshold I have seen a newline not getting rendered
so what should be two lines appear as one. This doesn't happen with the
second patch.


Rah Guzar

Rah Guzar <rahguzar@zohomail.eu> writes:

> Unfortunately I have very limited experience with building software and
> none at all with applying patches so testing this will be a non-trivial
> undertaking. I will try to do this next weekend.
>
> But what Visuwesh describes seem exactly the symptoms I was experiencing
> so I am pretty positive that it will work for me too. So thank you very
> much indeed!
>
> Rah Guzar
>
> Visuwesh <visuweshm@gmail.com> writes:
>
>> [புதன் செப்டம்பர் 07, 2022] Eli Zaretskii wrote:
>>
>>>> Date: Wed, 07 Sep 2022 15:18:21 +0900
>>>> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
>>>> Cc: Eli Zaretskii <eliz@gnu.org>,
>>>> 	larsi@gnus.org,
>>>> 	50951@debbugs.gnu.org
>>>>
>>>> Probably the following change will fix the problem:
>>>
>>> Thanks.  Rah Guzar, can you try one of these changes and see if they
>>> fix the problem?
>>
>> Not sure about Rah Guzar, but it squashes bug#54646 for me.  Thank you
>> so very much, Yamamoto Mitsuharu!
>>
>> With regards to the performance, my eyes say that patch no. 1 (i.e., the
>> one that patches src/hbfont.c) is slightly faster than patch no. 2 when
>> I continually zoomed to reproduce bug#54646.  The test material was this
>> random webpage https://www.dinamalar.com/news_detail.asp?id=3117719,
>> and M-x report-emacs-bug report on the Emacs with the patch is at the
>> end.
>> The test that I used is the same as I outlined in bug#54646, namely,
>> visit that webpage in eww, zoom in and out with C-scroll-wheel.  The
>> glyphs _do not_ bump into each other or get weirdly placed whereas
>> without the patch, glyphs can be oddly spaced.
>>
>> In GNU Emacs 29.0.50 (build 6, x86_64-pc-linux-gnu, X toolkit, cairo
>>  version 1.16.0, Xaw scroll bars) of 2022-09-08 built on astatine
>> Repository revision: 9219e83b3c0ef53df02caf4c8ba38f482937ab50
>> Repository branch: master
>> Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
>> System Description: Debian GNU/Linux bookworm/sid
>>
>> Configured using:
>>  'configure --with-sound=alsa --with-x-toolkit=lucid --with-json
>>  --without-xaw3d --without-gconf --without-libsystemd'
>>
>> Configured features:
>> ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
>> JSON LIBOTF LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
>> SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM
>> XINPUT2 XPM LUCID ZLIB
>>
>> Important settings:
>>   value of $LC_MONETARY: ta_IN.UTF-8
>>   value of $LC_NUMERIC: ta_IN.UTF-8
>>   value of $LANG: en_GB.UTF-8
>>   locale-coding-system: utf-8-unix

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> From: Rah Guzar <rahguzar@zohomail.eu>
> Cc: Visuwesh <visuweshm@gmail.com>, Eli Zaretskii <eliz@gnu.org>, YAMAMOTO
>  Mitsuharu <mituharu@math.s.chiba-u.ac.jp>, larsi@gnus.org,
>  50951@debbugs.gnu.org
> Date: Sat, 17 Sep 2022 18:37:39 +0200
> 
> 
> I finally tested the patches and both of them improve the situation by a
> lot but problems still remain. One word that is not rendered by
> accurately by them is
> 
> ہنگام
> 
> Where is problem is fourth character which is
>         character: گ‎ (displayed as گ‎) (codepoint 1711, #o3257, #x6af)
>         charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>         code point in charset: 0x06AF
>         script: arabic
> 
> This character should be rendered as a circle and two slanted lines
> which seem to get clipped.
> 
> Both the patches seem to have the same effect, except that with the
> first one above a threshold I have seen a newline not getting rendered
> so what should be two lines appear as one. This doesn't happen with the
> second patch.

Thanks, I've now installed the second patch on the master branch.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

On Sun, 18 Sep 2022 01:37:39 +0900,
Rah Guzar wrote:
> 
> 
> I finally tested the patches and both of them improve the situation by a
> lot but problems still remain. One word that is not rendered by
> accurately by them is
> 
> ہنگام
> 
> Where is problem is fourth character which is
>         character: گ‎ (displayed as گ‎) (codepoint 1711, #o3257, #x6af)
>         charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>         code point in charset: 0x06AF
>         script: arabic
> 
> This character should be rendered as a circle and two slanted lines
> which seem to get clipped.

Thanks for testing.

The width of grapheme cluster corresponding to U+06AF (ARABIC LETTER
GAF) is rounded to zero, and Emacs does not display such clusters:

xdisp.c:
 32424	      gstring = composition_gstring_from_id (it->cmp_it.id);
 32425	      it->pixel_width
 32426		= composition_gstring_width (gstring, it->cmp_it.from, it->cmp_it.to,
 32427					     &metrics);
 32428	      if (it->pixel_width == 0)
 32429		{
 32430		  it->glyph_not_available_p = true;
 32431		  it->phys_ascent = it->ascent;
 32432		  it->phys_descent = it->descent;
 32433		  it->pixel_width = face->font->space_width;
 32434		}
 32435	      else

The attached patch avoids zero-width grapheme clusters by adding 1 to
the width of the last glyph in such clusters.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

[-- Attachment #2: avoid-zero-width-grapheme-clusters.diff --]
[-- Type: application/octet-stream, Size: 2464 bytes --]

diff --git a/src/composite.c b/src/composite.c
index 249d7587f6..5e856fd5e5 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -800,6 +800,48 @@ composition_gstring_width (Lisp_Object gstring, ptrdiff_t from, ptrdiff_t to,
   return width;
 }
 
+/* Adjust the width of each grapheme cluster of GSTRING because
+   zero-width grapheme clusters are not displayed.  If the width is
+   zero, then the width of the last glyph in the cluster is
+   incremented.  */
+
+void
+composition_gstring_adjust_zero_width (Lisp_Object gstring)
+{
+  ptrdiff_t from = 0;
+  int width = 0;
+
+  for (ptrdiff_t i = 0; i < LGSTRING_GLYPH_LEN (gstring); i++)
+    {
+      Lisp_Object glyph = LGSTRING_GLYPH (gstring, i);
+
+      if (NILP (glyph) || from != LGLYPH_FROM (glyph))
+	{
+	  eassert (i > 0);
+	  Lisp_Object last = LGSTRING_GLYPH (gstring, i - 1);
+
+	  if (width == 0)
+	    {
+	      if (NILP (LGLYPH_ADJUSTMENT (last)))
+		LGLYPH_SET_ADJUSTMENT (last,
+				       CALLN (Fvector,
+					      make_fixnum (0), make_fixnum (0),
+					      make_fixnum (LGLYPH_WIDTH (last)
+							   + 1)));
+	      else
+		ASET (LGLYPH_ADJUSTMENT (last), 2,
+		      make_fixnum (LGLYPH_WADJUST (last) + 1));
+	    }
+	  if (NILP (glyph))
+	    break;
+	  from = LGLYPH_FROM (glyph);
+	  width = 0;
+	}
+      width += (NILP (LGLYPH_ADJUSTMENT (glyph))
+		? LGLYPH_WIDTH (glyph) : LGLYPH_WADJUST (glyph));
+    }
+}
+
 
 static Lisp_Object gstring_work;
 static Lisp_Object gstring_work_headers;
diff --git a/src/composite.h b/src/composite.h
index d77dd0d506..8a6fd203d0 100644
--- a/src/composite.h
+++ b/src/composite.h
@@ -340,6 +340,7 @@ #define LGLYPH_WADJUST(g) (VECTORP (LGLYPH_ADJUSTMENT (g)) \
 extern bool composition_gstring_p (Lisp_Object);
 extern int composition_gstring_width (Lisp_Object, ptrdiff_t, ptrdiff_t,
                                       struct font_metrics *);
+extern void composition_gstring_adjust_zero_width (Lisp_Object);
 
 extern bool find_automatic_composition (ptrdiff_t, ptrdiff_t, ptrdiff_t,
 					ptrdiff_t *, ptrdiff_t *,
diff --git a/src/font.c b/src/font.c
index 413cb381ee..defbb5084b 100644
--- a/src/font.c
+++ b/src/font.c
@@ -4678,6 +4678,7 @@ DEFUN ("font-shape-gstring", Ffont_shape_gstring, Sfont_shape_gstring, 2, 2, 0,
       from = LGLYPH_FROM (glyph);
       to = LGLYPH_TO (glyph);
     }
+  composition_gstring_adjust_zero_width (gstring);
   return composition_gstring_put_cache (gstring, XFIXNUM (n));
 
  shaper_error:

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> Date: Tue, 20 Sep 2022 12:41:40 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: Visuwesh <visuweshm@gmail.com>,
> 	Eli Zaretskii <eliz@gnu.org>,
> 	larsi@gnus.org,
> 	50951@debbugs.gnu.org
> 
> The width of grapheme cluster corresponding to U+06AF (ARABIC LETTER
> GAF) is rounded to zero, and Emacs does not display such clusters:
> 
> xdisp.c:
>  32424	      gstring = composition_gstring_from_id (it->cmp_it.id);
>  32425	      it->pixel_width
>  32426		= composition_gstring_width (gstring, it->cmp_it.from, it->cmp_it.to,
>  32427					     &metrics);
>  32428	      if (it->pixel_width == 0)
>  32429		{
>  32430		  it->glyph_not_available_p = true;
>  32431		  it->phys_ascent = it->ascent;
>  32432		  it->phys_descent = it->descent;
>  32433		  it->pixel_width = face->font->space_width;
>  32434		}
>  32435	      else
> 
> The attached patch avoids zero-width grapheme clusters by adding 1 to
> the width of the last glyph in such clusters.

If the problem is rounding, I think we should do this adjustment only
when the last glyph has a non-zero width that was rounded to zero, no?
Otherwise, we are inventing adjustments out of thin air, which could
adversely affect the displayed result, I think?

Or maybe we should have a variable that controls this heuristic?

Bottom line: I'm uneasy with messing with the grapheme cluster data
without some sound basis.  We delegate this job to a text-shaping
engine for a reason.  But if there is a sound basis for this
adjustment, could you please elaborate on it?

Thanks.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Rah Guzar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi,
  Thank you so much for this patch. Rendering is perfect in my cursory
  tests after applying this patch.

One last problem which is unrelated to this patch: the beginning of some
lines is often clipped. For example for me the word,
تنہائی
is shown clipped and is visible from the end of the second of two dots
at the top if  it is at the beginning of the line. This seems to be the
case even for fonts which don't misbehave otherwise. My solution has
been to set `bidi-paragraph-direction` to `left-to-right` but that is
not ideal.

Thanks a lot again,
Rah Guzar

YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

> On Sun, 18 Sep 2022 01:37:39 +0900,
> Rah Guzar wrote:
>>
>>
>> I finally tested the patches and both of them improve the situation by a
>> lot but problems still remain. One word that is not rendered by
>> accurately by them is
>>
>> ہنگام
>>
>> Where is problem is fourth character which is
>>         character: گ‎ (displayed as گ‎) (codepoint 1711, #o3257, #x6af)
>>         charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>>         code point in charset: 0x06AF
>>         script: arabic
>>
>> This character should be rendered as a circle and two slanted lines
>> which seem to get clipped.
>
> Thanks for testing.
>
> The width of grapheme cluster corresponding to U+06AF (ARABIC LETTER
> GAF) is rounded to zero, and Emacs does not display such clusters:
>
> xdisp.c:
>  32424	      gstring = composition_gstring_from_id (it->cmp_it.id);
>  32425	      it->pixel_width
>  32426		= composition_gstring_width (gstring, it->cmp_it.from, it->cmp_it.to,
>  32427					     &metrics);
>  32428	      if (it->pixel_width == 0)
>  32429		{
>  32430		  it->glyph_not_available_p = true;
>  32431		  it->phys_ascent = it->ascent;
>  32432		  it->phys_descent = it->descent;
>  32433		  it->pixel_width = face->font->space_width;
>  32434		}
>  32435	      else
>
> The attached patch avoids zero-width grapheme clusters by adding 1 to
> the width of the last glyph in such clusters.
>
> 				     YAMAMOTO Mitsuharu
> 				mituharu@math.s.chiba-u.ac.jp
>
> [2. text/x-patch; avoid-zero-width-grapheme-clusters.diff]...

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

[-- Attachment #1: Type: text/plain, Size: 2453 bytes --]

On Tue, 20 Sep 2022 20:07:12 +0900,
Eli Zaretskii wrote:
> 
> > Date: Tue, 20 Sep 2022 12:41:40 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: Visuwesh <visuweshm@gmail.com>,
> > 	Eli Zaretskii <eliz@gnu.org>,
> > 	larsi@gnus.org,
> > 	50951@debbugs.gnu.org
> > 
> > The width of grapheme cluster corresponding to U+06AF (ARABIC LETTER
> > GAF) is rounded to zero, and Emacs does not display such clusters:
> > 
> > xdisp.c:
> >  32424	      gstring = composition_gstring_from_id (it->cmp_it.id);
> >  32425	      it->pixel_width
> >  32426		= composition_gstring_width (gstring, it->cmp_it.from, it->cmp_it.to,
> >  32427					     &metrics);
> >  32428	      if (it->pixel_width == 0)
> >  32429		{
> >  32430		  it->glyph_not_available_p = true;
> >  32431		  it->phys_ascent = it->ascent;
> >  32432		  it->phys_descent = it->descent;
> >  32433		  it->pixel_width = face->font->space_width;
> >  32434		}
> >  32435	      else
> > 
> > The attached patch avoids zero-width grapheme clusters by adding 1 to
> > the width of the last glyph in such clusters.
> 
> If the problem is rounding, I think we should do this adjustment only
> when the last glyph has a non-zero width that was rounded to zero, no?
> Otherwise, we are inventing adjustments out of thin air, which could
> adversely affect the displayed result, I think?
> 
> Or maybe we should have a variable that controls this heuristic?
> 
> Bottom line: I'm uneasy with messing with the grapheme cluster data
> without some sound basis.  We delegate this job to a text-shaping
> engine for a reason.  But if there is a sound basis for this
> adjustment, could you please elaborate on it?
> 
> Thanks.

IIUC, the only "unsound" case is that the width of a grapheme cluster
is exactly 0 before rounding.  I think such a case is quite rare.  And
even for such a case, Emacs needs to put at least extra 1 pixel to
move the cursor to the position of the grapheme cluster.  So the
adjustment made by the patch is minimum and necessary.

The current (unpatched) master may put multiple pixels (space width of
the font as in Line 32433 above), and moreover the corresponding
glyphs are not displayed.  If we keep this behavior for the "unsound"
case, the result would be much more apart from the optimal.

(The attachment is for comparison between unpatched, patched, and the
output of hb-view (optimal).)

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp


[-- Attachment #2: unpatched.pdf --]
[-- Type: application/pdf, Size: 8073 bytes --]

[-- Attachment #3: patched.pdf --]
[-- Type: application/pdf, Size: 8307 bytes --]

[-- Attachment #4: hb-view.pdf --]
[-- Type: application/pdf, Size: 3467 bytes --]

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

On Wed, 21 Sep 2022 11:20:54 +0900,
YAMAMOTO Mitsuharu wrote:
> 
> (The attachment is for comparison between unpatched, patched, and the
> output of hb-view (optimal).)

Sorry, the unpatched one was wrong.  Below is the correct one.
 
				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

[-- Attachment #2: unpatched.pdf --]
[-- Type: application/pdf, Size: 8053 bytes --]

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> Date: Wed, 21 Sep 2022 11:20:54 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: rahguzar@zohomail.eu,
> 	visuweshm@gmail.com,
> 	larsi@gnus.org,
> 	50951@debbugs.gnu.org
> 
> > If the problem is rounding, I think we should do this adjustment only
> > when the last glyph has a non-zero width that was rounded to zero, no?
> > Otherwise, we are inventing adjustments out of thin air, which could
> > adversely affect the displayed result, I think?
> > 
> > Or maybe we should have a variable that controls this heuristic?
> > 
> > Bottom line: I'm uneasy with messing with the grapheme cluster data
> > without some sound basis.  We delegate this job to a text-shaping
> > engine for a reason.  But if there is a sound basis for this
> > adjustment, could you please elaborate on it?
> > 
> > Thanks.
> 
> IIUC, the only "unsound" case is that the width of a grapheme cluster
> is exactly 0 before rounding.  I think such a case is quite rare.  And
> even for such a case, Emacs needs to put at least extra 1 pixel to
> move the cursor to the position of the grapheme cluster.  So the
> adjustment made by the patch is minimum and necessary.
> 
> The current (unpatched) master may put multiple pixels (space width of
> the font as in Line 32433 above), and moreover the corresponding
> glyphs are not displayed.  If we keep this behavior for the "unsound"
> case, the result would be much more apart from the optimal.

Can you please point me to the place(s) in our code where this
rounding takes place?

Also, I asked whether you could elaborate on the rationale for
adjusting the zero width to be 1 pixel, and I don't think you answered
that particular question.  What you are saying (AFAIU) is that
heuristically the results of using this adjustment are better, at
least in this case.  I don't argue with that, but I wonder whether
there's some rationale for this that isn't just heuristics?  IOW, do
you know how come hb-view doesn't have this problem? what do we do
that produces the zero width where hb-view doesn't?

Thanks.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

On Thu, 22 Sep 2022 14:37:24 +0900,
Eli Zaretskii wrote:
> 
> > Date: Wed, 21 Sep 2022 11:20:54 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: rahguzar@zohomail.eu,
> > 	visuweshm@gmail.com,
> > 	larsi@gnus.org,
> > 	50951@debbugs.gnu.org
> > 
> > > If the problem is rounding, I think we should do this adjustment only
> > > when the last glyph has a non-zero width that was rounded to zero, no?
> > > Otherwise, we are inventing adjustments out of thin air, which could
> > > adversely affect the displayed result, I think?
> > > 
> > > Or maybe we should have a variable that controls this heuristic?
> > > 
> > > Bottom line: I'm uneasy with messing with the grapheme cluster data
> > > without some sound basis.  We delegate this job to a text-shaping
> > > engine for a reason.  But if there is a sound basis for this
> > > adjustment, could you please elaborate on it?
> > > 
> > > Thanks.
> > 
> > IIUC, the only "unsound" case is that the width of a grapheme cluster
> > is exactly 0 before rounding.  I think such a case is quite rare.  And
> > even for such a case, Emacs needs to put at least extra 1 pixel to
> > move the cursor to the position of the grapheme cluster.  So the
> > adjustment made by the patch is minimum and necessary.
> > 
> > The current (unpatched) master may put multiple pixels (space width of
> > the font as in Line 32433 above), and moreover the corresponding
> > glyphs are not displayed.  If we keep this behavior for the "unsound"
> > case, the result would be much more apart from the optimal.
> 
> Can you please point me to the place(s) in our code where this
> rounding takes place?

For the HarfBuzz shaper, the width rounding happens at Line 595
directly, and at the callee of Line 586 indirectly:

  hbfont.c:
   585	      unsigned code = info[i].codepoint;
   586	      font->driver->text_extents (font, &code, 1, &metrics);
   587	      LGLYPH_SET_WIDTH (lglyph, metrics.width);
   588	      LGLYPH_SET_LBEARING (lglyph, metrics.lbearing);
   589	      LGLYPH_SET_RBEARING (lglyph, metrics.rbearing);
   590	      LGLYPH_SET_ASCENT (lglyph, metrics.ascent);
   591	      LGLYPH_SET_DESCENT (lglyph, metrics.descent);
   592	
   593	      xoff = lround (pos[i].x_offset * position_unit);
   594	      yoff = - lround (pos[i].y_offset * position_unit);
   595	      wadjust = lround (pos[i].x_advance * position_unit);

The value of position_unit is usually 1.0 / 32.

For the callee of Line 586, rounding may happen either at the Emacs
side as in the ftcrhb font backend,

  ftcrfont.c:
    99	      cairo_scaled_font_glyph_extents (ftcrfont_info->cr_scaled_font,
   100					       &cr_glyph, 1, &extents);
   101	      cache->lbearing = floor (extents.x_bearing);
   102	      cache->rbearing = ceil (extents.width + extents.x_bearing);
   103	      cache->width = lround (extents.x_advance);

or at the library side as in the xfthb font backend.

  xftfont.c:
   469	  block_input ();
   470	  XftGlyphExtents (xftfont_info->display, xftfont_info->xftfont, code, nglyphs,
   471			   &extents);
   472	  unblock_input ();
   473	
   474	  metrics->lbearing = - extents.x;
   475	  metrics->rbearing = - extents.x + extents.width;
   476	  metrics->width = extents.xOff;

For the Uniscribe shaper, rounding seems to happen at the library
side:

  w32uniscribe.c:
   297	  int *advances;
    :
   346	  advances = alloca (max_glyphs * sizeof (int));
    :
   399		  result = ScriptPlace (context, (SCRIPT_CACHE) &(uniscribe_font->cache),
   400					glyphs, nglyphs, attributes, &(items[i].a),
   401					advances, offsets, &overall_metrics);
    :
   501			  LGLYPH_SET_WIDTH (lglyph, advances[j]);
    :
   563			      ASET (vec, 2, make_fixnum (advances[j]));
   564			      LGLYPH_SET_ADJUSTMENT (lglyph, vec);

If rounding happens at the library side, we don't know whether the
width before rounding was exactly 0 or not.

> Also, I asked whether you could elaborate on the rationale for
> adjusting the zero width to be 1 pixel, and I don't think you
> answered that particular question.  What you are saying (AFAIU) is
> that heuristically the results of using this adjustment are better,
> at least in this case.  I don't argue with that, but I wonder
> whether there's some rationale for this that isn't just heuristics?
> IOW, do you know how come hb-view doesn't have this problem? what do
> we do that produces the zero width where hb-view doesn't?

The output of hb-view was in PDF, and its coordinate system does not
directly correspond to the integral number of physical pixels unlike
in Emacs.

The display engine of Emacs only accepts positive integer as
pixel-width of a glyph (in Emacs terminology).  If the actual grapheme
cluster has width zero (after rounding), then it is replaced to some
positive integer (space width) in gui_produce_glyphs.  Because some
grapheme cluster in the result of shaping can be in very small width
and rounded to 0, adjusting it to 1 is almost the best approximation.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#50951: 28.0.50; Urdu text is not displayed correctly

[Eli Zaretskii]

> Date: Sun, 25 Sep 2022 16:18:26 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: rahguzar@zohomail.eu, visuweshm@gmail.com, larsi@gnus.org,
>   50951@debbugs.gnu.org
> 
> > Also, I asked whether you could elaborate on the rationale for
> > adjusting the zero width to be 1 pixel, and I don't think you
> > answered that particular question.  What you are saying (AFAIU) is
> > that heuristically the results of using this adjustment are better,
> > at least in this case.  I don't argue with that, but I wonder
> > whether there's some rationale for this that isn't just heuristics?
> > IOW, do you know how come hb-view doesn't have this problem? what do
> > we do that produces the zero width where hb-view doesn't?
> 
> The output of hb-view was in PDF, and its coordinate system does not
> directly correspond to the integral number of physical pixels unlike
> in Emacs.
> 
> The display engine of Emacs only accepts positive integer as
> pixel-width of a glyph (in Emacs terminology).  If the actual grapheme
> cluster has width zero (after rounding), then it is replaced to some
> positive integer (space width) in gui_produce_glyphs.  Because some
> grapheme cluster in the result of shaping can be in very small width
> and rounded to 0, adjusting it to 1 is almost the best approximation.

OK, thanks.  Please install your patch on the master branch.

bug#50951: 28.0.50; Urdu text is not displayed correctly

[YAMAMOTO Mitsuharu]

On Mon, 26 Sep 2022 16:18:50 +0900,
Eli Zaretskii wrote:
> 
> > Date: Sun, 25 Sep 2022 16:18:26 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: rahguzar@zohomail.eu, visuweshm@gmail.com, larsi@gnus.org,
> >   50951@debbugs.gnu.org
> > 
> > > Also, I asked whether you could elaborate on the rationale for
> > > adjusting the zero width to be 1 pixel, and I don't think you
> > > answered that particular question.  What you are saying (AFAIU) is
> > > that heuristically the results of using this adjustment are better,
> > > at least in this case.  I don't argue with that, but I wonder
> > > whether there's some rationale for this that isn't just heuristics?
> > > IOW, do you know how come hb-view doesn't have this problem? what do
> > > we do that produces the zero width where hb-view doesn't?
> > 
> > The output of hb-view was in PDF, and its coordinate system does not
> > directly correspond to the integral number of physical pixels unlike
> > in Emacs.
> > 
> > The display engine of Emacs only accepts positive integer as
> > pixel-width of a glyph (in Emacs terminology).  If the actual grapheme
> > cluster has width zero (after rounding), then it is replaced to some
> > positive integer (space width) in gui_produce_glyphs.  Because some
> > grapheme cluster in the result of shaping can be in very small width
> > and rounded to 0, adjusting it to 1 is almost the best approximation.
> 
> OK, thanks.  Please install your patch on the master branch.

I installed a slightly modified version because the previous one did
not adjust the last grapheme cluster when its width is zero.  Closing.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp