bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 102 bytes --]

The ZERO WIDTH NON-JOINER character (U+200C) is inserted, but displayed as
a regular space character.

[-- Attachment #2: Type: text/html, Size: 184 bytes --]

bug#54325: my emacs-bug-info.

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 3691 bytes --]

In GNU Emacs 28.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.32,
cairo version 1.17.4)
 of 2022-03-09 built on codeagers
Repository revision: d184773c2e2a69bea9b96190c83727b4e7a16542
Repository branch: makepkg
Windowing system distributor 'The X.Org Foundation', version 11.0.12201000
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-sound=alsa
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --without-gsettings --with-native-compilation --with-x-toolkit=gtk3
 --without-xaw3d --with-sound=alsa --without-gpm
 --without-compress-install
 '--program-transform-name=s/^ctags$/ctags.emacs/' 'CFLAGS=-march=x86-64
 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2
 -Wformat -Werror=format-security -fstack-clash-protection
 -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ JPEG JSON LCMS2
LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3
ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  fido-vertical-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  fido-mode: t
  text-scale-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cus-start
cus-load icomplete cl-loaddefs cl-lib face-remap iso-transl tooltip
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting font-render-setting cairo move-toolbar gtk
x-toolkit x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 79590 4955)
 (symbols 48 7677 0)
 (strings 32 21241 1575)
 (string-bytes 1 700971)
 (vectors 16 14350)
 (vector-slots 8 303357 16178)
 (floats 8 51 561)
 (intervals 56 592 1)
 (buffers 992 11))

[-- Attachment #2: Type: text/html, Size: 4129 bytes --]

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Thu, 10 Mar 2022 05:58:08 +0330
> 
> The ZERO WIDTH NON-JOINER character (U+200C) is inserted, but displayed as a regular space
> character.

Thanks, but please tell more.  What was the surrounding text in this
case, and why did you think what Emacs does was not correct in that
situation?

The U+200C character has several behaviors defined by the Unicode
Standard, and those behaviors depend on the characters surrounding
U+200C, so it is important to know the exact details of the case you
are talking about.

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

[Please use Reply All so that the bug tracker records this discussion.]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Thu, 10 Mar 2022 20:02:55 +0330
> 
> This character is not displayed correctly in Persian texts. (Font change does not make a difference) But it is
> not a problem in English texts and is displayed correctly. In Emacs 28 and 29, the length of this character in
> Persian texts is exactly the same as the space character.

Please also post the text you used, as plain text, not just as image.
It is important for investigating the issue to know exactly what you
typed.

Also, does this happen in "emacs -Q"?

> I have attached two screenshots to this email. Which indicates that these characters are displayed correctly
> in Emacs 27, but in Emacs 28 and 29 it is as long as the Space character.

Since you only wrote to me in private email, I attach those two images
below.


[-- Attachment #2: emacs 27.png --]
[-- Type: image/png, Size: 24439 bytes --]

[-- Attachment #3: image30523.png --]
[-- Type: image/png, Size: 21565 bytes --]

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[Eli Zaretskii]

[PLEASE keep the bug address on the CC list.]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Thu, 10 Mar 2022 20:53:49 +0330
> 
> > Also, does this happen in "emacs -Q"?
> Yes, it happens ...
> 
> این‌ها
> 
> ماه‌ها
> 
> رفته‌ای

OK, I see it now, thanks.

I think what you want is to customize the variable
arabic-shaper-ZWNJ-handling to the value 'as-space or maybe even
'absorb.  Then the display will be as you expect.

FTR: Emacs 27 had a bug in the character composition/shaping rules for
the Arabic script, and on my system the text you sent is displayed in
Emacs 27 with glitches (when I move the cursor across the ZWNJ
character, it leaves artifacts behind).  Emacs 28 fixes that problem,
and that's why you see the difference.  By default, Emacs isn't
supposed to display ZWNJ specially in Arabic shaping, but the above
variable can change that, so I suggest you use it.

Bottom line: I don't think there's a bug here.  You just don't like
the default behavior, but an optional behavior like you want is
available.

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]

You are right. This variable fixed everything.

Thank you very much for your attention.

On Thu, Mar 10, 2022 at 9:18 PM Eli Zaretskii <eliz@gnu.org> wrote:

> [PLEASE keep the bug address on the CC list.]
>
> > From: hossein valizadeh <valizadeh.ho@gmail.com>
> > Date: Thu, 10 Mar 2022 20:53:49 +0330
> >
> > > Also, does this happen in "emacs -Q"?
> > Yes, it happens ...
> >
> > این‌ها
> >
> > ماه‌ها
> >
> > رفته‌ای
>
> OK, I see it now, thanks.
>
> I think what you want is to customize the variable
> arabic-shaper-ZWNJ-handling to the value 'as-space or maybe even
> 'absorb.  Then the display will be as you expect.
>
> FTR: Emacs 27 had a bug in the character composition/shaping rules for
> the Arabic script, and on my system the text you sent is displayed in
> Emacs 27 with glitches (when I move the cursor across the ZWNJ
> character, it leaves artifacts behind).  Emacs 28 fixes that problem,
> and that's why you see the difference.  By default, Emacs isn't
> supposed to display ZWNJ specially in Arabic shaping, but the above
> variable can change that, so I suggest you use it.
>
> Bottom line: I don't think there's a bug here.  You just don't like
> the default behavior, but an optional behavior like you want is
> available.
>

[-- Attachment #2: Type: text/html, Size: 1856 bytes --]

bug#54325: Emacs Shows the ZWNJ character (U+200C) as a space character

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Thu, 10 Mar 2022 21:26:27 +0330
> Cc: 54325@debbugs.gnu.org
> 
> You are right. This variable fixed everything.
> 
> Thank you very much for your attention.

Thanks, I'm therefore closing this bug.