bug#54219: 26.3; Bidi and empty parentheses

bug#54219: 26.3; Bidi and empty parentheses

[Richard Wordingham via Bug reports for GNU Emacs, the Swiss army knife of text editors]

--Recipe starting from emacs-Q:--

; Create a new buffer for general text, e.g. c.txt
C-x C-f c.txt RET
; Enter RTL character
insert-char 644 RET
; Enter rest of line
x()y

--Problem--
The result is that a flush-right line is produced that appears to read

y()xل

It should appear to read

x()yل

See https://corp.unicode.org/pipermail/unicode/2022-February/009984.html
and reply from Ken Whistler for confirmation that the observed
behaviour is wrong.

This misbehaviour is still present in Emacs 28.0.91 as built from the
tarball.  (Configuration not included.)

For the usage that prompted this report, several work-arounds are
available, including defining end of line as a paragraph boundary.
The Emacs display in that usage is also rendered baffling by Bug #27525
'Line wrapping of bidi paragraphs'.

--Configuration Data--

In GNU Emacs 26.3 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.14)
 of 2020-03-26, modified by Debian built on lcy01-amd64-020
Windowing system distributor 'The X.Org Foundation', version
11.0.12013000 System Description:	Ubuntu 20.04.4 LTS

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/26.3/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/26.3/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/26.3/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/26.3/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-mEZBk7/emacs-26.3+1=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD LCMS2

Important settings:
  value of $LANG: en_GB.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils map seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib elec-pair time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 95749 11279)
 (symbols 48 31997 2)
 (miscs 40 76 118)
 (strings 32 84051 2222)
 (string-bytes 1 1876297)
 (vectors 16 14442)
 (vector-slots 8 898152 15448)
 (floats 8 49 68)
 (intervals 56 262 0)
 (buffers 992 12))

bug#54219: 26.3; Bidi and empty parentheses

[Eli Zaretskii]

> Date: Wed, 2 Mar 2022 05:26:37 +0000
> From:  Richard Wordingham via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> ; Create a new buffer for general text, e.g. c.txt
> C-x C-f c.txt RET
> ; Enter RTL character
> insert-char 644 RET
> ; Enter rest of line
> x()y
> 
> --Problem--
> The result is that a flush-right line is produced that appears to read
> 
> y()xل
> 
> It should appear to read
> 
> x()yل

Thanks, fixed for the upcoming Emacs 28.1.

> See https://corp.unicode.org/pipermail/unicode/2022-February/009984.html
> and reply from Ken Whistler for confirmation that the observed
> behaviour is wrong.

Unfortunately, that reply didn't really help, because it didn't
analyze the BPA details, which are crucial in this case.  (If you set
bidi-inhibit-bpa non-nil, the problem goes away.)  It just provided
the final output from Ken's program, which (as any other program) can
have bugs.

> For the usage that prompted this report, several work-arounds are
> available, including defining end of line as a paragraph boundary.
> The Emacs display in that usage is also rendered baffling by Bug #27525
> 'Line wrapping of bidi paragraphs'.

Not sure what paragraph boundaries have to do with this.  Sounds like
an unrelated issue.

bug#54219: 26.3; Bidi and empty parentheses

[Richard Wordingham via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Thu, 03 Mar 2022 15:59:32 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> > For the usage that prompted this report, several work-arounds are
> > available, including defining end of line as a paragraph boundary.
> > The Emacs display in that usage is also rendered baffling by Bug
> > #27525 'Line wrapping of bidi paragraphs'.  
> 
> Not sure what paragraph boundaries have to do with this.  Sounds like
> an unrelated issue.

It relates to how the issue came to light.  Program output included
sequences composed of blank line, line of Arabic and then a sentence
starting off in ASCII and containing an empty pair of parentheses.  The
point was that fixing the error was not urgent; there were other ways
of fixing the problem.  The Emacs' default paragraphing rules are
simply wrong for the application.

Richard.