bug#41005: problem with rendering Persian text in Emacs 27

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1.1: Type: text/plain, Size: 982 bytes --]

Hello,

I have been using GNU Emacs for a few years now, and I recently decided
to build Emacs 27.0.91 from the emacs-27 branch.  I use Arch GNU/Linux,
along with the i3 window manager.

I have noticed a new problem with rendering Persian text in Emacs 27,
which to my knowledge was not present in earlier Emacs versions.  If you
see the screenshots I have attached, some text seems to get garbled at
random.  The same words sometimes appear correctly and sometimes do not.
I recall John Wiegley mentioning in his EmacsConf 2019 talk that Emacs
has recently started using the HarfBuzz text shaping engine.  I wonder
if that's relevant and possibly the cause of this issue?

Thanks in advance for looking into this.  I really hope the issue can be
resolved, as I use Emacs and Org extensively for writing my documents,
and this bug effectively renders Emacs unusable for me.  I'll be happy
to try and provide any further information needed to debug this.

Regards,
Hossein Valizadeh

[-- Attachment #1.2: Type: text/html, Size: 1175 bytes --]

[-- Attachment #2: scr-1.png --]
[-- Type: image/png, Size: 62390 bytes --]

[-- Attachment #3: latest-screenshot.png --]
[-- Type: image/png, Size: 404309 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Fri, 1 May 2020 22:32:31 +0430
> 
> I have noticed a new problem with rendering Persian text in Emacs 27,
> which to my knowledge was not present in earlier Emacs versions.  If you
> see the screenshots I have attached, some text seems to get garbled at
> random.  The same words sometimes appear correctly and sometimes do not.
> I recall John Wiegley mentioning in his EmacsConf 2019 talk that Emacs
> has recently started using the HarfBuzz text shaping engine.  I wonder
> if that's relevant and possibly the cause of this issue?

I don't know.  Was your Emacs built with HarfBuzz?  You didn't attach
the data collected by "M-x report-emacs-bug", so I cannot know.

I suggest to start by attaching a sample file that exhibits the
problem, as text, not as an image, and then showing what it looked
like in previous versions of Emacs and what it looks like in Emacs
27.0.91 on your system.  Also, please tell which font was in use when
the problematic display was produced.  Then we can look into this
issue and see what could have caused it.

Thanks.

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

[Please keep the bug address on the CC list; use "Reply All".]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Wed, 3 Jun 2020 09:45:09 +0430
> 
> The Emacs I'm using is compiled by HarfBuzz. I also attached a video file containing information about this
> problem for a better understanding.
> 
> I checked my config file, line by line in different modes and in combination with different commands.
> I think it gets garbled in on of the following 2 situations:
> 
> 1. the auto-fill-mode is on
> 2. column-number-mode is on (or while I'm using %c modifier in custom mode line)
> 
> # These two seem to be related, As auto-fill-mode calculates the number of character in each line too.
> # I tried these in raw and un-configured Emacs; the results were the same and character got garbled.
> # The using font makes no different.(In raw Emacs I used "DejaVu Sans Mono" font )
> # Scale the buffer may help to show the word correctly but other words are still garbled.
> # It mostly happens when I'm trying to add something to existing line. (and sometimes it just normally
> happen)
> # I have deactivated these two options on my config file and Emacs worked properly since then.
> 
> *** I noticed another bug :
> 
> when I changed the input language to Persian(or any other language) by set-input-method command and
> scale the text with "C-x +", When I start typing, the first character in buffer ignores the input-method and an
> English character appears in buffer; But it won't happen for the next characters.

I asked you to please provide a test file with text that you think
looks incorrectly on display, and images that show how it looks in
Emacs 27 and in Emacs 26:

>  I suggest to start by attaching a sample file that exhibits the
>  problem, as text, not as an image, and then showing what it looked
>  like in previous versions of Emacs and what it looks like in Emacs
>  27.0.91 on your system.  Also, please tell which font was in use when
>  the problematic display was produced.  Then we can look into this
>  issue and see what could have caused it.

Please provide that stuff, it is important to make progress with this
bug report.  TIA.

bug#41005: problem with rendering Persian text in Emacs 27

[Nicholas Drozd]

Not certain, but this sounds like it could be related to bug#37683
[1]. That report was closed because it does not seem to be an EWW
problem, but the issue still shows up for me.

To recap, the problem there is that Arabic script letters, which are
typically joined together in a cursive style, show up separated and
unconnected, if and only if text scaling is at zero. If the text is
made larger or smaller, the text becomes properly joined. This leads
me to suspect that it is not a Harfbuzz issue, because the text is
displayed correctly under some conditions but not under others.

[1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-10/msg00921.html

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Nicholas Drozd <nicholasdrozd@gmail.com>
> Date: Wed, 3 Jun 2020 12:24:30 -0500
> 
> Not certain, but this sounds like it could be related to bug#37683
> [1]. That report was closed because it does not seem to be an EWW
> problem, but the issue still shows up for me.
> 
> To recap, the problem there is that Arabic script letters, which are
> typically joined together in a cursive style, show up separated and
> unconnected, if and only if text scaling is at zero. If the text is
> made larger or smaller, the text becomes properly joined. This leads
> me to suspect that it is not a Harfbuzz issue, because the text is
> displayed correctly under some conditions but not under others.

It happens to you with any font that supports Arabic, or just with
some?

And if you change the scale, then change it back to zero, does the
problem happen again, or does it only happen when you look at some
text for the first time?

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

Hello,

1. This happen with every font that supports Arabic.
2. When scale back to zero the problem happen again.

On Wed, Jun 3, 2020 at 10:32 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Nicholas Drozd <nicholasdrozd@gmail.com>
> > Date: Wed, 3 Jun 2020 12:24:30 -0500
> >
> > Not certain, but this sounds like it could be related to bug#37683
> > [1]. That report was closed because it does not seem to be an EWW
> > problem, but the issue still shows up for me.
> >
> > To recap, the problem there is that Arabic script letters, which are
> > typically joined together in a cursive style, show up separated and
> > unconnected, if and only if text scaling is at zero. If the text is
> > made larger or smaller, the text becomes properly joined. This leads
> > me to suspect that it is not a Harfbuzz issue, because the text is
> > displayed correctly under some conditions but not under others.
>
> It happens to you with any font that supports Arabic, or just with
> some?
>
> And if you change the scale, then change it back to zero, does the
> problem happen again, or does it only happen when you look at some
> text for the first time?
>

[-- Attachment #2: Type: text/html, Size: 1821 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1.1: Type: text/plain, Size: 1415 bytes --]

Hello,

1. This happen with every font that supports Arabic.
2. When scale back to zero the problem happen again.

I attached my emacs-bug-info.


On Thu, Jun 4, 2020 at 7:09 AM hossein valizadeh <valizadeh.ho@gmail.com>
wrote:

> Hello,
>
> 1. This happen with every font that supports Arabic.
> 2. When scale back to zero the problem happen again.
>
> On Wed, Jun 3, 2020 at 10:32 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Nicholas Drozd <nicholasdrozd@gmail.com>
>> > Date: Wed, 3 Jun 2020 12:24:30 -0500
>> >
>> > Not certain, but this sounds like it could be related to bug#37683
>> > [1]. That report was closed because it does not seem to be an EWW
>> > problem, but the issue still shows up for me.
>> >
>> > To recap, the problem there is that Arabic script letters, which are
>> > typically joined together in a cursive style, show up separated and
>> > unconnected, if and only if text scaling is at zero. If the text is
>> > made larger or smaller, the text becomes properly joined. This leads
>> > me to suspect that it is not a Harfbuzz issue, because the text is
>> > displayed correctly under some conditions but not under others.
>>
>> It happens to you with any font that supports Arabic, or just with
>> some?
>>
>> And if you change the scale, then change it back to zero, does the
>> problem happen again, or does it only happen when you look at some
>> text for the first time?
>>
>

[-- Attachment #1.2: Type: text/html, Size: 2642 bytes --]

[-- Attachment #2: Emacs-bug-info.txt --]
[-- Type: text/plain, Size: 18116 bytes --]

In GNU Emacs 27.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.17, cairo version 1.17.3)
 of 2020-04-25 built on Codeager
Repository revision: a76af88dd872091f78bb1a4716750934f6fbaab3
Repository branch: makepkg
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Arch Linux

Recent messages:
Checking 180 files in /usr/share/emacs/27.0.91/lisp/emacs-lisp...
Checking 24 files in /usr/share/emacs/27.0.91/lisp/cedet...
Checking 59 files in /usr/share/emacs/27.0.91/lisp/calendar...
Checking 87 files in /usr/share/emacs/27.0.91/lisp/calc...
Checking 113 files in /usr/share/emacs/27.0.91/lisp/obsolete...
Checking for load-path shadows...done
Quit
Mark set
Quit [3 times]
Buffer *unsent mail to bug-gnu-emacs@gnu.org* modified; kill anyway? (y or n) y
Quit
Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-sound=alsa --with-modules --without-gconf --without-gsettings
 --with-x-toolkit=gtk3 --without-xaw3d --without-m17n-flt --with-cairo
 --without-compress-install 'CFLAGS=-march=x86-64 -mtune=generic -O2
 -pipe -fno-plt -g -flto -g -flto -s -fuse-ld=gold'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3
X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  company-mode: t
  rainbow-delimiters-mode: t
  winner-mode: t
  global-hl-line-mode: t
  save-place-mode: t
  global-auto-revert-mode: t
  delete-selection-mode: t
  global-subword-mode: t
  subword-mode: t
  electric-pair-mode: t
  show-paren-mode: t
  ido-everywhere: t
  recentf-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: 1
  mouse-wheel-mode: t
  tool-bar-mode: t
  global-prettify-symbols-mode: t
  prettify-symbols-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t
  hs-minor-mode: t

Load-path shadows:
/home/hov/.emacs.d/elpa/org-20200427/org-version hides /usr/share/emacs/27.0.91/lisp/org/org-version
/home/hov/.emacs.d/elpa/org-20200427/ox-md hides /usr/share/emacs/27.0.91/lisp/org/ox-md
/home/hov/.emacs.d/elpa/org-20200427/ob-eval hides /usr/share/emacs/27.0.91/lisp/org/ob-eval
/home/hov/.emacs.d/elpa/org-20200427/org-indent hides /usr/share/emacs/27.0.91/lisp/org/org-indent
/home/hov/.emacs.d/elpa/org-20200427/org-attach hides /usr/share/emacs/27.0.91/lisp/org/org-attach
/home/hov/.emacs.d/elpa/org-20200427/ob-calc hides /usr/share/emacs/27.0.91/lisp/org/ob-calc
/home/hov/.emacs.d/elpa/org-20200427/org-lint hides /usr/share/emacs/27.0.91/lisp/org/org-lint
/home/hov/.emacs.d/elpa/org-20200427/org-attach-git hides /usr/share/emacs/27.0.91/lisp/org/org-attach-git
/home/hov/.emacs.d/elpa/org-20200427/org-crypt hides /usr/share/emacs/27.0.91/lisp/org/org-crypt
/home/hov/.emacs.d/elpa/org-20200427/ob-ledger hides /usr/share/emacs/27.0.91/lisp/org/ob-ledger
/home/hov/.emacs.d/elpa/org-20200427/ob-lisp hides /usr/share/emacs/27.0.91/lisp/org/ob-lisp
/home/hov/.emacs.d/elpa/org-20200427/ob-ditaa hides /usr/share/emacs/27.0.91/lisp/org/ob-ditaa
/home/hov/.emacs.d/elpa/org-20200427/ob-sed hides /usr/share/emacs/27.0.91/lisp/org/ob-sed
/home/hov/.emacs.d/elpa/org-20200427/ob-abc hides /usr/share/emacs/27.0.91/lisp/org/ob-abc
/home/hov/.emacs.d/elpa/org-20200427/org-entities hides /usr/share/emacs/27.0.91/lisp/org/org-entities
/home/hov/.emacs.d/elpa/org-20200427/ob-processing hides /usr/share/emacs/27.0.91/lisp/org/ob-processing
/home/hov/.emacs.d/elpa/org-20200427/ob-core hides /usr/share/emacs/27.0.91/lisp/org/ob-core
/home/hov/.emacs.d/elpa/org-20200427/ob-dot hides /usr/share/emacs/27.0.91/lisp/org/ob-dot
/home/hov/.emacs.d/elpa/org-20200427/ox-man hides /usr/share/emacs/27.0.91/lisp/org/ox-man
/home/hov/.emacs.d/elpa/org-20200427/org-tempo hides /usr/share/emacs/27.0.91/lisp/org/org-tempo
/home/hov/.emacs.d/elpa/org-20200427/ob-lob hides /usr/share/emacs/27.0.91/lisp/org/ob-lob
/home/hov/.emacs.d/elpa/org-20200427/ob-python hides /usr/share/emacs/27.0.91/lisp/org/ob-python
/home/hov/.emacs.d/elpa/org-20200427/ob-scheme hides /usr/share/emacs/27.0.91/lisp/org/ob-scheme
/home/hov/.emacs.d/elpa/org-20200427/ob-eshell hides /usr/share/emacs/27.0.91/lisp/org/ob-eshell
/home/hov/.emacs.d/elpa/org-20200427/ol-eshell hides /usr/share/emacs/27.0.91/lisp/org/ol-eshell
/home/hov/.emacs.d/elpa/org-20200427/ob-sql hides /usr/share/emacs/27.0.91/lisp/org/ob-sql
/home/hov/.emacs.d/elpa/org-20200427/org-compat hides /usr/share/emacs/27.0.91/lisp/org/org-compat
/home/hov/.emacs.d/elpa/org-20200427/org-capture hides /usr/share/emacs/27.0.91/lisp/org/org-capture
/home/hov/.emacs.d/elpa/org-20200427/ob-J hides /usr/share/emacs/27.0.91/lisp/org/ob-J
/home/hov/.emacs.d/elpa/org-20200427/ox-latex hides /usr/share/emacs/27.0.91/lisp/org/ox-latex
/home/hov/.emacs.d/elpa/org-20200427/org-keys hides /usr/share/emacs/27.0.91/lisp/org/org-keys
/home/hov/.emacs.d/elpa/org-20200427/org-table hides /usr/share/emacs/27.0.91/lisp/org/org-table
/home/hov/.emacs.d/elpa/org-20200427/ol-info hides /usr/share/emacs/27.0.91/lisp/org/ol-info
/home/hov/.emacs.d/elpa/org-20200427/ob-tangle hides /usr/share/emacs/27.0.91/lisp/org/ob-tangle
/home/hov/.emacs.d/elpa/org-20200427/org-faces hides /usr/share/emacs/27.0.91/lisp/org/org-faces
/home/hov/.emacs.d/elpa/org-20200427/org-timer hides /usr/share/emacs/27.0.91/lisp/org/org-timer
/home/hov/.emacs.d/elpa/org-20200427/ob-table hides /usr/share/emacs/27.0.91/lisp/org/ob-table
/home/hov/.emacs.d/elpa/org-20200427/ob-screen hides /usr/share/emacs/27.0.91/lisp/org/ob-screen
/home/hov/.emacs.d/elpa/org-20200427/org-datetree hides /usr/share/emacs/27.0.91/lisp/org/org-datetree
/home/hov/.emacs.d/elpa/org-20200427/ox-odt hides /usr/share/emacs/27.0.91/lisp/org/ox-odt
/home/hov/.emacs.d/elpa/org-20200427/ol-w3m hides /usr/share/emacs/27.0.91/lisp/org/ol-w3m
/home/hov/.emacs.d/elpa/org-20200427/org-loaddefs hides /usr/share/emacs/27.0.91/lisp/org/org-loaddefs
/home/hov/.emacs.d/elpa/org-20200427/org-list hides /usr/share/emacs/27.0.91/lisp/org/org-list
/home/hov/.emacs.d/elpa/org-20200427/ob-emacs-lisp hides /usr/share/emacs/27.0.91/lisp/org/ob-emacs-lisp
/home/hov/.emacs.d/elpa/org-20200427/ob-picolisp hides /usr/share/emacs/27.0.91/lisp/org/ob-picolisp
/home/hov/.emacs.d/elpa/org-20200427/ob-haskell hides /usr/share/emacs/27.0.91/lisp/org/ob-haskell
/home/hov/.emacs.d/elpa/org-20200427/org-element hides /usr/share/emacs/27.0.91/lisp/org/org-element
/home/hov/.emacs.d/elpa/org-20200427/ob-sqlite hides /usr/share/emacs/27.0.91/lisp/org/ob-sqlite
/home/hov/.emacs.d/elpa/org-20200427/ob-hledger hides /usr/share/emacs/27.0.91/lisp/org/ob-hledger
/home/hov/.emacs.d/elpa/org-20200427/org-clock hides /usr/share/emacs/27.0.91/lisp/org/org-clock
/home/hov/.emacs.d/elpa/org-20200427/ox-icalendar hides /usr/share/emacs/27.0.91/lisp/org/ox-icalendar
/home/hov/.emacs.d/elpa/org-20200427/org-duration hides /usr/share/emacs/27.0.91/lisp/org/org-duration
/home/hov/.emacs.d/elpa/org-20200427/org-id hides /usr/share/emacs/27.0.91/lisp/org/org-id
/home/hov/.emacs.d/elpa/org-20200427/ob hides /usr/share/emacs/27.0.91/lisp/org/ob
/home/hov/.emacs.d/elpa/org-20200427/ob-shen hides /usr/share/emacs/27.0.91/lisp/org/ob-shen
/home/hov/.emacs.d/elpa/org-20200427/ol-gnus hides /usr/share/emacs/27.0.91/lisp/org/ol-gnus
/home/hov/.emacs.d/elpa/org-20200427/org-src hides /usr/share/emacs/27.0.91/lisp/org/org-src
/home/hov/.emacs.d/elpa/org-20200427/ob-awk hides /usr/share/emacs/27.0.91/lisp/org/ob-awk
/home/hov/.emacs.d/elpa/org-20200427/ol-rmail hides /usr/share/emacs/27.0.91/lisp/org/ol-rmail
/home/hov/.emacs.d/elpa/org-20200427/org-habit hides /usr/share/emacs/27.0.91/lisp/org/org-habit
/home/hov/.emacs.d/elpa/org-20200427/ob-gnuplot hides /usr/share/emacs/27.0.91/lisp/org/ob-gnuplot
/home/hov/.emacs.d/elpa/org-20200427/org-colview hides /usr/share/emacs/27.0.91/lisp/org/org-colview
/home/hov/.emacs.d/elpa/org-20200427/ob-perl hides /usr/share/emacs/27.0.91/lisp/org/ob-perl
/home/hov/.emacs.d/elpa/org-20200427/org-footnote hides /usr/share/emacs/27.0.91/lisp/org/org-footnote
/home/hov/.emacs.d/elpa/org-20200427/ox hides /usr/share/emacs/27.0.91/lisp/org/ox
/home/hov/.emacs.d/elpa/org-20200427/ob-js hides /usr/share/emacs/27.0.91/lisp/org/ob-js
/home/hov/.emacs.d/elpa/org-20200427/ox-ascii hides /usr/share/emacs/27.0.91/lisp/org/ox-ascii
/home/hov/.emacs.d/elpa/org-20200427/ob-forth hides /usr/share/emacs/27.0.91/lisp/org/ob-forth
/home/hov/.emacs.d/elpa/org-20200427/org-goto hides /usr/share/emacs/27.0.91/lisp/org/org-goto
/home/hov/.emacs.d/elpa/org-20200427/org-protocol hides /usr/share/emacs/27.0.91/lisp/org/org-protocol
/home/hov/.emacs.d/elpa/org-20200427/ob-sass hides /usr/share/emacs/27.0.91/lisp/org/ob-sass
/home/hov/.emacs.d/elpa/org-20200427/ol hides /usr/share/emacs/27.0.91/lisp/org/ol
/home/hov/.emacs.d/elpa/org-20200427/org-agenda hides /usr/share/emacs/27.0.91/lisp/org/org-agenda
/home/hov/.emacs.d/elpa/org-20200427/ox-texinfo hides /usr/share/emacs/27.0.91/lisp/org/ox-texinfo
/home/hov/.emacs.d/elpa/org-20200427/ob-comint hides /usr/share/emacs/27.0.91/lisp/org/ob-comint
/home/hov/.emacs.d/elpa/org-20200427/ob-makefile hides /usr/share/emacs/27.0.91/lisp/org/ob-makefile
/home/hov/.emacs.d/elpa/org-20200427/org-pcomplete hides /usr/share/emacs/27.0.91/lisp/org/org-pcomplete
/home/hov/.emacs.d/elpa/org-20200427/ob-groovy hides /usr/share/emacs/27.0.91/lisp/org/ob-groovy
/home/hov/.emacs.d/elpa/org-20200427/ob-R hides /usr/share/emacs/27.0.91/lisp/org/ob-R
/home/hov/.emacs.d/elpa/org-20200427/ob-latex hides /usr/share/emacs/27.0.91/lisp/org/ob-latex
/home/hov/.emacs.d/elpa/org-20200427/org-macs hides /usr/share/emacs/27.0.91/lisp/org/org-macs
/home/hov/.emacs.d/elpa/org-20200427/ob-lua hides /usr/share/emacs/27.0.91/lisp/org/ob-lua
/home/hov/.emacs.d/elpa/org-20200427/ob-exp hides /usr/share/emacs/27.0.91/lisp/org/ob-exp
/home/hov/.emacs.d/elpa/org-20200427/ob-ebnf hides /usr/share/emacs/27.0.91/lisp/org/ob-ebnf
/home/hov/.emacs.d/elpa/org-20200427/ob-ref hides /usr/share/emacs/27.0.91/lisp/org/ob-ref
/home/hov/.emacs.d/elpa/org-20200427/ob-maxima hides /usr/share/emacs/27.0.91/lisp/org/ob-maxima
/home/hov/.emacs.d/elpa/org-20200427/ob-mscgen hides /usr/share/emacs/27.0.91/lisp/org/ob-mscgen
/home/hov/.emacs.d/elpa/org-20200427/ol-irc hides /usr/share/emacs/27.0.91/lisp/org/ol-irc
/home/hov/.emacs.d/elpa/org-20200427/ob-ocaml hides /usr/share/emacs/27.0.91/lisp/org/ob-ocaml
/home/hov/.emacs.d/elpa/org-20200427/ol-mhe hides /usr/share/emacs/27.0.91/lisp/org/ol-mhe
/home/hov/.emacs.d/elpa/org-20200427/ob-vala hides /usr/share/emacs/27.0.91/lisp/org/ob-vala
/home/hov/.emacs.d/elpa/org-20200427/org-feed hides /usr/share/emacs/27.0.91/lisp/org/org-feed
/home/hov/.emacs.d/elpa/org-20200427/ob-ruby hides /usr/share/emacs/27.0.91/lisp/org/ob-ruby
/home/hov/.emacs.d/elpa/org-20200427/ob-lilypond hides /usr/share/emacs/27.0.91/lisp/org/ob-lilypond
/home/hov/.emacs.d/elpa/org-20200427/org-mobile hides /usr/share/emacs/27.0.91/lisp/org/org-mobile
/home/hov/.emacs.d/elpa/org-20200427/ob-C hides /usr/share/emacs/27.0.91/lisp/org/ob-C
/home/hov/.emacs.d/elpa/org-20200427/org-macro hides /usr/share/emacs/27.0.91/lisp/org/org-macro
/home/hov/.emacs.d/elpa/org-20200427/ox-html hides /usr/share/emacs/27.0.91/lisp/org/ox-html
/home/hov/.emacs.d/elpa/org-20200427/ob-java hides /usr/share/emacs/27.0.91/lisp/org/ob-java
/home/hov/.emacs.d/elpa/org-20200427/ox-beamer hides /usr/share/emacs/27.0.91/lisp/org/ox-beamer
/home/hov/.emacs.d/elpa/org-20200427/ob-plantuml hides /usr/share/emacs/27.0.91/lisp/org/ob-plantuml
/home/hov/.emacs.d/elpa/org-20200427/org-mouse hides /usr/share/emacs/27.0.91/lisp/org/org-mouse
/home/hov/.emacs.d/elpa/org-20200427/ox-org hides /usr/share/emacs/27.0.91/lisp/org/ox-org
/home/hov/.emacs.d/elpa/org-20200427/ol-bbdb hides /usr/share/emacs/27.0.91/lisp/org/ol-bbdb
/home/hov/.emacs.d/elpa/org-20200427/ol-docview hides /usr/share/emacs/27.0.91/lisp/org/ol-docview
/home/hov/.emacs.d/elpa/org-20200427/org-install hides /usr/share/emacs/27.0.91/lisp/org/org-install
/home/hov/.emacs.d/elpa/org-20200427/org-inlinetask hides /usr/share/emacs/27.0.91/lisp/org/org-inlinetask
/home/hov/.emacs.d/elpa/org-20200427/org-plot hides /usr/share/emacs/27.0.91/lisp/org/org-plot
/home/hov/.emacs.d/elpa/org-20200427/ob-coq hides /usr/share/emacs/27.0.91/lisp/org/ob-coq
/home/hov/.emacs.d/elpa/org-20200427/ol-eww hides /usr/share/emacs/27.0.91/lisp/org/ol-eww
/home/hov/.emacs.d/elpa/org-20200427/org-ctags hides /usr/share/emacs/27.0.91/lisp/org/org-ctags
/home/hov/.emacs.d/elpa/org-20200427/ob-shell hides /usr/share/emacs/27.0.91/lisp/org/ob-shell
/home/hov/.emacs.d/elpa/org-20200427/ol-bibtex hides /usr/share/emacs/27.0.91/lisp/org/ol-bibtex
/home/hov/.emacs.d/elpa/org-20200427/ob-stan hides /usr/share/emacs/27.0.91/lisp/org/ob-stan
/home/hov/.emacs.d/elpa/org-20200427/ob-io hides /usr/share/emacs/27.0.91/lisp/org/ob-io
/home/hov/.emacs.d/elpa/org-20200427/ob-clojure hides /usr/share/emacs/27.0.91/lisp/org/ob-clojure
/home/hov/.emacs.d/elpa/org-20200427/ob-fortran hides /usr/share/emacs/27.0.91/lisp/org/ob-fortran
/home/hov/.emacs.d/elpa/org-20200427/ob-css hides /usr/share/emacs/27.0.91/lisp/org/ob-css
/home/hov/.emacs.d/elpa/org-20200427/org-num hides /usr/share/emacs/27.0.91/lisp/org/org-num
/home/hov/.emacs.d/elpa/org-20200427/ob-matlab hides /usr/share/emacs/27.0.91/lisp/org/ob-matlab
/home/hov/.emacs.d/elpa/org-20200427/ox-publish hides /usr/share/emacs/27.0.91/lisp/org/ox-publish
/home/hov/.emacs.d/elpa/org-20200427/ob-octave hides /usr/share/emacs/27.0.91/lisp/org/ob-octave
/home/hov/.emacs.d/elpa/org-20200427/ob-asymptote hides /usr/share/emacs/27.0.91/lisp/org/ob-asymptote
/home/hov/.emacs.d/elpa/org-20200427/org hides /usr/share/emacs/27.0.91/lisp/org/org
/home/hov/.emacs.d/elpa/org-20200427/org-archive hides /usr/share/emacs/27.0.91/lisp/org/org-archive
/home/hov/.emacs.d/elpa/org-20200427/ob-org hides /usr/share/emacs/27.0.91/lisp/org/ob-org

Features:
(shadow sort mail-extr emacsbug sendmail misearch multi-isearch
face-remap ox-odt rng-loc rng-uri rng-parse rng-match rng-dt rng-util
rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar ox-html table ox-ascii ox-publish ox image-file org-element
avl-tree disp-table ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir
gnus-sum url url-proxy url-privacy url-expand url-methods url-history
mailcap shr url-cookie url-domsuf url-util svg xml dom gnus-group
gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc
nnoo parse-time iso8601 gnus-spec gnus-int gnus-range message rmc puny
rfc822 mml mml-sec epa derived epg epg-config mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win
gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr ol-docview doc-view jka-compr image-mode
exif ol-bibtex bibtex ol-bbdb ol-w3m ob-sqlite ob-sql ob-css ob-C
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs ob-shell shell ob-js ob-python ob-clojure ob-latex org
ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote
org-src ob-comint org-pcomplete pcomplete comint org-list org-faces
org-entities time-date noutline outline easy-mmode org-version
ob-emacs-lisp ob-core ob-eval org-table ol org-keys org-compat org-macs
org-loaddefs format-spec cal-menu calendar cal-loaddefs goto-addr
thingatpt bookmark text-property-search pp hideshow company-oddmuse
company-keywords company-etags etags fileloop generator xref project
company-gtags company-dabbrev-code company-dabbrev company-files
company-capf company-cmake company-xcode company-clang company-semantic
company-eclim company-template company-bbdb company pcase
rainbow-delimiters winner ring crt-dark-black-theme dired-x dired
dired-loaddefs hl-line saveplace autorevert filenotify delsel cap-words
superword subword elec-pair paren ido recentf tree-widget wid-edit
skeleton windmove flycheck ansi-color find-func help-mode rx dash
edmacro kmacro advice finder-inf info slime-autoloads package easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
font-render-setting cairo move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 392485 26219)
 (symbols 48 32832 1)
 (strings 32 141714 3614)
 (string-bytes 1 4387163)
 (vectors 16 52134)
 (vector-slots 8 611810 21734)
 (floats 8 285 420)
 (intervals 56 324 0)
 (buffers 1000 19))

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

On June 4, 2020 6:01:21 AM GMT+03:00, hossein valizadeh <valizadeh.ho@gmail.com> wrote:
> Hello,
> 
> 1. This happen with every font that supports Arabic.
> 2. When scale back to zero the problem happen again.
> 
> I attached my emacs-bug-info.
> 
> 
> On Thu, Jun 4, 2020 at 7:09 AM hossein valizadeh
> <valizadeh.ho@gmail.com>
> wrote:
> 
> > Hello,
> >
> > 1. This happen with every font that supports Arabic.
> > 2. When scale back to zero the problem happen again.
> >


Thanks.

What is your version of HarfBuzz?
And what happens if you unset XMODIFIERS, i.e. disable the ibus input method framework?

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

On June 4, 2020 7:01:30 AM GMT+03:00, Eli Zaretskii <eliz@gnu.org> wrote:
> On June 4, 2020 6:01:21 AM GMT+03:00, hossein valizadeh
> <valizadeh.ho@gmail.com> wrote:
> > Hello,
> > 
> > 1. This happen with every font that supports Arabic.
> > 2. When scale back to zero the problem happen again.
> > 
> > I attached my emacs-bug-info.
> > 
> > 
> > On Thu, Jun 4, 2020 at 7:09 AM hossein valizadeh
> > <valizadeh.ho@gmail.com>
> > wrote:
> > 
> > > Hello,
> > >
> > > 1. This happen with every font that supports Arabic.
> > > 2. When scale back to zero the problem happen again.
> > >
> 
> 
> Thanks.
> 
> What is your version of HarfBuzz?
> And what happens if you unset XMODIFIERS, i.e. disable the ibus input
> method framework?

Also, please go to the problematic place in the text and type "C-u C-x =", then post everything that Emacs shows in the *Help* buffer as result.  Please do this both at text scale zero, when shaping is incorrect, and at non-zero scale, and post the contents of *Help* in both cases.

Screenshots of both displays as well as the text of the buffer used for these experiments will also help, as mentioned earlier 

And finally, please try this with the version on the master branch, where a few fixes were installed lately.

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1.1: Type: text/plain, Size: 1619 bytes --]

HarfBuzz version is  2.6.7
XMODIFIERS=''  make no difference.
Text used in these screenshots are  https://time.ir  in  EWW mode. (help
buffer content displayed in screenshots)

i will try on master branch and  report the result.

On Thu, Jun 4, 2020 at 8:40 AM Eli Zaretskii <eliz@gnu.org> wrote:

> On June 4, 2020 7:01:30 AM GMT+03:00, Eli Zaretskii <eliz@gnu.org> wrote:
> > On June 4, 2020 6:01:21 AM GMT+03:00, hossein valizadeh
> > <valizadeh.ho@gmail.com> wrote:
> > > Hello,
> > >
> > > 1. This happen with every font that supports Arabic.
> > > 2. When scale back to zero the problem happen again.
> > >
> > > I attached my emacs-bug-info.
> > >
> > >
> > > On Thu, Jun 4, 2020 at 7:09 AM hossein valizadeh
> > > <valizadeh.ho@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > 1. This happen with every font that supports Arabic.
> > > > 2. When scale back to zero the problem happen again.
> > > >
> >
> >
> > Thanks.
> >
> > What is your version of HarfBuzz?
> > And what happens if you unset XMODIFIERS, i.e. disable the ibus input
> > method framework?
>
> Also, please go to the problematic place in the text and type "C-u C-x =",
> then post everything that Emacs shows in the *Help* buffer as result.
> Please do this both at text scale zero, when shaping is incorrect, and at
> non-zero scale, and post the contents of *Help* in both cases.
>
> Screenshots of both displays as well as the text of the buffer used for
> these experiments will also help, as mentioned earlier
>
> And finally, please try this with the version on the master branch, where
> a few fixes were installed lately.
>

[-- Attachment #1.2: Type: text/html, Size: 2561 bytes --]

[-- Attachment #2: zero-scale.png --]
[-- Type: image/png, Size: 162650 bytes --]

[-- Attachment #3: none-zero-scale.png --]
[-- Type: image/png, Size: 177299 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

hossein valizadeh <valizadeh.ho@gmail.com> writes:

>  And finally, please try this with the version on the master branch, where a few fixes were
>  installed lately.

I can reproduce the very similar issue described (Farsi Wikipedia entry
for Emacs), on current master. I believe I've figured it out, but I can
also debug further if required.

What happens is that font-shape-gstring is called with direction == L2R,
mis-shapes the text, then caches that version in the composition gstring
cache. The cache doesn't distinguish directions, and it's never cleared,
so this "infects" other buffers, but only if they're opened afterwards,
and only for the same words.

shr appears to force bidi-display-reordering off. Removing that let
binding from shr-insert-document avoids the problem.

We should consider adding direction to our gstrings, on master. While
we're there, let's also add script, language, and harfbuzz features to
the gstrings so we know we've captured the precise harfbuzz parameters?

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  41005@debbugs.gnu.org,
>   nicholasdrozd@gmail.com
> Date: Thu, 04 Jun 2020 08:28:24 +0000
> 
> What happens is that font-shape-gstring is called with direction == L2R,
> mis-shapes the text, then caches that version in the composition gstring
> cache. The cache doesn't distinguish directions, and it's never cleared,
> so this "infects" other buffers, but only if they're opened afterwards,
> and only for the same words.
> 
> shr appears to force bidi-display-reordering off. Removing that let
> binding from shr-insert-document avoids the problem.

Right, thanks.  When I added that binding of bidi-display-reordering,
we didn't yet have HarfBuzz, and the other shapers' Arabic shaping
wasn't affected by the local text direction like HarfBuzz is.

> We should consider adding direction to our gstrings, on master. While
> we're there, let's also add script, language, and harfbuzz features to
> the gstrings so we know we've captured the precise harfbuzz parameters?

On emacs-27, I can fix this by a simpler band-aid below.

On master, I think we should indeed add direction to the cached
gstrings, as there could be other much more subtle situations where
looking at bidi-display-reordering is not enough, and we could then
still cache a composition with the wrong direction.  Patches welcome.

As for script and language, for now adding them would just waste
memory, since we don't yet have any meaningful support for
buffer-local, let-alone paragraph-local, scripts and languages.  When
we added HarfBuzz support, I considered adding some functionality to
determine language and/or script from the codepoints, but decided that
it made little sense, since HarfBuzz already does exactly that in
hb_buffer_guess_segment_properties.  So I think we should add to the
FIXME in hbfont.c the fact that when we do support those internally in
Emacs, we should add these attributes to cached gstrings, but not yet
actually add them for now.

Here's the patch I propose for emacs-27:

diff --git a/src/hbfont.c b/src/hbfont.c
index 576c5fe..4b3f64e 100644
--- a/src/hbfont.c
+++ b/src/hbfont.c
@@ -26,6 +26,7 @@
 #include "composite.h"
 #include "font.h"
 #include "dispextern.h"
+#include "buffer.h"
 
 #ifdef HAVE_NTGUI
 
@@ -438,7 +439,11 @@ hbfont_shape (Lisp_Object lgstring, Lisp_Object direction)
 
   /* If the caller didn't provide a meaningful DIRECTION, let HarfBuzz
      guess it. */
-  if (!NILP (direction))
+  if (!NILP (direction)
+      /* If they bind bidi-display-reordering to nil, the DIRECTION
+	 they provide is meaningless, and we should let HarfBuzz guess
+	 the real direction.  */
+      && !NILP (BVAR (current_buffer, bidi_display_reordering)))
     {
       hb_direction_t dir = HB_DIRECTION_LTR;
       if (EQ (direction, QL2R))

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  41005@debbugs.gnu.org,
>>   nicholasdrozd@gmail.com
>> Date: Thu, 04 Jun 2020 08:28:24 +0000
>> 
>> What happens is that font-shape-gstring is called with direction == L2R,
>> mis-shapes the text, then caches that version in the composition gstring
>> cache. The cache doesn't distinguish directions, and it's never cleared,
>> so this "infects" other buffers, but only if they're opened afterwards,
>> and only for the same words.
>> 
>> shr appears to force bidi-display-reordering off. Removing that let
>> binding from shr-insert-document avoids the problem.

>> We should consider adding direction to our gstrings, on master. While
>> we're there, let's also add script, language, and harfbuzz features to
>> the gstrings so we know we've captured the precise harfbuzz parameters?
>
> On emacs-27, I can fix this by a simpler band-aid below.

> On master, I think we should indeed add direction to the cached
> gstrings, as there could be other much more subtle situations where
> looking at bidi-display-reordering is not enough, and we could then
> still cache a composition with the wrong direction.  Patches welcome.

Sure, such subtle situations exist.

> As for script and language, for now adding them would just waste
> memory, since we don't yet have any meaningful support for
> buffer-local, let-alone paragraph-local, scripts and languages.  When
> we added HarfBuzz support, I considered adding some functionality to
> determine language and/or script from the codepoints, but decided that
> it made little sense, since HarfBuzz already does exactly that in
> hb_buffer_guess_segment_properties.  So I think we should add to the
> FIXME in hbfont.c the fact that when we do support those internally in
> Emacs, we should add these attributes to cached gstrings, but not yet
> actually add them for now.

We're going to have to change the lgstring structure, though, right?

Could we maybe get away with just doing so once? My suggestion would be
to add a single field to gstrings which would currently be L2R or R2L
but might become an alist or something when we get around to adding
those features?

> Here's the patch I propose for emacs-27:

Let's try that.

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

I tried the patch.
The eww problem is solved, but the problem still there, when I enable
auto-fill-mode or column-number-mode.

Please look at this video file for a better understanding:
http://s13.picofile.com/d/8399189550/7eeb413f-0df7-4da6-9db1-1632c9fc749f/out.mkv
https://filebin.net/mzmjm74lp7wsxr8e
https://gofile.io/d/H8xk26


For example, if you type in the following sentence:
این نام است که می‌ماند


Then go back a few characters in the same line and type words randomly. You
will see that the letters in some words are displayed separately. I type a
few words at random, after the word این and before the word نام :‌

این فراموشی را به همه اینکه فرمت مراتب افتتاح گرامی گرایش سراسیمه نام است
که می‌ماند

This line should look like this:

http://s12.picofile.com/file/8399190550/correct.png

‌But if one of the auto-fill-mode or column-number-mode is enabled. it will
be displayed this way:

http://s13.picofile.com/file/8399190584/malformed.png

[-- Attachment #2: Type: text/html, Size: 1774 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Fri, 5 Jun 2020 09:16:53 +0430
> Cc: Eli Zaretskii <eliz@gnu.org>, 41005@debbugs.gnu.org, 
> 	Nicholas Drozd <nicholasdrozd@gmail.com>
> 
> I tried the patch.

Please tell which patch was that, and in what Emacs version you tried
it.  (Please understand that you are generally talking to people some
of whom don't read Arabic or Persian, so the more details you supply
the less misunderstanding and confusion will follow, and the faster
this problem will be solved.)

> The eww problem is solved, but the problem still there, when I enable auto-fill-mode or
> column-number-mode.
> 
> Please look at this video file for a better understanding:
> http://s13.picofile.com/d/8399189550/7eeb413f-0df7-4da6-9db1-1632c9fc749f/out.mkv
> https://filebin.net/mzmjm74lp7wsxr8e
> https://gofile.io/d/H8xk26

I cannot play this on my system, I see a bunch of ads (or what looks
like ads), and the name of a .mkv file.

> For example, if you type in the following sentence:
> این نام است که می‌ماند
> 
> Then go back a few characters in the same line and type words randomly. You will see that the letters in
> some words are displayed separately. I type a few words at random, after the word این and before the word
> نام :‌
> 
> این فراموشی را به همه اینکه فرمت مراتب افتتاح گرامی گرایش سراسیمه نام است که می‌ماند
> 
> This line should look like this:
> 
> http://s12.picofile.com/file/8399190550/correct.png
> 
> ‌But if one of the auto-fill-mode or column-number-mode is enabled. it will be displayed this way: 
> 
> http://s13.picofile.com/file/8399190584/malformed.png

Please show a full recipe for reproducing this, starting from
"emacs -Q", and describing every step to reproduce the result.  Please
tell the codepoint of each character you type to reproduce the problem
and each Emacs command.  I'd also greatly appreciate if you
specifically point out at which parts of the display to look at what
differences to pay attention to.  This is needed to fully understand
the problem and analyze its root cause(s), given that not all of us
can read the Arabic script.

The master branch has recently got a few improvements in this area, so
please use that version of the code if you can.  And in any case,
please always state in what version you see which problem.

Thanks.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

hossein valizadeh <valizadeh.ho@gmail.com> writes:
> I tried the patch.
> The eww problem is solved, but the problem still there, when I enable auto-fill-mode or
> column-number-mode.

I only see it with column-number-mode so far (but, again, I can
reproduce it and debug further).

It's this code in indent.c

      /* Check composition sequence.  */
      if (cmp_it.id >= 0
	  || (scan == cmp_it.stop_pos
	      && composition_reseat_it (&cmp_it, scan, scan_byte, end,
					w, NEUTRAL_DIR, NULL, Qnil)))

which appears to think the sixth argument to composition_reseat_it is a
direction (it probably should be). It's actually a bidi level, and
passing NEUTRAL_DIR, which is 0, results in L2R layout being used by
hbfont_shape, as in the eww bug.

Will write a patch soon if no one beats me to it.

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
> Date: Thu, 04 Jun 2020 19:52:42 +0000
> 
> > As for script and language, for now adding them would just waste
> > memory, since we don't yet have any meaningful support for
> > buffer-local, let-alone paragraph-local, scripts and languages.  When
> > we added HarfBuzz support, I considered adding some functionality to
> > determine language and/or script from the codepoints, but decided that
> > it made little sense, since HarfBuzz already does exactly that in
> > hb_buffer_guess_segment_properties.  So I think we should add to the
> > FIXME in hbfont.c the fact that when we do support those internally in
> > Emacs, we should add these attributes to cached gstrings, but not yet
> > actually add them for now.
> 
> We're going to have to change the lgstring structure, though, right?

I think so.  We should probably add one more element to the vector in
LGSTRING_HEADER, because the header is the hash key of the composition
cache.

> Could we maybe get away with just doing so once? My suggestion would be
> to add a single field to gstrings which would currently be L2R or R2L
> but might become an alist or something when we get around to adding
> those features?

We could add an element that would currently be a symbol or an
integer, but could later become a vector of several elements.  Is that
what you had in mind?  (I think we should prefer vectors to lists in
this case, because consing them is slightly faster, and the number of
elements is known in advance and fixed.)

> > Here's the patch I propose for emacs-27:
> 
> Let's try that.

Pushed to the emacs-27 branch.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
>> Date: Thu, 04 Jun 2020 19:52:42 +0000
>> 
>> > As for script and language, for now adding them would just waste
>> > memory, since we don't yet have any meaningful support for
>> > buffer-local, let-alone paragraph-local, scripts and languages.  When
>> > we added HarfBuzz support, I considered adding some functionality to
>> > determine language and/or script from the codepoints, but decided that
>> > it made little sense, since HarfBuzz already does exactly that in
>> > hb_buffer_guess_segment_properties.  So I think we should add to the
>> > FIXME in hbfont.c the fact that when we do support those internally in
>> > Emacs, we should add these attributes to cached gstrings, but not yet
>> > actually add them for now.
>> 
>> We're going to have to change the lgstring structure, though, right?
>
> I think so.  We should probably add one more element to the vector in
> LGSTRING_HEADER, because the header is the hash key of the composition
> cache.

Do we have to maintain compatibility? If so, I suggest we change

[FONT-OBJECT CHAR ...]

to

[FONT-OBJECT [CHAR ...] DIRECTION], and use ARRAYP (AREF (..., 2)) to
decide whether the new format is in effect.  I actually thought about
suggesting the format be [FONT-OBJECT STRING DIRECTION], but that would
make debugging harder when pretty-printing the string in a failed
composition re-attempts that composition.

But of course it would be easier not to maintain compatibility, and then
we could order the elements any way we choose.

>> Could we maybe get away with just doing so once? My suggestion would be
>> to add a single field to gstrings which would currently be L2R or R2L
>> but might become an alist or something when we get around to adding
>> those features?
>
> We could add an element that would currently be a symbol or an
> integer, but could later become a vector of several elements.  Is that
> what you had in mind?

Yes, a vector was what I meant by "or something".

> (I think we should prefer vectors to lists in
> this case, because consing them is slightly faster, and the number of
> elements is known in advance and fixed.)

No argument there, though harfbuzz features, if we ever add them,
probably should be added as a list inside the vector. I'm talking about
things like "kern=0" passed to hb_feature_from_string, then to hb_shape,
to disable kerning.

bug#41005: problem with rendering Persian text in Emacs 27

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hossein valizadeh <valizadeh.ho@gmail.com>
>> Date: Fri, 5 Jun 2020 09:16:53 +0430
>> Cc: Eli Zaretskii <eliz@gnu.org>, 41005@debbugs.gnu.org, 
>> 	Nicholas Drozd <nicholasdrozd@gmail.com>
>> 
>> The eww problem is solved, but the problem still there, when I enable auto-fill-mode or
>> column-number-mode.
>> 
>> Please look at this video file for a better understanding:
>> http://s13.picofile.com/d/8399189550/7eeb413f-0df7-4da6-9db1-1632c9fc749f/out.mkv
>> https://filebin.net/mzmjm74lp7wsxr8e
>> https://gofile.io/d/H8xk26
>
> I cannot play this on my system, I see a bunch of ads (or what looks
> like ads), and the name of a .mkv file.

On the first page, you need to click twice the red button with the link
icon that according to my beginner-level reading of Perso-Arabic script
says "[something] link download".

That said, the easiest way for me to download the file was from the
third gofile link.  (All three links point to the same file AFAICT.)

If you're not able to play .mkv on Windows I think VLC supports it OOTB
(but I haven't used Windows in almost a decade so YMMV).

The large screen resolution of the screencast makes it a bit hard for me
to tell what's going on on my 14" laptop, though.

HTH,

-- 
Basil

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
> Date: Fri, 05 Jun 2020 08:41:19 +0000
> 
> >> We're going to have to change the lgstring structure, though, right?
> >
> > I think so.  We should probably add one more element to the vector in
> > LGSTRING_HEADER, because the header is the hash key of the composition
> > cache.
> 
> Do we have to maintain compatibility? If so, I suggest we change
> 
> [FONT-OBJECT CHAR ...]
> 
> to
> 
> [FONT-OBJECT [CHAR ...] DIRECTION], and use ARRAYP (AREF (..., 2)) to
> decide whether the new format is in effect.  I actually thought about
> suggesting the format be [FONT-OBJECT STRING DIRECTION], but that would
> make debugging harder when pretty-printing the string in a failed
> composition re-attempts that composition.
> 
> But of course it would be easier not to maintain compatibility, and then
> we could order the elements any way we choose.

We don't have to be backward-compatible here, I think, as the
structure of the header is an internal implementation detail.  So
something like [FONT-OBJECT DIRECTION CHAR ...] is also a possibility.

We could also put DIRECTION elsewhere and just modify the code that
passes the hash key to the hash function.

> > (I think we should prefer vectors to lists in
> > this case, because consing them is slightly faster, and the number of
> > elements is known in advance and fixed.)
> 
> No argument there, though harfbuzz features, if we ever add them,
> probably should be added as a list inside the vector. I'm talking about
> things like "kern=0" passed to hb_feature_from_string, then to hb_shape,
> to disable kerning.

Maybe.  Something to consider when we actually support those features.

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1: Type: text/plain, Size: 4384 bytes --]

> Please tell which patch was that, and in what Emacs version you tried
> it.  (Please understand that you are generally talking to people some
> of whom don't read Arabic or Persian, so the more details you supply
> the less misunderstanding and confusion will follow, and the faster
> this problem will be solved.)

I'm sorry, my English is so bad, at least in writing. That's why it's
hard for me to give a full explanation.
My Emacs version:  27.0.91
I applied this patch:

diff --git a/src/hbfont.c b/src/hbfont.c
index 576c5fe..4b3f64e 100644
--- a/src/hbfont.c
+++ b/src/hbfont.c
@@ -26,6 +26,7 @@
 #include "composite.h"
 #include "font.h"
 #include "dispextern.h"
+#include "buffer.h"

 #ifdef HAVE_NTGUI

@@ -438,7 +439,11 @@ hbfont_shape (Lisp_Object lgstring, Lisp_Object
direction)

   /* If the caller didn't provide a meaningful DIRECTION, let HarfBuzz
      guess it. */
-  if (!NILP (direction))
+  if (!NILP (direction)
+      /* If they bind bidi-display-reordering to nil, the DIRECTION
+        they provide is meaningless, and we should let HarfBuzz guess
+        the real direction.  */
+      && !NILP (BVAR (current_buffer, bidi_display_reordering)))
     {
       hb_direction_t dir = HB_DIRECTION_LTR;
       if (EQ (direction, QL2R))

--------------------------------------------------------------------------------
> I cannot play this on my system, I see a bunch of ads (or what looks
> like ads), and the name of a .mkv file.

video file reuploaded (.mp4 and .ogg):
https://srv-file9.gofile.io/download/cz7P41/out.mp4
https://srv-file4.gofile.io/download/Mwv8k4/out.ogg
--------------------------------------------------------------------------------

This patch solved the problems of eww, newsticker, (and possibly some
other major modes) in displaying Persian/Arabic words. However, if one
of the auto-fill-mode or column-number-mode is enabled, there is still
the same problem in files that use Persian or Arabic characters.
Especially when you want to go back a few characters in a line and add
something to that line.

--------------------------------------------------------------------------------
> Please tell the codepoint of each character you type to reproduce
> the problem and each Emacs command.

For example, if you type in the following sentence:
این نام است که می‌ماند

codepoint:
\u0627\u06cc\u0646 \u0646\u0627\u0645 \u0627\u0633\u062a \u06a9\u0647
\u0645\u06cc\u200c\u0645\u0627\u0646\u062f


Then go back a few characters in the same line and type words randomly. You
will see that the letters in some words are displayed separately. I type a
few words at random, after the word این and before the word نام :‌

این فراموشی را به همه اینکه فرمت مراتب افتتاح گرامی گرایش سراسیمه نام است
که می‌ماند

codepoint:
\u0627\u06cc\u0646 \u0641\u0631\u0627\u0645\u0648\u0634\u06cc \u0631\u0627
\u0628\u0647 \u0647\u0645\u0647 \u0627\u06cc\u0646\u06a9\u0647
\u0641\u0631\u0645\u062a \u0645\u0631\u0627\u062a\u0628
\u0627\u0641\u062a\u062a\u0627\u062d \u06af\u0631\u0627\u0645\u06cc
\u06af\u0631\u0627\u06cc\u0634 \u0633\u0631\u0627\u0633\u06cc\u0645\u0647
\u0646\u0627\u0645 \u0627\u0633\u062a \u06a9\u0647
\u0645\u06cc\u200c\u0645\u0627\u0646\u062f

This line should look like this:
http://s12.picofile.com/file/8399190550/correct.png

But if one of the auto-fill-mode or column-number-mode is enabled. it will
be displayed this way:
http://s13.picofile.com/file/8399190584/malformed.png

--------------------------------------------------------------------------------
> Please show a full recipe for reproducing this, starting from
> "emacs -Q", and describing every step to reproduce the result.

All steps are displayed in the video file.

> I'd also greatly appreciate if you specifically point out at which
> parts of the display to look at what differences to pay attention
> to. This is needed to fully understand the problem and analyze its
> root cause(s), given that not all of us can read the Arabic script.

The letters that should normally be connected will be displayed separately:
http://s12.picofile.com/file/8399222618/latest_screenshot.png
--------------------------------------------------------------------------------

[-- Attachment #2: Type: text/html, Size: 5159 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Fri, 5 Jun 2020 17:02:55 +0430
> Cc: pipcet@gmail.com, 41005@debbugs.gnu.org, 
> 	Nicholas Drozd <nicholasdrozd@gmail.com>
> 
> I'm sorry, my English is so bad, at least in writing. That's why it's
> hard for me to give a full explanation.

Your English is entirely adequate, there's nothing for you to be
ashamed of.

Thanks for the details, I think they point to the code identified by
Pip Cet, which is called from current-column and similar APIs.  A fix
will probably be available soon.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hossein valizadeh <valizadeh.ho@gmail.com>
>> Date: Fri, 5 Jun 2020 17:02:55 +0430
>> Cc: pipcet@gmail.com, 41005@debbugs.gnu.org, 
>> 	Nicholas Drozd <nicholasdrozd@gmail.com>
>> 
>> I'm sorry, my English is so bad, at least in writing. That's why it's
>> hard for me to give a full explanation.
>
> Your English is entirely adequate, there's nothing for you to be
> ashamed of.
>
> Thanks for the details, I think they point to the code identified by
> Pip Cet, which is called from current-column and similar APIs.  A fix
> will probably be available soon.

I think the attached patch is a fairly minimal fix; it's against master,
applies to emacs-27 but I haven't tested it there.

Given these two bugs, I wonder whether it wouldn't be more reasonable
always to let HarfBuzz guess the direction, at least for Emacs-27:
scripts which change direction, if they are supported by HarfBuzz, won't
work anyway.

Or am I missing something?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Test-patch-for-bug-41005.patch --]
[-- Type: text/x-diff, Size: 1605 bytes --]

From 18d0e15ac298f40951ddeeec56e9d87c01f51798 Mon Sep 17 00:00:00 2001
From: Pip Cet <pipcet@gmail.com>
Date: Fri, 5 Jun 2020 12:54:01 +0000
Subject: [PATCH] Test patch for bug#41005

---
 src/composite.c | 4 +++-
 src/indent.c    | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/composite.c b/src/composite.c
index 2c589e4f3a..13421a80da 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -1213,7 +1213,9 @@ composition_reseat_it (struct composition_it *cmp_it, ptrdiff_t charpos,
 	continue;
       if (charpos < endpos)
 	{
-	  if ((bidi_level & 1) == 0)
+	  if (bidi_level < 0)
+	    direction = Qnil;
+	  else if ((bidi_level & 1) == 0)
 	    direction = QL2R;
 	  else
 	    direction = QR2L;
diff --git a/src/indent.c b/src/indent.c
index c0b4c13b2c..581323b91e 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -596,7 +596,7 @@ scan_for_column (ptrdiff_t *endpos, EMACS_INT *goalcol, ptrdiff_t *prevcol)
       if (cmp_it.id >= 0
 	  || (scan == cmp_it.stop_pos
 	      && composition_reseat_it (&cmp_it, scan, scan_byte, end,
-					w, NEUTRAL_DIR, NULL, Qnil)))
+					w, -1, NULL, Qnil)))
 	composition_update_it (&cmp_it, scan, scan_byte, Qnil);
       if (cmp_it.id >= 0)
 	{
@@ -1504,7 +1504,7 @@ compute_motion (ptrdiff_t from, ptrdiff_t frombyte, EMACS_INT fromvpos,
 	  if (cmp_it.id >= 0
 	      || (pos == cmp_it.stop_pos
 		  && composition_reseat_it (&cmp_it, pos, pos_byte, to, win,
-					    NEUTRAL_DIR, NULL, Qnil)))
+					    -1, NULL, Qnil)))
 	    composition_update_it (&cmp_it, pos, pos_byte, Qnil);
 	  if (cmp_it.id >= 0)
 	    {
-- 
2.27.0.rc0

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: hossein valizadeh <valizadeh.ho@gmail.com>,  41005@debbugs.gnu.org,
>   nicholasdrozd@gmail.com
> Date: Fri, 05 Jun 2020 13:05:43 +0000
> 
> I think the attached patch is a fairly minimal fix; it's against master,
> applies to emacs-27 but I haven't tested it there.

Thanks, it LGTM.

I think we should put this on emacs-27, because this is a regression
caused by Emacs 27's support for HarfBuzz as the default shaping
engine.  The other shapers didn't want us to provide the direction,
they determined it internally.  We added the DIRECTION argument as
part of integrating HarfBuzz.

We could do better than your patch by actually computing the resolved
bidi level there, which would require start_display followed by
move_it_to, in which case we probably won't need to call
composition_reseat_it by hand at all, and could just pick up the
result produced by move_it_to.  Or maybe we should just use
Fvertical_motion instead (which does all that internally).  But these
ideas are for the master branch, not for emacs-27.

> Given these two bugs, I wonder whether it wouldn't be more reasonable
> always to let HarfBuzz guess the direction, at least for Emacs-27:
> scripts which change direction, if they are supported by HarfBuzz, won't
> work anyway.

Please explain "scripts that change direction" and "won't work
anyway", I don't think I understand that part.

The reason we don't let HarfBuzz guess in all cases is because the
resolved bidi level, when we have it, is a more accurate indication of
the required direction.  For example, if you have RTL characters
inside the LRO..PDF embedding, it would be wrong to let the shaper
guess, because it could (and usually will) guess wrongly that the
direction is R2L.  It is true that these are rare and unusual use
cases, but they do exist, and Emacs does want to support them,
including with scripts that must use the shaping engine.

bug#41005: problem with rendering Persian text in Emacs 27

[hossein valizadeh]

[-- Attachment #1.1: Type: text/plain, Size: 212 bytes --]

I tested  Pip Cet's patch on Emacs 27.0.91 and everything was fine.
In both cases (auto-fill-mode & column-number-mode) There was no
confusion in the letters; and all the words were displayed correctly.

Thanks.

[-- Attachment #1.2: Type: text/html, Size: 424 bytes --]

[-- Attachment #2: 0001-Test-patch-for-bug-41005.patch --]
[-- Type: text/x-patch, Size: 1655 bytes --]

From 18d0e15ac298f40951ddeeec56e9d87c01f51798 Mon Sep 17 00:00:00 2001
From: Pip Cet <pipcet@gmail.com>
Date: Fri, 5 Jun 2020 12:54:01 +0000
Subject: [PATCH] Test patch for bug#41005

---
 src/composite.c | 4 +++-
 src/indent.c    | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/composite.c b/src/composite.c
index 2c589e4f3a..13421a80da 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -1213,7 +1213,9 @@ composition_reseat_it (struct composition_it *cmp_it, ptrdiff_t charpos,
 	continue;
       if (charpos < endpos)
 	{
-	  if ((bidi_level & 1) == 0)
+	  if (bidi_level < 0)
+	    direction = Qnil;
+	  else if ((bidi_level & 1) == 0)
 	    direction = QL2R;
 	  else
 	    direction = QR2L;
diff --git a/src/indent.c b/src/indent.c
index c0b4c13b2c..581323b91e 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -596,7 +596,7 @@ scan_for_column (ptrdiff_t *endpos, EMACS_INT *goalcol, ptrdiff_t *prevcol)
       if (cmp_it.id >= 0
 	  || (scan == cmp_it.stop_pos
 	      && composition_reseat_it (&cmp_it, scan, scan_byte, end,
-					w, NEUTRAL_DIR, NULL, Qnil)))
+					w, -1, NULL, Qnil)))
 	composition_update_it (&cmp_it, scan, scan_byte, Qnil);
       if (cmp_it.id >= 0)
 	{
@@ -1504,7 +1504,7 @@ compute_motion (ptrdiff_t from, ptrdiff_t frombyte, EMACS_INT fromvpos,
 	  if (cmp_it.id >= 0
 	      || (pos == cmp_it.stop_pos
 		  && composition_reseat_it (&cmp_it, pos, pos_byte, to, win,
-					    NEUTRAL_DIR, NULL, Qnil)))
+					    -1, NULL, Qnil)))
 	    composition_update_it (&cmp_it, pos, pos_byte, Qnil);
 	  if (cmp_it.id >= 0)
 	    {
-- 
2.27.0.rc0

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: hossein valizadeh <valizadeh.ho@gmail.com>
> Date: Fri, 5 Jun 2020 18:53:40 +0430
> Cc: Eli Zaretskii <eliz@gnu.org>, 41005@debbugs.gnu.org, 
> 	Nicholas Drozd <nicholasdrozd@gmail.com>
> 
> I tested  Pip Cet's patch on Emacs 27.0.91 and everything was fine.
> In both cases (auto-fill-mode & column-number-mode) There was no
> confusion in the letters; and all the words were displayed correctly.

Great, thanks for testing it.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: hossein valizadeh <valizadeh.ho@gmail.com>,  41005@debbugs.gnu.org,
>>   nicholasdrozd@gmail.com
>> Date: Fri, 05 Jun 2020 13:05:43 +0000
>> 
>> I think the attached patch is a fairly minimal fix; it's against master,
>> applies to emacs-27 but I haven't tested it there.
>
> Thanks, it LGTM.
>
> I think we should put this on emacs-27, because this is a regression
> caused by Emacs 27's support for HarfBuzz as the default shaping
> engine.  The other shapers didn't want us to provide the direction,
> they determined it internally.  We added the DIRECTION argument as
> part of integrating HarfBuzz.

Okay, will do that unless someone objects.

> We could do better than your patch by actually computing the resolved
> bidi level there, which would require start_display followed by
> move_it_to, in which case we probably won't need to call
> composition_reseat_it by hand at all, and could just pick up the
> result produced by move_it_to.  Or maybe we should just use
> Fvertical_motion instead (which does all that internally).  But these
> ideas are for the master branch, not for emacs-27.

That sounds good.

>> Given these two bugs, I wonder whether it wouldn't be more reasonable
>> always to let HarfBuzz guess the direction, at least for Emacs-27:
>> scripts which change direction, if they are supported by HarfBuzz, won't
>> work anyway.
>
> Please explain "scripts that change direction" and "won't work
> anyway", I don't think I understand that part.

I think your example (RLO..PDF in RTL text) is better: that won't work
anyway, right now, because if, for example, you type

<HEBREW LETTER SHIN> <RIGHT-TO-LEFT OVERRIDE> f i

and have set the char table to treat "fi" as a ligature, the result will
(at least sometimes) be an "fi" ligature, but it should look like the
word "if".

> The reason we don't let HarfBuzz guess in all cases is because the
> resolved bidi level, when we have it, is a more accurate indication of
> the required direction.

Yes, but we'll still cache the wrong direction. If we let HarfBuzz guess
in all cases, output will be consistent and usually correct

> For example, if you have RTL characters
> inside the LRO..PDF embedding, it would be wrong to let the shaper
> guess, because it could (and usually will) guess wrongly that the
> direction is R2L.  It is true that these are rare and unusual use
> cases, but they do exist, and Emacs does want to support them,
> including with scripts that must use the shaping engine.

As I described, I don't think RLO..PDF works with shaping right now,
because other code might have already cached the non-overridden glyph
string.

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
> Date: Sat, 06 Jun 2020 08:38:39 +0000
> 
> >> Given these two bugs, I wonder whether it wouldn't be more reasonable
> >> always to let HarfBuzz guess the direction, at least for Emacs-27:
> >> scripts which change direction, if they are supported by HarfBuzz, won't
> >> work anyway.
> >
> > Please explain "scripts that change direction" and "won't work
> > anyway", I don't think I understand that part.
> 
> I think your example (RLO..PDF in RTL text) is better: that won't work
> anyway, right now, because if, for example, you type
> 
> <HEBREW LETTER SHIN> <RIGHT-TO-LEFT OVERRIDE> f i
> 
> and have set the char table to treat "fi" as a ligature, the result will
> (at least sometimes) be an "fi" ligature, but it should look like the
> word "if".

That's not how shaping engines work, at least not how HarfBuzz does
AFAIU.  It gets the characters in the logical order, so it always
wants to see "fi", even if the directionality of the characters was
overridden, and it also wants to know the local text directionality.
What is produced from that depends on the font: if it has different
ligatures for "fi" in different directions, then HarfBuzz should give
us back the ligature appropriate for the direction it was passed.

(Personally, I think that when some text uses a directional override,
they don't intend to see ligatures, because the override is mostly for
treating characters as independent of the surrounding context.  But
this is eventually up to the font to specify.  AFAIU, Arabic shaping
works differently in different directional contexts, for example.)

> > The reason we don't let HarfBuzz guess in all cases is because the
> > resolved bidi level, when we have it, is a more accurate indication of
> > the required direction.
> 
> Yes, but we'll still cache the wrong direction.

Why "wrong"?  We will cache the same direction as we passed to
HarfBuzz, and thus the produced glyphs will be consistent with the
cached direction.  And if we ever need to display the same sequence of
characters with a different direction, the cached sequence will fail
to match, and we will call HarfBuzz again to produce glyphs for this
other direction.  That sounds TRT to me.

> If we let HarfBuzz guess in all cases, output will be consistent and
> usually correct

We want the direction to be _always_ correct, not just "usually".  The
shapers we used before HarfBuzz didn't allow to pass the direction,
they always guessed it.  HarfBuzz lets us specify the direction, which
is progress, since Emacs now has better control on the glyphs that are
produced, and HarfBuzz developers tell us the difference sometimes
matters.

> > For example, if you have RTL characters
> > inside the LRO..PDF embedding, it would be wrong to let the shaper
> > guess, because it could (and usually will) guess wrongly that the
> > direction is R2L.  It is true that these are rare and unusual use
> > cases, but they do exist, and Emacs does want to support them,
> > including with scripts that must use the shaping engine.
> 
> As I described, I don't think RLO..PDF works with shaping right now,
> because other code might have already cached the non-overridden glyph
> string.

I was saying that under the assumption that the direction will be
cached.  You are right that currently this doesn't work correctly, but
that's exactly why we agreed to cache the direction with the other
composition information.  Once the caching of direction is
implemented, my point is that passing the direction to HarfBuzz and
caching it will produce better results for text in a directional
override than if we let HarfBuzz guess the direction.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
>> Date: Sat, 06 Jun 2020 08:38:39 +0000
>> 
>> >> Given these two bugs, I wonder whether it wouldn't be more reasonable
>> >> always to let HarfBuzz guess the direction, at least for Emacs-27:
                                                  ~~~~~~~~~~~~~~~~~~~~~

I should have been clearer here and said I was only concerned with Emacs 27.

>> >> scripts which change direction, if they are supported by HarfBuzz, won't
>> >> work anyway.
>> >
>> > Please explain "scripts that change direction" and "won't work
>> > anyway", I don't think I understand that part.
>> 
>> I think your example (RLO..PDF in RTL text) is better: that won't work
>> anyway, right now, because if, for example, you type
>> 
>> <HEBREW LETTER SHIN> <RIGHT-TO-LEFT OVERRIDE> f i
>> 
>> and have set the char table to treat "fi" as a ligature, the result will
>> (at least sometimes) be an "fi" ligature, but it should look like the
>> word "if".
>
> That's not how shaping engines work, at least not how HarfBuzz does
> AFAIU.  It gets the characters in the logical order, so it always
> wants to see "fi", even if the directionality of the characters was
> overridden, and it also wants to know the local text directionality.
> What is produced from that depends on the font: if it has different
> ligatures for "fi" in different directions, then HarfBuzz should give
> us back the ligature appropriate for the direction it was passed.
>
> (Personally, I think that when some text uses a directional override,
> they don't intend to see ligatures, because the override is mostly for
> treating characters as independent of the surrounding context.  But
> this is eventually up to the font to specify.  AFAIU, Arabic shaping
> works differently in different directional contexts, for example.)
>
>> > The reason we don't let HarfBuzz guess in all cases is because the
>> > resolved bidi level, when we have it, is a more accurate indication of
>> > the required direction.
>> 
>> Yes, but we'll still cache the wrong direction.
>
> Why "wrong"?  We will cache the same direction as we passed to
> HarfBuzz, and thus the produced glyphs will be consistent with the
> cached direction.  And if we ever need to display the same sequence of
> characters with a different direction, the cached sequence will fail
> to match, and we will call HarfBuzz again to produce glyphs for this
> other direction.  That sounds TRT to me.

You're absolutely correct, sorry for wasting so much of your time with
this: caching directions is the right thing, I was just concerned about
what to do in Emacs 27 where AIUI we don't want to cache directions...

> Once the caching of direction is
> implemented, my point is that passing the direction to HarfBuzz and
> caching it will produce better results for text in a directional
> override than if we let HarfBuzz guess the direction.

Again, I agree. Sorry for the misunderstanding.

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
> Date: Sat, 06 Jun 2020 09:11:36 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Pip Cet <pipcet@gmail.com>
> >> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
> >> Date: Sat, 06 Jun 2020 08:38:39 +0000
> >> 
> >> >> Given these two bugs, I wonder whether it wouldn't be more reasonable
> >> >> always to let HarfBuzz guess the direction, at least for Emacs-27:
>                                                   ~~~~~~~~~~~~~~~~~~~~~
> 
> I should have been clearer here and said I was only concerned with Emacs 27.

Ah, okay.  That settles some of the misunderstanding.

But even in emacs-27, I think passing the right direction where we
know it is better.  For example, the use case with directional
override is only problematic in emacs-27 if the following conditions
are both true:

  . the same sequence of characters is used elsewhere, but without the
    override
  . the font glyphs produced by the shaper are different for different
    directions

The first is somewhat likely to happen, but the second happens only
for some specific scripts, such as Arabic (again, if we consider the
scope of what is supported well by Emacs 27, which, for example,
excludes ligatures).

And the use case with directional overrides is itself very rare.  The
more frequent use cases which hit on this deficiency in emacs-27 are
those which you just fixed on emacs-27, and the fix is indeed to let
HarfBuzz guess.

bug#41005: problem with rendering Persian text in Emacs 27

[Pip Cet]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pip Cet <pipcet@gmail.com>
>> Cc: valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,  nicholasdrozd@gmail.com
>> Date: Sat, 06 Jun 2020 09:11:36 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Pip Cet <pipcet@gmail.com>
>> >> Cc: valizadeh.ho@gmail.com, 41005@debbugs.gnu.org,
>> >> nicholasdrozd@gmail.com
>> >> Date: Sat, 06 Jun 2020 08:38:39 +0000
>> >> 
>> >> >> Given these two bugs, I wonder whether it wouldn't be more reasonable
>> >> >> always to let HarfBuzz guess the direction, at least for Emacs-27:
>>                                                   ~~~~~~~~~~~~~~~~~~~~~
>> 
>> I should have been clearer here and said I was only concerned with Emacs 27.
>
> Ah, okay.  That settles some of the misunderstanding.
>
> But even in emacs-27, I think passing the right direction where we
> know it is better.  For example, the use case with directional
> override is only problematic in emacs-27 if the following conditions
> are both true:
>
>   . the same sequence of characters is used elsewhere, but without the
>     override
>   . the font glyphs produced by the shaper are different for different
>     directions

Thanks, you've convinced me.  Let's do it like that.

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

Hi Eli, Pip,

Thank you for working on this and fixing it on emacs-27 per the reports.

I have not been able to keep a close eye on GNU lists and trackers for a
while now, but I was wondering if there's been any more progress/updates
on this issue on master?  I sometimes read/write in Persian, and this
bug is quite an annoyance. :-)  Changing the text scale (C-x C-=) and/or
filling the paragraph (M-q) seems to sometimes help, but not always; and
the issue remains.

Thanks again for your work on this.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Robert Pluim]

>>>>> On Tue, 21 Jul 2020 08:40:56 -0400, Amin Bandali <bandali@gnu.org> said:

    Amin> Hi Eli, Pip,
    Amin> Thank you for working on this and fixing it on emacs-27 per the reports.

    Amin> I have not been able to keep a close eye on GNU lists and trackers for a
    Amin> while now, but I was wondering if there's been any more progress/updates
    Amin> on this issue on master?  I sometimes read/write in Persian, and this
    Amin> bug is quite an annoyance. :-)  Changing the text scale (C-x C-=) and/or
    Amin> filling the paragraph (M-q) seems to sometimes help, but not always; and
    Amin> the issue remains.

emacs-27 gets merged to master on a regular basis, and the relevant
commit is present in both:

$ git branch --contains 30a7ee505a
* emacs-27
+ master

Are you still seeing issues with the latest master?

Robert

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 21 Jul 2020 08:40:56 -0400, Amin Bandali <bandali@gnu.org> said:
>
>     Amin> Hi Eli, Pip,
>     Amin> Thank you for working on this and fixing it on emacs-27 per the reports.
>
>     Amin> I have not been able to keep a close eye on GNU lists and trackers for a
>     Amin> while now, but I was wondering if there's been any more progress/updates
>     Amin> on this issue on master?  I sometimes read/write in Persian, and this
>     Amin> bug is quite an annoyance. :-)  Changing the text scale (C-x C-=) and/or
>     Amin> filling the paragraph (M-q) seems to sometimes help, but not always; and
>     Amin> the issue remains.
>
> emacs-27 gets merged to master on a regular basis, and the relevant
> commit is present in both:
>
> $ git branch --contains 30a7ee505a
> * emacs-27
> + master
>
> Are you still seeing issues with the latest master?
>
> Robert
>

Ah, great point!  This got me suspicious, so I went ahead and actually
tried both emacs-27 and master in a few different configurations myself.

It seems that with emacs-27, Xft rendering of Arabic/Persian text works
just fine, like it used to: <https://p.bndl.org/persian-emacs-xft.png>.

However, using Cairo+HarfBuzz, on both emacs-27 and master the issue is
still present: <https://p.bndl.org/persian-emacs-ftcrhb.png>.

Excerpts from M-x describe-char RET on a Persian character from Xft and
Cairo+HarfBuzz are at <https://p.bndl.org/persian-emacs-xft.txt> and
<https://p.bndl.org/persian-emacs-ftcrhb.txt> respectively, in case that
might be useful somehow.  The font used for typesetting the Persian text
is Vazir v22.1.0, available from
<https://github.com/rastikerdar/vazir-font/releases/download/v22.1.0/vazir-font-v22.1.0.zip>.

My .Xresources configuration for the Xft and Cairo+HarfBuzz setups, the
former only for emacs-27, and the latter for both emacs-27 and master:

Emacs.FontBackend: xft,x   
Emacs.font: Source Code Pro Medium:size=14

Emacs.FontBackend: ftcrhb,x
Emacs.font: Source Code Pro Medium:size=14

Lastly, it might be worth mentioning that if I recall correctly, when
using xfthb with emacs-27, I observe the same issue.  Which may suggest
that perhaps the issue is related to Emacs's HarfBuzz support.

Hope this is useful.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  valizadeh.ho@gmail.com,
>   41005@debbugs.gnu.org,  Pip Cet <pipcet@gmail.com>
> Date: Tue, 21 Jul 2020 13:53:25 -0400
> 
> Lastly, it might be worth mentioning that if I recall correctly, when
> using xfthb with emacs-27, I observe the same issue.  Which may suggest
> that perhaps the issue is related to Emacs's HarfBuzz support.

But that's exactly the configuration that was fixed...

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Amin Bandali <bandali@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  valizadeh.ho@gmail.com,
>>   41005@debbugs.gnu.org,  Pip Cet <pipcet@gmail.com>
>> Date: Tue, 21 Jul 2020 13:53:25 -0400
>> 
>> Lastly, it might be worth mentioning that if I recall correctly, when
>> using xfthb with emacs-27, I observe the same issue.  Which may suggest
>> that perhaps the issue is related to Emacs's HarfBuzz support.
>
> But that's exactly the configuration that was fixed...
>

It is strange.  I did some more testing.  Whether with xfthb (emacs-27)
or ftcrhb (master), it seems like typing in Persian in *scratch* works
okay.  However, if I paste (yank) Persian text, e.g. from Wikipedia,
into *scratch*, the issue surfaces and yanked text is garbled.  Further,
Persian text in Gnus's article-mode and in message-mode is always
garbled to begin with.  It does seem like a HarfBuzz issue to me.
Please feel free to grab the Vazir font and test for yourself.

Hossein, do you also get this behaviour with Emacs's HarfBuzz?

Any other users using Emacs for Persian or Arabic?  I wonder if this
issue happens for other right-to-left languages, e.g. Hebrew.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>   pipcet@gmail.com
> Date: Tue, 21 Jul 2020 22:12:19 -0400
> 
> >> Lastly, it might be worth mentioning that if I recall correctly, when
> >> using xfthb with emacs-27, I observe the same issue.  Which may suggest
> >> that perhaps the issue is related to Emacs's HarfBuzz support.
> >
> > But that's exactly the configuration that was fixed...
> >
> 
> It is strange.  I did some more testing.  Whether with xfthb (emacs-27)
> or ftcrhb (master), it seems like typing in Persian in *scratch* works
> okay.  However, if I paste (yank) Persian text, e.g. from Wikipedia,
> into *scratch*, the issue surfaces and yanked text is garbled.  Further,
> Persian text in Gnus's article-mode and in message-mode is always
> garbled to begin with.  It does seem like a HarfBuzz issue to me.
> Please feel free to grab the Vazir font and test for yourself.

Does this happen only with that font, or does it happen with any font
that supports Persian?

If it's only that font, then I don't think we should try solving this
in Emacs; please report this to the font developers.

If the problem happens with any Persian-supporting font, then please
tell the details: which page you copy/paste from, what browser did you
use to copy that text, detailed steps for how to reproduce in Gnus,
etc.  This bug report described a problem that happened in different
situation (the Emacs EWW browser), so it's hard to know what exactly
happens in your case.

Thanks.

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 2801 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Amin Bandali <bandali@gnu.org>
>> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>>   pipcet@gmail.com
>> Date: Tue, 21 Jul 2020 22:12:19 -0400
>> 
>> >> Lastly, it might be worth mentioning that if I recall correctly, when
>> >> using xfthb with emacs-27, I observe the same issue.  Which may suggest
>> >> that perhaps the issue is related to Emacs's HarfBuzz support.
>> >
>> > But that's exactly the configuration that was fixed...
>> >
>> 
>> It is strange.  I did some more testing.  Whether with xfthb (emacs-27)
>> or ftcrhb (master), it seems like typing in Persian in *scratch* works
>> okay.  However, if I paste (yank) Persian text, e.g. from Wikipedia,
>> into *scratch*, the issue surfaces and yanked text is garbled.  Further,
>> Persian text in Gnus's article-mode and in message-mode is always
>> garbled to begin with.  It does seem like a HarfBuzz issue to me.
>> Please feel free to grab the Vazir font and test for yourself.
>
> Does this happen only with that font, or does it happen with any font
> that supports Persian?
>
> If it's only that font, then I don't think we should try solving this
> in Emacs; please report this to the font developers.
>

It appears to happen with any font supporting Persian/Arabic.  Examples
include DejaVu Sans and Noto Sans Arabic.

>
> If the problem happens with any Persian-supporting font, then please
> tell the details: which page you copy/paste from, what browser did you
> use to copy that text, detailed steps for how to reproduce in Gnus,
> etc.  This bug report described a problem that happened in different
> situation (the Emacs EWW browser), so it's hard to know what exactly
> happens in your case.
>
> Thanks.
>

Examples of pages I copied excerpts from include the front page of the
Persian Wikipedia <https://fa.wikipedia.org>, as well as the Persian
translation of GNU's homepage <https://www.gnu.org/home.fa.html>.  The
issue does not seem to be specific to a particular text, and appears to
occur when pasting any Persian text into a buffer.  As for the browser,
I tried with both GNU IceCat and Emacs's EWW; same results.  I am able
to reproduce by pasting any random Persian text into any Emacs buffer.
In the case of Gnus, simply pressing RET on the subject of an email in
gnus-summary-mode to have it displayed using gnus-article-mode shows the
Persian text in the email body as garbled.

Hope this helps.

                                 * * *

Actually, as I was about to hit send, it /just/ occurred to me to try
with -Q or -q, and in both cases I do not see this bug!  How strange!
I'll start bisecting my Emacs configuration and will report back if I
manage to find the cause.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>   pipcet@gmail.com
> Date: Fri, 24 Jul 2020 00:11:30 -0400
> 
> Actually, as I was about to hit send, it /just/ occurred to me to try
> with -Q or -q, and in both cases I do not see this bug!  How strange!
> I'll start bisecting my Emacs configuration and will report back if I
> manage to find the cause.

Yes, that would be most useful.

If the result still points into some valid Emacs use pattern, please
describe in more detail a small part of the Wikipedia page that needs
to be copy/pasted, and the precise place where the result of pasting
is rendered incorrectly.  Please keep in mind that for people who
don't read Persian it is hard to find those issues in vast amounts of
text, so any measure that makes this significantly easier will allow
debugging and finding solutions to be much more efficient.

Thanks.

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

[...]
>
> Yes, that would be most useful.
>
> If the result still points into some valid Emacs use pattern, please
> describe in more detail a small part of the Wikipedia page that needs
> to be copy/pasted, and the precise place where the result of pasting
> is rendered incorrectly.  Please keep in mind that for people who
> don't read Persian it is hard to find those issues in vast amounts of
> text, so any measure that makes this significantly easier will allow
> debugging and finding solutions to be much more efficient.
>
> Thanks.
>

Having done some digging, I've found at least one reliable way to
reproduce the issue with -Q:

1. launch Emacs using `emacs -Q';
2. switch to *scratch* if not there already;
3. do M-x column-number-mode RET;
4. paste the following Persian text into *scratch*:

   ویکی‌پدیا دانشنامه‌ای اینترنتی با بیش از ۲۸۰ زبان با محتوای آزاد است که با همکاری افراد داوطلب نوشته می‌شود و هر کس که به اینترنت دسترسی داشته باشد می‌تواند مقاله‌های آن را ویرایش کند.

   It is from the first sentence of the "دربارهٔ ویکی‌پدیا" (About
   Wikipedia) section on <https://fa.wikipedia.org>.

   The text should appear garbled.  However, if you omit step 3, the
   pasted text should appear just fine.  It appears that if Persian text
   is pasted before column-number-mode is enabled, even if the paste is
   subsequently undone before enabling column-number-mode, the issue
   does *not* surface.

The above works for pasting into a message-mode buffer as well.  I have
not tried to find a recipe specific to gnus-article-mode yet.

Hope this helps.

P.S. if I did not mention earlier, all of this is on a Debian Buster
GNU/Linux system.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>   pipcet@gmail.com
> Date: Sat, 25 Jul 2020 00:19:35 -0400
> 
> > If the result still points into some valid Emacs use pattern, please
> > describe in more detail a small part of the Wikipedia page that needs
> > to be copy/pasted, and the precise place where the result of pasting
> > is rendered incorrectly.  Please keep in mind that for people who
> > don't read Persian it is hard to find those issues in vast amounts of
> > text, so any measure that makes this significantly easier will allow
> > debugging and finding solutions to be much more efficient.
> >
> > Thanks.
> >
> 
> Having done some digging, I've found at least one reliable way to
> reproduce the issue with -Q:
> 
> 1. launch Emacs using `emacs -Q';
> 2. switch to *scratch* if not there already;
> 3. do M-x column-number-mode RET;
> 4. paste the following Persian text into *scratch*:
 
>    ویکی‌پدیا دانشنامه‌ای اینترنتی با بیش از ۲۸۰ زبان با محتوای آزاد است که با همکاری افراد داوطلب نوشته می‌شود و هر کس که به اینترنت دسترسی داشته باشد می‌تواند مقاله‌های آن را ویرایش کند.
 
>    It is from the first sentence of the "دربارهٔ ویکی‌پدیا" (About
>    Wikipedia) section on <https://fa.wikipedia.org>.
> 
>    The text should appear garbled.  However, if you omit step 3, the
>    pasted text should appear just fine.  It appears that if Persian text
>    is pasted before column-number-mode is enabled, even if the paste is
>    subsequently undone before enabling column-number-mode, the issue
>    does *not* surface.

Please help me understand what exactly do you mean by "garbled".  The
text you show is still quite long, and I cannot easily locate it in
that page (I don't see any "About Wikipedia" in the English version of
that page), nor do I understand what you mean by "garbled".

Would it be possible instead to paste only a very small portion of the
text, and tell exactly which part(s) of that short text are garbled,
and how they are garbled?  Can you, for example, post a screenshot
showing exactly which part of the Wikipedia page should be copied, and
another screenshot of the garbled text in Emacs showing which part(s)
are displayed incorrectly?  And please keep the pasted text as short
as possible, because locating the garbled part(s) in text I cannot
read which is displayed in a different font from what's in the
screenshot can be a very frustrating and error-prone experience.

Also, do you copy this from EWW or from some other Web browser?

(I tried to explain all of this in my previous message, but
unfortunately this reproduction recipe again presents the same
difficulties as I tried to avoid by explaining how to provide
information that would be easy to follow up.)

Please help me understand the problem, without that I see no way of
making any progress here.

Thanks.

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 3673 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

[...]
>
> Please help me understand what exactly do you mean by "garbled".  The
> text you show is still quite long, and I cannot easily locate it in
> that page (I don't see any "About Wikipedia" in the English version of
> that page), nor do I understand what you mean by "garbled".
>

Choosing the Persian Wikipedia was probably not the best idea, given how
busy their page is and how much material is on there, which does not
help at all when trying to find an excerpt of text in a language one
doesn't know.

Also, sorry for not being clear about what I mean by "garbled" text.
Please see <https://en.wikipedia.org/wiki/Persian_alphabet#Letters>.
In Persian alphabet, letters take different forms depending on where in
a word they appear.  The "overview table" in the above page includes
examples of the possible contextual forms for each letter.  The issue
described in this bug report is basically about Emacs using the wrong
contextual form of letters when rendering Persian text.

>
> Would it be possible instead to paste only a very small portion of the
> text, and tell exactly which part(s) of that short text are garbled,
> and how they are garbled?  Can you, for example, post a screenshot
> showing exactly which part of the Wikipedia page should be copied, and
> another screenshot of the garbled text in Emacs showing which part(s)
> are displayed incorrectly?  And please keep the pasted text as short
> as possible, because locating the garbled part(s) in text I cannot
> read which is displayed in a different font from what's in the
> screenshot can be a very frustrating and error-prone experience.
>
> Also, do you copy this from EWW or from some other Web browser?
>

Certainly.  Instead of Persian Wikipedia, let's use the Persian
translation of the GNU homepage: <https://www.gnu.org/home.fa.html>;
specifically, the first part of the first sentence of the first
paragraph (up to and including the semicolon):

گنو یک سیستم‌عامل بر مبنای نرم‌افزار آزاد است؛

I have created a very short video screencast of me walking through
reproducing the issue, by opening <https://www.gnu.org/home.fa.html> in
Debian Buster's firefox-esr (68.10.0esr (64-bit)), copying the above
excerpt from the page, and pasting into an Emacs *scratch* using C-y:

    https://p.bndl.org/emacs-persian-wrong-contextual-forms.webm

The two open Emacs instances were both launched with "emacs -Q" and are
identical; except for the second one, I did M-x column-number-mode RET
before pasting the Persian text.

I also meant to include the following in my video in case they might be
useful, but forgot to.

,----[ M-x emacs-version RET ]
| GNU Emacs 28.0.50 (build 8, x86_64-pc-linux-gnu, X toolkit, cairo
| version 1.16.0, Xaw3d scroll bars) of 2020-07-20
`----

,----[ C-h v system-configuration-options RET ]
| "--with-modules --without-gconf --without-gsettings
| --with-x-toolkit=lucid --with-xft --with-xaw3d --without-gpm
| --with-imagemagick --with-harfbuzz --prefix=/data/bandali/usr/local"
`----

>
> (I tried to explain all of this in my previous message, but
> unfortunately this reproduction recipe again presents the same
> difficulties as I tried to avoid by explaining how to provide
> information that would be easy to follow up.)
>
> Please help me understand the problem, without that I see no way of
> making any progress here.
>
> Thanks.
>

I really am trying. :-) Thank you for baring with me here, and for
trying to help find the issue, Eli; I appreciate it.

Hope this helps.

Thanks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>   pipcet@gmail.com
> Date: Sat, 25 Jul 2020 11:53:19 -0400
> 
> > Also, do you copy this from EWW or from some other Web browser?
> >
> 
> Certainly.  Instead of Persian Wikipedia, let's use the Persian
> translation of the GNU homepage: <https://www.gnu.org/home.fa.html>;
> specifically, the first part of the first sentence of the first
> paragraph (up to and including the semicolon):
> 
> گنو یک سیستم‌عامل بر مبنای نرم‌افزار آزاد است؛
> 
> I have created a very short video screencast of me walking through
> reproducing the issue, by opening <https://www.gnu.org/home.fa.html> in
> Debian Buster's firefox-esr (68.10.0esr (64-bit)), copying the above
> excerpt from the page, and pasting into an Emacs *scratch* using C-y:
> 
>     https://p.bndl.org/emacs-persian-wrong-contextual-forms.webm
> 
> The two open Emacs instances were both launched with "emacs -Q" and are
> identical; except for the second one, I did M-x column-number-mode RET
> before pasting the Persian text.

OK, thanks.  Now everything is clear: at the time we discussed this,
almost 2 months ago, Pip Cet proposed a patch, which I asked him to
install.  But it turns out it was never actually installed, and so
this particular situation was indeed not fixed.

I have now installed that patch on the emacs-27 branch, and the
problem is gone.

bug#41005: problem with rendering Persian text in Emacs 27

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

[...]
>
> OK, thanks.  Now everything is clear: at the time we discussed this,
> almost 2 months ago, Pip Cet proposed a patch, which I asked him to
> install.  But it turns out it was never actually installed, and so
> this particular situation was indeed not fixed.
>

Oh, I see!

>
> I have now installed that patch on the emacs-27 branch, and the
> problem is gone.
>

I tried cherry picking them onto master in my local checkout, and they
do seem to fix the issue!  Looking forward to them eventually getting
properly merged into master using the script.  Thanks again for your
help and patience, Eli.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#41005: problem with rendering Persian text in Emacs 27

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Cc: rpluim@gmail.com,  valizadeh.ho@gmail.com,  41005@debbugs.gnu.org,
>   pipcet@gmail.com
> Date: Sat, 25 Jul 2020 12:44:36 -0400
> 
> I tried cherry picking them onto master in my local checkout, and they
> do seem to fix the issue!  Looking forward to them eventually getting
> properly merged into master using the script.  Thanks again for your
> help and patience, Eli.

Thanks, so I think we can finally close this bug.