bug#36030: Changing the region or the point changes the way characters are displayed

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

[-- Attachment #1: Type: text/plain, Size: 4790 bytes --]

Hi all,

I'm seeing puzzling display changes when I change the selected region.  I have attached pictures.

This is the recipe I use:
* emacs -Q
* C-h h
* M-g c 767 (or go to the beginning of 'Здравствуйте')
* C-SPC C-f C-f C-f C-f C-f C-f C-f

As I press C-f to extend the region, the display of the strings on the 'Arabic (العربيّة)' line changes.
I see this problem fairly commonly when looking at non-ASCII text in Emacs; sometimes just moving the point around is enough.

The problem is the same with xft and xfthb. In fact, if I have one xfthb frame and one xft frame both displaying HELLO, changing the region in one alters the display of the characters in both.

Here is the output of C-u x =

---
             position: 874 of 3518 (25%), column: 14
            character: ة‎ (displayed as ة‎) (codepoint 1577, #o3051, #x629)
              charset: mule-unicode-0100-24ff (Unicode characters of the range U+0100..U+24FF.)
code point in charset: 0x2D69
               script: arabic
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "C-x 8 RET 629" or "C-x 8 RET ARABIC LETTER TEH MARBUTA"
          buffer code: #xD8 #xA9
            file code: #xD8 #xA9 (encoded by coding system utf-8-unix)
              display: composed to form "ة" (see below)

Composed using this font:
  xft:-kcst-KacstScreen-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1
by these glyphs:
  [7 7 0 102 7 0 8 11 1 nil]

Character code properties: customize what to show
  name: ARABIC LETTER TEH MARBUTA
  old-name: ARABIC LETTER TAA MARBUTAH
  general-category: Lo (Letter, Other)
  decomposition: (1577) ('ة')

There are text properties here:
  charset              mule-unicode-0100-24ff
---

And here is some debug information:

---
Configured using:
 'configure -C --with-harfbuzz'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD JSON PDUMPER
LCMS2 GMP

Important settings:
  value of $LC_MONETARY: en_DK.UTF-8
  value of $LC_NUMERIC: en_DK.UTF-8
  value of $LC_TIME: sv_SE.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  enriched-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  use-hard-newlines: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  view-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils pp wid-edit descr-text
help-mode cl-loaddefs cl-lib time-date thai-util thai-word lao-util
vc-git diff-mode easymenu easy-mmode disp-table enriched view elec-pair
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 148491 17252)
 (symbols 48 19615 1)
 (strings 32 78745 2506)
 (string-bytes 1 1949992)
 (vectors 16 20937)
 (vector-slots 8 1267056 145604)
 (floats 8 32 105)
 (intervals 56 479 3)
 (buffers 992 13))
---

I hope this is not a duplicate. I have a vague memory of a related bug, but I couldn't find it.

Clément.

[-- Attachment #2: display-bug-0.png --]
[-- Type: image/png, Size: 37450 bytes --]

[-- Attachment #3: display-bug-1.png --]
[-- Type: image/png, Size: 36318 bytes --]

[-- Attachment #4: display-bug-2.png --]
[-- Type: image/png, Size: 36479 bytes --]

[-- Attachment #5: display-bug-3.png --]
[-- Type: image/png, Size: 36557 bytes --]

[-- Attachment #6: display-bug-4.png --]
[-- Type: image/png, Size: 36749 bytes --]

[-- Attachment #7: display-bug-5.png --]
[-- Type: image/png, Size: 36685 bytes --]

[-- Attachment #8: display-bug-6.png --]
[-- Type: image/png, Size: 36844 bytes --]

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 31 May 2019 11:21:51 -0400
> 
> * emacs -Q
> * C-h h
> * M-g c 767 (or go to the beginning of 'Здравствуйте')
> * C-SPC C-f C-f C-f C-f C-f C-f C-f
> 
> As I press C-f to extend the region, the display of the strings on the 'Arabic (العربيّة)' line changes.
> I see this problem fairly commonly when looking at non-ASCII text in Emacs; sometimes just moving the point around is enough.

This is a duplicate of bug#35811, AFAIU.

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-05-31 11:36, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 31 May 2019 11:21:51 -0400
>>
>> * emacs -Q
>> * C-h h
>> * M-g c 767 (or go to the beginning of 'Здравствуйте')
>> * C-SPC C-f C-f C-f C-f C-f C-f C-f
>>
>> As I press C-f to extend the region, the display of the strings on the 'Arabic (العربيّة)' line changes.
>> I see this problem fairly commonly when looking at non-ASCII text in Emacs; sometimes just moving the point around is enough.
> 
> This is a duplicate of bug#35811, AFAIU.

Agh, sorry for the noise.  I should note, though, that the problem isn't specific to Arabic: I see it with many other scripts.

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 31 May 2019 13:06:32 -0400
> 
> > This is a duplicate of bug#35811, AFAIU.
> 
> Agh, sorry for the noise.  I should note, though, that the problem isn't specific to Arabic: I see it with many other scripts.

Which ones?  I only see this on the Arabic line, at least in "emacs -Q".

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-05-31 15:02, Eli Zaretskii wrote:
>> Cc: 36030@debbugs.gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 31 May 2019 13:06:32 -0400
>>
>>> This is a duplicate of bug#35811, AFAIU.
>>
>> Agh, sorry for the noise.  I should note, though, that the problem isn't specific to Arabic: I see it with many other scripts.
>
> Which ones?  I only see this on the Arabic line, at least in "emacs -Q".

I see it with Hebrew as well, but not in emacs -Q.  With this string:

  'ՙա-ևא-תװ-ײؠ-يٮ-ٯٱ-ۓەۥ-ۦۮ-ۯۺ-ۼۿܐܒ-ܯݍ-ޥޱߊ-ߪߴ-ߵߺࠀ-ࠕࠚࠤࠨࡀ-ࡘࢠ-ࢴࢶ'

(taken from a regular expression)

  if I place the point at the beginning of the line, I see this:

             position: 21767 of 29526 (74%), column: 25
            character: ת‎ (displayed as ת‎) (codepoint 1514, #o2752, #x5ea)
              charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
code point in charset: 0x05EA
               script: hebrew
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong)
             to input: type "C-x 8 RET 5ea" or "C-x 8 RET HEBREW LETTER TAV"
          buffer code: #xD7 #xAA
            file code: #xD7 #xAA (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-GNU -FreeSerif-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x587)

  Character code properties: customize what to show
    name: HEBREW LETTER TAV
    general-category: Lo (Letter, Other)
    decomposition: (1514) ('ת')

  There are text properties here:
    face                 font-lock-string-face
    fontified            t

But if I place it at the ܒ, then run (save-excursion (goto-char <position of the beginning of the string>) (what-cursor-position t)) I see xft:-GNU -FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x509) instead.

Moving the point to a different line, the font changes again, this time to xft:-GNU -FreeMono-normal-normal-normal-*-16-*-*-*-m-0-iso10646-1 (#x4FF) 

The rest of the what-cursor-position buffer is unchanged.

With the string 'ਸ-ਹਖ਼-ੜਫ਼ੲ-ੴઅ-ઍએ-ઑઓ-નપ-રલ-ળવ-હઽૐૠ-ૡૹଅ-ଌଏ-ଐଓ-ନପ-ରଲ-ଳଵ-ହଽଡ଼-ଢ଼ୟ-', it is enough to just resize the frame to change from xft:-GNU -FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x663) to xft:-GNU -FreeSerif-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x7F2) (but I had to set a timer to investigate this one, because moving the point reverted the font changes caused by resizing the frame.

Additionally, this seems to be linked to face properties.  In the what-cursor-position buffer, I see this:

  ਸ (displayed as ਸ) (codepoint 2616, #o5070, #xa38)
  ^ using one font.
                  ^ using a different font, with font-lock-string-face on it.

… and moving around the what-cursor-position buffer changes the second one, but not the first one.


Clément.

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 31 May 2019 15:28:46 -0400
> 
> I see it with many other scripts.
> >
> > Which ones?  I only see this on the Arabic line, at least in "emacs -Q".
> 
> I see it with Hebrew as well, but not in emacs -Q.  With this string:
> 
>   'ՙա-ևא-תװ-ײؠ-يٮ-ٯٱ-ۓەۥ-ۦۮ-ۯۺ-ۼۿܐܒ-ܯݍ-ޥޱߊ-ߪߴ-ߵߺࠀ-ࠕࠚࠤࠨࡀ-ࡘࢠ-ࢴࢶ'
> 
> (taken from a regular expression)

So this is not at all in the HELLO buffer?  Please be sure to tell
this important detail in the future: I just spent some time trying to
reproduce the problem in HELLO.

> But if I place it at the ܒ, then run (save-excursion (goto-char <position of the beginning of the string>) (what-cursor-position t)) I see xft:-GNU -FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x509) instead.
> 
> Moving the point to a different line, the font changes again, this time to xft:-GNU -FreeMono-normal-normal-normal-*-16-*-*-*-m-0-iso10646-1 (#x4FF) 

The font doesn't change, just the glyph number reported by Emacs
changes (that's what that hex code in parentheses means).

> The rest of the what-cursor-position buffer is unchanged.
> 
> With the string 'ਸ-ਹਖ਼-ੜਫ਼ੲ-ੴઅ-ઍએ-ઑઓ-નપ-રલ-ળવ-હઽૐૠ-ૡૹଅ-ଌଏ-ଐଓ-ନପ-ରଲ-ଳଵ-ହଽଡ଼-ଢ଼ୟ-', it is enough to just resize the frame to change from xft:-GNU -FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x663) to xft:-GNU -FreeSerif-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x7F2) (but I had to set a timer to investigate this one, because moving the point reverted the font changes caused by resizing the frame.
> 
> Additionally, this seems to be linked to face properties.  In the what-cursor-position buffer, I see this:
> 
>   ਸ (displayed as ਸ) (codepoint 2616, #o5070, #xa38)
>   ^ using one font.
>                   ^ using a different font, with font-lock-string-face on it.
> 
> … and moving around the what-cursor-position buffer changes the second one, but not the first one.

I see none of that in "emacs -Q".  I also don't understand how
resizing the frame could cause these effects.  If the recipe is more
complicated than just copy/paste the strings you show into *scratch*,
please be sure to show the whole recipe.  E.g., where did the face
properties come from?

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-05-31 16:02, Eli Zaretskii wrote:
>> Cc: 36030@debbugs.gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 31 May 2019 15:28:46 -0400
>>
>> I see it with many other scripts.
>>>
>>> Which ones?  I only see this on the Arabic line, at least in "emacs -Q".
>>
>> I see it with Hebrew as well, but not in emacs -Q.  With this string:
>>
>>   'ՙա-ևא-תװ-ײؠ-يٮ-ٯٱ-ۓەۥ-ۦۮ-ۯۺ-ۼۿܐܒ-ܯݍ-ޥޱߊ-ߪߴ-ߵߺࠀ-ࠕࠚࠤࠨࡀ-ࡘࢠ-ࢴࢶ'
>>
>> (taken from a regular expression)
> 
> So this is not at all in the HELLO buffer?  Please be sure to tell
> this important detail in the future: I just spent some time trying to
> reproduce the problem in HELLO.

Sorry about this.  It is in a Python buffer.

>> But if I place it at the ܒ, then run (save-excursion (goto-char <position of the beginning of the string>) (what-cursor-position t)) I see xft:-GNU -FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x509) instead.
>>
>> Moving the point to a different line, the font changes again, this time to xft:-GNU -FreeMono-normal-normal-normal-*-16-*-*-*-m-0-iso10646-1 (#x4FF) 
> 
> The font doesn't change, just the glyph number reported by Emacs
> changes (that's what that hex code in parentheses means).

But it also changes from FreeSans to FreeMono; isn't that a font change?

> I see none of that in "emacs -Q".  I also don't understand how
> resizing the frame could cause these effects.  If the recipe is more
> complicated than just copy/paste the strings you show into *scratch*,
> please be sure to show the whole recipe.  E.g., where did the face
> properties come from?

This is python-mode's syntax highlighting.  Unfortunately, I don't have a reproducible recipe for these issues yet; they happen on my machine, but I have many custom fonts and Emacs customization here.  I will try to find a recipe.

Clément.

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 31 May 2019 16:51:10 -0400
> 
> > The font doesn't change, just the glyph number reported by Emacs
> > changes (that's what that hex code in parentheses means).
> 
> But it also changes from FreeSans to FreeMono; isn't that a font change?

Right, missed that among all the noise, and similar font names.

> > I see none of that in "emacs -Q".  I also don't understand how
> > resizing the frame could cause these effects.  If the recipe is more
> > complicated than just copy/paste the strings you show into *scratch*,
> > please be sure to show the whole recipe.  E.g., where did the face
> > properties come from?
> 
> This is python-mode's syntax highlighting.  Unfortunately, I don't have a reproducible recipe for these issues yet; they happen on my machine, but I have many custom fonts and Emacs customization here.  I will try to find a recipe.

A recipe is very important to solve this issue, because something very
strange happens here.  Are your Python faces specify fonts or font
families, per chance?

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 31 May 2019 16:51:10 -0400
> 
> > I see none of that in "emacs -Q".  I also don't understand how
> > resizing the frame could cause these effects.  If the recipe is more
> > complicated than just copy/paste the strings you show into *scratch*,
> > please be sure to show the whole recipe.  E.g., where did the face
> > properties come from?
> 
> This is python-mode's syntax highlighting.  Unfortunately, I don't have a reproducible recipe for these issues yet; they happen on my machine, but I have many custom fonts and Emacs customization here.  I will try to find a recipe.

Could you please try the latest master and see if these problems are
solved?  If not, could you please try applying the patch I posted in
bug#28312, and see if that fixes your problems?

Thanks.

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-06-06 10:18, Eli Zaretskii wrote:
>> Cc: 36030@debbugs.gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 31 May 2019 16:51:10 -0400
>>
>>> I see none of that in "emacs -Q".  I also don't understand how
>>> resizing the frame could cause these effects.  If the recipe is more
>>> complicated than just copy/paste the strings you show into *scratch*,
>>> please be sure to show the whole recipe.  E.g., where did the face
>>> properties come from?
>>
>> This is python-mode's syntax highlighting.  Unfortunately, I don't have a reproducible recipe for these issues yet; they happen on my machine, but I have many custom fonts and Emacs customization here.  I will try to find a recipe.
> 
> Could you please try the latest master and see if these problems are
> solved?  If not, could you please try applying the patch I posted in
> bug#28312, and see if that fixes your problems?

I think it might fix part of the problem, but I still see some flickering and font changes.
I'm still working on finding a repro.
Thanks a lot for your help.

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 7 Jun 2019 13:45:01 -0400
> 
> > Could you please try the latest master and see if these problems are
> > solved?  If not, could you please try applying the patch I posted in
> > bug#28312, and see if that fixes your problems?
> 
> I think it might fix part of the problem, but I still see some flickering and font changes.

Which of the two changes fixed part of the problem? the current master
or the additional patch in bug#28312?  The latter is not installed on
master.

> I'm still working on finding a repro.

Thanks.

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-06-07 15:50, Eli Zaretskii wrote:
>> Cc: 36030@debbugs.gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 7 Jun 2019 13:45:01 -0400
>>
>>> Could you please try the latest master and see if these problems are
>>> solved?  If not, could you please try applying the patch I posted in
>>> bug#28312, and see if that fixes your problems?
>>
>> I think it might fix part of the problem, but I still see some flickering and font changes.
> 
> Which of the two changes fixed part of the problem? the current master
> or the additional patch in bug#28312?  The latter is not installed on
> master.

The current master, I think.  But please don't give too much credence to my vague hunches (things seem to be better, but I need a good repro to be definitive).

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 7 Jun 2019 17:11:59 -0400
> 
> >> I think it might fix part of the problem, but I still see some flickering and font changes.
> > 
> > Which of the two changes fixed part of the problem? the current master
> > or the additional patch in bug#28312?  The latter is not installed on
> > master.
> 
> The current master, I think.  But please don't give too much credence to my vague hunches (things seem to be better, but I need a good repro to be definitive).

OK, thanks.  If you could afford trying the other patch, I'd be
interested to know whether it has any effect on your problem.

bug#36030: Changing the region or the point changes the way characters are displayed

[Clément Pit-Claudel]

On 2019-06-08 01:54, Eli Zaretskii wrote:
>> Cc: 36030@debbugs.gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 7 Jun 2019 17:11:59 -0400
>>
>>>> I think it might fix part of the problem, but I still see some flickering and font changes.
>>>
>>> Which of the two changes fixed part of the problem? the current master
>>> or the additional patch in bug#28312?  The latter is not installed on
>>> master.
>>
>> The current master, I think.  But please don't give too much credence to my vague hunches (things seem to be better, but I need a good repro to be definitive).
> 
> OK, thanks.  If you could afford trying the other patch, I'd be
> interested to know whether it has any effect on your problem.

I tried it, but I didn't see further improvements.  I think there is a slight performance degradation, but that's subjective too.

bug#36030: Changing the region or the point changes the way characters are displayed

[Eli Zaretskii]

> Cc: 36030@debbugs.gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 10 Jun 2019 16:32:07 -0400
> 
> > OK, thanks.  If you could afford trying the other patch, I'd be
> > interested to know whether it has any effect on your problem.
> 
> I tried it, but I didn't see further improvements.  I think there is a slight performance degradation, but that's subjective too.

Thanks.  I guess this means the problems discussed in this bug report
have different cause(s).