bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Дилян Палаузов]

Hello,

I have a text unix file u.txt where I insert this line:


t.add('🌐︎ ' + 'abcd')

That is a globe U+1F310 followed by Unicode Variation Selector 15 to remove the colour of the globe.

$ hexdump -C  u.txt
00000000  74 2e 61 64 64 28 27 f0  9f 8c 90 ef b8 8e 20 27  |t.add('....... '|
00000010  20 2b 20 27 61 62 63 64  27 29 0a                 | + 'abcd').|
0000001b

I open the file in text mode, emacs-fedora-28.3 -nw, and press the right arrow repeatedly to go over c.  Then I press x.  emacs shows that c is deleted and on its place appears x. I press Ctrl+L to refresh.  Now c appears before x.

I close emacs, repeat the above steps and get consistent results.  I try this on a different system with self-compiled emacs 29.1, same results.

The problem report is that when that character sequence is contained in a text file, modifying more complex lines gets practically impossible with emacs, as the latter stops showing the actual text.  I have to refresh, close, open emacs several times in order to get modifications correctly on the lines, containing the above sequence.  I literally  cannot see what I am changing on such lines, as what is displayed is different to what is actually modified.

Kind regards
  Dilyan

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Eli Zaretskii]

> From: Дилян Палаузов
>  <dilyan.palauzov@aegee.org>
> Date: Sat, 21 Oct 2023 14:07:52 +0200
> 
> Hello,
> 
> I have a text unix file u.txt where I insert this line:
> 
> 
> t.add('🌐︎ ' + 'abcd')
> 
> That is a globe U+1F310 followed by Unicode Variation Selector 15 to remove the colour of the globe.
> 
> $ hexdump -C  u.txt
> 00000000  74 2e 61 64 64 28 27 f0  9f 8c 90 ef b8 8e 20 27  |t.add('....... '|
> 00000010  20 2b 20 27 61 62 63 64  27 29 0a                 | + 'abcd').|
> 0000001b
> 
> I open the file in text mode, emacs-fedora-28.3 -nw, and press the right arrow repeatedly to go over c.  Then I press x.  emacs shows that c is deleted and on its place appears x. I press Ctrl+L to refresh.  Now c appears before x.
> 
> I close emacs, repeat the above steps and get consistent results.  I try this on a different system with self-compiled emacs 29.1, same results.

What is the terminal emulator you are using, and on what OS?

Basically, what the above means is that the text terminal you are
using doesn't support the sequence of U+1F310 followed by VS-15 as
Emacs expects it.  On such a terminal you need to disable
auto-composition-mode to be able to edit such text.

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Дилян Палаузов]

Hello Eli,

thanks for your answer. I am using Gnome Terminal 3.48.1 on Fedora 38, but with MATE Terminal 1.26.1 the same happens.  I tried to follow your advice.  I start emacs -nw u.txt , then M-x auto-composition-mode.  

Emacs changes the displayed text from

t.add('🌐︎ ' + 'abcd')

to 

t.add('🌐   ' + 'abcd')

and prints “Auto-Composition mode disabled in current buffer”.  After the change the globe is coloured in the terminal and there is an underscore after the globe.  I again press several times the right arrow to go over the letter c and then press x. Emacs again deletes (actually hides) the letter c and shows on its place the letter x.

So in this example turning off auto-composition-mode does not help.

I expect that the letter c stays visible.

What terminals and OSes do you recommend, which work good with enabled auto-composition-mode?

Kind regards
  Дилян

-----Original Message-----
From: Eli Zaretskii <eliz@gnu.org>
To: Дилян Палаузов <dilyan.palauzov@aegee.org>
Cc: 66664@debbugs.gnu.org
Subject: Re: bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines
Date: 10/21/2023 02:44:11 PM

> From: Дилян Палаузов
>  <dilyan.palauzov@aegee.org>
> Date: Sat, 21 Oct 2023 14:07:52 +0200
> 
> Hello,
> 
> I have a text unix file u.txt where I insert this line:
> 
> 
> t.add('🌐︎ ' + 'abcd')
> 
> That is a globe U+1F310 followed by Unicode Variation Selector 15 to remove the colour of the globe.
> 
> $ hexdump -C  u.txt
> 00000000  74 2e 61 64 64 28 27 f0  9f 8c 90 ef b8 8e 20 27  |t.add('....... '|
> 00000010  20 2b 20 27 61 62 63 64  27 29 0a                 | + 'abcd').|
> 0000001b
> 
> I open the file in text mode, emacs-fedora-28.3 -nw, and press the right arrow repeatedly to go over c.  Then I press x.  emacs shows that c is deleted and on its place appears x. I press Ctrl+L to refresh.  Now c appears before x.
> 
> I close emacs, repeat the above steps and get consistent results.  I try this on a different system with self-compiled emacs 29.1, same results.

What is the terminal emulator you are using, and on what OS?

Basically, what the above means is that the text terminal you are
using doesn't support the sequence of U+1F310 followed by VS-15 as
Emacs expects it.  On such a terminal you need to disable
auto-composition-mode to be able to edit such text.

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Eli Zaretskii]

> From: Дилян Палаузов
> 	 <dilyan.palauzov@aegee.org>
> Cc: 66664@debbugs.gnu.org
> Date: Sat, 21 Oct 2023 15:12:03 +0200
> 
> thanks for your answer. I am using Gnome Terminal 3.48.1 on Fedora 38, but with MATE Terminal 1.26.1 the same happens.  I tried to follow your advice.  I start emacs -nw u.txt , then M-x auto-composition-mode.  
> 
> Emacs changes the displayed text from
> 
> t.add('🌐︎ ' + 'abcd')
> 
> to 
> 
> t.add('🌐   ' + 'abcd')
> 
> and prints “Auto-Composition mode disabled in current buffer”.  After the change the globe is coloured in the terminal and there is an underscore after the globe.  I again press several times the right arrow to go over the letter c and then press x. Emacs again deletes (actually hides) the letter c and shows on its place the letter x.

Then I cannot reproduce this, nether with nor without
auto-composition-mode.  What happens if you use xterm instead?

Also, does this happen in "emacs -Q -nw"?  And what is your
locale-environment and language-environment?

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Eli Zaretskii]

> Cc: 66664@debbugs.gnu.org
> Date: Sat, 21 Oct 2023 16:36:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > and prints “Auto-Composition mode disabled in current buffer”.  After the change the globe is coloured in the terminal and there is an underscore after the globe.  I again press several times the right arrow to go over the letter c and then press x. Emacs again deletes (actually hides) the letter c and shows on its place the letter x.
> 
> Then I cannot reproduce this, nether with nor without
> auto-composition-mode.

Scratch that, I was mis-reading your description.  I can reproduce,
and I know the reason: Emacs considers the U+1F310 character to take 2
columns on display, whereas your terminal thinks it takes only one
column.  That's why you see these problems.

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Eli Zaretskii]

[Please use Reply All, to keep the bug tracker on the CC list.]

> From: Дилян Палаузов
> 	 <dilyan.palauzov@aegee.org>
> Date: Sat, 21 Oct 2023 17:25:11 +0200
> 
> when I load the file the globe appears below “it” from “file edIT options” menu.  So at least visually I see two columns.
> 
> Now that you can reproduce the problem, what shall I do?

Try

  M-: (set-char-table-range char-width-table '(#x1f310 . #x1f310) 1) RET

But in general, I don't think this problem has a solution, except to
use GUI frames.  Text-mode terminals don't give a damn, to put it
mildly, about Emacs expectations and needs, when display of such
advanced character sequences is concerned.  They invent their own
implementations and de-facto standards, completely disregarding the
fact that Emacs was there and did that many years before them.  When
approached, they basically tell us "our way or the highway".  "Their
way" isn't going to happen in Emacs, for several good reasons, so here
we are.

Why are you using a text-mode terminal to edit files with these
characters?  Why not use a GUI session instead?

bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines

[Дилян Палаузов]

Hello Eli,

thanks for your answer.  I start  emacs -nw  then M-: shows Eval:   I enter 

(set-char-table-range char-width-table '(#x1f310 . #x1f310) 1) RET

then the mini-buffer shows

1 (#o1, #x1, ?\C-a)

I open then the problematic file, go to the problematic line, then to its end (Ctrl+E).  Emacs should show the cursor after the last letter on the line, but it does show the cursor over the last letter of the line.

> Why are you using a text-mode terminal to edit files with these characters?  Why not use a GUI session instead?

I used connect over SSH to some systems, then start emacs and finally change the files there.  I see I can first start Emacs-GUI and then  “find a file” starting with /ssh:, which open locally ssh-remote files.  So this is a valid work around.

Greetings
  Дилян


-----Original Message-----
From: Eli Zaretskii <eliz@gnu.org>
To: Дилян Палаузов <dilyan.palauzov@aegee.org>
Cc: 66664@debbugs.gnu.org
Subject: Re: bug#66664: On modified lines containing 🌐︎ and Unicode VS15 in text mode emacs does not show the real content of the lines
Date: 10/21/2023 05:39:53 PM

[Please use Reply All, to keep the bug tracker on the CC list.]

> From: Дилян Палаузов
> 	 <dilyan.palauzov@aegee.org>
> Date: Sat, 21 Oct 2023 17:25:11 +0200
> 
> when I load the file the globe appears below “it” from “file edIT options” menu.  So at least visually I see two columns.
> 
> Now that you can reproduce the problem, what shall I do?

Try

  M-: (set-char-table-range char-width-table '(#x1f310 . #x1f310) 1) RET

But in general, I don't think this problem has a solution, except to
use GUI frames.  Text-mode terminals don't give a damn, to put it
mildly, about Emacs expectations and needs, when display of such
advanced character sequences is concerned.  They invent their own
implementations and de-facto standards, completely disregarding the
fact that Emacs was there and did that many years before them.  When
approached, they basically tell us "our way or the highway".  "Their
way" isn't going to happen in Emacs, for several good reasons, so here
we are.

Why are you using a text-mode terminal to edit files with these
characters?  Why not use a GUI session instead?