* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters @ 2021-10-02 22:50 Rudi C 2021-10-03 5:51 ` Eli Zaretskii 2021-10-03 9:08 ` Lars Ingebrigtsen 0 siblings, 2 replies; 21+ messages in thread From: Rudi C @ 2021-10-02 22:50 UTC (permalink / raw) To: 50983 [-- Attachment #1: Type: text/plain, Size: 2263 bytes --] I have two display bugs to report, one a regression that is not present in emacs 27. I start with this regression. 1. `curl https://files.lilf.ir/tmp/weird.txt > weird.txt` 2. `emacs -Q -nw weird.txt` 3. try editing the text, deleting characters, etc. The character display will get messed up. Here is a screenshot of emacs before editing the file: https://files.lilf.ir/tmp/tmp.kik6vbBw8S.png And here is a screenshot after I do `backspace a`: https://files.lilf.ir/tmp/tmp.Twz5ZXVbR6.png I have tried this bug with emacs 27 (both myself and some other user on IRC), and it is not present there. The second bug: 1. `curl https://files.lilf.ir/tmp/bug.txt > bug.txt` 2. do `cat bug.txt` and note the output: https://files.lilf.ir/tmp/tmp.HKfKc9PUds.png 3. `emacs -Q -nw bug.txt` As you can see, emacs is displaying the file incorrectly: https://files.lilf.ir/tmp/tmp.0yKbCbB80R.png In particular, the line `#+TITLE: sharif/contact info` is not displayed at all. I could reproduce this bug on both emacs 27 and 28. Additional info: In GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin20.3.0, NS appkit-2022.30 Version 11.2.1 (Build 20D75)) of 2021-09-21 built on Fereidoons-MacBook-Pro.local System Description: macOS 11.2.1 Configured using: 'configure --disable-dependency-tracking --disable-silent-rules --enable-locallisppath=/usr/local/share/emacs/site-lisp --infodir=/usr/local/Cellar/emacs-plus@28/28.0.50/share/info/emacs --prefix=/usr/local/Cellar/emacs-plus@28/28.0.50 --with-xml2 --with-gnutls --with-native-compilation --without-dbus --with-imagemagick --with-modules --with-rsvg --with-xwidgets --with-ns --disable-ns-self-contained 'CFLAGS=-I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include -I/usr/local/opt/gmp/include -I/usr/local/opt/jpeg/include' 'LDFLAGS=-L/usr/local/lib/gcc/11 -I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include -I/usr/local/opt/gmp/include -I/usr/local/opt/jpeg/include'' Configured features: ACL GIF GLIB GMP GNUTLS IMAGEMAGICK JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY KQUEUE NS PDUMPER PNG RSVG THREADS TIFF TOOLKIT_SCROLL_BARS XIM XWIDGETS ZLIB Important settings: value of $LC_ALL: en_US.UTF-8 value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix [-- Attachment #2: Type: text/html, Size: 4286 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-02 22:50 bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters Rudi C @ 2021-10-03 5:51 ` Eli Zaretskii 2021-10-03 6:47 ` Rudi C ` (2 more replies) 2021-10-03 9:08 ` Lars Ingebrigtsen 1 sibling, 3 replies; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 5:51 UTC (permalink / raw) To: Rudi C; +Cc: 50983 > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > Date: Sun, 3 Oct 2021 02:20:24 +0330 > > I have two display bugs to report, one a regression that is not present in emacs 27. I start with this > regression. > > 1. `curl https://files.lilf.ir/tmp/weird.txt > weird.txt` > 2. `emacs -Q -nw weird.txt` > 3. try editing the text, deleting characters, etc. The character display will get messed up. > > Here is a screenshot of emacs before editing the file: > > https://files.lilf.ir/tmp/tmp.kik6vbBw8S.png > > And here is a screenshot after I do `backspace a`: > https://files.lilf.ir/tmp/tmp.Twz5ZXVbR6.png > > I have tried this bug with emacs 27 (both myself and some other user on IRC), and it is not present there. > > The second bug: > 1. `curl https://files.lilf.ir/tmp/bug.txt > bug.txt` > 2. do `cat bug.txt` and note the output: > https://files.lilf.ir/tmp/tmp.HKfKc9PUds.png > > 3. `emacs -Q -nw bug.txt` > As you can see, emacs is displaying the file incorrectly: > https://files.lilf.ir/tmp/tmp.0yKbCbB80R.png > > In particular, the line `#+TITLE: sharif/contact info` is not displayed at all. > > I could reproduce this bug on both emacs 27 and 28. I'm unable to reproduce any of this on my system. Both files display correctly, and the problems after deleting character and/or after displaying the file in a -nw session don't happen. This could be specific to macOS, where AFAIK the display is implemented slightly differently from the other platforms. Or maybe something else is at work here. For the -nw problems, this could perhaps be related to the terminal emulator you are using (just a guess, I have no real explanation how that could hide entire portions of the file's display). P.S. The site which you use to post the files is problematic: its certificate is expired or invalid, and at least on one of my systems wget said the TLS handshake failed, perhaps for the same reason. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 5:51 ` Eli Zaretskii @ 2021-10-03 6:47 ` Rudi C 2021-10-03 9:01 ` Eli Zaretskii 2021-10-03 9:11 ` Lars Ingebrigtsen 2021-10-03 9:54 ` Alan Third 2 siblings, 1 reply; 21+ messages in thread From: Rudi C @ 2021-10-03 6:47 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983 [-- Attachment #1: Type: text/plain, Size: 3476 bytes --] > The site which you use to post the files is problematic: its certificate is expired or invalid. I use caddy to automatically manage its certificates, and I don't get any cert errors myself. Can you be more specific? Perhaps you need newer versions of wget? https://www.ssllabs.com/ssltest/analyze.html?d=files.lilf.ir also says everything is okay. > This could be specific to macOS I tested `bug.txt` via SSH on an Ubuntu server with emacs 27.2, and it was no different: https://files.lilf.ir/tmp/tmp.mZxt5Pilap.png Testing it with other terminal apps, none of the bugs occur with `terminal.app`. The RTL is all wrong on `terminal.app` though ( https://files.lilf.ir/tmp/tmp.1UjK8TYGoG.png) , but I guess it's unrelated. Alacritty doesn't show the bug, and it also doesn't mess up the RTL shaping. However, the bug is probably an interaction between both emacs and the terminal app Kitty, as `vim` does not have this problem. Interestingly, neovim does. (This is true for both of the bugs; vim doesn't have them, emacs and nvim do, and only on Kitty.) BTW, I tested using `command kitty --config=/dev/null`, so the bug did not have anything with my Kitty config. (A screenshot in unconfigured Kitty: https://files.lilf.ir/tmp/tmp.UcEXWQkTwn.png) If you think the issue is to be upstreamed to Kitty, can you open an issue on their Github? (https://github.com/kovidgoyal/kitty/issues) Thanks. On Sun, Oct 3, 2021 at 9:21 AM Eli Zaretskii <eliz@gnu.org> wrote: > > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > > Date: Sun, 3 Oct 2021 02:20:24 +0330 > > > > I have two display bugs to report, one a regression that is not present > in emacs 27. I start with this > > regression. > > > > 1. `curl https://files.lilf.ir/tmp/weird.txt > weird.txt` > > 2. `emacs -Q -nw weird.txt` > > 3. try editing the text, deleting characters, etc. The character display > will get messed up. > > > > Here is a screenshot of emacs before editing the file: > > > > https://files.lilf.ir/tmp/tmp.kik6vbBw8S.png > > > > And here is a screenshot after I do `backspace a`: > > https://files.lilf.ir/tmp/tmp.Twz5ZXVbR6.png > > > > I have tried this bug with emacs 27 (both myself and some other user on > IRC), and it is not present there. > > > > The second bug: > > 1. `curl https://files.lilf.ir/tmp/bug.txt > bug.txt` > > 2. do `cat bug.txt` and note the output: > > https://files.lilf.ir/tmp/tmp.HKfKc9PUds.png > > > > 3. `emacs -Q -nw bug.txt` > > As you can see, emacs is displaying the file incorrectly: > > https://files.lilf.ir/tmp/tmp.0yKbCbB80R.png > > > > In particular, the line `#+TITLE: sharif/contact info` is not displayed > at all. > > > > I could reproduce this bug on both emacs 27 and 28. > > I'm unable to reproduce any of this on my system. Both files display > correctly, and the problems after deleting character and/or after > displaying the file in a -nw session don't happen. > > This could be specific to macOS, where AFAIK the display is > implemented slightly differently from the other platforms. Or maybe > something else is at work here. For the -nw problems, this could > perhaps be related to the terminal emulator you are using (just a > guess, I have no real explanation how that could hide entire portions > of the file's display). > > P.S. The site which you use to post the files is problematic: its > certificate is expired or invalid, and at least on one of my systems > wget said the TLS handshake failed, perhaps for the same reason. > [-- Attachment #2: Type: text/html, Size: 5223 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 6:47 ` Rudi C @ 2021-10-03 9:01 ` Eli Zaretskii 0 siblings, 0 replies; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 9:01 UTC (permalink / raw) To: Rudi C; +Cc: 50983 > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > Date: Sun, 3 Oct 2021 10:17:21 +0330 > Cc: 50983@debbugs.gnu.org > > > The site which you use to post the files is problematic: its > certificate is expired or invalid. > > I use caddy to automatically manage its certificates, and I don't get any cert errors myself. Can you be more > specific? Perhaps you need newer versions of wget? No, I don't think so. Anyway, this is a tangent; if you think everything is okay with the site, I can fetch files regardless. > https://www.ssllabs.com/ssltest/analyze.html?d=files.lilf.ir also says everything is okay. > > > This could be specific to macOS > > I tested `bug.txt` via SSH on an Ubuntu server with emacs 27.2, and it was no different: > https://files.lilf.ir/tmp/tmp.mZxt5Pilap.png How exactly did you do that? where was Emacs running and where was the display running? Was that with or without X forwarding? Also, this is the second file; what about the first one? Do you see on Ubuntu problems with deleting characters in it, and if so, which characters and what problems this causes? > Testing it with other terminal apps, none of the bugs occur with `terminal.app`. > > The RTL is all wrong on `terminal.app` though (https://files.lilf.ir/tmp/tmp.1UjK8TYGoG.png) > , but I guess it's unrelated. Alacritty doesn't show the bug, and it also doesn't mess up the RTL shaping. > To display RTL text on a terminal, you need to turn off bidirectional features of the terminal, if it has them, because Emacs performs the bidirectional processing by itself. > If you think the issue is to be upstreamed to Kitty, can you open an issue on their Github? > (https://github.com/kovidgoyal/kitty/issues) Sorry, I wouldn't know what to write there, and cannot present any data as I don't have Kitty installed. I think it's better that you do it. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 5:51 ` Eli Zaretskii 2021-10-03 6:47 ` Rudi C @ 2021-10-03 9:11 ` Lars Ingebrigtsen 2021-10-03 9:54 ` Alan Third 2 siblings, 0 replies; 21+ messages in thread From: Lars Ingebrigtsen @ 2021-10-03 9:11 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, Rudi C Eli Zaretskii <eliz@gnu.org> writes: > P.S. The site which you use to post the files is problematic: its > certificate is expired or invalid, and at least on one of my systems > wget said the TLS handshake failed, perhaps for the same reason. It's not expired, but if you're getting messages saying that it is, it probably means that your wget is too old. It's due to the Let's Encrypt meltdown on October 1st. Here's one of the many, many threads about it; basically you have to delete the X3 certificate: https://stackoverflow.com/questions/69387175/git-for-windows-ssl-certificate-problem-certificate-has-expired -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 5:51 ` Eli Zaretskii 2021-10-03 6:47 ` Rudi C 2021-10-03 9:11 ` Lars Ingebrigtsen @ 2021-10-03 9:54 ` Alan Third 2021-10-03 10:04 ` Eli Zaretskii 2 siblings, 1 reply; 21+ messages in thread From: Alan Third @ 2021-10-03 9:54 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, Rudi C On Sun, Oct 03, 2021 at 08:51:39AM +0300, Eli Zaretskii wrote: > > This could be specific to macOS, where AFAIK the display is > implemented slightly differently from the other platforms. As far as I'm aware the "-nw" display is implemented the same as any other platform. For the record I can't see these problems on GUI emacs on the mac, but I do see them in the terminal using iTerm2. It looks like sometimes the display is incorrect, and other times the action actually changes the underlying buffer contents in an unexpected way. -- Alan Third ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 9:54 ` Alan Third @ 2021-10-03 10:04 ` Eli Zaretskii 2021-10-03 10:24 ` Alan Third 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 10:04 UTC (permalink / raw) To: Alan Third; +Cc: 50983, rudiwillalwaysloveyou > Date: Sun, 3 Oct 2021 10:54:01 +0100 > From: Alan Third <alan@idiocy.org> > Cc: Rudi C <rudiwillalwaysloveyou@gmail.com>, 50983@debbugs.gnu.org > > On Sun, Oct 03, 2021 at 08:51:39AM +0300, Eli Zaretskii wrote: > > > > This could be specific to macOS, where AFAIK the display is > > implemented slightly differently from the other platforms. > > As far as I'm aware the "-nw" display is implemented the same as any > other platform. I meant the GUI part of the report. > For the record I can't see these problems on GUI emacs > on the mac, but I do see them in the terminal using iTerm2. Does iTerm2 have some support for bidirectional text? If so, you need to disable it for Emacs -nw to display bidirectional text correctly. > It looks like sometimes the display is incorrect, and other times the > action actually changes the underlying buffer contents in an > unexpected way. Any idea what could cause that? Does it happen with plain-ASCII text as well? ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 10:04 ` Eli Zaretskii @ 2021-10-03 10:24 ` Alan Third 2021-10-03 10:49 ` Eli Zaretskii 2021-10-03 11:00 ` Rudi C 0 siblings, 2 replies; 21+ messages in thread From: Alan Third @ 2021-10-03 10:24 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, rudiwillalwaysloveyou On Sun, Oct 03, 2021 at 01:04:36PM +0300, Eli Zaretskii wrote: > > Date: Sun, 3 Oct 2021 10:54:01 +0100 > > From: Alan Third <alan@idiocy.org> > > Cc: Rudi C <rudiwillalwaysloveyou@gmail.com>, 50983@debbugs.gnu.org > > > > For the record I can't see these problems on GUI emacs > > on the mac, but I do see them in the terminal using iTerm2. > > Does iTerm2 have some support for bidirectional text? If so, you need > to disable it for Emacs -nw to display bidirectional text correctly. I don't see any options (but there are a LOT of options). > > It looks like sometimes the display is incorrect, and other times the > > action actually changes the underlying buffer contents in an > > unexpected way. > > Any idea what could cause that? Does it happen with plain-ASCII text > as well? Actually, I was wrong. If I follow the instructions for the first example, by removing the character indicated by an underscore in Rudi's first screenshot, it actually deletes the previous "o" in "note", and displays the rest wrongly, as shown in his second screenshot. If I put the cursor over that underscore character and do describe-char, it tells me it's an "o", so the problem exists even before editing. I don't see these problems in normal ascii text, or even normal UTF-8 text, even RTL. For example the Hebrew text in HELLO behaves exactly as I'd expect. -- Alan Third ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 10:24 ` Alan Third @ 2021-10-03 10:49 ` Eli Zaretskii 2021-10-03 11:26 ` Alan Third 2021-10-03 11:00 ` Rudi C 1 sibling, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 10:49 UTC (permalink / raw) To: Alan Third; +Cc: 50983, alan, rudiwillalwaysloveyou > Date: Sun, 3 Oct 2021 11:24:59 +0100 > From: Alan Third <alan@idiocy.org> > Cc: rudiwillalwaysloveyou@gmail.com, 50983@debbugs.gnu.org > > > > It looks like sometimes the display is incorrect, and other times the > > > action actually changes the underlying buffer contents in an > > > unexpected way. > > > > Any idea what could cause that? Does it happen with plain-ASCII text > > as well? > > Actually, I was wrong. If I follow the instructions for the first > example, by removing the character indicated by an underscore in > Rudi's first screenshot, it actually deletes the previous "o" in > "note", and displays the rest wrongly, as shown in his second > screenshot. > > If I put the cursor over that underscore character and do > describe-char, it tells me it's an "o", so the problem exists even > before editing. Is this in a GUI frame or a TTY frame? And what do you mean by "underscore character"? What is its Unicode codepoint? ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 10:49 ` Eli Zaretskii @ 2021-10-03 11:26 ` Alan Third 2021-10-03 12:02 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Alan Third @ 2021-10-03 11:26 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, rudiwillalwaysloveyou On Sun, Oct 03, 2021 at 01:49:29PM +0300, Eli Zaretskii wrote: > > Date: Sun, 3 Oct 2021 11:24:59 +0100 > > From: Alan Third <alan@idiocy.org> > > Cc: rudiwillalwaysloveyou@gmail.com, 50983@debbugs.gnu.org > > > > > > It looks like sometimes the display is incorrect, and other times the > > > > action actually changes the underlying buffer contents in an > > > > unexpected way. > > > > > > Any idea what could cause that? Does it happen with plain-ASCII text > > > as well? > > > > Actually, I was wrong. If I follow the instructions for the first > > example, by removing the character indicated by an underscore in > > Rudi's first screenshot, it actually deletes the previous "o" in > > "note", and displays the rest wrongly, as shown in his second > > screenshot. > > > > If I put the cursor over that underscore character and do > > describe-char, it tells me it's an "o", so the problem exists even > > before editing. > > Is this in a GUI frame or a TTY frame? All TTY, GUI works fine. > And what do you mean by "underscore character"? What is its Unicode > codepoint? In the screenshot (and in my own iTerm2 session) there is an underscore character after "note-". I think it's inserted by the terminal as a placeholder for something it doesn't understand. In GUI Emacs that position in the file has a zero width character. If I do describe-char on the underscore it says it's a plain ascii "o", which is clearly incorrect. In GUI it says it's 8203 (0x200B), "ZERO WIDTH SPACE", and as I said it displays as a zero width space. I think I agree with your other email that it's down to the terminal doing something strange with characters it doesn't understand. -- Alan Third ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 11:26 ` Alan Third @ 2021-10-03 12:02 ` Eli Zaretskii 2021-10-03 12:54 ` Alan Third 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 12:02 UTC (permalink / raw) To: Alan Third; +Cc: 50983, rudiwillalwaysloveyou > Date: Sun, 3 Oct 2021 12:26:22 +0100 > From: Alan Third <alan@idiocy.org> > Cc: rudiwillalwaysloveyou@gmail.com, 50983@debbugs.gnu.org > > > And what do you mean by "underscore character"? What is its Unicode > > codepoint? > > In the screenshot (and in my own iTerm2 session) there is an > underscore character after "note-". I think it's inserted by the > terminal as a placeholder for something it doesn't understand. No, it's a special face we use to display some characters that may look like ASCII, but aren't. See nobreak-char-display. > In GUI Emacs that position in the file has a zero width character. > > If I do describe-char on the underscore it says it's a plain ascii > "o", which is clearly incorrect. In GUI it says it's 8203 (0x200B), > "ZERO WIDTH SPACE", and as I said it displays as a zero width space. Can you show the output of "C-x =" on all the characters, one by one, starting from "n" in "note" and ending with "t" in "taking" after it? Are they all incorrect, i.e. do not correspond to the place the cursor is on? That is, does the corruption start around there or does it start much earlier (and if the latter, where does it start)? > I think I agree with your other email that it's down to the terminal > doing something strange with characters it doesn't understand. If this is the case, the only way to fix the display is to use us-ascii as terminal encoding. Or maybe set up the terminal for a "simpler" encoding, like latin-1, and then set up Emacs to that using set-terminal-coding-system. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 12:02 ` Eli Zaretskii @ 2021-10-03 12:54 ` Alan Third 2021-10-03 14:48 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Alan Third @ 2021-10-03 12:54 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, rudiwillalwaysloveyou On Sun, Oct 03, 2021 at 03:02:20PM +0300, Eli Zaretskii wrote: > Can you show the output of "C-x =" on all the characters, one by one, > starting from "n" in "note" and ending with "t" in "taking" after it? > Are they all incorrect, i.e. do not correspond to the place the cursor > is on? That is, does the corruption start around there or does it > start much earlier (and if the latter, where does it start)? n -> i o -> t t -> h e -> SPC - -> n _ -> o t -> t So it's off-set by some 4 characters. Looking at the raw file, there are 4 0xAD (SOFT HYPHEN) characters before "note", and after each one the offset increases by one. I do not see them displayed in the terminal. > > I think I agree with your other email that it's down to the terminal > > doing something strange with characters it doesn't understand. > > If this is the case, the only way to fix the display is to use > us-ascii as terminal encoding. Or maybe set up the terminal for a > "simpler" encoding, like latin-1, and then set up Emacs to that using > set-terminal-coding-system. Indeed, changing the "character encoding" setting in iTerm to ASCII displays the soft-hyphens as a red "A" and everything seems to work right. The default is UTF-8 and it reports itself as "xterm-256color". I suspect most terminal applications on macOS will default to UTF-8 since that's the default everywhere else, which might help explain why this seems to be limited to macOS. -- Alan Third ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 12:54 ` Alan Third @ 2021-10-03 14:48 ` Eli Zaretskii 0 siblings, 0 replies; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 14:48 UTC (permalink / raw) To: Alan Third; +Cc: 50983, rudiwillalwaysloveyou > Date: Sun, 3 Oct 2021 13:54:33 +0100 > From: Alan Third <alan@idiocy.org> > Cc: 50983@debbugs.gnu.org, rudiwillalwaysloveyou@gmail.com > > n -> i > o -> t > t -> h > e -> SPC > - -> n > _ -> o > t -> t > > So it's off-set by some 4 characters. > > Looking at the raw file, there are 4 0xAD (SOFT HYPHEN) characters > before "note", and after each one the offset increases by one. > > I do not see them displayed in the terminal. So this terminal seems to be unable to display those SOFT HYPHEN characters (or maybe it's a "feature"?), and since Emacs knows nothing about that, the relation between cursor position and buffer position is disrupted. > > If this is the case, the only way to fix the display is to use > > us-ascii as terminal encoding. Or maybe set up the terminal for a > > "simpler" encoding, like latin-1, and then set up Emacs to that using > > set-terminal-coding-system. > > Indeed, changing the "character encoding" setting in iTerm to ASCII > displays the soft-hyphens as a red "A" and everything seems to work > right. > > The default is UTF-8 and it reports itself as "xterm-256color". I > suspect most terminal applications on macOS will default to UTF-8 > since that's the default everywhere else, which might help explain why > this seems to be limited to macOS. OK, thanks. I think we can say this is not an Emacs problem. I recommend to file a bug with the developers of the terminal, and maybe they will tell how to avoid that by some setting. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 10:24 ` Alan Third 2021-10-03 10:49 ` Eli Zaretskii @ 2021-10-03 11:00 ` Rudi C 2021-10-03 11:11 ` Eli Zaretskii 1 sibling, 1 reply; 21+ messages in thread From: Rudi C @ 2021-10-03 11:00 UTC (permalink / raw) To: Alan Third, Eli Zaretskii, Rudi C, 50983 [-- Attachment #1: Type: text/plain, Size: 2198 bytes --] > I don't see these problems in normal ascii text, or even normal UTF-8 text Are the text files I have attached not normal UTf-8? (I have no idea how to notice if they are normal or not.) > support for bidirectional text This is irrelevant to these bugs, I think. If your terminal does RTL reordering, then emacs does that, too, so the ordering will get reversed, but it shouldn't have anything to do with these two bugs. Also, can you be more specific about where you do observe the bugs? In TUI emacs on iTerm? I can confirm that the bug with `weird.txt` happens on iTerm, too, again with both emacs and neovim! But the bug with `bug.txt` does not happen in iTerm, only on Kitty. On Sun, Oct 3, 2021 at 1:55 PM Alan Third <alan@idiocy.org> wrote: > On Sun, Oct 03, 2021 at 01:04:36PM +0300, Eli Zaretskii wrote: > > > Date: Sun, 3 Oct 2021 10:54:01 +0100 > > > From: Alan Third <alan@idiocy.org> > > > Cc: Rudi C <rudiwillalwaysloveyou@gmail.com>, 50983@debbugs.gnu.org > > > > > > For the record I can't see these problems on GUI emacs > > > on the mac, but I do see them in the terminal using iTerm2. > > > > Does iTerm2 have some support for bidirectional text? If so, you need > > to disable it for Emacs -nw to display bidirectional text correctly. > > I don't see any options (but there are a LOT of options). > > > > It looks like sometimes the display is incorrect, and other times the > > > action actually changes the underlying buffer contents in an > > > unexpected way. > > > > Any idea what could cause that? Does it happen with plain-ASCII text > > as well? > > Actually, I was wrong. If I follow the instructions for the first > example, by removing the character indicated by an underscore in > Rudi's first screenshot, it actually deletes the previous "o" in > "note", and displays the rest wrongly, as shown in his second > screenshot. > > If I put the cursor over that underscore character and do > describe-char, it tells me it's an "o", so the problem exists even > before editing. > > I don't see these problems in normal ascii text, or even normal UTF-8 > text, even RTL. For example the Hebrew text in HELLO behaves exactly > as I'd expect. > -- > Alan Third > [-- Attachment #2: Type: text/html, Size: 3005 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 11:00 ` Rudi C @ 2021-10-03 11:11 ` Eli Zaretskii 2021-10-04 8:05 ` Rudi C 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-03 11:11 UTC (permalink / raw) To: Rudi C; +Cc: 50983, alan, rudiwillalwaysloveyou > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > Date: Sun, 3 Oct 2021 14:30:56 +0330 > > Also, can you be more specific about where you do observe the bugs? In TUI emacs on iTerm? > > I can confirm that the bug with `weird.txt` happens on iTerm, too, again with both emacs and neovim! But the > bug with `bug.txt` does not happen in iTerm, only on Kitty. This sounds like the terminal emulators have a problem in supporting unusual Unicode characters, such as zero-width or double-width characters, perhaps? I see no Emacs problem here, since it happens only on some terminal emulators and not on others. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 11:11 ` Eli Zaretskii @ 2021-10-04 8:05 ` Rudi C 2021-10-04 12:40 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Rudi C @ 2021-10-04 8:05 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, Alan Third [-- Attachment #1: Type: text/plain, Size: 1341 bytes --] > I see no Emacs problem here But the problem does not happen with vim (nor with emacs 27 for `weird.txt`), so it is clearly an interaction of different elements. Anyhow, I have opened an [upstream issue]( https://github.com/kovidgoyal/kitty/issues/4094). Please subscribe to it so that you might offer your emacs expertise there, if needed. > changing the "character encoding" setting in iTerm to ASCII This is a most loath workaround. I do want UTF-8, as I use mathematical symbols, emojis, and non-English languages. Anyhow, making the text full of random unrecognized characters is not much better than the current behavior. On Sun, Oct 3, 2021 at 2:42 PM Eli Zaretskii <eliz@gnu.org> wrote: > > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > > Date: Sun, 3 Oct 2021 14:30:56 +0330 > > > > Also, can you be more specific about where you do observe the bugs? In > TUI emacs on iTerm? > > > > I can confirm that the bug with `weird.txt` happens on iTerm, too, again > with both emacs and neovim! But the > > bug with `bug.txt` does not happen in iTerm, only on Kitty. > > This sounds like the terminal emulators have a problem in supporting > unusual Unicode characters, such as zero-width or double-width > characters, perhaps? I see no Emacs problem here, since it happens > only on some terminal emulators and not on others. > [-- Attachment #2: Type: text/html, Size: 1916 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-04 8:05 ` Rudi C @ 2021-10-04 12:40 ` Eli Zaretskii 2021-10-04 13:50 ` Eli Zaretskii 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-04 12:40 UTC (permalink / raw) To: Rudi C; +Cc: 50983, alan > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > Date: Mon, 4 Oct 2021 11:35:41 +0330 > Cc: Alan Third <alan@idiocy.org>, 50983@debbugs.gnu.org > > But the problem does not happen with vim (nor with emacs 27 for `weird.txt`), so it is clearly an interaction > of different elements. > > Anyhow, I have opened an [upstream issue](https://github.com/kovidgoyal/kitty/issues/4094). Please > subscribe to it so that you might offer your emacs expertise there, if needed. I subscribed and posted the following comment: Emacs uses character width tables computed from the latest Unicode Standard version 14.0.0, using the data in the file EastAsianWidth.txt. In that text, the U+00AD SOFT HYPHEN character, which caused the problems in your file, has the East Asian Width property value of A, which stands for "Ambiguous". The definition of this value in the Unicode Standard Annex 11 (UAX#11) is as follows: East Asian Ambiguous (A): All characters that can be sometimes wide and sometimes narrow. Ambiguous characters require additional information not contained in the character code to further resolve their width. Ambiguous characters occur in East Asian legacy character sets as wide characters, but as narrow (i.e., normal-width) characters in non-East Asian usage. And since the file you show didn't have any East Asian legacy characters, treating SOFT HYPHEN as narrow is IMO correct. > > changing the "character encoding" setting in iTerm to ASCII > > This is a most loath workaround. I do want UTF-8, as I use mathematical symbols, emojis, and non-English > languages. Anyhow, making the text full of random unrecognized characters is not much better than the > current behavior. It is better because it doesn't confuse the user regarding which character is he or she editing. But I agree with you that the results are hardly satisfactory, so my recommendation is not to use Kitty in conjunction with Emacs. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-04 12:40 ` Eli Zaretskii @ 2021-10-04 13:50 ` Eli Zaretskii 2022-09-04 21:37 ` Lars Ingebrigtsen 0 siblings, 1 reply; 21+ messages in thread From: Eli Zaretskii @ 2021-10-04 13:50 UTC (permalink / raw) To: rudiwillalwaysloveyou; +Cc: 50983, alan > Date: Mon, 04 Oct 2021 15:40:05 +0300 > From: Eli Zaretskii <eliz@gnu.org> > Cc: 50983@debbugs.gnu.org, alan@idiocy.org > > > From: Rudi C <rudiwillalwaysloveyou@gmail.com> > > Date: Mon, 4 Oct 2021 11:35:41 +0330 > > Cc: Alan Third <alan@idiocy.org>, 50983@debbugs.gnu.org > > > > But the problem does not happen with vim (nor with emacs 27 for `weird.txt`), so it is clearly an interaction > > of different elements. > > > > Anyhow, I have opened an [upstream issue](https://github.com/kovidgoyal/kitty/issues/4094). Please > > subscribe to it so that you might offer your emacs expertise there, if needed. > > I subscribed and posted the following comment: > > Emacs uses character width tables computed from the latest Unicode > Standard version 14.0.0, using the data in the file > EastAsianWidth.txt. In that text, the U+00AD SOFT HYPHEN character, > which caused the problems in your file, has the East Asian Width > property value of A, which stands for "Ambiguous". The definition of > this value in the Unicode Standard Annex 11 (UAX#11) is as follows: > > East Asian Ambiguous (A): All characters that can be sometimes wide > and sometimes narrow. Ambiguous characters require additional > information not contained in the character code to further resolve > their width. > > Ambiguous characters occur in East Asian legacy character sets as > wide characters, but as narrow (i.e., normal-width) characters in > non-East Asian usage. > > And since the file you show didn't have any East Asian legacy > characters, treating SOFT HYPHEN as narrow is IMO correct. To summarize the comments there: The problematic character in the first example is U+00AD SOFT HYPHEN. Kitty assumes that character is never rendered, and therefore effectively treats it as zero-width character. I don't see how Emacs display can possibly work correctly on such a terminal, so I think we should close this bug report as "notabug". For the second example, I think there could be an issue with character compositions on this terminal, so the OP is advised to try turning off auto-composition-mode. If that solves the problem, fine; if not, I guess Kitty once again assumes something about how such sequences are rendered, and those assumptions don't fit how Emacs displays them in reality, and if so, that problem, too, has no satisfactory solution (and isn't an Emacs bug). ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-04 13:50 ` Eli Zaretskii @ 2022-09-04 21:37 ` Lars Ingebrigtsen 0 siblings, 0 replies; 21+ messages in thread From: Lars Ingebrigtsen @ 2022-09-04 21:37 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 50983, rudiwillalwaysloveyou, alan Eli Zaretskii <eliz@gnu.org> writes: > I don't see how Emacs display can possibly work correctly on such a > terminal, so I think we should close this bug report as "notabug". > > For the second example, I think there could be an issue with character > compositions on this terminal, so the OP is advised to try turning off > auto-composition-mode. If that solves the problem, fine; if not, I > guess Kitty once again assumes something about how such sequences are > rendered, and those assumptions don't fit how Emacs displays them in > reality, and if so, that problem, too, has no satisfactory solution > (and isn't an Emacs bug). So I guess the conclusion here is that there's nothing to be done on the Emacs side here, and I'm therefore closing this bug report. ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-02 22:50 bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters Rudi C 2021-10-03 5:51 ` Eli Zaretskii @ 2021-10-03 9:08 ` Lars Ingebrigtsen 2021-10-03 10:48 ` Rudi C 1 sibling, 1 reply; 21+ messages in thread From: Lars Ingebrigtsen @ 2021-10-03 9:08 UTC (permalink / raw) To: Rudi C; +Cc: 50983 [-- Attachment #1: Type: text/plain, Size: 557 bytes --] Rudi C <rudiwillalwaysloveyou@gmail.com> writes: > I have two display bugs to report, one a regression that is not present in > emacs 27. I start with this regression. > > 1. `curl https://files.lilf.ir/tmp/weird.txt > weird.txt` > 2. `emacs -Q -nw weird.txt` > 3. try editing the text, deleting characters, etc. The character display will get > messed up. I'm unable to reproduce this problem, but my Emacs looks very different from yours -- I'm thinking of the line breaking in particular. Here's what mine look with "emacs -Q" under Debian/bullseye: [-- Attachment #2: Type: image/png, Size: 35490 bytes --] [-- Attachment #3: Type: text/plain, Size: 105 bytes --] -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no ^ permalink raw reply [flat|nested] 21+ messages in thread
* bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters 2021-10-03 9:08 ` Lars Ingebrigtsen @ 2021-10-03 10:48 ` Rudi C 0 siblings, 0 replies; 21+ messages in thread From: Rudi C @ 2021-10-03 10:48 UTC (permalink / raw) To: Lars Ingebrigtsen; +Cc: 50983 [-- Attachment #1: Type: text/plain, Size: 1022 bytes --] > I'm unable to reproduce this problem, but my Emacs looks very different from yours -- I'm thinking of the line breaking in particular. Aren't you using emacs 27? Mine also looks like that in 27. It also doesn't have the bug in that version. On Sun, Oct 3, 2021 at 12:38 PM Lars Ingebrigtsen <larsi@gnus.org> wrote: > Rudi C <rudiwillalwaysloveyou@gmail.com> writes: > > > I have two display bugs to report, one a regression that is not present > in > > emacs 27. I start with this regression. > > > > 1. `curl https://files.lilf.ir/tmp/weird.txt > weird.txt` > > 2. `emacs -Q -nw weird.txt` > > 3. try editing the text, deleting characters, etc. The character display > will get > > messed up. > > I'm unable to reproduce this problem, but my Emacs looks very different > from yours -- I'm thinking of the line breaking in particular. Here's > what mine look with "emacs -Q" under Debian/bullseye: > > > > -- > (domestic pets only, the antidote for overdose, milk.) > bloggy blog: http://lars.ingebrigtsen.no > [-- Attachment #2: Type: text/html, Size: 1711 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2022-09-04 21:37 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-02 22:50 bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters Rudi C 2021-10-03 5:51 ` Eli Zaretskii 2021-10-03 6:47 ` Rudi C 2021-10-03 9:01 ` Eli Zaretskii 2021-10-03 9:11 ` Lars Ingebrigtsen 2021-10-03 9:54 ` Alan Third 2021-10-03 10:04 ` Eli Zaretskii 2021-10-03 10:24 ` Alan Third 2021-10-03 10:49 ` Eli Zaretskii 2021-10-03 11:26 ` Alan Third 2021-10-03 12:02 ` Eli Zaretskii 2021-10-03 12:54 ` Alan Third 2021-10-03 14:48 ` Eli Zaretskii 2021-10-03 11:00 ` Rudi C 2021-10-03 11:11 ` Eli Zaretskii 2021-10-04 8:05 ` Rudi C 2021-10-04 12:40 ` Eli Zaretskii 2021-10-04 13:50 ` Eli Zaretskii 2022-09-04 21:37 ` Lars Ingebrigtsen 2021-10-03 9:08 ` Lars Ingebrigtsen 2021-10-03 10:48 ` Rudi C
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).
