bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

I noticed that when running emacs 27 in client/server mode I often
open a file to find out that the string "[I]" (it's an upper-case i,
not a lower-case L) has been added to the beginning of the buffer. I
don't have this problem with 26.3.

Best regards
--
Carlos

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

I'm closing this because it seems to be related to the interaction of
emacs-27 with the doom-themes package, and not to emacs-27 per se. If
it turns out to be a problem in emacs I will report it again.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

> If it turns out to be a problem in emacs I will report it again.

Ok, it turned out to be a problem in emacs :).

A minimal configuration to reproduce it would be: (load-theme 'wombat).

Then launch `emacs --daemon` and finally `emacsclient -t`.

Btw, it only happens in the terminal.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Wed, 16 Oct 2019 23:33:50 -0300
> 
> > If it turns out to be a problem in emacs I will report it again.
> 
> Ok, it turned out to be a problem in emacs :).
> 
> A minimal configuration to reproduce it would be: (load-theme 'wombat).
> 
> Then launch `emacs --daemon` and finally `emacsclient -t`.
> 
> Btw, it only happens in the terminal.

Most probably comes from escape sequences Emacs sends to the terminal
to detect support of advanced features.  This particular one seems to
come from detecting focus-in support, see xterm.el:

    ;; These translation functions actually call the focus handlers
    ;; internally and return an empty sequence, causing us to go on to
    ;; read the next event.
    (define-key map "\e[I" #'xterm-translate-focus-in)
    (define-key map "\e[O" #'xterm-translate-focus-out)

Please debug to find out why that particular theme causes this
failure.

Thanks.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

The problem doesn't show when TERM=xterm, but only when TERM=xterm-24bit.

I created a terminfo definition as described in
https://www.gnu.org/software/emacs/manual/html_node/efaq/Colors-on-a-TTY.html
and was using it without any problem in emacs26.

Besides the aforementioned issue, I get true 24-bit color and
everything else works just fine.

The "\[I" sequence is defined for rxvt in xterm.el and it seems to me
weird that it's being used since my terminal is gnome-terminal and, as
I said, TERM=xterm-24bit.

Any idea?

Thanks!

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

Moreover, in emacs 26 those two focus-in/out sequences are not defined at all.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

I don't have time to further debug this right now but I believe it
might be related to

(defun xterm--init-focus-tracking ()
  "Terminal initialization for focus tracking mode."
  (send-string-to-terminal "\e[?1004h")

somehow not being correctly or timely initialized.

See this similar issue
https://superuser.com/questions/931873/o-and-i-appearing-on-iterm2-when-focus-lost

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Thu, 17 Oct 2019 09:08:40 -0300
> Cc: 37782@debbugs.gnu.org
> 
> Any idea?

Well, since the difference is between xterm and xterm-24bit, I'd start
by looking at what these two settings change, and why.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Thu, 17 Oct 2019 09:26:26 -0300
> Cc: 37782@debbugs.gnu.org
> 
> Moreover, in emacs 26 those two focus-in/out sequences are not defined at all.

Which probably explains why you didn't see the problem in Emacs 26,
right?

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Thu, 17 Oct 2019 10:01:45 -0300
> Cc: 37782@debbugs.gnu.org
> 
> I don't have time to further debug this right now but I believe it
> might be related to
> 
> (defun xterm--init-focus-tracking ()
>   "Terminal initialization for focus tracking mode."
>   (send-string-to-terminal "\e[?1004h")
> 
> somehow not being correctly or timely initialized.

Most probably (which is why I pointed to the corresponding escape
sequences), but the question is exactly why does this not work as
expected?  Is the terminal you are using too fast or too slow to
respond?  Is something else wrong?  We need to answer these questions
to make some progress with this issue.

Maybe someone else can reproduce and debug this.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

> Most probably (which is why I pointed to the corresponding escape
> sequences), but the question is exactly why does this not work as
> expected?  Is the terminal you are using too fast or too slow to
> respond?  Is something else wrong?  We need to answer these questions
> to make some progress with this issue.
>
> Maybe someone else can reproduce and debug this.

I've had a hard time debugging this issue but the cause is still unclear to me.

I can easily reproduce the problem with an empty configuration, see
the attached screencast. There I'm just running

emacs --daemon
emacsclient -t

with an empty configuration. Since the behavior is not very
"deterministic" I was many times mislead to believe that the cause was
this or that theme or package, and maybe some packages that make heavy
use of color or other escape sequences do might tend to produce the
artifact more often, I don't know.

Things that I can say for (99%) sure:

1. It only happens in a terminal.
2. It only happens in client/server mode.
3. It only happens the first time I open a client in the terminal.
4. Commenting the  `(send-string-to-terminal "\e[?1004h")` line in
xterm.el suppresses the `[I` sequence (but this of course by disabling
focus tracking).
5. After the initial artifact, focus is indeed tracked correctly, i.e.
xterm-translate-focus-in/out are called as expected for `[I` and `[O`
sequences. But the first time xterm-translate-focus-in is NOT called.
6. When focus tracking is activated xterm-function-map was already
pushed to input-decode-map so AFAICS focus handlers should already be
properly installed.
7. Sometimes it is `[I`, sometimes it is `[I]`.

My hunch is that the escape sequence is not correctly parsed, that the
`[I` part is taken as normal input from the keyboard because the
escape prefix got lost or was interspersed with another sequence.
Probably the answer is in some part of read_key_sequence, but the
combination of client/server mode and low-level tty input is beyond my
debugging abilities.

Best regards
--
Carlos

[-- Attachment #2: Screencast from 18-10-19 01:01:46.webm --]
[-- Type: video/webm, Size: 659386 bytes --]

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Carlos Pita]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

After hours and hours (and hours) of life sucking debugging I found
out the offending code (in read_key_sequence):

    if ((FIXNUMP (key) && XFIXNUM (key) == -2) /* wrong_kboard_jmpbuf */
/* When switching to a new tty (with a new keyboard),
   read_char returns the new buffer, rather than -2
   (Bug#5095).  This is because `terminal-init-xterm'
   calls read-char, which eats the wrong_kboard_jmpbuf
   return.  Any better way to fix this? -- cyd  */
|| (interrupted_kboard != current_kboard))
  {
   ....
   mock_input = 0;
   goto replay_entire_sequence;
  }

The problem with the fix for #5095 is that it's dropping the current
event, which is a valid char event and not -2. Thus, the first esc
char sent by the terminal after turning on focus tracking is lost and
we get `[`, `I` events separately inserted into the buffer afterwards.

Attached is a patch that replays key as mock input when it's not -2.

Can I ask for some of my hidpi fringe or colorized comint/python
history patches to be merged in exchange ;) ?

Best regards
--
Carlos

[-- Attachment #2: 0001-Replay-key-if-kboard-is-interrupted-while-initializi.patch --]
[-- Type: text/x-patch, Size: 1238 bytes --]

From 0e98bdf5aa485c2b4164c9627ec34b7b54892352 Mon Sep 17 00:00:00 2001
From: memeplex <carlosjosepita@gmail.com>
Date: Sat, 19 Oct 2019 00:13:15 -0300
Subject: [PATCH] Replay key if kboard is interrupted while initializing
 (Bug#37782)

* src/keyboard.c (read_key_sequence): Add key as mock_input and replay
sequence using new keyboard.
---
 src/keyboard.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/src/keyboard.c b/src/keyboard.c
index d67d18a801..5a214833ed 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -9580,7 +9580,16 @@ read_key_sequence (Lisp_Object *keybuf, Lisp_Object prompt,
 		       Fcons (make_lispy_switch_frame (frame),
 			      KVAR (interrupted_kboard, kbd_queue)));
 		  }
-		mock_input = 0;
+                if (FIXNUMP (key) && XFIXNUM (key) == -2)
+                  mock_input = 0;
+                else
+                  {
+                    /* If interrupted while initializing terminal, we
+                       need to replay the interrupting key.  See
+                       Bug#5095 and Bug#37782.  */
+                    mock_input = 1;
+                    keybuf[0] = key;
+                  }
 		goto replay_entire_sequence;
 	      }
 	  }
-- 
2.20.1

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Sat, 19 Oct 2019 00:20:05 -0300
> Cc: 37782@debbugs.gnu.org
> 
> Can I ask for some of my hidpi fringe or colorized comint/python
> history patches to be merged in exchange ;) ?

They are in my queue; sorry for not yet having time to do a thorough
review.  Thank you for your patience.

bug#37782: Emacs 27 client adding [I] to the beginning of the buffer

[Eli Zaretskii]

> From: Carlos Pita <carlosjosepita@gmail.com>
> Date: Sat, 19 Oct 2019 00:20:05 -0300
> Cc: 37782@debbugs.gnu.org
> 
> The problem with the fix for #5095 is that it's dropping the current
> event, which is a valid char event and not -2. Thus, the first esc
> char sent by the terminal after turning on focus tracking is lost and
> we get `[`, `I` events separately inserted into the buffer afterwards.
> 
> Attached is a patch that replays key as mock input when it's not -2.

Thanks, pushed, and closing the bug.

bug#37782: Emacs 27 client adding [I] to the beginning of the

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Hello Carlos and Eli,

The fix for bug#37782 in commit e3cebbb (2019-10-19) introduced an error
when starting emacsclient.

    undefined: KEY must be an integer, cons, symbol, or string

I think the message is from keymap.c around line 2279.

This error is reproducible and is happening on FreeBSD (versions 12.0
and 12.1 confirmed).

Regards,

Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]