bug#20444: paste error in emacs -nw

bug#20444: paste error in emacs -nw

[Glenn Morris]

Package: emacs
Version: 25.0.50

Using RHEL 7.1, with XFCE 4.12, xterm "X.Org 7.6.0(295)", current master:

emacs -Q -nw

Double mouse-1 click to select "text" from the scratch buffer.
Middle mouse click to paste.

The terminal flashes (as it does on eg ctrl-g).
The pasted content is: "~text~" (with extra '~' characters.)

No such issue with 24.5.

bug#20444: paste error in emacs -nw

[Stefan Monnier]

> Using RHEL 7.1, with XFCE 4.12, xterm "X.Org 7.6.0(295)", current master:
> emacs -Q -nw
> Double mouse-1 click to select "text" from the scratch buffer.
> Middle mouse click to paste.
> The terminal flashes (as it does on eg ctrl-g).
> The pasted content is: "~text~" (with extra '~' characters.)

Hmm... I can't reproduce this on Debian with xterm-312.
Could you give us the C-h l output, to see in more detail what keys were
sent (and insert a few C-f/C-b between the mouse clicks so we can
better separate the various elements)?

Also could you try setting xterm-extra-capabilities to (modifyOtherKeys
reportBackground), and also to (modifyOtherKeys reportBackground setSelection)
to see if it's related to the new `setSelection' feature?


        Stefan

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

>> Using RHEL 7.1, with XFCE 4.12, xterm "X.Org 7.6.0(295)", current master:
>> emacs -Q -nw
>> Double mouse-1 click to select "text" from the scratch buffer.
>> Middle mouse click to paste.
>> The terminal flashes (as it does on eg ctrl-g).
>> The pasted content is: "~text~" (with extra '~' characters.)

> Could you give us the C-h l output, to see in more detail what keys were
> sent (and insert a few C-f/C-b between the mouse clicks so we can
> better separate the various elements)?

 ESC [ > 4 1 ; 2 9 5 ; 0 c ESC ] 1 1 ; r g b : 0 0 0 0 / 0 0 0 0 / 0 0
 0 0 ESC \ a [self-insert-command]
 b [self-insert-command]
 c [self-insert-command]
 d [self-insert-command]
 e [self-insert-command]
 f [self-insert-command]
 RET [newline]                           ; copied after this
 h [self-insert-command]
 i [self-insert-command]
 j [self-insert-command]
 k [self-insert-command]
 l [self-insert-command]
 m [self-insert-command]
 RET [newline]                           ; pasted after this
 ESC [ 2 0 0 [nil]
 ~ [self-insert-command]
 t [self-insert-command]
 e [self-insert-command]
 x [self-insert-command]
 t [self-insert-command]
 ESC [ 2 0 1 [nil]
 ~ [self-insert-command]
 RET [newline]                            ; after pasting
 n [self-insert-command]
 o [self-insert-command]
 p [self-insert-command]
 q [self-insert-command]
 r [self-insert-command]
 C-h l [view-lossage]


So, something to do with bracketed paste?
Indeed, commenting out the line

  (terminal-init-xterm-bracketed-paste-mode)

in terminal-init-xterm makes the issue go away.

> Also could you try setting xterm-extra-capabilities to
> (modifyOtherKeys reportBackground), and also to (modifyOtherKeys
> reportBackground setSelection) to see if it's related to the new
> `setSelection' feature?

Makes no difference (nor does setting it to nil).

bug#20444: paste error in emacs -nw

[Stefan Monnier]

>  RET [newline]                           ; pasted after this
>  ESC [ 2 0 0 [nil]
>  ~ [self-insert-command]

Hmm... I get

 RET [newline]
 ESC [ 2 0 0 ~ [xterm-paste]
 t e x t ESC [ 2 0 1 ~ RET [newline]

So, somehow your xterm is put into the mode that uses the "bracketed
paste" feature, yet Emacs seems to fail to parse that escape sequence properly.

What does (lookup-key input-decode-map "\e[200~") return?
What about (lookup-key input-decode-map "\e[200")
and (lookup-key input-decode-map "\e[20")?


        Stefan

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

> What does (lookup-key input-decode-map "\e[200~") return?

5

> What about (lookup-key input-decode-map "\e[200")

nil

> and (lookup-key input-decode-map "\e[20")?

(keymap
 (keymap
  (59 keymap
      (51 keymap
	  (126 .
	       [f57]))
      (54 keymap
	  (126 .
	       [f45]))
      (53 keymap
	  (126 .
	       [f33]))
      (50 keymap
	  (126 .
	       [f21])))
  (126 .
       [f9]))
 keymap
 (94 .
     [C-f9])
 (126 .
      [f9]))





BTW, I could reproduce similar symptoms on Debian testing with:
emacs -Q -nw
double-mouse-1 on "text"
C-h k mouse-2

But that doesn't seem to work right in earlier versions of Emacs either
(it does "C-h k t" and inserts "ext"). In master it does the "~" thing.

bug#20444: paste error in emacs -nw

[Stefan Monnier]

>> What does (lookup-key input-decode-map "\e[200~") return?
> 5

Hmm... 

>> What about (lookup-key input-decode-map "\e[200")
> nil
>> and (lookup-key input-decode-map "\e[20")?
> (keymap

Hmm... even more weird.  So "\e[20" is bound to a keymap and doesn't
hide any potential "\e[200~" binding: good!

So why isn't the "\e[200~" bindings added by term/xterm.el present?

How 'bout

   (lookup-key xterm-function-map "\e[200~")
and
   (lookup-key (keymap-parent input-decode-map) "\e[200~")

And then same thing for "\e[200" and "\e[20"?

> BTW, I could reproduce similar symptoms on Debian testing with:
> emacs -Q -nw
> double-mouse-1 on "text"
> C-h k mouse-2

That's a very different issue.  In response to `mouse-2' the xterm sends
a bunch of bytes which turn into an <xterm-paste> event followed by the
pasted text (followed by the "end of paste" event).

I guess we could change the way we handle this "bracketed paste" so that
it generates a single event (an `xterm-paste') and add the text to be
pasted as "payload" directly inside the `xterm-paste' event, but it's
not a very serious issue I think.


        Stefan

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

>    (lookup-key xterm-function-map "\e[200~")

[xterm-paste]

>    (lookup-key (keymap-parent input-decode-map) "\e[200~")

5

> And then same thing for "\e[200"

(keymap (126 . [xterm-paste]))
and
nil

> and "\e[20"?

(keymap
 (48 keymap (126 . [xterm-paste]))
 (59 keymap
     (52 keymap
	 (126 . [M-S-f9]))
     (51 keymap
	 (126 . [M-f9]))
     (54 keymap
	 (126 . [C-S-f9]))
     (53 keymap
	 (126 . [C-f9]))
     (50 keymap
	 (126 . [S-f9])))
 (126 . [f9]))

and

(keymap (94 . [C-f9]) (126 . [f9]))

bug#20444: paste error in emacs -nw

[Stefan Monnier]

>> (lookup-key xterm-function-map "\e[200~")
> [xterm-paste]
[...]

Thanks, so xterm-function-map is fine.
The problem is that it's not properly added to input-decode-map for
some reason.

Could you try to add:

   (add-hook (quote terminal-init-xterm-hook)
             (lambda ()
               (message "t-i-x-h: %S"
                        (lookup-key input-decode-map
                                    "\e[200~"))))
   
   (advice-add (quote xterm-register-default-colors)
               :before
               (lambda (&rest _)
                 (message "x-r-d-c: %S"
                          (lookup-key input-decode-map
                                      "\e[200~"))))

to your ~/.emacs and then run "emacs -nw" (without the -Q), then check
the messages in *Messages*.


        Stefan

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

>    (add-hook (quote terminal-init-xterm-hook)
>              (lambda ()
>                (message "t-i-x-h: %S"
>                         (lookup-key input-decode-map
>                                     "\e[200~"))))
>    
>    (advice-add (quote xterm-register-default-colors)
>                :before
>                (lambda (&rest _)
>                  (message "x-r-d-c: %S"
>                           (lookup-key input-decode-map
>                                       "\e[200~"))))

x-r-d-c: 5
t-i-x-h: 5

bug#20444: paste error in emacs -nw

[Stefan Monnier]

>> (add-hook (quote terminal-init-xterm-hook)
>> (lambda ()
>> (message "t-i-x-h: %S"
>> (lookup-key input-decode-map
>> "\e[200~"))))
>> 
>> (advice-add (quote xterm-register-default-colors)
>> :before
>> (lambda (&rest _)
>> (message "x-r-d-c: %S"
>> (lookup-key input-decode-map
>> "\e[200~"))))

> x-r-d-c: 5
> t-i-x-h: 5

Hmm...

OK, next step:

(advice-add (quote xterm-register-default-colors) :before
            (lambda (&rest _)
              (message "x-r-d-c-1: %S"
                       (lookup-key xterm-function-map
                                   "\e[200~"))
              (message "x-r-d-c-2: %S"
                       (lookup-key (copy-keymap xterm-function-map)
                                   "\e[200~"))
              (message "x-r-d-c-3: %S"
                       (lookup-key (keymap-parent input-decode-map)
                                   "\e[200~"))
              (message "x-r-d-c-4: %S"
                       (lookup-key (let ((map (copy-keymap xterm-function-map)))
                                     (set-keymap-parent map (keymap-parent input-decode-map))
                                     map)
                                   "\e[200~"))
            ))

-- Stefan

PS: While you're at it, could you try the patch below, which shouldn't
make any difference to your problem, but would be good anyway.


diff --git a/lisp/term/xterm.el b/lisp/term/xterm.el
index 79699c6..ec8b1ef 100644
--- a/lisp/term/xterm.el
+++ b/lisp/term/xterm.el
@@ -696,17 +696,16 @@ We run the first FUNCTION whose STRING matches the input events."
 	   (string-match "\\`rxvt" (getenv "COLORTERM" (selected-frame))))
       (tty-run-terminal-initialization (selected-frame) "rxvt")
 
-    (let ((map (copy-keymap xterm-alternatives-map)))
-      (set-keymap-parent map (keymap-parent local-function-key-map))
-      (set-keymap-parent local-function-key-map map))
-
-    (let ((map (copy-keymap xterm-function-map)))
-
-      ;; Use inheritance to let the main keymap override those defaults.
-      ;; This way we don't override terminfo-derived settings or settings
-      ;; made in the init file.
-      (set-keymap-parent map (keymap-parent input-decode-map))
-      (set-keymap-parent input-decode-map map)))
+    (let ((push-map
+           ;; Use inheritance to let the main keymaps override those defaults.
+           ;; This way we don't override terminfo-derived settings or settings
+           ;; made in the init file.
+           (lambda (map basemap)
+             (set-keymap-parent
+              basemap
+              (make-composed-keymap map (keymap-parent basemap))))))
+      (funcall push-map xterm-alternatives-map local-function-key-map)
+      (funcall push-map xterm-function-map     input-decode-map)))
 
   (xterm-register-default-colors)
   (tty-set-up-initial-frame-faces)

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

> (advice-add (quote xterm-register-default-colors) :before
>             (lambda (&rest _)
>               (message "x-r-d-c-1: %S"
>                        (lookup-key xterm-function-map
>                                    "\e[200~"))
>               (message "x-r-d-c-2: %S"
>                        (lookup-key (copy-keymap xterm-function-map)
>                                    "\e[200~"))
>               (message "x-r-d-c-3: %S"
>                        (lookup-key (keymap-parent input-decode-map)
>                                    "\e[200~"))
>               (message "x-r-d-c-4: %S"
>                        (lookup-key (let ((map (copy-keymap xterm-function-map)))
>                                      (set-keymap-parent map (keymap-parent input-decode-map))
>                                      map)
>                                    "\e[200~"))
>             ))


x-r-d-c-1: [xterm-paste]
x-r-d-c-2: [xterm-paste]
x-r-d-c-3: 5
x-r-d-c-4: [xterm-paste]


> PS: While you're at it, could you try the patch below, which shouldn't
> make any difference to your problem, but would be good anyway.

As you say, it doesn't make any difference.

bug#20444: paste error in emacs -nw

[Stefan Monnier]

> x-r-d-c-1: [xterm-paste]
> x-r-d-c-2: [xterm-paste]
> x-r-d-c-3: 5
> x-r-d-c-4: [xterm-paste]

Hmm... I'm really lost here.
One more try:

apply the patch below, and then show me the content of *Messages*.


        Stefan

bug#20444: paste error in emacs -nw

[Glenn Morris]

Stefan Monnier wrote:

> apply the patch below, and then show me the content of *Messages*.

There was no patch.

bug#20444: paste error in emacs -nw

[Stefan Monnier]

>> apply the patch below, and then show me the content of *Messages*.
> There was no patch.

Oh, come on Glenn!  What kind of lame excuse is that!


        Stefan


diff --git a/lisp/term/xterm.el b/lisp/term/xterm.el
index 79699c6..0f3a2ab 100644
--- a/lisp/term/xterm.el
+++ b/lisp/term/xterm.el
@@ -700,6 +700,8 @@ We run the first FUNCTION whose STRING matches the input events."
       (set-keymap-parent map (keymap-parent local-function-key-map))
       (set-keymap-parent local-function-key-map map))
 
+    (message "before, i-d-m = %S" input-decode-map)
+
     (let ((map (copy-keymap xterm-function-map)))
 
       ;; Use inheritance to let the main keymap override those defaults.
@@ -708,6 +710,8 @@ We run the first FUNCTION whose STRING matches the input events."
       (set-keymap-parent map (keymap-parent input-decode-map))
       (set-keymap-parent input-decode-map map)))
 
+  (message "after, i-d-m = %S" input-decode-map)
+
   (xterm-register-default-colors)
   (tty-set-up-initial-frame-faces)
 

bug#20444: paste error in emacs -nw

[Glenn Morris]

I was just going to guess that the answer was "5",
but it turns out it isn't...

Stefan Monnier wrote:

> --- a/lisp/term/xterm.el
> +++ b/lisp/term/xterm.el
> @@ -700,6 +700,8 @@ We run the first FUNCTION whose STRING matches the input events."
>        (set-keymap-parent map (keymap-parent local-function-key-map))
>        (set-keymap-parent local-function-key-map map))
>  
> +    (message "before, i-d-m = %S" input-decode-map)


This bit didn't run. It turns out that I have COLORTERM=rxvt set in my
environment. So I end up in the other branch, that's based on
xterm-alternatives-map rather than xterm-function-map.

Maybe this is a local problem. I'm still using the venerable aterm,
which sets COLORTERM=rxvt. When I start an xterm from that, it inherits
that setting and does not replace it with anything.

(But perhaps xterm-paste also needs to be in xterm-alternatives-map?)

bug#20444: paste error in emacs -nw

[Stefan Monnier]

> This bit didn't run.  It turns out that I have COLORTERM=rxvt set in my
> environment.  So I end up in the other branch, that's based on
> xterm-alternatives-map rather than xterm-function-map.

Aha!

I thought I was going crazy.  OK, I'll take a new look at the code,


        Stefan

bug#20444: paste error in emacs -nw

[Stefan Monnier]

> This bit didn't run. It turns out that I have COLORTERM=rxvt set in my
> environment. So I end up in the other branch, that's based on
> xterm-alternatives-map rather than xterm-function-map.

I could now reproduce the problem by setting COLORTERM=rxvt.
I've pushed a fix for it.  Thanks,


        Stefan