bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

I have observed this behavior from MacOS, with an Emacs running either
locally on MacOS, or over SSH (running on Linux).  Without any
modifications, a -Q invocation causes "xterm--pasted-text: Failed
select: Invalid argument", but without -Q it simply hangs.  It can
attach emacsclient sessions, but they do not accept input.

1. Start "emacs -nw"
2. M-x term <ret>
3. In another application, copy some text into the clipboard, it
doesn't matter what.
4. Return to the Emacs session, and paste into the terminal (I used Command-V).
4a. Emacs no longer responds.  You have to kill (not kill -9, mind
you).  Running strace makes it seem like it's missing some pselect
call, but I haven't quite sniffed out the issue.
4b. With -Q, it  more helpfully prints "xterm--pasted-text: Failed
select: Invalid argument" and continues functioning, but this is
clearly not the desired behavior either.

I no longer have (direct) access to a Linux, XTerm, so I've been
running into the problem since Emacs 26.3 on Linux *through* the Mac
OS Terminal application.  Emacs doesn't have to be the version listed
below, in other words.

I will try to update this bug when I figure out what thing in my dot
file causes the hang instead of the delay+error message.  All I have
relating to terminal-mode is a hook that sets two key-bindings.  When
I manually add this in -Q, it still doesn't hang.  Perhaps it's some
other mode that loads outside my dot file.

This problem is actually super annoying :) as I will have hundreds of
files open, as well as a bunch of terminals in an emacs daemon
process, and accidentally I will forget and paste some text into the
session, at which point I will have to open a second terminal to this
remote host, find and kill the emacs process, then rearrange all my
things back to what I was doing.  It happens once a day at this
point...

In GNU Emacs 27.2 (build 1, x86_64-apple-darwin18.7.0, NS
appkit-1671.60 Version 10.14.6 (Build 18G95))
 of 2021-03-27 built on builder10-14.porkrind.org
Windowing system distributor 'Apple', version 10.3.2022
System Description:  macOS 11.4

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...
Quit [2 times]
Making completion list...

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS JSON PDUMPER GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  global-whitespace-mode: t
  display-battery-mode: t
  show-paren-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq gv mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils disp-table whitespace
battery paren time byte-opt bytecomp byte-compile cconv advice tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information:
((conses 16 48031 7233)
 (symbols 48 6582 1)
 (strings 32 17172 1862)
 (string-bytes 1 572362)
 (vectors 16 10205)
 (vector-slots 8 127980 10714)
 (floats 8 28 24)
 (intervals 56 200 0)
 (buffers 1000 12))

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 10:28:59 -0400
> 
> I have observed this behavior from MacOS, with an Emacs running either
> locally on MacOS, or over SSH (running on Linux).  Without any
> modifications, a -Q invocation causes "xterm--pasted-text: Failed
> select: Invalid argument", but without -Q it simply hangs.  It can
> attach emacsclient sessions, but they do not accept input.

What kind of terminal emulator is actually being used, and does it
support the xterm X selection protocol?

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

This is the Mac Terminal application that comes out of the box with
MacOS.  It fully supports xterm-256color as the TERM type, and I can
actually paste just fine in other parts of Emacs.  Although I'm having
trouble getting to a Linux terminal emulator these days, I realized
that I do have a machine running WIndows.  I ran the Cygwin terminal
emulator, MinTTY, started (terminal, Cygwin -- i.e., not the Win32
native) Emacs, and it exhibits the same behavior.  Something more
fundamental seems to be going wrong.

On Mon, Jun 28, 2021 at 12:57 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Matt Bisson <bisson.m@gmail.com>
> > Date: Mon, 28 Jun 2021 10:28:59 -0400
> >
> > I have observed this behavior from MacOS, with an Emacs running either
> > locally on MacOS, or over SSH (running on Linux).  Without any
> > modifications, a -Q invocation causes "xterm--pasted-text: Failed
> > select: Invalid argument", but without -Q it simply hangs.  It can
> > attach emacsclient sessions, but they do not accept input.
>
> What kind of terminal emulator is actually being used, and does it
> support the xterm X selection protocol?

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 13:18:04 -0400
> Cc: 49253@debbugs.gnu.org
> 
> This is the Mac Terminal application that comes out of the box with
> MacOS.  It fully supports xterm-256color as the TERM type, and I can
> actually paste just fine in other parts of Emacs.  Although I'm having
> trouble getting to a Linux terminal emulator these days, I realized
> that I do have a machine running WIndows.  I ran the Cygwin terminal
> emulator, MinTTY, started (terminal, Cygwin -- i.e., not the Win32
> native) Emacs, and it exhibits the same behavior.  Something more
> fundamental seems to be going wrong.

I think you'll have to run this under GDB and see what kind of
"invalid argument" this scenario triggers on your systems.  The same
issue on multiple system with several terminal emulators almost always
means some local setting problem.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

Looking at the (E-Lisp) function, it's not obvious to me where I
should put a (native-code) breakpoint.  Any thought?  Of course I can
figure something out, but if you happen to know...

On Mon, Jun 28, 2021 at 1:27 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Matt Bisson <bisson.m@gmail.com>
> > Date: Mon, 28 Jun 2021 13:18:04 -0400
> > Cc: 49253@debbugs.gnu.org
> >
> > This is the Mac Terminal application that comes out of the box with
> > MacOS.  It fully supports xterm-256color as the TERM type, and I can
> > actually paste just fine in other parts of Emacs.  Although I'm having
> > trouble getting to a Linux terminal emulator these days, I realized
> > that I do have a machine running WIndows.  I ran the Cygwin terminal
> > emulator, MinTTY, started (terminal, Cygwin -- i.e., not the Win32
> > native) Emacs, and it exhibits the same behavior.  Something more
> > fundamental seems to be going wrong.
>
> I think you'll have to run this under GDB and see what kind of
> "invalid argument" this scenario triggers on your systems.  The same
> issue on multiple system with several terminal emulators almost always
> means some local setting problem.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

Given that it happens on two different windowing systems, with two
different terminal emulators, and two different builds of Emacs, and I
get the error message with "-Q", I'm really not sure what local issue
that could be, given that there's very little in common between the
two sites -- except Emacs itself.  Again, I will look, but I'm not
sure what your hypothesis is.

On Mon, Jun 28, 2021 at 1:30 PM Matt Bisson <bisson.m@gmail.com> wrote:
>
> Looking at the (E-Lisp) function, it's not obvious to me where I
> should put a (native-code) breakpoint.  Any thought?  Of course I can
> figure something out, but if you happen to know...
>
> On Mon, Jun 28, 2021 at 1:27 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Matt Bisson <bisson.m@gmail.com>
> > > Date: Mon, 28 Jun 2021 13:18:04 -0400
> > > Cc: 49253@debbugs.gnu.org
> > >
> > > This is the Mac Terminal application that comes out of the box with
> > > MacOS.  It fully supports xterm-256color as the TERM type, and I can
> > > actually paste just fine in other parts of Emacs.  Although I'm having
> > > trouble getting to a Linux terminal emulator these days, I realized
> > > that I do have a machine running WIndows.  I ran the Cygwin terminal
> > > emulator, MinTTY, started (terminal, Cygwin -- i.e., not the Win32
> > > native) Emacs, and it exhibits the same behavior.  Something more
> > > fundamental seems to be going wrong.
> >
> > I think you'll have to run this under GDB and see what kind of
> > "invalid argument" this scenario triggers on your systems.  The same
> > issue on multiple system with several terminal emulators almost always
> > means some local setting problem.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 13:34:41 -0400
> Cc: 49253@debbugs.gnu.org
> 
> Given that it happens on two different windowing systems, with two
> different terminal emulators, and two different builds of Emacs, and I
> get the error message with "-Q", I'm really not sure what local issue
> that could be

Some input Emacs gets fed at startup, which it doesn't expect, and
which looks like a beginning of a paste sequence?  This does happen
right at startup, yes?

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 13:30:19 -0400
> Cc: 49253@debbugs.gnu.org
> 
> Looking at the (E-Lisp) function, it's not obvious to me where I
> should put a (native-code) breakpoint.  Any thought?  Of course I can
> figure something out, but if you happen to know...

No, it's the other way around: you start Emacs from GDB.  There are
some instructions in etc/DEBUG.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

> Some input Emacs gets fed at startup, which it doesn't expect, and
> which looks like a beginning of a paste sequence?  This does happen
> right at startup, yes?

No.  (Unless I'm misunderstanding what you're asking) the problem can
occur at any time.  It's in response to simply pasting from the
clipboard into a terminal application.  I can start Emacs, run for
days normally -- using terminal-mode, editing files, and so forth.
The second I go into the terminal-mode buffer and paste from the
(Windows, Mac, whatever) clipboard (NOT using the Emacs yank command),
the problem happens.  If I never paste from the windowing system into
my terminal emulator, there are not problems, but invariably I find
some huge chunk of text, go to paste it into Emacs, and forget that
this will be a problem, and everything locks up irreversibly.  It is
as if there is some race with multiple parties asking for select(),
but I don't know the Emacs threading model yet.  TBH, I assumed it was
kind of single-threaded. :)

If your statement is more that the beginning of the sequence retrieved
from the xterm paste incantation occurs "at the start", then for that
I will have to debug into GDB, as we talked about.

> No, it's the other way around: you start Emacs from GDB.  There are
> some instructions in etc/DEBUG.

Yes, I do know that's what you mean :) but it's not as if Emacs is
going to crash, and stop in the debugger.  So my question is
basically, what src/*.c line should I set a breakpoint on to observe
the thing you would like me to observe?  If you can't say, that's
perfectly reasonable.  That said, as I type this, I can try to
interrupt Emacs when it's hung and see anything that's going on, but I
believe it will be after the problematic event has occurred.

On Mon, Jun 28, 2021 at 2:00 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Matt Bisson <bisson.m@gmail.com>
> > Date: Mon, 28 Jun 2021 13:30:19 -0400
> > Cc: 49253@debbugs.gnu.org
> >
> > Looking at the (E-Lisp) function, it's not obvious to me where I
> > should put a (native-code) breakpoint.  Any thought?  Of course I can
> > figure something out, but if you happen to know...
>
> No, it's the other way around: you start Emacs from GDB.  There are
> some instructions in etc/DEBUG.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 14:11:56 -0400
> Cc: 49253@debbugs.gnu.org
> 
> No.  (Unless I'm misunderstanding what you're asking) the problem can
> occur at any time.  It's in response to simply pasting from the
> clipboard into a terminal application.  I can start Emacs, run for
> days normally -- using terminal-mode, editing files, and so forth.
> The second I go into the terminal-mode buffer and paste from the
> (Windows, Mac, whatever) clipboard (NOT using the Emacs yank command),
> the problem happens.

Wait a minute: what do you mean by "terminal-mode buffer" into which
you need to go?  More generally, can you describe a detailed recipe
for reproducing the problem on your system, keystroke by keystroke?

> > No, it's the other way around: you start Emacs from GDB.  There are
> > some instructions in etc/DEBUG.
> 
> Yes, I do know that's what you mean :) but it's not as if Emacs is
> going to crash, and stop in the debugger.

You are investigating the error message saying "Failed select".
Search the *.c files for that text, and you will find only 3 instances
of it.  Put a breakpoint in all these 3 places, then do whatever it
takes to reproduce the problem, and you will be able to collect the
information I asked about.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

That's described in the initial report.  I see it's called "term-mode"
not "terminal-mode" which was a brain-fart on my part.  The steps
(term-mode in step 2):

1. Start "emacs -nw" <ret>
2. M-x term <ret> <ret>
3. In another application (for the purposes of "exact keystroke, let's
say, Alt-TAB to Firefox"), copy some text into the clipboard, it
doesn't matter what.
4. Return to the Emacs session (click on the Terminal emulator,
Alt-TAB, whatever*), and paste into the terminal (I used Command-V).
4* To be clear, return the cursor to the term-mode buffer (e.g., C-b
*terminal* <ret>).

Inject any mount of delays or operations between steps 1, 2, 3, and 4.
The bug essentially is "if you ever paste into term-mode buffers, you
lose.




On Mon, Jun 28, 2021 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Matt Bisson <bisson.m@gmail.com>
> > Date: Mon, 28 Jun 2021 14:11:56 -0400
> > Cc: 49253@debbugs.gnu.org
> >
> > No.  (Unless I'm misunderstanding what you're asking) the problem can
> > occur at any time.  It's in response to simply pasting from the
> > clipboard into a terminal application.  I can start Emacs, run for
> > days normally -- using terminal-mode, editing files, and so forth.
> > The second I go into the terminal-mode buffer and paste from the
> > (Windows, Mac, whatever) clipboard (NOT using the Emacs yank command),
> > the problem happens.
>
> Wait a minute: what do you mean by "terminal-mode buffer" into which
> you need to go?  More generally, can you describe a detailed recipe
> for reproducing the problem on your system, keystroke by keystroke?
>
> > > No, it's the other way around: you start Emacs from GDB.  There are
> > > some instructions in etc/DEBUG.
> >
> > Yes, I do know that's what you mean :) but it's not as if Emacs is
> > going to crash, and stop in the debugger.
>
> You are investigating the error message saying "Failed select".
> Search the *.c files for that text, and you will find only 3 instances
> of it.  Put a breakpoint in all these 3 places, then do whatever it
> takes to reproduce the problem, and you will be able to collect the
> information I asked about.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> From: Matt Bisson <bisson.m@gmail.com>
> Date: Mon, 28 Jun 2021 16:09:24 -0400
> Cc: 49253@debbugs.gnu.org
> 
> 1. Start "emacs -nw" <ret>
> 2. M-x term <ret> <ret>
> 3. In another application (for the purposes of "exact keystroke, let's
> say, Alt-TAB to Firefox"), copy some text into the clipboard, it
> doesn't matter what.
> 4. Return to the Emacs session (click on the Terminal emulator,
> Alt-TAB, whatever*), and paste into the terminal (I used Command-V).
> 4* To be clear, return the cursor to the term-mode buffer (e.g., C-b
> *terminal* <ret>).

Maybe term-mode is simply incompatible with bracketed-paste feature
(because they both call low-level keyboard input functions)?

Does the problem go away if you switch to term-line-mode?  If not, my
suggestion is to disable the bracketed-paste support.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

Ah, I didn't think about just term-line-mode.  I'd have expected it to
work, and indeed it does.  No problems in that mode.

I'm still working on debugging, fwiw (not a lot of time today).  My
ToT emacs repo wasn't building.

On Tue, Jun 29, 2021 at 8:09 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Matt Bisson <bisson.m@gmail.com>
> > Date: Mon, 28 Jun 2021 16:09:24 -0400
> > Cc: 49253@debbugs.gnu.org
> >
> > 1. Start "emacs -nw" <ret>
> > 2. M-x term <ret> <ret>
> > 3. In another application (for the purposes of "exact keystroke, let's
> > say, Alt-TAB to Firefox"), copy some text into the clipboard, it
> > doesn't matter what.
> > 4. Return to the Emacs session (click on the Terminal emulator,
> > Alt-TAB, whatever*), and paste into the terminal (I used Command-V).
> > 4* To be clear, return the cursor to the term-mode buffer (e.g., C-b
> > *terminal* <ret>).
>
> Maybe term-mode is simply incompatible with bracketed-paste feature
> (because they both call low-level keyboard input functions)?
>
> Does the problem go away if you switch to term-line-mode?  If not, my
> suggestion is to disable the bracketed-paste support.

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Lars Ingebrigtsen]

Matt Bisson <bisson.m@gmail.com> writes:

> I have observed this behavior from MacOS, with an Emacs running either
> locally on MacOS, or over SSH (running on Linux).  Without any
> modifications, a -Q invocation causes "xterm--pasted-text: Failed
> select: Invalid argument", but without -Q it simply hangs.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Are you still seeing this problem in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

For sure.  Even in emacs 28.2, if I want to paste something into the
terminal buffer, I have to switch to line mode, and back when I'm done.
You just can't paste into the terminal buffer in terminal emacs in key mode.

On Sat, Jul 16, 2022, 8:21 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Matt Bisson <bisson.m@gmail.com> writes:
>
> > I have observed this behavior from MacOS, with an Emacs running either
> > locally on MacOS, or over SSH (running on Linux).  Without any
> > modifications, a -Q invocation causes "xterm--pasted-text: Failed
> > select: Invalid argument", but without -Q it simply hangs.
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> Are you still seeing this problem in recent Emacs versions?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: Type: text/html, Size: 1392 bytes --]

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

I've finally had some time to look at this and what's happening is
term/xterm.el is fighting with term.el.  The bracketed paste command comes
in with "\e[200~", arrives in term/xterm.el's xterm--pasted-text function
first.  It correctly reads the text from the clipboard.  This is because
xterm-translate-bracketed-paste is registered as a key binding for
[xterm-paste] (or "\e[200~" on RXVT).

Immediately thereafter, term--xterm-paste in term.el notices the
[xterm-paste] key as well (it has also registered a binding for "raw"
mode), and begins the same process of xterm--pasted-text.  At this point,
there's nothing to read from read-event, and it hangs for
most-positive-fixnum (basically, forever).

So this is the root cause analysis.  Unfortunately, naively removing the
key binding from term.el does not work, because the pasted text must be
inserted via term-send-raw-string.

-Matt

On Sat, Jul 16, 2022 at 3:13 PM Matt Bisson <bisson.m@gmail.com> wrote:

> For sure.  Even in emacs 28.2, if I want to paste something into the
> terminal buffer, I have to switch to line mode, and back when I'm done.
> You just can't paste into the terminal buffer in terminal emacs in key mode.
>
> On Sat, Jul 16, 2022, 8:21 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> Matt Bisson <bisson.m@gmail.com> writes:
>>
>> > I have observed this behavior from MacOS, with an Emacs running either
>> > locally on MacOS, or over SSH (running on Linux).  Without any
>> > modifications, a -Q invocation causes "xterm--pasted-text: Failed
>> > select: Invalid argument", but without -Q it simply hangs.
>>
>> (I'm going through old bug reports that unfortunately weren't resolved
>> at the time.)
>>
>> Are you still seeing this problem in recent Emacs versions?
>>
>> --
>> (domestic pets only, the antidote for overdose, milk.)
>>    bloggy blog: http://lars.ingebrigtsen.no
>>
>

[-- Attachment #2: Type: text/html, Size: 2817 bytes --]

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

Right, so this silly little fix works, but it's an obvious hack.  I
clearly don't know enough about how the key-bindings work, because I
would have expected those set by term-mode to override those in the
global keybindings (from xterm.el), but they don't.  The xterm.el one
runs first.  Here's the diff:

diff --git a/lisp/term.el b/lisp/term.el
index 81746e0c20d..e047fa767e8 100644
--- a/lisp/term.el
+++ b/lisp/term.el
@@ -956,6 +956,7 @@ term-raw-map
     (define-key map [?\C- ] #'term-send-C-@)
     (define-key map [?\C-\M-/] #'term-send-C-M-_)
     (define-key map [?\C-\M- ] #'term-send-C-M-@)
+    (define-key map "\e[200~" #'term--xterm-paste)

     (when term-bind-function-keys
       (dotimes (key 21)
diff --git a/lisp/term/xterm.el b/lisp/term/xterm.el
index 5ed4e46e0a5..117bd131123 100644
--- a/lisp/term/xterm.el
+++ b/lisp/term/xterm.el
@@ -149,7 +149,8 @@ xterm--suspend-tty-function
 ;; looping on read-key and buffering input for later processing.

 (defun xterm-translate-bracketed-paste (_prompt)
-  (vector (list 'xterm-paste (xterm--pasted-text))))
+  (unless (and (eq major-mode 'term-mode) (term-in-char-mode))
+    (vector (list 'xterm-paste (xterm--pasted-text)))))

 (defvar xterm-rxvt-function-map
   (let ((map (make-sparse-keymap)))


On Fri, Dec 8, 2023 at 2:06 PM Matt Bisson <bisson.m@gmail.com> wrote:
>
> I've finally had some time to look at this and what's happening is term/xterm.el is fighting with term.el.  The bracketed paste command comes in with "\e[200~", arrives in term/xterm.el's xterm--pasted-text function first.  It correctly reads the text from the clipboard.  This is because xterm-translate-bracketed-paste is registered as a key binding for [xterm-paste] (or "\e[200~" on RXVT).
>
> Immediately thereafter, term--xterm-paste in term.el notices the [xterm-paste] key as well (it has also registered a binding for "raw" mode), and begins the same process of xterm--pasted-text.  At this point, there's nothing to read from read-event, and it hangs for most-positive-fixnum (basically, forever).
>
> So this is the root cause analysis.  Unfortunately, naively removing the key binding from term.el does not work, because the pasted text must be inserted via term-send-raw-string.
>
> -Matt
>
> On Sat, Jul 16, 2022 at 3:13 PM Matt Bisson <bisson.m@gmail.com> wrote:
>>
>> For sure.  Even in emacs 28.2, if I want to paste something into the terminal buffer, I have to switch to line mode, and back when I'm done.  You just can't paste into the terminal buffer in terminal emacs in key mode.
>>
>> On Sat, Jul 16, 2022, 8:21 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>>>
>>> Matt Bisson <bisson.m@gmail.com> writes:
>>>
>>> > I have observed this behavior from MacOS, with an Emacs running either
>>> > locally on MacOS, or over SSH (running on Linux).  Without any
>>> > modifications, a -Q invocation causes "xterm--pasted-text: Failed
>>> > select: Invalid argument", but without -Q it simply hangs.
>>>
>>> (I'm going through old bug reports that unfortunately weren't resolved
>>> at the time.)
>>>
>>> Are you still seeing this problem in recent Emacs versions?
>>>
>>> --
>>> (domestic pets only, the antidote for overdose, milk.)
>>>    bloggy blog: http://lars.ingebrigtsen.no

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

Jared, can I ask you to look into this and share your thoughts and
comments to the proposed change?  TIA.

> Cc: 49253@debbugs.gnu.org
> From: Matt Bisson <bisson.m@gmail.com>
> Date: Fri, 8 Dec 2023 15:35:36 -0500
> 
> Right, so this silly little fix works, but it's an obvious hack.  I
> clearly don't know enough about how the key-bindings work, because I
> would have expected those set by term-mode to override those in the
> global keybindings (from xterm.el), but they don't.  The xterm.el one
> runs first.  Here's the diff:
> 
> diff --git a/lisp/term.el b/lisp/term.el
> index 81746e0c20d..e047fa767e8 100644
> --- a/lisp/term.el
> +++ b/lisp/term.el
> @@ -956,6 +956,7 @@ term-raw-map
>      (define-key map [?\C- ] #'term-send-C-@)
>      (define-key map [?\C-\M-/] #'term-send-C-M-_)
>      (define-key map [?\C-\M- ] #'term-send-C-M-@)
> +    (define-key map "\e[200~" #'term--xterm-paste)
> 
>      (when term-bind-function-keys
>        (dotimes (key 21)
> diff --git a/lisp/term/xterm.el b/lisp/term/xterm.el
> index 5ed4e46e0a5..117bd131123 100644
> --- a/lisp/term/xterm.el
> +++ b/lisp/term/xterm.el
> @@ -149,7 +149,8 @@ xterm--suspend-tty-function
>  ;; looping on read-key and buffering input for later processing.
> 
>  (defun xterm-translate-bracketed-paste (_prompt)
> -  (vector (list 'xterm-paste (xterm--pasted-text))))
> +  (unless (and (eq major-mode 'term-mode) (term-in-char-mode))
> +    (vector (list 'xterm-paste (xterm--pasted-text)))))
> 
>  (defvar xterm-rxvt-function-map
>    (let ((map (make-sparse-keymap)))
> 
> 
> On Fri, Dec 8, 2023 at 2:06 PM Matt Bisson <bisson.m@gmail.com> wrote:
> >
> > I've finally had some time to look at this and what's happening is term/xterm.el is fighting with term.el.  The bracketed paste command comes in with "\e[200~", arrives in term/xterm.el's xterm--pasted-text function first.  It correctly reads the text from the clipboard.  This is because xterm-translate-bracketed-paste is registered as a key binding for [xterm-paste] (or "\e[200~" on RXVT).
> >
> > Immediately thereafter, term--xterm-paste in term.el notices the [xterm-paste] key as well (it has also registered a binding for "raw" mode), and begins the same process of xterm--pasted-text.  At this point, there's nothing to read from read-event, and it hangs for most-positive-fixnum (basically, forever).
> >
> > So this is the root cause analysis.  Unfortunately, naively removing the key binding from term.el does not work, because the pasted text must be inserted via term-send-raw-string.
> >
> > -Matt
> >
> > On Sat, Jul 16, 2022 at 3:13 PM Matt Bisson <bisson.m@gmail.com> wrote:
> >>
> >> For sure.  Even in emacs 28.2, if I want to paste something into the terminal buffer, I have to switch to line mode, and back when I'm done.  You just can't paste into the terminal buffer in terminal emacs in key mode.
> >>
> >> On Sat, Jul 16, 2022, 8:21 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
> >>>
> >>> Matt Bisson <bisson.m@gmail.com> writes:
> >>>
> >>> > I have observed this behavior from MacOS, with an Emacs running either
> >>> > locally on MacOS, or over SSH (running on Linux).  Without any
> >>> > modifications, a -Q invocation causes "xterm--pasted-text: Failed
> >>> > select: Invalid argument", but without -Q it simply hangs.
> >>>
> >>> (I'm going through old bug reports that unfortunately weren't resolved
> >>> at the time.)
> >>>
> >>> Are you still seeing this problem in recent Emacs versions?
> >>>
> >>> --
> >>> (domestic pets only, the antidote for overdose, milk.)
> >>>    bloggy blog: http://lars.ingebrigtsen.no
> 
> 
> 
> 

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Jared Finder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 2023-12-09 03:10, Eli Zaretskii wrote:
> Jared, can I ask you to look into this and share your thoughts and
> comments to the proposed change?  TIA.
> 
>> Cc: 49253@debbugs.gnu.org
>> From: Matt Bisson <bisson.m@gmail.com>
>> Date: Fri, 8 Dec 2023 15:35:36 -0500
>> 
>> Right, so this silly little fix works, but it's an obvious hack.  I
>> clearly don't know enough about how the key-bindings work, because I
>> would have expected those set by term-mode to override those in the
>> global keybindings (from xterm.el), but they don't.  The xterm.el one
>> runs first.  Here's the diff:
>> 
>> diff --git a/lisp/term.el b/lisp/term.el
>> index 81746e0c20d..e047fa767e8 100644
>> --- a/lisp/term.el
>> +++ b/lisp/term.el
>> @@ -956,6 +956,7 @@ term-raw-map
>>      (define-key map [?\C- ] #'term-send-C-@)
>>      (define-key map [?\C-\M-/] #'term-send-C-M-_)
>>      (define-key map [?\C-\M- ] #'term-send-C-M-@)
>> +    (define-key map "\e[200~" #'term--xterm-paste)
>> 
>>      (when term-bind-function-keys
>>        (dotimes (key 21)

Thanks for the investigation.  I don't think this is the right approach. 
  The xterm escape codes all go through input-decode-map and I would 
expect to preserve that.

Looking at code, the current behavior in xterm.el is the following:

Step 1: \e[200~ is put on input-decode-map, using 
xterm-translate-backeted-paste to decode.

Step 2: The function xterm-translate-bracketed-paste reads the pasted 
text and creates an event (xterm-paste "PASTED TEXT HERE")

Step 3: Globally, the event xterm-paste is bound to the function also 
named xterm-paste, which grabs the pasted text and puts it on the kill 
ring, then runs the function insert-for-yank.

Step 3a: In Term mode, the event xterm-paste is instead bound to the 
function term--xterm-paste.  However, instead of reading the pasted text 
off the event, it calls (xterm-pasted-text) again.

I think the correct fix is to change term--xterm-paste to read the 
pasted text off of the event generated in Step 2.  So something like the 
following (a real fix would add error checking like in xterm-paste):

(defun term--xterm-paste (event)
   "Insert the text pasted in an XTerm bracketed paste operation."
   (interactive "e")
   (term-send-raw-string (nth 1 event)))

Matt, can you try this change locally?  It worked for me.

   -- MJF

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

TL;DR: It works!

> I don't think this is the right approach.

Totally! :)  The patch was definitely not a serious suggestion of what
should be submitted, but just a demonstration that the interplay
between the two functions is causing the issue, and it can be solved
by touching only that.  I haven't built up the knowledge of how these
events function like you have, and was hoping you'd teach me a bit in
response, and you did!

The solution works for XTerm pasting into a scratch buffer, into a
term buffer with line-mode enabled, and most notably, with char-mode
enabled.  As I use term-mode for basically everything I do in a
terminal on a daily basis, this will make my life significantly nicer,
so thanks for the fix!

On Sat, Dec 9, 2023 at 4:38 PM Jared Finder <jared@finder.org> wrote:
>
> On 2023-12-09 03:10, Eli Zaretskii wrote:
> > Jared, can I ask you to look into this and share your thoughts and
> > comments to the proposed change?  TIA.
> >
> >> Cc: 49253@debbugs.gnu.org
> >> From: Matt Bisson <bisson.m@gmail.com>
> >> Date: Fri, 8 Dec 2023 15:35:36 -0500
> >>
> >> Right, so this silly little fix works, but it's an obvious hack.  I
> >> clearly don't know enough about how the key-bindings work, because I
> >> would have expected those set by term-mode to override those in the
> >> global keybindings (from xterm.el), but they don't.  The xterm.el one
> >> runs first.  Here's the diff:
> >>
> >> diff --git a/lisp/term.el b/lisp/term.el
> >> index 81746e0c20d..e047fa767e8 100644
> >> --- a/lisp/term.el
> >> +++ b/lisp/term.el
> >> @@ -956,6 +956,7 @@ term-raw-map
> >>      (define-key map [?\C- ] #'term-send-C-@)
> >>      (define-key map [?\C-\M-/] #'term-send-C-M-_)
> >>      (define-key map [?\C-\M- ] #'term-send-C-M-@)
> >> +    (define-key map "\e[200~" #'term--xterm-paste)
> >>
> >>      (when term-bind-function-keys
> >>        (dotimes (key 21)
>
> Thanks for the investigation.  I don't think this is the right approach.
>   The xterm escape codes all go through input-decode-map and I would
> expect to preserve that.
>
> Looking at code, the current behavior in xterm.el is the following:
>
> Step 1: \e[200~ is put on input-decode-map, using
> xterm-translate-backeted-paste to decode.
>
> Step 2: The function xterm-translate-bracketed-paste reads the pasted
> text and creates an event (xterm-paste "PASTED TEXT HERE")
>
> Step 3: Globally, the event xterm-paste is bound to the function also
> named xterm-paste, which grabs the pasted text and puts it on the kill
> ring, then runs the function insert-for-yank.
>
> Step 3a: In Term mode, the event xterm-paste is instead bound to the
> function term--xterm-paste.  However, instead of reading the pasted text
> off the event, it calls (xterm-pasted-text) again.
>
> I think the correct fix is to change term--xterm-paste to read the
> pasted text off of the event generated in Step 2.  So something like the
> following (a real fix would add error checking like in xterm-paste):
>
> (defun term--xterm-paste (event)
>    "Insert the text pasted in an XTerm bracketed paste operation."
>    (interactive "e")
>    (term-send-raw-string (nth 1 event)))
>
> Matt, can you try this change locally?  It worked for me.
>
>    -- MJF

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Jared Finder via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On 2023-12-11 08:10, Matt Bisson wrote:
> TL;DR: It works!

Thank you for the testing!

Eli, I think the fully correct definition for term--xterm-paste is the 
following, which has additional error checking:

(defun term--xterm-paste (event)
   "Insert the text pasted in an XTerm bracketed paste operation."
   (interactive "e")
   (unless (eq (car-safe event) 'xterm-paste)
     (error "term--xterm-paste must be found to xterm-paste event"))
   (let ((str (nth 1 event)))
     (unless (stringp str)
       (error "term--xterm-paste provided event does not contain paste 
text"))
     (term-send-raw-string str)))

>> I don't think this is the right approach.
> 
> Totally! :)  The patch was definitely not a serious suggestion of what
> should be submitted, but just a demonstration that the interplay
> between the two functions is causing the issue, and it can be solved
> by touching only that.  I haven't built up the knowledge of how these
> events function like you have, and was hoping you'd teach me a bit in
> response, and you did!

No worries!  I hope I didn't come across as judgemental here. :)  If you 
want a bit more detail around the architecture here, read on (Eli, 
please correct if I get anything wrong):

In Emacs, in addition to the local and global keymaps, there are also 
translation keymaps. 
(https://www.gnu.org/software/emacs/manual/html_node/elisp/Translation-Keymaps.html) 
  The docs are accurate but a bit hard to understand without concrete 
examples.  So let me provide an example for each of the translation 
keymaps.  There are three of them:

1. input-decode-map.  This exists to translate terminal escape sequences 
into the their proper events.  That can be for keys (like the PF1 key 
mentioned in the docs) but it also an be for mouse clicks or the xterm 
paste operation.

EXAMPLE:
This bug!  In this bug, input-decode-map is what translates the 
character sequence "ESC [ 2 0 0 ~ PASTED TEXT HERE ESC [ 2 0 1 ~"  into 
a single <xterm-paste> event, (xterm-paste "PASTED TEXT HERE").  It does 
this by mapping "ESC [ 2 0 0 ~" to a function that reads until 
encountering the "ESC [ 2 0 1 ~" sequence and returns that new event.

2. (local-)function-key-map.  This exists to rename keys to more 
preferred names that allow keybindings to be shared.  The docs say "the 
remapping only applies if the original key sequence would otherwise not 
have any binding", this is to allow you to use the native keynames on 
your keyboard if you do want to distinguish.

EXAMPLE:
Many keyboards have a separate keypad number area.  These keys get their 
own events, e.g. "<kp-0>" for the zero character on the keypad.  Emacs 
uses function-key-map to translate "<kp-0>" into "0", so pressing the 
keypad 0 acts the same as the number row 0.  If you wanted to map the 
keypad numbers to a different set of commands, you can still do so with 
the original "<kp-0>" key.  If you do so, that also implicitly overrides 
this translation.

3. key-translation-map.  I've never actually used this directly so I'm 
not entirely clear on its intent.  The docs make it sound like it is 
intended for changing keyboard layouts.  Which explains why I have never 
used it -- I generally use OSes that allow keyboard layout translating 
natively.

EXAMPLE:
Based on my guess above, key-translation-map would let me map between 
QWERTY and Dvorak inside Emacs.  This would be nice if I was running 
directly in a Linux terminal and did not have permission to run "sudo 
loadkeys".

As a final note, notice that all three of these examples involve 
changing a key into a different key.  None of them are for binding 
commands.  Commands being bound to a key always is the job of the local 
and global keymaps.

I hope these examples help you understand the setup here!

   -- MJF

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Matt Bisson]

I definitely want to give the feedback here that this was an
outstanding explanation of what's going on inside Emacs, and thanks
very much for taking the time to write it.  I didn't even know of the
existence of some of these maps, and I now know not only that, but
their intended purpose.  Again, thanks a lot for this!

On Mon, Dec 11, 2023 at 5:13 PM Jared Finder <jared@finder.org> wrote:
>
> On 2023-12-11 08:10, Matt Bisson wrote:
> > TL;DR: It works!
>
> Thank you for the testing!
>
> Eli, I think the fully correct definition for term--xterm-paste is the
> following, which has additional error checking:
>
> (defun term--xterm-paste (event)
>    "Insert the text pasted in an XTerm bracketed paste operation."
>    (interactive "e")
>    (unless (eq (car-safe event) 'xterm-paste)
>      (error "term--xterm-paste must be found to xterm-paste event"))
>    (let ((str (nth 1 event)))
>      (unless (stringp str)
>        (error "term--xterm-paste provided event does not contain paste
> text"))
>      (term-send-raw-string str)))
>
> >> I don't think this is the right approach.
> >
> > Totally! :)  The patch was definitely not a serious suggestion of what
> > should be submitted, but just a demonstration that the interplay
> > between the two functions is causing the issue, and it can be solved
> > by touching only that.  I haven't built up the knowledge of how these
> > events function like you have, and was hoping you'd teach me a bit in
> > response, and you did!
>
> No worries!  I hope I didn't come across as judgemental here. :)  If you
> want a bit more detail around the architecture here, read on (Eli,
> please correct if I get anything wrong):
>
> In Emacs, in addition to the local and global keymaps, there are also
> translation keymaps.
> (https://www.gnu.org/software/emacs/manual/html_node/elisp/Translation-Keymaps.html)
>   The docs are accurate but a bit hard to understand without concrete
> examples.  So let me provide an example for each of the translation
> keymaps.  There are three of them:
>
> 1. input-decode-map.  This exists to translate terminal escape sequences
> into the their proper events.  That can be for keys (like the PF1 key
> mentioned in the docs) but it also an be for mouse clicks or the xterm
> paste operation.
>
> EXAMPLE:
> This bug!  In this bug, input-decode-map is what translates the
> character sequence "ESC [ 2 0 0 ~ PASTED TEXT HERE ESC [ 2 0 1 ~"  into
> a single <xterm-paste> event, (xterm-paste "PASTED TEXT HERE").  It does
> this by mapping "ESC [ 2 0 0 ~" to a function that reads until
> encountering the "ESC [ 2 0 1 ~" sequence and returns that new event.
>
> 2. (local-)function-key-map.  This exists to rename keys to more
> preferred names that allow keybindings to be shared.  The docs say "the
> remapping only applies if the original key sequence would otherwise not
> have any binding", this is to allow you to use the native keynames on
> your keyboard if you do want to distinguish.
>
> EXAMPLE:
> Many keyboards have a separate keypad number area.  These keys get their
> own events, e.g. "<kp-0>" for the zero character on the keypad.  Emacs
> uses function-key-map to translate "<kp-0>" into "0", so pressing the
> keypad 0 acts the same as the number row 0.  If you wanted to map the
> keypad numbers to a different set of commands, you can still do so with
> the original "<kp-0>" key.  If you do so, that also implicitly overrides
> this translation.
>
> 3. key-translation-map.  I've never actually used this directly so I'm
> not entirely clear on its intent.  The docs make it sound like it is
> intended for changing keyboard layouts.  Which explains why I have never
> used it -- I generally use OSes that allow keyboard layout translating
> natively.
>
> EXAMPLE:
> Based on my guess above, key-translation-map would let me map between
> QWERTY and Dvorak inside Emacs.  This would be nice if I was running
> directly in a Linux terminal and did not have permission to run "sudo
> loadkeys".
>
> As a final note, notice that all three of these examples involve
> changing a key into a different key.  None of them are for binding
> commands.  Commands being bound to a key always is the job of the local
> and global keymaps.
>
> I hope these examples help you understand the setup here!
>
>    -- MJF

bug#49253: 27.2; Emacs non-responsive when pasting into terminal-mode

[Eli Zaretskii]

> Date: Mon, 11 Dec 2023 14:13:24 -0800
> From: Jared Finder <jared@finder.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, 49253@debbugs.gnu.org
> 
> On 2023-12-11 08:10, Matt Bisson wrote:
> > TL;DR: It works!
> 
> Thank you for the testing!
> 
> Eli, I think the fully correct definition for term--xterm-paste is the 
> following, which has additional error checking:
> 
> (defun term--xterm-paste (event)
>    "Insert the text pasted in an XTerm bracketed paste operation."
>    (interactive "e")
>    (unless (eq (car-safe event) 'xterm-paste)
>      (error "term--xterm-paste must be found to xterm-paste event"))
>    (let ((str (nth 1 event)))
>      (unless (stringp str)
>        (error "term--xterm-paste provided event does not contain paste 
> text"))
>      (term-send-raw-string str)))

Thanks, installed on the emacs-29 branch, and closing the bug.