bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Jim Paris]

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':


Since upgrading to emacs-23, the terminal-init-xterm function in term/xterm.el
performs a "discard-input" followed by a couple of xterm queries.  You
can see the effect of this in the "recent input" line below -- all I did
was start emacs and do M-x report-bug, the preceding input was generated
by the xterm in response to terminal-init-xterm's queries.

This causes several problems when I start emacs in a terminal:

- Everything that I type while emacs is still loading gets discarded.
  This problem is more common than you might initially think, because
  I very frequently load emacs and immediately hit C-s to search for
  something.  Now, the C-s gets lost and my search terms go into the
  buffer instead.  To reproduce this, just start emacs (preferably on
  a slow machine, or one with delays in ~/.emacs, for example)
  and start typing before emacs has finished loading.  Everything typed
  is lost.

- There are race conditions here.  If I'm typing when this
  query-response happens, my typing is discarded by terminal-init-xterm,
  and the actual xterm response gets inserted directly into my buffer as if
  I had typed it.  Since I type quickly, I see this quite frequently:
  I load emacs, start typing, and the buffer ends up with "0;251;0c" 
  rather than what I had typed.  This one is harder to reproduce, but
  it is not difficult for me if I just run "emacs -nw" from the
  commandline and quickly hit "asdfasdf" while it's loading.

I understand that the modify-other-keys feature is useful, but this
makes emacs-23 difficult to use.  Can an option be provided to disable
these probes?  Maybe we can just skip the discard-input and
query/response if the user provides, say, predetermined "xterm-version"
and "xterm-background-color" variables in their ~/.emacs?
That seems to be the quickest and easiest approach, although it requires
user intervention to work around the issue.

I can also imagine a more complete fix that would involve not flushing
the input buffer, and interpreting the Xterm responses in a more
asynchronous fashion:
- Don't discard input
- Send the \e[>0c probe
- Allow input into the buffer as usual, but for the next few seconds,
  interpret \e[>0;251;0c responses and perform
  (xterm-turn-on-modify-other-keys) etc. as necessary.

-jim


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/23.2/etc/DEBUG.


In GNU Emacs 23.2.1 (x86_64-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2010-05-16 on barber, modified by Debian
configured using `configure  '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.2/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.2/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: en_US
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: POSIX
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Text

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC [ > 0 ; 2 5 1 ; 0 c ESC ] 1 1 ; r g b : 0 0 0 0 
/ 0 0 0 0 / 0 0 0 0 ESC \ ESC x r e p o r t TAB RE
T

Recent messages:
Loading /etc/emacs/site-start.d/50namazu2.el (source)...done
Loading /etc/emacs/site-start.d/50php-elisp.el (source)...done
Loading /etc/emacs/site-start.d/50psvn.el (source)...done
Loading /etc/emacs/site-start.d/51debian-el.el (source)...done
Loading /etc/emacs/site-start.d/70jim.el (source)...
Toggling menu-bar-mode off; better pass an explicit argument.
Ready.
Loading /etc/emacs/site-start.d/70jim.el (source)...done
Loading quail/latin-ltx...done
Ready.

Load-path shadows:
/usr/share/emacs/23.2/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs23/site-lisp/easypg/epg hides /usr/share/emacs/23.2/lisp/epg
/usr/share/emacs23/site-lisp/easypg/epa-mail hides /usr/share/emacs/23.2/lisp/epa-mail
/usr/share/emacs23/site-lisp/easypg/epa-dired hides /usr/share/emacs/23.2/lisp/epa-dired
/usr/share/emacs23/site-lisp/flim/sha1 hides /usr/share/emacs/23.2/lisp/sha1
/usr/share/emacs23/site-lisp/easypg/epa-file hides /usr/share/emacs/23.2/lisp/epa-file
/usr/share/emacs23/site-lisp/emacs-goodies-el/ido hides /usr/share/emacs/23.2/lisp/ido
/usr/share/emacs23/site-lisp/flim/md4 hides /usr/share/emacs/23.2/lisp/md4
/usr/share/emacs23/site-lisp/emacs-goodies-el/ibuffer hides /usr/share/emacs/23.2/lisp/ibuffer
/usr/share/emacs23/site-lisp/emacs-goodies-el/wdired hides /usr/share/emacs/23.2/lisp/wdired
/usr/share/emacs23/site-lisp/flim/hex-util hides /usr/share/emacs/23.2/lisp/hex-util
/usr/share/emacs23/site-lisp/easypg/epg-config hides /usr/share/emacs/23.2/lisp/epg-config
/usr/share/emacs23/site-lisp/easypg/epa hides /usr/share/emacs/23.2/lisp/epa
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/23.2/lisp/textmodes/ispell
/usr/share/emacs23/site-lisp/emacs-goodies-el/table hides /usr/share/emacs/23.2/lisp/textmodes/table
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/23.2/lisp/textmodes/flyspell
/usr/share/emacs23/site-lisp/emacs-goodies-el/cfengine hides /usr/share/emacs/23.2/lisp/progmodes/cfengine
/usr/share/emacs23/site-lisp/flim/ntlm hides /usr/share/emacs/23.2/lisp/net/ntlm
/usr/share/emacs23/site-lisp/flim/hmac-def hides /usr/share/emacs/23.2/lisp/net/hmac-def
/usr/share/emacs23/site-lisp/flim/sasl hides /usr/share/emacs/23.2/lisp/net/sasl
/usr/share/emacs23/site-lisp/emacs-goodies-el/newsticker hides /usr/share/emacs/23.2/lisp/net/newsticker
/usr/share/emacs23/site-lisp/flim/sasl-digest hides /usr/share/emacs/23.2/lisp/net/sasl-digest
/usr/share/emacs23/site-lisp/flim/hmac-md5 hides /usr/share/emacs/23.2/lisp/net/hmac-md5
/usr/share/emacs23/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/23.2/lisp/net/sasl-ntlm
/usr/share/emacs23/site-lisp/flim/sasl-cram hides /usr/share/emacs/23.2/lisp/net/sasl-cram
/usr/share/emacs23/site-lisp/latex-cjk-thai/thai-word hides /usr/share/emacs/23.2/lisp/language/thai-word

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
mml-sec password-cache mm-decode mm-bodies mm-encode mailcap mail-parse
rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util
netrc time-date mm-util mail-prsvr gmm-utils wid-edit mailheader canlock
sha1 sha1-el hex-util hashcash mail-utils warnings emacsbug quail
help-mode easymenu view debian-el debian-el-loaddefs path-util byte-opt
bytecomp byte-compile advice help-fns advice-preload poe pym static
apel-ver product emacs-goodies-el emacs-goodies-custom
emacs-goodies-loaddefs epa-setup epa-file epa derived epg epg-config
epg-package-info dpkg-dev-el dpkg-dev-el-loaddefs tooltip ediff-hook
vc-hooks lisp-float-type mwheel x-win x-dnd font-setting tool-bar dnd
fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mldrag mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev loaddefs button minibuffer faces
cus-face files text-properties overlay md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote
make-network-process dbusbind system-font-setting font-render-setting
gtk x-toolkit x multi-tty emacs)

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Stefan Monnier]

> I can also imagine a more complete fix that would involve not flushing
> the input buffer, and interpreting the Xterm responses in a more
> asynchronous fashion:
> - Don't discard input
> - Send the \e[>0c probe
> - Allow input into the buffer as usual, but for the next few seconds,
>   interpret \e[>0;251;0c responses and perform
>   (xterm-turn-on-modify-other-keys) etc. as necessary.

Time-dependent processing is considered brittle (especially when
working remotely via something like SSH), so we prefer to avoid it.
Especially since coding it right could prove pretty darn tricky.

OTOH, I thought we had agreed that we need to add a configuration
variable xterm-turn-on-modify-other-keys which could be set to t or nil
to force the modifyOtherKeys feature of Xterm ON or OFF without first
checking whether the current xterm indeed supports it or not.
Its default value would be `auto', which would first check for support
and then turn it ON if applicable.
Since the "discard input" is needed for the check, this would solve
your problem.

Now, I don't see such a variable, so either I misremeber, or the patch
was never written, or never committed, ...


        Stefan

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Johan Bockgård]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> OTOH, I thought we had agreed that we need to add a configuration
> variable xterm-turn-on-modify-other-keys which could be set to t or nil
> to force the modifyOtherKeys feature of Xterm ON or OFF without first
> checking whether the current xterm indeed supports it or not.
> Its default value would be `auto', which would first check for support
> and then turn it ON if applicable.
> Since the "discard input" is needed for the check, this would solve
> your problem.
>
> Now, I don't see such a variable, so either I misremeber, or the patch
> was never written, or never committed, ...

http://lists.gnu.org/archive/html/emacs-devel/2009-08/msg01276.html

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Jim Paris]

Stefan Monnier wrote:
> > I can also imagine a more complete fix that would involve not flushing
> > the input buffer, and interpreting the Xterm responses in a more
> > asynchronous fashion:
> > - Don't discard input
> > - Send the \e[>0c probe
> > - Allow input into the buffer as usual, but for the next few seconds,
> >   interpret \e[>0;251;0c responses and perform
> >   (xterm-turn-on-modify-other-keys) etc. as necessary.
> 
> Time-dependent processing is considered brittle (especially when
> working remotely via something like SSH), so we prefer to avoid it.
> Especially since coding it right could prove pretty darn tricky.

Forget the time-dependent part of what I said, let's just send the
query and handle the response whenever it happens to come in.  See
the below patch.  This fixes everything -- it still detects both
modifyOtherKeys and the background color, but doesn't require flushing
input or rely on any timeouts at all, unlike the existing code.
And there's nothing tricky.  What do you think?


> OTOH, I thought we had agreed that we need to add a configuration
> variable xterm-turn-on-modify-other-keys which could be set to t or nil
> to force the modifyOtherKeys feature of Xterm ON or OFF without first
> checking whether the current xterm indeed supports it or not.
> Its default value would be `auto', which would first check for support
> and then turn it ON if applicable.

This may still be useful for people who don't want any queries sent to
their terminal.

-jim


--- xterm.el-orig	2010-04-03 18:26:04.000000000 -0400
+++ xterm.el	2010-08-02 16:51:45.000000000 -0400
@@ -440,6 +440,65 @@
 ;; List of terminals for which modify-other-keys has been turned on.
 (defvar xterm-modify-other-keys-terminal-list nil)
 
+(defun xterm-osc-translate (event)
+  "Read and handle a Operating System Controls response"
+  (let* ((str "")
+	 (chr nil)
+	 (recompute-faces nil))
+
+    ;; The reply should be of the form: \e ] 11 ; rgb: NUMBER1 / NUMBER2 / NUMBER3 \e \\
+    (while (not (equal (setq chr (read-char)) ?\\))
+      (setq str (concat str (string chr))))
+
+    (when (string-match "rgb:\\([a-f0-9]+\\)/\\([a-f0-9]+\\)/\\([a-f0-9]+\\)" str)
+      (setq recompute-faces
+	    (xterm-maybe-set-dark-background-mode
+	     (string-to-number (match-string 1 str) 16)
+	     (string-to-number (match-string 2 str) 16)
+	     (string-to-number (match-string 3 str) 16))))
+
+    (when recompute-faces
+      (tty-set-up-initial-frame-faces))
+
+    ""))
+
+(defun xterm-secondary-da-translate (event)
+  "Read and handle a Secondary Device Attributes response"
+  (let* ((str "")
+	 (chr nil)
+	 version)
+
+    ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
+    (while (not (equal (setq chr (read-char)) ?c))
+      (setq str (concat str (string chr))))
+
+    (when (string-match "0;\\([0-9]+\\);0" str)
+      (setq version (string-to-number
+		     (substring str (match-beginning 1) (match-end 1))))
+      ;; NUMBER2 is the xterm version number, look for something
+      ;; greater than 216, the version when modifyOtherKeys was
+      ;; introduced.
+      (when (>= version 216)
+	;; Make sure that the modifyOtherKeys state is restored when
+	;; suspending, resuming and exiting.
+	(add-hook 'suspend-hook 'xterm-turn-off-modify-other-keys)
+	(add-hook 'suspend-resume-hook 'xterm-turn-on-modify-other-keys)
+	(add-hook 'kill-emacs-hook 'xterm-remove-modify-other-keys)
+	(add-hook 'delete-terminal-functions 'xterm-remove-modify-other-keys)
+	;; Add the selected frame to the list of frames that
+	;; need to deal with modify-other-keys.
+	(push (frame-terminal (selected-frame))
+	      xterm-modify-other-keys-terminal-list)
+	(xterm-turn-on-modify-other-keys))
+
+      ;; xterm version 235 supports reporting the background
+      ;; color, maybe earlier versions do too...
+      (when (>= version 235)
+	(define-key input-decode-map "\e]11;" 'xterm-osc-translate)
+	(send-string-to-terminal "\e]11;?\e\\")))
+    
+    ""))
+
 (defun terminal-init-xterm ()
   "Terminal initialization function for xterm."
   ;; rxvt terminals sometimes set the TERM variable to "xterm", but
@@ -469,71 +528,12 @@
     ;; C-. C-, etc.
     ;; To do that we need to find out if the current terminal supports
     ;; modifyOtherKeys. At this time only xterm does.
-    (let ((coding-system-for-read 'binary)
-	  (chr nil)
-	  (str nil)
-	  (recompute-faces nil)
-	  version)
-      ;; Pending input can be mistakenly returned by the calls to
-      ;; read-event below.  Discard it.
-      (discard-input)
-      ;; Try to find out the type of terminal by sending a "Secondary
-      ;; Device Attributes (DA)" query.
-      (send-string-to-terminal "\e[>0c")
-
-      ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
-      ;; If the timeout is completely removed for read-event, this
-      ;; might hang for terminals that pretend to be xterm, but don't
-      ;; respond to this escape sequence.  RMS' opinion was to remove
-      ;; it completely.  That might be right, but let's first try to
-      ;; see if by using a longer timeout we get rid of most issues.
-      (when (equal (read-event nil nil 2) ?\e)
-	(when (equal (read-event nil nil 2) ?\[)
-	  (while (not (equal (setq chr (read-event nil nil 2)) ?c))
-	    (setq str (concat str (string chr))))
-	  (when (string-match ">0;\\([0-9]+\\);0" str)
-	    (setq version (string-to-number
-			   (substring str (match-beginning 1) (match-end 1))))
-	    ;; xterm version 242 supports reporting the background
-	    ;; color, maybe earlier versions do too...
-	    (when (>= version 242)
-	      (send-string-to-terminal "\e]11;?\e\\")
-	      (when (equal (read-event nil nil 2) ?\e)
-		(when (equal (read-event nil nil 2) ?\])
-		  (setq str "")
-		  (while (not (equal (setq chr (read-event nil nil 2)) ?\\))
-		    (setq str (concat str (string chr))))
-		  (when (string-match "11;rgb:\\([a-f0-9]+\\)/\\([a-f0-9]+\\)/\\([a-f0-9]+\\)" str)
-		    (setq recompute-faces
-			  (xterm-maybe-set-dark-background-mode
-			   (string-to-number (match-string 1 str) 16)
-			   (string-to-number (match-string 2 str) 16)
-			   (string-to-number (match-string 3 str) 16)))))))
-	    ;; NUMBER2 is the xterm version number, look for something
-	    ;; greater than 216, the version when modifyOtherKeys was
-	    ;; introduced.
-	    (when (>= version 216)
-	      ;; Make sure that the modifyOtherKeys state is restored when
-	      ;; suspending, resuming and exiting.
-	      (add-hook 'suspend-hook 'xterm-turn-off-modify-other-keys)
-	      (add-hook 'suspend-resume-hook 'xterm-turn-on-modify-other-keys)
-	      (add-hook 'kill-emacs-hook 'xterm-remove-modify-other-keys)
-	      (add-hook 'delete-terminal-functions 'xterm-remove-modify-other-keys)
-	      ;; Add the selected frame to the list of frames that
-	      ;; need to deal with modify-other-keys.
-	      (push (frame-terminal (selected-frame))
-		    xterm-modify-other-keys-terminal-list)
-	      (xterm-turn-on-modify-other-keys))
-
-	    ;; Recompute faces here in case the background mode was
-	    ;; set to dark.  We used to call
-	    ;; `tty-set-up-initial-frame-faces' only once, but that
-	    ;; caused the light background faces to be computed
-	    ;; incorrectly.  See:
-	    ;; http://permalink.gmane.org/gmane.emacs.devel/119627
-	    (when recompute-faces
-	      (tty-set-up-initial-frame-faces))))))
 
+    ;; Try to find out the type of terminal by sending a "Secondary
+    ;; Device Attributes (DA)" query.
+    (define-key input-decode-map "\e[>" 'xterm-secondary-da-translate)
+    (send-string-to-terminal "\e[>0c")
+    
     (run-hooks 'terminal-init-xterm-hook))
 
 ;; Set up colors, for those versions of xterm that support it.

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Andreas Schwab]

Jim Paris <jim@jtan.com> writes:

> +    ;; The reply should be of the form: \e ] 11 ; rgb: NUMBER1 / NUMBER2 / NUMBER3 \e \\
> +    (while (not (equal (setq chr (read-char)) ?\\))
> +      (setq str (concat str (string chr))))

At least this loop needs to have a timeout, in case the terminator
character gets lost for whatever reason.

> +    ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
> +    (while (not (equal (setq chr (read-char)) ?c))
> +      (setq str (concat str (string chr))))

Likewise.  (It isn't all that hard to type ESC [ > by accident).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Jim Paris]

Andreas Schwab wrote:
> Jim Paris <jim@jtan.com> writes:
> 
> > +    ;; The reply should be of the form: \e ] 11 ; rgb: NUMBER1 / NUMBER2 / NUMBER3 \e \\
> > +    (while (not (equal (setq chr (read-char)) ?\\))
> > +      (setq str (concat str (string chr))))
> 
> At least this loop needs to have a timeout, in case the terminator
> character gets lost for whatever reason.
> 
> > +    ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
> > +    (while (not (equal (setq chr (read-char)) ?c))
> > +      (setq str (concat str (string chr))))
> 
> Likewise.  (It isn't all that hard to type ESC [ > by accident).

Ok, (read-char) can be replaced with (read-event nil nil 2).  There
may also need to be a (let (coding-system-for-read 'binary)) in there
somewhere -- I'm really not familiar with emacs programming, this is
mostly just based on other code I've found (like xt-mouse.el)

-jim

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Stefan Monnier]

> Forget the time-dependent part of what I said, let's just send the
> query and handle the response whenever it happens to come in.  See
> the below patch.  This fixes everything -- it still detects both
> modifyOtherKeys and the background color, but doesn't require flushing
> input or rely on any timeouts at all, unlike the existing code.
> And there's nothing tricky.  What do you think?

Looks pretty good from here, thank you.
We may want to get rid of the "\e[>" (and "\e]11;") decode rules after
they've been used, just in case (or better yet: make them more
robust).

Any objection?


        Stefan

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Jim Paris]

Stefan Monnier wrote:
> > Forget the time-dependent part of what I said, let's just send the
> > query and handle the response whenever it happens to come in.  See
> > the below patch.  This fixes everything -- it still detects both
> > modifyOtherKeys and the background color, but doesn't require flushing
> > input or rely on any timeouts at all, unlike the existing code.
> > And there's nothing tricky.  What do you think?
> 
> Looks pretty good from here, thank you.
> We may want to get rid of the "\e[>" (and "\e]11;") decode rules after
> they've been used, just in case (or better yet: make them more
> robust).
> 
> Any objection?

I found a problem: the Mac OS X terminal sets TERM=xterm-color, but it
responds to the "Secondary DA" query as if it were a "Primary DA" query.
Instead of a response like xterm's "\e[>0;253;0c", it sends "\e[?1;2c".

The below patch accounts for this.  It also adds the 2-second timeout
when reading responses, and unsets the decode rules once they're
unused.

-jim

--- xterm.el-orig	2010-04-03 18:26:04.000000000 -0400
+++ xterm.el	2010-08-03 16:05:49.000000000 -0400
@@ -440,6 +440,86 @@
 ;; List of terminals for which modify-other-keys has been turned on.
 (defvar xterm-modify-other-keys-terminal-list nil)
 
+(defun xterm-osc-translate (event)
+  "Read and handle a Operating System Controls response"
+  (let* ((str "")
+	 (chr nil)
+	 (recompute-faces nil))
+
+    ;; The reply should be of the form: \e ] 11 ; rgb: NUMBER1 / NUMBER2 / NUMBER3 \e \\
+    (while (not (equal (setq chr (read-event nil nil 2)) ?\\))
+      (setq str (concat str (string chr))))
+
+    (when (string-match "rgb:\\([a-f0-9]+\\)/\\([a-f0-9]+\\)/\\([a-f0-9]+\\)" str)
+      (setq recompute-faces
+	    (xterm-maybe-set-dark-background-mode
+	     (string-to-number (match-string 1 str) 16)
+	     (string-to-number (match-string 2 str) 16)
+	     (string-to-number (match-string 3 str) 16))))
+
+    (when recompute-faces
+      (tty-set-up-initial-frame-faces))
+
+    ;; We no longer expect the OSC response
+    (define-key input-decode-map "\e]11;" nil)
+    ""))
+
+(defun xterm-primary-da-translate (event)
+  "Read and handle a Primary Device Attributes response"
+  (let* ((str "")
+	 (chr nil))
+
+    ;; The reply should be of the form: \e [ ? Pd c
+    (while (not (equal (setq chr (read-event nil nil 2)) ?c))
+      (setq str (concat str (string chr))))
+
+    ;; No need to do anything with this response.
+
+    ;; Undefine both DA responses, as we don't expect either anymore
+    (define-key input-decode-map "\e[?" nil)
+    (define-key input-decode-map "\e[>" nil)
+    ""))
+	 
+(defun xterm-secondary-da-translate (event)
+  "Read and handle a Secondary Device Attributes response"
+  (let* ((str "")
+	 (chr nil)
+	 version)
+
+    ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
+    (while (not (equal (setq chr (read-event nil nil 2)) ?c))
+      (setq str (concat str (string chr))))
+
+    (when (string-match "0;\\([0-9]+\\);0" str)
+      (setq version (string-to-number
+		     (substring str (match-beginning 1) (match-end 1))))
+      ;; NUMBER2 is the xterm version number, look for something
+      ;; greater than 216, the version when modifyOtherKeys was
+      ;; introduced.
+      (when (>= version 216)
+	;; Make sure that the modifyOtherKeys state is restored when
+	;; suspending, resuming and exiting.
+	(add-hook 'suspend-hook 'xterm-turn-off-modify-other-keys)
+	(add-hook 'suspend-resume-hook 'xterm-turn-on-modify-other-keys)
+	(add-hook 'kill-emacs-hook 'xterm-remove-modify-other-keys)
+	(add-hook 'delete-terminal-functions 'xterm-remove-modify-other-keys)
+	;; Add the selected frame to the list of frames that
+	;; need to deal with modify-other-keys.
+	(push (frame-terminal (selected-frame))
+	      xterm-modify-other-keys-terminal-list)
+	(xterm-turn-on-modify-other-keys))
+
+      ;; xterm version 235 supports reporting the background
+      ;; color, maybe earlier versions do too...
+      (when (>= version 235)
+	(define-key input-decode-map "\e]11;" 'xterm-osc-translate)
+	(send-string-to-terminal "\e]11;?\e\\")))
+    
+    ;; Undefine both DA responses, as we don't expect either anymore
+    (define-key input-decode-map "\e[?" nil)
+    (define-key input-decode-map "\e[>" nil)
+    ""))
+
 (defun terminal-init-xterm ()
   "Terminal initialization function for xterm."
   ;; rxvt terminals sometimes set the TERM variable to "xterm", but
@@ -469,71 +549,15 @@
     ;; C-. C-, etc.
     ;; To do that we need to find out if the current terminal supports
     ;; modifyOtherKeys. At this time only xterm does.
-    (let ((coding-system-for-read 'binary)
-	  (chr nil)
-	  (str nil)
-	  (recompute-faces nil)
-	  version)
-      ;; Pending input can be mistakenly returned by the calls to
-      ;; read-event below.  Discard it.
-      (discard-input)
-      ;; Try to find out the type of terminal by sending a "Secondary
-      ;; Device Attributes (DA)" query.
-      (send-string-to-terminal "\e[>0c")
-
-      ;; The reply should be of the form: \e [ > NUMBER1 ; NUMBER2 ; NUMBER3 c
-      ;; If the timeout is completely removed for read-event, this
-      ;; might hang for terminals that pretend to be xterm, but don't
-      ;; respond to this escape sequence.  RMS' opinion was to remove
-      ;; it completely.  That might be right, but let's first try to
-      ;; see if by using a longer timeout we get rid of most issues.
-      (when (equal (read-event nil nil 2) ?\e)
-	(when (equal (read-event nil nil 2) ?\[)
-	  (while (not (equal (setq chr (read-event nil nil 2)) ?c))
-	    (setq str (concat str (string chr))))
-	  (when (string-match ">0;\\([0-9]+\\);0" str)
-	    (setq version (string-to-number
-			   (substring str (match-beginning 1) (match-end 1))))
-	    ;; xterm version 242 supports reporting the background
-	    ;; color, maybe earlier versions do too...
-	    (when (>= version 242)
-	      (send-string-to-terminal "\e]11;?\e\\")
-	      (when (equal (read-event nil nil 2) ?\e)
-		(when (equal (read-event nil nil 2) ?\])
-		  (setq str "")
-		  (while (not (equal (setq chr (read-event nil nil 2)) ?\\))
-		    (setq str (concat str (string chr))))
-		  (when (string-match "11;rgb:\\([a-f0-9]+\\)/\\([a-f0-9]+\\)/\\([a-f0-9]+\\)" str)
-		    (setq recompute-faces
-			  (xterm-maybe-set-dark-background-mode
-			   (string-to-number (match-string 1 str) 16)
-			   (string-to-number (match-string 2 str) 16)
-			   (string-to-number (match-string 3 str) 16)))))))
-	    ;; NUMBER2 is the xterm version number, look for something
-	    ;; greater than 216, the version when modifyOtherKeys was
-	    ;; introduced.
-	    (when (>= version 216)
-	      ;; Make sure that the modifyOtherKeys state is restored when
-	      ;; suspending, resuming and exiting.
-	      (add-hook 'suspend-hook 'xterm-turn-off-modify-other-keys)
-	      (add-hook 'suspend-resume-hook 'xterm-turn-on-modify-other-keys)
-	      (add-hook 'kill-emacs-hook 'xterm-remove-modify-other-keys)
-	      (add-hook 'delete-terminal-functions 'xterm-remove-modify-other-keys)
-	      ;; Add the selected frame to the list of frames that
-	      ;; need to deal with modify-other-keys.
-	      (push (frame-terminal (selected-frame))
-		    xterm-modify-other-keys-terminal-list)
-	      (xterm-turn-on-modify-other-keys))
-
-	    ;; Recompute faces here in case the background mode was
-	    ;; set to dark.  We used to call
-	    ;; `tty-set-up-initial-frame-faces' only once, but that
-	    ;; caused the light background faces to be computed
-	    ;; incorrectly.  See:
-	    ;; http://permalink.gmane.org/gmane.emacs.devel/119627
-	    (when recompute-faces
-	      (tty-set-up-initial-frame-faces))))))
 
+    ;; Try to find out the type of terminal by sending a "Secondary
+    ;; Device Attributes (DA)" query.  Some terminals (like OS X's
+    ;; Terminal.app) respond to this query as if it were a "Primary
+    ;; Device Attributes" query instead, so we should handle that too.
+    (define-key input-decode-map "\e[?" 'xterm-primary-da-translate)
+    (define-key input-decode-map "\e[>" 'xterm-secondary-da-translate)
+    (send-string-to-terminal "\e[>0c")
+    
     (run-hooks 'terminal-init-xterm-hook))
 
 ;; Set up colors, for those versions of xterm that support it.

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Jim Paris]

Any thoughts on this patch?

I've found further problems with doing aynchronous terminal queries:
minibuffer prompts get aborted by the xterm response.  For example,
when opening up a large file:

1) Emacs loads and sends "\e[>0c"
2) Emacs prints the "File x is large (123MB), really open? (y or n)" prompt
3) Xterm sends the "\e[>0;253;0c" response, aborting the prompt

Is there a way to make the (define-key input-decode-map "\e[?" nil)
stuff take effect in the minibuffer too so the response can be
properly handled?  Or handle the xterm responses at a higher level
so it doesn't matter where they get sent?

Sigh, this whole terminal response stuff is a real mess and new
failure modes keep popping up every month...  I don't see how this
can be enabled by default for anyone in its current form.

-jim

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Stefan Monnier]

> Any thoughts on this patch?
> I've found further problems with doing aynchronous terminal queries:
> minibuffer prompts get aborted by the xterm response.  For example,
> when opening up a large file:

> 1) Emacs loads and sends "\e[>0c"
> 2) Emacs prints the "File x is large (123MB), really open? (y or n)" prompt
> 3) Xterm sends the "\e[>0;253;0c" response, aborting the prompt

Indeed.  But this one is "easy": y-or-n-p should use read-key rather
than read-event, so that input-decode-map will be obeyed.  Of course,
that probably requires moving y-or-n-p from C to Elisp.


        Stefan

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Stefan Monnier]

> Any thoughts on this patch?
> I've found further problems with doing aynchronous terminal queries:
> minibuffer prompts get aborted by the xterm response.  For example,
> when opening up a large file:

> 1) Emacs loads and sends "\e[>0c"
> 2) Emacs prints the "File x is large (123MB), really open? (y or n)" prompt
> 3) Xterm sends the "\e[>0;253;0c" response, aborting the prompt

You can try the patch below (may require manual application, it's made
against a non-vanilla tree) to see if it fixes your problem,


        Stefan


=== modified file 'lisp/subr.el'
--- lisp/subr.el	2010-09-11 09:14:23 +0000
+++ lisp/subr.el	2010-09-11 14:57:41 +0000
@@ -3680,6 +3680,52 @@
   buffer)
 
 
+(defun y-or-n-p (prompt)
+  "Ask user a \"y or n\" question.  Return t if answer is \"y\".
+The argument PROMPT is the string to display to ask the question.
+It should end in a space; `y-or-n-p' adds `(y or n) ' to it.
+No confirmation of the answer is requested; a single character is enough.
+Also accepts Space to mean yes, or Delete to mean no.  \(Actually, it uses
+the bindings in `query-replace-map'; see the documentation of that variable
+for more information.  In this case, the useful bindings are `act', `skip',
+`recenter', and `quit'.\)
+
+Under a windowing system a dialog box will be used if `last-nonmenu-event'
+is nil and `use-dialog-box' is non-nil."
+  (let ((answer 'none)
+        (xprompt (setq prompt (propertize (concat prompt "(y or n) ")
+                                          'face 'minibuffer-prompt))))
+    (if (and (display-popup-menus-p)
+             (listp last-nonmenu-event)
+             use-dialog-box)
+        (progn
+          (setq answer
+                (x-popup-dialog t `(,prompt ("yes" . t) ("No" . nil)))))
+      (while
+          (progn
+            (when minibuffer-auto-raise
+              (raise-frame (window-frame (minibuffer-window))))
+            (let* ((key
+                    (let ((cursor-in-echo-area t))
+                      (read-key xprompt))))
+              (setq answer (lookup-key query-replace-map (vector key) t))
+              (case answer
+                ((skip act) nil)
+                (recenter (recenter) t)
+                ((exit-prefix quit) (signal 'quit nil) t)
+                (t t))))
+        (ding)
+        (discard-input)
+        (setq xprompt
+              (if (eq answer 'recenter) prompt
+                (propertize (concat "Please answer y or n.  " prompt)
+                            'face 'minibuffer-prompt))))
+
+      (let ((ret (eq answer 'act)))
+        (unless noninteractive
+          (message "%s %s" prompt (if ret "y" "n")))
+        ret))))
+
 ;; (defun get-doc-string (pos &optional unibyte definition)
 ;;   (let ((file (or (car-safe pos) internal-doc-file-name))
 ;; 	(position (or (cdr-safe pos) pos))

=== modified file 'src/fileio.c'
--- src/fileio.c	2010-08-10 10:46:59 +0000
+++ src/fileio.c	2010-09-11 14:22:33 +0000
@@ -1842,7 +1842,7 @@
       tem = format2 ("File %s already exists; %s anyway? ",
 		     absname, build_string (querystring));
       if (quick)
-	tem = Fy_or_n_p (tem);
+	tem = call1 (intern ("y-or-n-p"), tem);
       else
 	tem = do_yes_or_no_p (tem);
       UNGCPRO;

=== modified file 'src/fns.c'
--- src/fns.c	2010-08-14 22:35:37 +0000
+++ src/fns.c	2010-09-11 14:10:27 +0000
@@ -2437,146 +2437,6 @@
   return sequence;
 }
 \f
-/* Anything that calls this function must protect from GC!  */
-
-DEFUN ("y-or-n-p", Fy_or_n_p, Sy_or_n_p, 1, 1, 0,
-       doc: /* Ask user a "y or n" question.  Return t if answer is "y".
-Takes one argument, which is the string to display to ask the question.
-It should end in a space; `y-or-n-p' adds `(y or n) ' to it.
-No confirmation of the answer is requested; a single character is enough.
-Also accepts Space to mean yes, or Delete to mean no.  \(Actually, it uses
-the bindings in `query-replace-map'; see the documentation of that variable
-for more information.  In this case, the useful bindings are `act', `skip',
-`recenter', and `quit'.\)
-
-Under a windowing system a dialog box will be used if `last-nonmenu-event'
-is nil and `use-dialog-box' is non-nil.  */)
-  (Lisp_Object prompt)
-{
-  register Lisp_Object obj, key, def, map;
-  register int answer;
-  Lisp_Object xprompt;
-  Lisp_Object args[2];
-  struct gcpro gcpro1, gcpro2;
-  int count = SPECPDL_INDEX ();
-
-  specbind (Qcursor_in_echo_area, Qt);
-
-  map = Fsymbol_value (intern ("query-replace-map"));
-
-  CHECK_STRING (prompt);
-  xprompt = prompt;
-  GCPRO2 (prompt, xprompt);
-
-#ifdef HAVE_WINDOW_SYSTEM
-  if (display_hourglass_p)
-    cancel_hourglass ();
-#endif
-
-  while (1)
-    {
-
-#ifdef HAVE_MENUS
-      if (FRAME_WINDOW_P (SELECTED_FRAME ())
-          && (NILP (last_nonmenu_event) || CONSP (last_nonmenu_event))
-	  && use_dialog_box
-	  && have_menus_p ())
-	{
-	  Lisp_Object pane, menu;
-	  redisplay_preserve_echo_area (3);
-	  pane = Fcons (Fcons (build_string ("Yes"), Qt),
-			Fcons (Fcons (build_string ("No"), Qnil),
-			       Qnil));
-	  menu = Fcons (prompt, pane);
-	  obj = Fx_popup_dialog (Qt, menu, Qnil);
-	  answer = !NILP (obj);
-	  break;
-	}
-#endif /* HAVE_MENUS */
-      cursor_in_echo_area = 1;
-      choose_minibuf_frame ();
-
-      {
-	Lisp_Object pargs[3];
-
-	/* Colorize prompt according to `minibuffer-prompt' face.  */
-	pargs[0] = build_string ("%s(y or n) ");
-	pargs[1] = intern ("face");
-	pargs[2] = intern ("minibuffer-prompt");
-	args[0] = Fpropertize (3, pargs);
-	args[1] = xprompt;
-	Fmessage (2, args);
-      }
-
-      if (minibuffer_auto_raise)
-	{
-	  Lisp_Object mini_frame;
-
-	  mini_frame = WINDOW_FRAME (XWINDOW (minibuf_window));
-
-	  Fraise_frame (mini_frame);
-	}
-
-      temporarily_switch_to_single_kboard (SELECTED_FRAME ());
-      obj = read_filtered_event (1, 0, 0, 0, Qnil);
-      cursor_in_echo_area = 0;
-      /* If we need to quit, quit with cursor_in_echo_area = 0.  */
-      QUIT;
-
-      key = Fmake_vector (make_number (1), obj);
-      def = Flookup_key (map, key, Qt);
-
-      if (EQ (def, intern ("skip")))
-	{
-	  answer = 0;
-	  break;
-	}
-      else if (EQ (def, intern ("act")))
-	{
-	  answer = 1;
-	  break;
-	}
-      else if (EQ (def, intern ("recenter")))
-	{
-	  Frecenter (Qnil);
-	  xprompt = prompt;
-	  continue;
-	}
-      else if (EQ (def, intern ("quit")))
-	Vquit_flag = Qt;
-      /* We want to exit this command for exit-prefix,
-	 and this is the only way to do it.  */
-      else if (EQ (def, intern ("exit-prefix")))
-	Vquit_flag = Qt;
-
-      QUIT;
-
-      /* If we don't clear this, then the next call to read_char will
-	 return quit_char again, and we'll enter an infinite loop.  */
-      Vquit_flag = Qnil;
-
-      Fding (Qnil);
-      Fdiscard_input ();
-      if (EQ (xprompt, prompt))
-	{
-	  args[0] = build_string ("Please answer y or n.  ");
-	  args[1] = prompt;
-	  xprompt = Fconcat (2, args);
-	}
-    }
-  UNGCPRO;
-
-  if (! noninteractive)
-    {
-      cursor_in_echo_area = -1;
-      message_with_string (answer ? "%s(y or n) y" : "%s(y or n) n",
-			   xprompt, 0);
-    }
-
-  unbind_to (count, Qnil);
-  return answer ? Qt : Qnil;
-}
-\f
 /* This is how C code calls `yes-or-no-p' and allows the user
    to redefined it.
 
@@ -5057,7 +4917,6 @@
   defsubr (&Smapcar);
   defsubr (&Smapc);
   defsubr (&Smapconcat);
-  defsubr (&Sy_or_n_p);
   defsubr (&Syes_or_no_p);
   defsubr (&Sload_average);
   defsubr (&Sfeaturep);

=== modified file 'src/lisp.h'
--- src/lisp.h	2010-09-10 23:24:48 +0000
+++ src/lisp.h	2010-09-11 14:09:41 +0000
@@ -2586,7 +2586,6 @@
 EXFUN (Fnconc, MANY);
 EXFUN (Fmapcar, 2);
 EXFUN (Fmapconcat, 3);
-EXFUN (Fy_or_n_p, 1);
 extern Lisp_Object do_yes_or_no_p (Lisp_Object);
 EXFUN (Frequire, 3);
 EXFUN (Fprovide, 2);

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Stefan Monnier]

> Any thoughts on this patch?

I installed a similar patch (tho rewritten based on the latest version
of the code), except that it only uses your asynchronous approach if
there's pending input.  This way, the old synchronous method is still
used in "most" cases and the new riskier approach is only used when we
would otherwise have had to discard-input.

Thank you for your help,


        Stefan

bug#6758: 23.2; xterm.el: please provide an option to not discard input in terminal-init-xterm

[Chong Yidong]

Jim Paris <jim@jtan.com> writes:

> Since upgrading to emacs-23, the terminal-init-xterm function in
> term/xterm.el performs a "discard-input" followed by a couple of xterm
> queries.
>
> Can an option be provided to disable these probes?

Emacs 24.1 has `xterm-extra-capabilities' for this.  Closing.

bug#6758: So what do I need to do to keep emacs from eating my typeahead?

[Karl O. Pinc]

Hi,

Thanks for fixing this, but I'm left unclear on what
I would need to do to keep emacs from discarding
my typeahead on startup.  (Bug report #8482,
merged with bug #6758.)

I'll also have to get a new emacs (24.1), which might
be why this is not clear.  Sorry if this is the
case, but I remain nervous that the solution will
be obvious; `xterm-extra-capabilities' does not
scream out to me as something that fixes a problem
with typeahead.

Thanks again for your work.

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

bug#6758: So what do I need to do to keep emacs from eating my typeahead?

[Glenn Morris]

"Karl O. Pinc" wrote:

> Thanks for fixing this, but I'm left unclear on what I would need to
> do to keep emacs from discarding my typeahead on startup. (Bug report
> #8482, merged with bug #6758.)

Set xterm-extra-capabilities to a value other than `check' (the
default). See that variable's documentation (in term/xterm.el) for more
details. Eg you may set it to nil.