bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Glyph]

I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
terminal (running with -nw) results in garbage characters on either side
of the paste.

For example, if I paste the URL, it looks like
200~http://emacsformacosx.com/201~, or causes jumping around within the
buffer (for example, when filing this bug, it jumped to the beginning of
the buffer and pasted the URL and the "201~" before the From:).



In GNU Emacs 24.4.50.1 (x86_64-apple-darwin, NS apple-appkit-1038.36)
of 2014-04-26 on bob.porkrind.org
Repository revision: monnier@iro.umontreal.ca-20140425192226-frpjpmgs0r8ad9wc
Configured using:
`configure --host=x86_64-apple-darwin --build=i686-apple-darwin
--with-ns'

Configured features:
ACL LIBXML2 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  quick-hack-emacs-lisp-mode: t
  shell-dirtrack-mode: t
  helm-match-plugin-mode: t
  helm-occur-match-plugin-mode: t
  global-auto-complete-mode: t
  auto-complete-mode: t
  global-rainbow-delimiters-mode: t
  rainbow-delimiters-mode: t
  obb-mode: t
  which-function-mode: t
  show-paren-mode: t
  server-mode: t
  icomplete-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC [ ? 1 ; 2 c ESC [ 2 0 0 ~ h t t p : / / e m a c 
s f o r m a c o s x . c o m / ESC [ 2 0 1 ~ C-_ C-_ 
C-_ C-_ ESC x e m a c s b DEL - b u g C-n RET

Recent messages:
Loading paren...done
Loading which-func...done
Loading /Users/glyph/Settings/Emacs/bootstrap/customizations.el (source)...done
Loading /Users/glyph/Settings/Emacs/firstparty/quickhack-autoload.el (source)...done
Loading /Users/glyph/Settings/Emacs/firstparty/flyflakes-autoload.el (source)...done
ad-handle-definition: `tramp-read-passwd' got redefined
Pymacs loading ropemacs...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Nowhere to go back to. [2 times]
Undo! [4 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils help-at-pt xterm ropemacs
pymacs rx python-patches python quickhack glyph-setup adaptive-wrap
helm-C-x-b helm-imenu helm-command helm-elisp helm-eval eldoc helm-mode
helm-cmd-t helm-files image-dired tramp tramp-compat tramp-loaddefs
trampver shell pcomplete format-spec dired-x dired-aux ffap thingatpt
helm-buffers helm-elscreen helm-tags helm-bookmark helm-adaptative
helm-info helm-net browse-url xml url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
auth-source eieio byte-opt bytecomp byte-compile cconv eieio-core
gnus-util mm-util mail-prsvr password-cache url-vars mailcap helm-plugin
bookmark pp helm-help helm-match-plugin helm-grep helm-regexp grep
helm-external helm-utils dired helm-locate helm helm-config helm-aliases
auto-complete-config auto-complete edmacro kmacro popup pelican-mode
backandforth advice help-fns rainbow-delimiters column-marker easy-mmode
obb-mode cl-macs combinator cl cl-loaddefs cl-lib epytext
goto-definition gv info easymenu package time-date jka-compr
tango-dark-theme which-func imenu paren server avoid icomplete linum
cus-start cus-load flymake compile comint ansi-color ring tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 215726 5957)
(symbols 48 31136 0)
(miscs 40 58 122)
(strings 32 54984 6179)
(string-bytes 1 1481420)
(vectors 16 21509)
(vector-slots 8 504251 2686)
(floats 8 157 651)
(intervals 56 364 18)
(buffers 960 14))

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Jan Djärv]

Hello.

27 apr 2014 kl. 09:47 skrev Glyph <glyph@twistedmatrix.com>:

> 
> I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
> between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
> terminal (running with -nw) results in garbage characters on either side
> of the paste.
> 
> For example, if I paste the URL, it looks like
> 200~http://emacsformacosx.com/201~, or causes jumping around within the
> buffer (for example, when filing this bug, it jumped to the beginning of
> the buffer and pasted the URL and the "201~" before the From:).
> 
> 

I can't reproduce this.  Did you start emacs with -Q?
What application did you copy from?
What key combination did you use to paste?

	Jan D.

> 
> In GNU Emacs 24.4.50.1 (x86_64-apple-darwin, NS apple-appkit-1038.36)
> of 2014-04-26 on bob.porkrind.org
> Repository revision: monnier@iro.umontreal.ca-20140425192226-frpjpmgs0r8ad9wc
> Configured using:
> `configure --host=x86_64-apple-darwin --build=i686-apple-darwin
> --with-ns'
> 
> Configured features:
> ACL LIBXML2 ZLIB
> 
> Important settings:
>  value of $LANG: en_US.UTF-8
>  locale-coding-system: utf-8-unix
> 
> Major mode: Lisp Interaction
> 
> Minor modes in effect:
>  quick-hack-emacs-lisp-mode: t
>  shell-dirtrack-mode: t
>  helm-match-plugin-mode: t
>  helm-occur-match-plugin-mode: t
>  global-auto-complete-mode: t
>  auto-complete-mode: t
>  global-rainbow-delimiters-mode: t
>  rainbow-delimiters-mode: t
>  obb-mode: t
>  which-function-mode: t
>  show-paren-mode: t
>  server-mode: t
>  icomplete-mode: t
>  tooltip-mode: t
>  mouse-wheel-mode: t
>  menu-bar-mode: t
>  file-name-shadow-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  column-number-mode: t
>  line-number-mode: t
>  transient-mark-mode: t
> 
> Recent input:
> ESC [ ? 1 ; 2 c ESC [ 2 0 0 ~ h t t p : / / e m a c 
> s f o r m a c o s x . c o m / ESC [ 2 0 1 ~ C-_ C-_ 
> C-_ C-_ ESC x e m a c s b DEL - b u g C-n RET
> 
> Recent messages:
> Loading paren...done
> Loading which-func...done
> Loading /Users/glyph/Settings/Emacs/bootstrap/customizations.el (source)...done
> Loading /Users/glyph/Settings/Emacs/firstparty/quickhack-autoload.el (source)...done
> Loading /Users/glyph/Settings/Emacs/firstparty/flyflakes-autoload.el (source)...done
> ad-handle-definition: `tramp-read-passwd' got redefined
> Pymacs loading ropemacs...done
> For information about GNU Emacs and the GNU system, type C-h C-a.
> Nowhere to go back to. [2 times]
> Undo! [4 times]
> 
> Load-path shadows:
> None found.
> 
> Features:
> (shadow sort mail-extr emacsbug message rfc822 mml mml-sec mm-decode
> mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
> sendmail rfc2047 rfc2045 ietf-drums mail-utils help-at-pt xterm ropemacs
> pymacs rx python-patches python quickhack glyph-setup adaptive-wrap
> helm-C-x-b helm-imenu helm-command helm-elisp helm-eval eldoc helm-mode
> helm-cmd-t helm-files image-dired tramp tramp-compat tramp-loaddefs
> trampver shell pcomplete format-spec dired-x dired-aux ffap thingatpt
> helm-buffers helm-elscreen helm-tags helm-bookmark helm-adaptative
> helm-info helm-net browse-url xml url url-proxy url-privacy url-expand
> url-methods url-history url-cookie url-domsuf url-util url-parse
> auth-source eieio byte-opt bytecomp byte-compile cconv eieio-core
> gnus-util mm-util mail-prsvr password-cache url-vars mailcap helm-plugin
> bookmark pp helm-help helm-match-plugin helm-grep helm-regexp grep
> helm-external helm-utils dired helm-locate helm helm-config helm-aliases
> auto-complete-config auto-complete edmacro kmacro popup pelican-mode
> backandforth advice help-fns rainbow-delimiters column-marker easy-mmode
> obb-mode cl-macs combinator cl cl-loaddefs cl-lib epytext
> goto-definition gv info easymenu package time-date jka-compr
> tango-dark-theme which-func imenu paren server avoid icomplete linum
> cus-start cus-load flymake compile comint ansi-color ring tooltip
> electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win
> tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
> lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
> scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
> georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
> korean japanese hebrew greek romanian slovak czech european ethiopic
> indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
> abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files
> text-properties overlay sha1 md5 base64 format env code-pages mule
> custom widget hashtable-print-readable backquote make-network-process
> cocoa ns multi-tty emacs)
> 
> Memory information:
> ((conses 16 215726 5957)
> (symbols 48 31136 0)
> (miscs 40 58 122)
> (strings 32 54984 6179)
> (string-bytes 1 1481420)
> (vectors 16 21509)
> (vector-slots 8 504251 2686)
> (floats 8 157 651)
> (intervals 56 364 18)
> (buffers 960 14))
> 
> 

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

On 04/27/2014 10:27 AM, Jan Djärv wrote:
> 27 apr 2014 kl. 09:47 skrev Glyph <glyph@twistedmatrix.com>:
> 
>>
>> I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
>> between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
>> terminal (running with -nw) results in garbage characters on either side
>> of the paste.
>>
>> For example, if I paste the URL, it looks like
>> 200~http://emacsformacosx.com/201~, or causes jumping around within the
>> buffer (for example, when filing this bug, it jumped to the beginning of
>> the buffer and pasted the URL and the "201~" before the From:).
>>
>>
> 
> I can't reproduce this.  Did you start emacs with -Q?
> What application did you copy from?
> What key combination did you use to paste?

That's the new bracketed paste system supoport. Emacs is correctly
sending the terminal the sequence to enable bracketed paste mode, but
isn't correctly the start sequence. I hope we don't have to make this
feature conditional.

OP, what's in *Messages*? If you M-x trace-function-background
xterm-paste, do you see messages in *trace-output*? What happens if you
manually type ESC [ 2 0 0 ~ ?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2004 bytes --]

On 04/27/2014 10:44 AM, Daniel Colascione wrote:
> On 04/27/2014 10:27 AM, Jan Djärv wrote:
>> 27 apr 2014 kl. 09:47 skrev Glyph <glyph@twistedmatrix.com>:
>>
>>>
>>> I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
>>> between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
>>> terminal (running with -nw) results in garbage characters on either side
>>> of the paste.
>>>
>>> For example, if I paste the URL, it looks like
>>> 200~http://emacsformacosx.com/201~, or causes jumping around within the
>>> buffer (for example, when filing this bug, it jumped to the beginning of
>>> the buffer and pasted the URL and the "201~" before the From:).
>>>
>>>
>>
>> I can't reproduce this.  Did you start emacs with -Q?
>> What application did you copy from?
>> What key combination did you use to paste?
> 
> That's the new bracketed paste system supoport. Emacs is correctly
> sending the terminal the sequence to enable bracketed paste mode, but
> isn't correctly the start sequence. I hope we don't have to make this
> feature conditional.
> 
> OP, what's in *Messages*? If you M-x trace-function-background
> xterm-paste, do you see messages in *trace-output*? What happens if you
> manually type ESC [ 2 0 0 ~ ?
> 

I have an idea, actually. What if Terminal.app, in its infinite glory,
is sending ESC twice, once because bracketed paste support is enabled,
and once because Command is held down? Does the following patch help or
change the behavior in any way?

=== modified file 'lisp/term/xterm.el'
--- lisp/term/xterm.el	2014-04-27 23:26:42 +0000
+++ lisp/term/xterm.el	2014-04-27 23:30:39 +0000
@@ -428,6 +428,8 @@
     ;; Recognize the start of a bracketed paste sequence.  The handler
     ;; internally recognizes the end.
     (define-key map "\e[200~" [xterm-paste])
+    ;; Work around Terminal.app bug?
+    (define-key map "\e\e[200~" [xterm-paste])

     map)
   "Function key map overrides for xterm.")



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Glyph]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]


On Apr 27, 2014, at 10:44 AM, Daniel Colascione <dancol@dancol.org> wrote:

> On 04/27/2014 10:27 AM, Jan Djärv wrote:
>> 27 apr 2014 kl. 09:47 skrev Glyph <glyph@twistedmatrix.com>:
>> 
>>> 
>>> I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
>>> between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
>>> terminal (running with -nw) results in garbage characters on either side
>>> of the paste.
>>> 
>>> For example, if I paste the URL, it looks like
>>> 200~http://emacsformacosx.com/201~, or causes jumping around within the
>>> buffer (for example, when filing this bug, it jumped to the beginning of
>>> the buffer and pasted the URL and the "201~" before the From:).
>>> 
>>> 
>> 
>> I can't reproduce this.  Did you start emacs with -Q?
>> What application did you copy from?
>> What key combination did you use to paste?
> 
> That's the new bracketed paste system supoport. Emacs is correctly
> sending the terminal the sequence to enable bracketed paste mode, but
> isn't correctly the start sequence. I hope we don't have to make this
> feature conditional.

I did figure this out a couple of hours after I discovered the weird behavior.

> OP, what's in *Messages*? If you M-x trace-function-background
> xterm-paste, do you see messages in *trace-output*? What happens if you
> manually type ESC [ 2 0 0 ~ ?

I can save you some trouble - bracketed paste support works fine, the terminal sends the right control codes :-).

The problem is that I had this key-binding in some emacs-lisp code I wrote a while ago:

(global-set-key [(meta \])] 'something)
(global-set-key [(meta \[)] 'something-else)

After removing those global key bindings, it works great.

It might be worth putting some nicer error-reporting around those though, since M-] and M-[ are perfectly cromulent keys to want to bind (and I still bind them, I just have an (if (not (eq window-system nil))) around it now.

Thanks,

-g

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4124 bytes --]

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2499 bytes --]

On 04/27/2014 05:03 PM, Glyph wrote:
> 
> On Apr 27, 2014, at 10:44 AM, Daniel Colascione <dancol@dancol.org> wrote:
> 
>> On 04/27/2014 10:27 AM, Jan Djärv wrote:
>>> 27 apr 2014 kl. 09:47 skrev Glyph <glyph@twistedmatrix.com>:
>>>
>>>>
>>>> I'm using snapshot builds from http://emacsformacosx.com/.  Somewhere
>>>> between the 4/13 snapshot and the 4/24 snapshot, "paste" in an OS X
>>>> terminal (running with -nw) results in garbage characters on either side
>>>> of the paste.
>>>>
>>>> For example, if I paste the URL, it looks like
>>>> 200~http://emacsformacosx.com/201~, or causes jumping around within the
>>>> buffer (for example, when filing this bug, it jumped to the beginning of
>>>> the buffer and pasted the URL and the "201~" before the From:).
>>>>
>>>>
>>>
>>> I can't reproduce this.  Did you start emacs with -Q?
>>> What application did you copy from?
>>> What key combination did you use to paste?
>>
>> That's the new bracketed paste system supoport.

Dammit. Why do I only notice typos in my emails after I send them?

> Emacs is correctly
>> sending the terminal the sequence to enable bracketed paste mode, but
>> isn't correctly the start sequence. I hope we don't have to make this
>> feature conditional.
> 
> I did figure this out a couple of hours after I discovered the weird behavior.

Eh, at least I found an unrelated bug in the bracketed paste support as
a result of your message.

>> OP, what's in *Messages*? If you M-x trace-function-background
>> xterm-paste, do you see messages in *trace-output*? What happens if you
>> manually type ESC [ 2 0 0 ~ ?
> 
> I can save you some trouble - bracketed paste support works fine, the terminal sends the right control codes :-).
> 
> The problem is that I had this key-binding in some emacs-lisp code I wrote a while ago:
> 
> (global-set-key [(meta \])] 'something)
> (global-set-key [(meta \[)] 'something-else)

Your numeric keypad bindings should have broken too.

Also, we bind these keys in xterm-function-map. Shouldn't that map
override global-map?

> After removing those global key bindings, it works great.
> 
> It might be worth putting some nicer error-reporting around those
> though, since M-] and M-[ are perfectly cromulent keys to want to bind
> (and I still bind them, I just have an (if (not (eq window-system nil)))
> around it now.

I'd advise something a bit more subtle, in case you want to use emacs
server with tty frames.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Stefan Monnier]

>> (global-set-key [(meta \])] 'something)
>> (global-set-key [(meta \[)] 'something-else)
> Your numeric keypad bindings should have broken too.
> Also, we bind these keys in xterm-function-map. Shouldn't that map
> override global-map?

No.  The way input decoding takes place is very delicate and what "takes
priority" when is quite delicate.

Yes, a ?\M-\] binding in input-decode-map takes precedence over such
a binding in global-map.

But after reading ?\M-\] we don't have enough input (yet) to translate
that to something else, so Emacs has a choice to make:
- either wait for more input, in the hope that it will match one of the
  bindings under the ?\M-\] prefix in input-decode-map.
- or stop right here since we do have a valid binding already (in the
  global map).
And Emacs chooses option 2 because option 1 is too risky.
E.g. Option 1 would prevent ESC ESC ESC from working, since there's
pretty much always an ESC prefix in input-decode-map, so we'd always
"wait for more" after an ESC.

Maybe it would make sense to try and improve the input-decode-map
support with a way to say: if we've seen "ESC [", we have enough
evidence that there is a real escape sequence coming, so keep waiting
for input as long as it matches something in input-decode-map.


        Stefan

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Glyph Lefkowitz]

[-- Attachment #1.1: Type: text/plain, Size: 408 bytes --]

On Apr 27, 2014, at 5:11 PM, Daniel Colascione <dancol@dancol.org> wrote:

> I'd advise something a bit more subtle, in case you want to use emacs
> server with tty frames.

Good point.  In fact, I frequently do exactly that, so finding a better solution would be good.

Is there a frame-local map that I can use / that Emacs can recommend other people use for similar types of bindings?

-glyph


[-- Attachment #1.2: Type: text/html, Size: 1885 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4124 bytes --]

bug#17357: 24.4.50; garbage characters when pasting in an OS X terminal

[Lars Ingebrigtsen]

Glyph <glyph@twistedmatrix.com> writes:

> The problem is that I had this key-binding in some emacs-lisp code I
> wrote a while ago:
>
> (global-set-key [(meta \])] 'something)
> (global-set-key [(meta \[)] 'something-else)
>
> After removing those global key bindings, it works great.
>
> It might be worth putting some nicer error-reporting around those
> though, since M-] and M-[ are perfectly cromulent keys to want to bind
> (and I still bind them, I just have an (if (not (eq window-system
> nil))) around it now.

This was apparently not an Emacs bug, but then the discussion turned to
possibly adding a new feature that would warn the user if binding these
keys in a terminal.  I'm not sure that's really warranted -- there's a
bunch of key bindings that work under a window system that don't work in
a terminal.

It would be nice to have key bindings that work under different terminal
types (and is that what overriding-terminal-local-map kinda sorta
does?), but this would in any case be a new feature request, so I'm
closing this bug report.  (If this is something that should be worked
upon, please open a new wishlist bug report.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no