bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Kaloian Doganov]

0. Start Emacs using `emacs -Q'.

1. Open some other X application (gedit, GNOME Terminal, etc.) and enter
   the following text:

       Проба

2. Select the text using the mouse and yank it using mouse-2 in a
   scratch buffer in Emacs.

3. The result is:

       \u041f\u0440\u043e\u0431\u0430

   instead of the expected:

       Проба


In GNU Emacs 24.0.50.1 (i486-pc-linux-gnu, GTK+ Version 2.12.9)
 of 2010-08-05 on gana, modified for gNewSense
 (Unofficial gNewSense emacs-snapshot package, version 1:20100805-gns1)
Windowing system distributor `The X.Org Foundation', version 11.0.10400090
configured using `configure  '--build=i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/24.0.50/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.0.50/site-lisp:/usr/share/emacs/site-lisp' '--without-compress-info' '--with-x=yes' '--with-x-toolkit=gtk' 'build_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -Wall -Wno-pointer-sign -fno-crossjumping -I/usr/include/GNUstep -O2' 'LDFLAGS=-g -Wl,-znocombreloc -Wl,-z,defs -Wl,--as-needed' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: bg_BG.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Org

Minor modes in effect:
  shell-dirtrack-mode: t
  type-break-mode-line-message-mode: t
  show-paren-mode: t
  iswitchb-mode: t
  display-battery-mode: t
  desktop-save-mode: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> M-x r e p o r t - e m <tab> <return>

Recent messages:
Using schema /usr/share/emacs/24.0.50/etc/schema/xslt.rnc [2 times]
Using vacuous schema
Using schema /usr/share/emacs/24.0.50/etc/schema/xslt.rnc [6 times]
Using vacuous schema
Loading arc-mode...done
Parsing archive file...done.
Using vacuous schema
Wrote /home/kaloian/.emacs.desktop.lock
Desktop: 128 buffers restored.
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
/usr/share/emacs-snapshot/site-lisp/mit-scheme/xscheme hides /usr/share/emacs/site-lisp/mit-scheme/xscheme
/usr/share/emacs/24.0.50/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/local/share/emacs/site-lisp/remember hides /usr/share/emacs/24.0.50/lisp/textmodes/remember
/usr/share/emacs-snapshot/site-lisp/mit-scheme/xscheme hides /usr/share/emacs/24.0.50/lisp/progmodes/xscheme

Features:
(shadow sort gnus-cite mail-extr emacsbug arc-mode archive-mode sql
parse-time vc-cvs tex-mode compile shell scheme sh-script executable
view tar-mode flyspell ispell dired-aux dired make-mode python-21 python
jka-compr nxml-uchnm rng-xsd xsd-regexp rng-cmpct rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn
nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc
xmltok imenu cc-mode cc-fonts cc-menus cc-cmds vc-dispatcher vc-svn
org-wl org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo
org-infojs org-html org-exp ob-exp org-exp-blocks org-agenda org-info
org-gnus org-docview org-bibtex org-bbdb package type-break paren
iswitchb battery desktop cus-start cus-load cc-styles cc-align cc-engine
cc-vars cc-defs org-nnml nnml org byte-opt warnings bytecomp
byte-compile ob-emacs-lisp ob-keys ob-comint comint ring ob-tangle
ob-ref ob-lob ob-table ob org-footnote org-src org-list org-faces
org-compat org-entities org-macs noutline outline easy-mmode remember
info-look info advice help-fns advice-preload spam spam-stat bbdb-com cl
cl-19 bbdb timezone gnus-uu yenc gnus-msg gnus-art mm-uu mml2015
epg-config mm-view smime password-cache dig mailcap gnus-sum nnoo
gnus-group time-date gnus-undo nnmail mail-source format-spec gnus-start
gnus-spec gnus-int gnus-range gnus-win gnus gnus-ems nnheader gnus-util
wid-edit message sendmail rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mailabbrev mail-utils gmm-utils mailheader edmacro kmacro
ps-mule ps-print ps-def lpr regexp-opt cal-menu easymenu calendar
cal-loaddefs time server bbdb-autoloads tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)

-- 
Protect your digital freedom and privacy, eliminate DRM, learn more at
http://www.defectivebydesign.org/what_is_drm

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[David De La Harpe Golden]

On 05/08/10 14:56, Kaloian Doganov wrote:
> 0. Start Emacs using `emacs -Q'.
>
> 1. Open some other X application (gedit, GNOME Terminal, etc.) and enter
>     the following text:
>
>         Проба
>
> 2. Select the text using the mouse and yank it using mouse-2 in a
>     scratch buffer in Emacs.
>
> 3. The result is:
>
>         \u041f\u0440\u043e\u0431\u0430
>
>     instead of the expected:
>
>         Проба
>

This is a known issue (#6635) and there is a patch, but I guess it 
hasn't made it in-tree yet. See link below for a file that you can load 
to fix this issue:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=6720#17

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Kenichi Handa]

In article <87mxt1go33.fsf@doganov.org>, Kaloian Doganov <kaloian@gnu.org> writes:

> 0. Start Emacs using `emacs -Q'.
> 1. Open some other X application (gedit, GNOME Terminal, etc.) and enter
>    the following text:

>        Проба

> 2. Select the text using the mouse and yank it using mouse-2 in a
>    scratch buffer in Emacs.

> 3. The result is:

>        \u041f\u0440\u043e\u0431\u0430

>    instead of the expected:

>        Проба

I can reproduce this bug.  In Emacs 23 mouse-2 was bound to
mouse-yank-at-click which calls (x-get-selection 'PRIMARY
'UTF8_STRING) with this calling sequence.

mouse-yank-at-click
  ->yank->current-kill
    ->x-cut-buffer-or-selection-value
      ->x-get-buffer

But, now mouse-2 is bound to mouse-yank-primary which
directly calls (x-get-selection 'PRIMARY).  As the second
optional arg DATA-TYPE is omitted, x-get-selection uses
'STRING for that.  And, for such a data-type request, Gnome
applications returns something like this:
  \u041f\u0440\u043e\u0431\u0430

There are various ways to fix it, and each is fairly easy
once we decide the precise spec of each functions.  But as I
have not followed the discussions in the thread "Cut and
paste with C-w/mouse-2 not working?", I don't know which is
the best.  One easy way is this:

=== modified file 'lisp/mouse.el'
--- lisp/mouse.el	2010-08-08 21:57:34 +0000
+++ lisp/mouse.el	2010-08-09 02:54:23 +0000
@@ -1265,7 +1265,7 @@
     ;; the middle of an active region.
     (deactivate-mark))
   (or mouse-yank-at-point (mouse-set-point click))
-  (let ((primary (x-get-selection 'PRIMARY)))
+  (let ((primary (x-get-selection 'PRIMARY 'TEXT)))
     (if primary
         (insert primary)
       (error "No primary selection"))))

---
Kenichi Handa
handa@m17n.org

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicodeescapes

[Drew Adams]

> In Emacs 23 mouse-2 was bound to mouse-yank-at-click 
> But, now mouse-2 is bound to mouse-yank-primary

It is not so nice to change the binding.

That breaks 3rd-party and user-customization code.

Consider key-binding code that uses [remap], for example.
Or code that advises `mouse-yank-at-click'.

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Mon, 09 Aug 2010 11:57:12 +0900
> Cc: 6802@debbugs.gnu.org
> 
> I can reproduce this bug.  In Emacs 23 mouse-2 was bound to
> mouse-yank-at-click which calls (x-get-selection 'PRIMARY
> 'UTF8_STRING) with this calling sequence.
> 
> mouse-yank-at-click
>   ->yank->current-kill
>     ->x-cut-buffer-or-selection-value
>       ->x-get-buffer
> 
> But, now mouse-2 is bound to mouse-yank-primary which
> directly calls (x-get-selection 'PRIMARY).  As the second
> optional arg DATA-TYPE is omitted, x-get-selection uses
> 'STRING for that.

Why does x-get-selection default to 'STRING?  Is that sensible
behavior?

> -  (let ((primary (x-get-selection 'PRIMARY)))
> +  (let ((primary (x-get-selection 'PRIMARY 'TEXT)))

Would it make sense to have x-get-selection default to 'TEXT instead?

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Jan Djärv]

Eli Zaretskii skrev 2010-08-09 18.34:

> Why does x-get-selection default to 'STRING?  Is that sensible
> behavior?
>
>> -  (let ((primary (x-get-selection 'PRIMARY)))
>> +  (let ((primary (x-get-selection 'PRIMARY 'TEXT)))
>
> Would it make sense to have x-get-selection default to 'TEXT instead?

TEXT can be in any encoding which can make it diffcult.  Nowdays I think the 
default should be UTF8_STRING, and if that fails, try TEXT and then STRING.

But I think the application should not offer STRING (which implies Latin-1) if 
the data isn't in latin-1.

	Jan D.

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Kenichi Handa]

In article <4C603A4F.20006@swipnet.se>, Jan Djärv <jan.h.d@swipnet.se> writes:

> Eli Zaretskii skrev 2010-08-09 18.34:

> > Why does x-get-selection default to 'STRING?  Is that sensible
> > behavior?

I just didn't change it from Emacs 19.  Perhaps it's not
sensible nowadays.

>>> -  (let ((primary (x-get-selection 'PRIMARY)))
>>> +  (let ((primary (x-get-selection 'PRIMARY 'TEXT)))
> >
> > Would it make sense to have x-get-selection default to 'TEXT instead?

> TEXT can be in any encoding which can make it diffcult.

Ah, yes.

> Nowdays I think the 
> default should be UTF8_STRING, and if that fails, try TEXT and then STRING.

Why not COMPOUND_TEXT instead of TEXT above?  Please see the
docstring of x-select-request-type.

Anyway, the current selection-related codes mix an abstract
layer (something like interprogram-cut/paste-function) and
X-specific layer in a chaos manner.  It seems that overhaul
and re-design is necessary.

> But I think the application should not offer STRING (which
> implies Latin-1) if the data isn't in latin-1.

Fully agreed.

---
Kenichi Handa
handa@m17n.org

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Jan D.]

On 2010-08-10 04:29, Kenichi Handa wrote:
> In article<4C603A4F.20006@swipnet.se>, Jan Djärv<jan.h.d@swipnet.se>  writes:
>> Nowdays I think the
>> default should be UTF8_STRING, and if that fails, try TEXT and then STRING.
>
> Why not COMPOUND_TEXT instead of TEXT above?  Please see the
> docstring of x-select-request-type.

I forgot about that.  It is better to try that before TEXT.

>
> Anyway, the current selection-related codes mix an abstract
> layer (something like interprogram-cut/paste-function) and
> X-specific layer in a chaos manner.  It seems that overhaul
> and re-design is necessary.
>

It is a bit of a mess.  Separation of interprogram cut/paste and X 
selections would be nice.

	Jan D.

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Štěpán Němec]

"Jan D." <jan.h.d@swipnet.se> writes:

> On 2010-08-10 04:29, Kenichi Handa wrote:
>> In article<4C603A4F.20006@swipnet.se>, Jan Djärv<jan.h.d@swipnet.se> writes:
>>> Nowdays I think the
>>> default should be UTF8_STRING, and if that fails, try TEXT and then STRING.
>>
>> Why not COMPOUND_TEXT instead of TEXT above?  Please see the
>> docstring of x-select-request-type.
>
> I forgot about that.  It is better to try that before TEXT.
>
>>
>> Anyway, the current selection-related codes mix an abstract
>> layer (something like interprogram-cut/paste-function) and
>> X-specific layer in a chaos manner.  It seems that overhaul
>> and re-design is necessary.
>>
>
> It is a bit of a mess.  Separation of interprogram cut/paste and X selections
> would be nice.
>
> 	Jan D.

This is (and has been for quite a while now) a rather show-stopperish
bug. Is anybody working on fixing it?

(A refresher: you now get \u0ca0_\u0ca0 instead of ಠ_ಠ when pasting from
another X app.)

  Štěpán

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=iso-2022-jp-2, Size: 950 bytes --]

In article <87fwvq4ix4.fsf@gmail.com>, ^[$(D*^^[(Bt^[$(D+5^[(Bp^[$(D+!^[(Bn N^[$(D+5^[(Bmec <stepnem@gmail.com> writes:
>>> Anyway, the current selection-related codes mix an abstract
>>> layer (something like interprogram-cut/paste-function) and
>>> X-specific layer in a chaos manner.  It seems that overhaul
>>> and re-design is necessary.
>>> 
> >
> > It is a bit of a mess.  Separation of interprogram cut/paste and X selections
> > would be nice.
> >
> > 	Jan D.

> This is (and has been for quite a while now) a rather show-stopperish
> bug. Is anybody working on fixing it?

As I have been using my private version of
mouse-yank-primary (calling (x-selection-value-internal nil)
instead of (x-get-selection ...)), I didn't notice that this
problem has not yet been solved.  :-(

I wrote:

> It seems that overhaul and re-design is necessary.

but I don't have a time to work on it at the moment.  Any
volunteer?

---
Kenichi Handa
handa@m17n.org

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[David De La Harpe Golden]

On 29/10/10 03:18, Kenichi Handa wrote:
> I wrote:
>
>> It seems that overhaul and re-design is necessary.
>
> but I don't have a time to work on it at the moment.  Any
> volunteer?

(I suppose I have some familiarity with the general area at this stage, 
but my emacs time is currently rather limited too, until late december. 
I'm quite likely to make some suggestion then ...at least if problems 
remain - from thread "Re: moving x-select*" on emacs-devel a few days 
ago it looks like there is some amount of source cleanup/refactoring 
ongoing.)

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Jan Djärv]

2010-10-31 02:38, David De La Harpe Golden skrev:
> On 29/10/10 03:18, Kenichi Handa wrote:
>> I wrote:
>>
>>> It seems that overhaul and re-design is necessary.
>>
>> but I don't have a time to work on it at the moment. Any
>> volunteer?
>
> (I suppose I have some familiarity with the general area at this stage, but my
> emacs time is currently rather limited too, until late december. I'm quite
> likely to make some suggestion then ...at least if problems remain - from
> thread "Re: moving x-select*" on emacs-devel a few days ago it looks like
> there is some amount of source cleanup/refactoring ongoing.)

For the purpose of fixing this bug, such an overhaul isn't needed.  I made a 
simple fix that makes mouse-2 use the same mechanism as yank from clipboard 
for X, i.e. try UTF8_STRING and COMPOUND_TEXT before STRING.

	Jan D.

bug#6802: 24.0.50; Yanking non-ASCII text from other X application leads to unicode escapes

[Kenichi Handa]

In article <4CCD93DF.5030509@swipnet.se>, Jan Djärv <jan.h.d@swipnet.se> writes:

> For the purpose of fixing this bug, such an overhaul isn't needed.

That's always true.

> I made a 
> simple fix that makes mouse-2 use the same mechanism as yank from clipboard 
> for X, i.e. try UTF8_STRING and COMPOUND_TEXT before STRING.

Ah, thank you.  It is better to update the comment in
mouse-yank-primary.

Anyway, we have these functions:
  x-get-selection
  x-get-selection-internal
  x-get-selection-value
  x-selection-value
  x-selection-value-internal
What a mess...

---
Kenichi Handa
handa@m17n.org