bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Stefan Monnier]

Package: Emacs
Version: 27.0.50


(gui-get-selection nil 'text/html)

returns utf-16 text when the primary selection is owned by Mozilla, but
we decode it as latin-1 instead, so it looks like garbage.

I don't know why we're getting utf-16.  Is that what standards say it
should do?  If so, we should adjust our code (which currently knows
nothing about the `text/html` target-type).

As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
using something else because he's getting something with a `charset`
property which I don't get here) because:
- selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
  the property `foreign-selection` set to `STRING` when the actual
  string type is not known (as opposed to COMPOUND-TEXT and
  UTF8-STRING, basically).
- in gui-get-selection we then have a mapping from `STRING` to
  `iso-8859-1` (which is apparently the right thing for the official
  `STRING` target-type in X11).

I can't figure out if/where these kinds of things about the X11
selection protocol is described, but at least in `xclip` they have
a hack specifically for this case:

    [...]
    if (html != None && sel_type == html) {
	/* if the buffer contains UCS-2 (UTF-16), convert to
	 * UTF-8.  Mozilla-based browsers do this for the
	 * text/html target.
	 */
    [...]

and according to the subsequent code it's not even always the
same endianness.

I don't know what is the difference between the `target-type` passed to
x-get-selection-internal and the `foreign-selection` property we get on
the returned string (they seem to be the same in my tests, except when
the type is not one of the known ones, and where we then force
`foreign-selection` to be `STRING`).


        Stefan



In GNU Emacs 27.0.50 (build 1, i686-pc-linux-gnu, GTK+ Version 2.24.32)
 of 2018-03-23 built on ceviche
Repository revision: ef4cd3805771e2cccd395d0f0b35f56816940508
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Debian GNU/Linux buster/sid

Recent messages:
Saving file /home/monnier/src/emacs/work/src/xselect.c...
Wrote /home/monnier/src/emacs/work/src/xselect.c
Mark set
user-error: Minibuffer window is not active
Mark set
Mark saved where search started
Mark set
Making completion list... [2 times]
Quit [2 times]
Mark set

Configured using:
 'configure -C --enable-checking --with-modules --enable-check-lisp-object-type
 'CFLAGS=-Wall -g3 -Og -Wno-pointer-sign'
 PKG_CONFIG_PATH=/home/monnier/lib/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS NOTIFY GNUTLS
LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11
MODULES THREADS

Important settings:
  value of $LANG: fr_CH.UTF-8
  locale-coding-system: utf-8-unix

Major mode: InactiveMinibuffer

Minor modes in effect:
  csv-field-index-mode: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  electric-pair-mode: t
  global-reveal-mode: t
  reveal-mode: t
  auto-insert-mode: t
  savehist-mode: t
  minibuffer-electric-default-mode: t
  global-compact-docstrings-mode: t
  url-handler-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/monnier/src/emacs/elpa/packages/svg/svg hides /home/monnier/src/emacs/work/lisp/svg
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-xref hides /home/monnier/src/emacs/work/lisp/progmodes/ada-xref
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-mode hides /home/monnier/src/emacs/work/lisp/progmodes/ada-mode
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-stmt hides /home/monnier/src/emacs/work/lisp/progmodes/ada-stmt
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-prj hides /home/monnier/src/emacs/work/lisp/progmodes/ada-prj
/home/monnier/src/emacs/elpa/packages/hyperbole/set hides /home/monnier/src/emacs/work/lisp/emacs-lisp/set
/home/monnier/src/emacs/elpa/packages/landmark/landmark hides /home/monnier/src/emacs/work/lisp/obsolete/landmark
/home/monnier/src/emacs/elpa/packages/crisp/crisp hides /home/monnier/src/emacs/work/lisp/obsolete/crisp

Features:
(mule-diag csv-mode mailcap reporter debian-bug debian-el-loaddefs
image-file iimage skeleton html5-schema rng-xsd xsd-regexp rng-cmpct
rng-nxml nxml-mode nxml-outln nxml-rap sgml-mode dom reftex-dcr reftex
reftex-loaddefs reftex-vars latexenc sort mail-extr emacsbug tildify rst
rng-valid refer refer-to-bibtex refbib printing picture nroff-mode
enriched ebnf2ps ps-print ps-print-loaddefs ps-def lpr delim-col
bib-mode view cal-china lunar solar cal-dst cal-bahai cal-islam
cal-hebrew holidays hol-loaddefs cal-french diary-lib diary-loaddefs
cal-move battery log-view srecode/document semantic/doc srecode/semantic
semantic/senator semantic/decorate semantic/ctxt semantic/format
srecode/extract srecode/insert srecode/filters srecode/find srecode/map
srecode/ctxt semantic/tag-ls semantic/find srecode/compile
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw srecode/args ede/speedbar ede/files ede ede/detect ede/base
ede/auto ede/source eieio-speedbar eieio-custom cedet srecode/dictionary
srecode/table eieio-base srecode mode-local informat texinfo tex-mode
vc-dir grep rect gdb-mi bindat gud ffap cl-print ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox org-protocol org-mouse org-mobile org-agenda org-indent
org-feed org-crypt org-capture org-attach org-id org-rmail org-mhe
org-irc org-info org-gnus nnir gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range gnus-win gnus nnheader
org-docview org-bibtex bibtex org-bbdb org-w3m org-element avl-tree
generator org org-macro org-footnote org-pcomplete org-list org-faces
org-entities org-version ob-emacs-lisp ob ob-tangle org-src ob-ref
ob-lob ob-table ob-keys ob-exp ob-comint ob-core ob-eval org-compat
org-macs org-loaddefs cal-menu calendar cal-loaddefs autorevert
filenotify doc-view jka-compr image-mode vc-bzr vc-src vc-sccs vc-svn
vc-cvs vc-rcs dabbrev log-edit message sendmail rmc puny dired
dired-loaddefs format-spec rfc822 mml mml-sec gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils mailheader
pcvs-util bug-reference add-log sh-script make-mode autoload shell
pcomplete pulse etags xref project epa-file epa derived epg sm-c-mode
smie whitespace misearch multi-isearch eieio-opt speedbar sb-image
ezimage dframe cl-extra help-fns radix-tree executable copyright
lisp-mnt xscheme unsafep trace testcover shadow scheme re-builder
profiler inf-lisp ielm gmm-utils ert pp find-func ewoc debug elp edebug
cl-indent cus-edit cus-start cus-load wid-edit vc vc-dispatcher
smerge-mode vc-git diff-mode filecache server time-date flymake-proc
flymake compile comint ansi-color ring warnings noutline outline
easy-mmode flyspell ispell checkdoc thingatpt help-mode load-dir
elec-pair reveal autoinsert proof-site proof-autoloads cl pg-vars
savehist minibuf-eldef disp-table compact-docstrings cl-seq inline
kotl-autoloads advice info realgud-recursive-autoloads finder-inf
url-auth package easymenu epg-config url-handlers url-parse auth-source
eieio eieio-core cl-macs eieio-loaddefs password-cache json map url-vars
seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 8 904625 146270)
 (symbols 24 56914 156) (miscs 20 15608 1993) (strings 16 269351 14086)
 (string-bytes 1 8339699)
 (vectors 12 109056) (vector-slots 4 3333709 279700) (floats 8 1341 1410)
 (intervals 28 57426 412)
 (buffers 536 153))

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> using something else because he's getting something with a `charset`
> property which I don't get here) because:

I'm also running under GNU/Linux -- it's the latest Debian (9, which
is...  stretch?), but not with Gnome.  Instead I'm using xfce -- I guess
Gnome could get involved with the selection stuff somehow.

Another data point: If I select some HTML in Chromium,
(gui-get-selection nil 'text/html) returns nil.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Fri, 13 Apr 2018 16:55:26 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>
> 
> (gui-get-selection nil 'text/html)
> 
> returns utf-16 text when the primary selection is owned by Mozilla, but
> we decode it as latin-1 instead, so it looks like garbage.
> 
> I don't know why we're getting utf-16.  Is that what standards say it
> should do?  If so, we should adjust our code (which currently knows
> nothing about the `text/html` target-type).
> 
> As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> using something else because he's getting something with a `charset`
> property which I don't get here) because:
> - selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
>   the property `foreign-selection` set to `STRING` when the actual
>   string type is not known (as opposed to COMPOUND-TEXT and
>   UTF8-STRING, basically).
> - in gui-get-selection we then have a mapping from `STRING` to
>   `iso-8859-1` (which is apparently the right thing for the official
>   `STRING` target-type in X11).
> 
> I can't figure out if/where these kinds of things about the X11
> selection protocol is described, but at least in `xclip` they have
> a hack specifically for this case:
> 
>     [...]
>     if (html != None && sel_type == html) {
> 	/* if the buffer contains UCS-2 (UTF-16), convert to
> 	 * UTF-8.  Mozilla-based browsers do this for the
> 	 * text/html target.
> 	 */
>     [...]
> 
> and according to the subsequent code it's not even always the
> same endianness.
> 
> I don't know what is the difference between the `target-type` passed to
> x-get-selection-internal and the `foreign-selection` property we get on
> the returned string (they seem to be the same in my tests, except when
> the type is not one of the known ones, and where we then force
> `foreign-selection` to be `STRING`).

I Hope Handa-san (CC'ed) could comment on this.

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

Ping!

> Date: Sat, 14 Apr 2018 09:32:41 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 31149@debbugs.gnu.org
> 
> > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > Date: Fri, 13 Apr 2018 16:55:26 -0400
> > Cc: Lars Ingebrigtsen <larsi@gnus.org>
> > 
> > (gui-get-selection nil 'text/html)
> > 
> > returns utf-16 text when the primary selection is owned by Mozilla, but
> > we decode it as latin-1 instead, so it looks like garbage.
> > 
> > I don't know why we're getting utf-16.  Is that what standards say it
> > should do?  If so, we should adjust our code (which currently knows
> > nothing about the `text/html` target-type).
> > 
> > As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> > using something else because he's getting something with a `charset`
> > property which I don't get here) because:
> > - selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
> >   the property `foreign-selection` set to `STRING` when the actual
> >   string type is not known (as opposed to COMPOUND-TEXT and
> >   UTF8-STRING, basically).
> > - in gui-get-selection we then have a mapping from `STRING` to
> >   `iso-8859-1` (which is apparently the right thing for the official
> >   `STRING` target-type in X11).
> > 
> > I can't figure out if/where these kinds of things about the X11
> > selection protocol is described, but at least in `xclip` they have
> > a hack specifically for this case:
> > 
> >     [...]
> >     if (html != None && sel_type == html) {
> > 	/* if the buffer contains UCS-2 (UTF-16), convert to
> > 	 * UTF-8.  Mozilla-based browsers do this for the
> > 	 * text/html target.
> > 	 */
> >     [...]
> > 
> > and according to the subsequent code it's not even always the
> > same endianness.
> > 
> > I don't know what is the difference between the `target-type` passed to
> > x-get-selection-internal and the `foreign-selection` property we get on
> > the returned string (they seem to be the same in my tests, except when
> > the type is not one of the known ones, and where we then force
> > `foreign-selection` to be `STRING`).
> 
> I hope Handa-san (CC'ed) could comment on this.

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

Ping! Ping!

> Date: Tue, 24 Apr 2018 21:11:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> 
> Ping!
> 
> > Date: Sat, 14 Apr 2018 09:32:41 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, 31149@debbugs.gnu.org
> > 
> > > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > > Date: Fri, 13 Apr 2018 16:55:26 -0400
> > > Cc: Lars Ingebrigtsen <larsi@gnus.org>
> > > 
> > > (gui-get-selection nil 'text/html)
> > > 
> > > returns utf-16 text when the primary selection is owned by Mozilla, but
> > > we decode it as latin-1 instead, so it looks like garbage.
> > > 
> > > I don't know why we're getting utf-16.  Is that what standards say it
> > > should do?  If so, we should adjust our code (which currently knows
> > > nothing about the `text/html` target-type).
> > > 
> > > As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> > > using something else because he's getting something with a `charset`
> > > property which I don't get here) because:
> > > - selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
> > >   the property `foreign-selection` set to `STRING` when the actual
> > >   string type is not known (as opposed to COMPOUND-TEXT and
> > >   UTF8-STRING, basically).
> > > - in gui-get-selection we then have a mapping from `STRING` to
> > >   `iso-8859-1` (which is apparently the right thing for the official
> > >   `STRING` target-type in X11).
> > > 
> > > I can't figure out if/where these kinds of things about the X11
> > > selection protocol is described, but at least in `xclip` they have
> > > a hack specifically for this case:
> > > 
> > >     [...]
> > >     if (html != None && sel_type == html) {
> > > 	/* if the buffer contains UCS-2 (UTF-16), convert to
> > > 	 * UTF-8.  Mozilla-based browsers do this for the
> > > 	 * text/html target.
> > > 	 */
> > >     [...]
> > > 
> > > and according to the subsequent code it's not even always the
> > > same endianness.
> > > 
> > > I don't know what is the difference between the `target-type` passed to
> > > x-get-selection-internal and the `foreign-selection` property we get on
> > > the returned string (they seem to be the same in my tests, except when
> > > the type is not one of the known ones, and where we then force
> > > `foreign-selection` to be `STRING`).
> > 
> > I hope Handa-san (CC'ed) could comment on this.
> 
> 
> 
> 

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

Ping! Ping! Ping!

> Date: Sat, 05 May 2018 12:37:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> 
> Ping! Ping!
> 
> > Date: Tue, 24 Apr 2018 21:11:10 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> > 
> > Ping!
> > 
> > > Date: Sat, 14 Apr 2018 09:32:41 +0300
> > > From: Eli Zaretskii <eliz@gnu.org>
> > > Cc: larsi@gnus.org, 31149@debbugs.gnu.org
> > > 
> > > > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > > > Date: Fri, 13 Apr 2018 16:55:26 -0400
> > > > Cc: Lars Ingebrigtsen <larsi@gnus.org>
> > > > 
> > > > (gui-get-selection nil 'text/html)
> > > > 
> > > > returns utf-16 text when the primary selection is owned by Mozilla, but
> > > > we decode it as latin-1 instead, so it looks like garbage.
> > > > 
> > > > I don't know why we're getting utf-16.  Is that what standards say it
> > > > should do?  If so, we should adjust our code (which currently knows
> > > > nothing about the `text/html` target-type).
> > > > 
> > > > As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> > > > using something else because he's getting something with a `charset`
> > > > property which I don't get here) because:
> > > > - selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
> > > >   the property `foreign-selection` set to `STRING` when the actual
> > > >   string type is not known (as opposed to COMPOUND-TEXT and
> > > >   UTF8-STRING, basically).
> > > > - in gui-get-selection we then have a mapping from `STRING` to
> > > >   `iso-8859-1` (which is apparently the right thing for the official
> > > >   `STRING` target-type in X11).
> > > > 
> > > > I can't figure out if/where these kinds of things about the X11
> > > > selection protocol is described, but at least in `xclip` they have
> > > > a hack specifically for this case:
> > > > 
> > > >     [...]
> > > >     if (html != None && sel_type == html) {
> > > > 	/* if the buffer contains UCS-2 (UTF-16), convert to
> > > > 	 * UTF-8.  Mozilla-based browsers do this for the
> > > > 	 * text/html target.
> > > > 	 */
> > > >     [...]
> > > > 
> > > > and according to the subsequent code it's not even always the
> > > > same endianness.
> > > > 
> > > > I don't know what is the difference between the `target-type` passed to
> > > > x-get-selection-internal and the `foreign-selection` property we get on
> > > > the returned string (they seem to be the same in my tests, except when
> > > > the type is not one of the known ones, and where we then force
> > > > `foreign-selection` to be `STRING`).
> > > 
> > > I hope Handa-san (CC'ed) could comment on this.
> > 

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

Ping! Ping! Ping! Ping!

> Date: Fri, 11 May 2018 12:18:13 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> 
> Ping! Ping! Ping!
> 
> > Date: Sat, 05 May 2018 12:37:24 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> > 
> > Ping! Ping!
> > 
> > > Date: Tue, 24 Apr 2018 21:11:10 +0300
> > > From: Eli Zaretskii <eliz@gnu.org>
> > > Cc: larsi@gnus.org, 31149@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> > > 
> > > Ping!
> > > 
> > > > Date: Sat, 14 Apr 2018 09:32:41 +0300
> > > > From: Eli Zaretskii <eliz@gnu.org>
> > > > Cc: larsi@gnus.org, 31149@debbugs.gnu.org
> > > > 
> > > > > From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> > > > > Date: Fri, 13 Apr 2018 16:55:26 -0400
> > > > > Cc: Lars Ingebrigtsen <larsi@gnus.org>
> > > > > 
> > > > > (gui-get-selection nil 'text/html)
> > > > > 
> > > > > returns utf-16 text when the primary selection is owned by Mozilla, but
> > > > > we decode it as latin-1 instead, so it looks like garbage.
> > > > > 
> > > > > I don't know why we're getting utf-16.  Is that what standards say it
> > > > > should do?  If so, we should adjust our code (which currently knows
> > > > > nothing about the `text/html` target-type).
> > > > > 
> > > > > As for why we decode it as latin-1, it's (under GNU/Linux; Lars may be
> > > > > using something else because he's getting something with a `charset`
> > > > > property which I don't get here) because:
> > > > > - selection_data_to_lisp_data (in xselect.c) makes a unibyte string with
> > > > >   the property `foreign-selection` set to `STRING` when the actual
> > > > >   string type is not known (as opposed to COMPOUND-TEXT and
> > > > >   UTF8-STRING, basically).
> > > > > - in gui-get-selection we then have a mapping from `STRING` to
> > > > >   `iso-8859-1` (which is apparently the right thing for the official
> > > > >   `STRING` target-type in X11).
> > > > > 
> > > > > I can't figure out if/where these kinds of things about the X11
> > > > > selection protocol is described, but at least in `xclip` they have
> > > > > a hack specifically for this case:
> > > > > 
> > > > >     [...]
> > > > >     if (html != None && sel_type == html) {
> > > > > 	/* if the buffer contains UCS-2 (UTF-16), convert to
> > > > > 	 * UTF-8.  Mozilla-based browsers do this for the
> > > > > 	 * text/html target.
> > > > > 	 */
> > > > >     [...]
> > > > > 
> > > > > and according to the subsequent code it's not even always the
> > > > > same endianness.
> > > > > 
> > > > > I don't know what is the difference between the `target-type` passed to
> > > > > x-get-selection-internal and the `foreign-selection` property we get on
> > > > > the returned string (they seem to be the same in my tests, except when
> > > > > the type is not one of the known ones, and where we then force
> > > > > `foreign-selection` to be `STRING`).
> > > > 
> > > > I hope Handa-san (CC'ed) could comment on this.
> > > 
> 
> 
> 
> 

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> (gui-get-selection nil 'text/html)
>
> returns utf-16 text when the primary selection is owned by Mozilla, but
> we decode it as latin-1 instead, so it looks like garbage.

This is still the case on the trunk:

#("ÿþM^@e^@r^@g^@e^@d^@" 0 14 (foreign-selection STRING charset iso-8859-1))

[...]

> I can't figure out if/where these kinds of things about the X11
> selection protocol is described, but at least in `xclip` they have
> a hack specifically for this case:
>
>     [...]
>     if (html != None && sel_type == html) {
> 	/* if the buffer contains UCS-2 (UTF-16), convert to
> 	 * UTF-8.  Mozilla-based browsers do this for the
> 	 * text/html target.
> 	 */
>     [...]
>
> and according to the subsequent code it's not even always the
> same endianness.

I think it would make sense for us to do the same here.  It should be
easy enough for us to detect that the string is utf-16, I think?  The
data has a BOM and everything...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 29 Sep 2019 10:44:48 +0200
> Cc: 31149@debbugs.gnu.org
> 
> >     if (html != None && sel_type == html) {
> > 	/* if the buffer contains UCS-2 (UTF-16), convert to
> > 	 * UTF-8.  Mozilla-based browsers do this for the
> > 	 * text/html target.
> > 	 */
> >     [...]
> >
> > and according to the subsequent code it's not even always the
> > same endianness.
> 
> I think it would make sense for us to do the same here.  It should be
> easy enough for us to detect that the string is utf-16, I think?

I think you want to use auto-coding-regexp-alist-lookup.

> The data has a BOM

Does it?  It doesn't have to, at least not in principle.

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think it would make sense for us to do the same here.  It should be
>> easy enough for us to detect that the string is utf-16, I think?
>
> I think you want to use auto-coding-regexp-alist-lookup.

Ah, thanks.  So should I go ahead and make this change?  It looks pretty
trivial, but I guess there could be interop problems with code that
assumes the current odd behaviour.

>> The data has a BOM
>
> Does it?  It doesn't have to, at least not in principle.

It doesn't have to, but it always does on the systems I've tested on.
But I guess it doesn't matter, `auto-coding-regexp-alist-lookup' will
figure it out in any case...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@IRO.UMontreal.CA,  31149@debbugs.gnu.org
> Date: Sun, 29 Sep 2019 11:37:28 +0200
> 
> > I think you want to use auto-coding-regexp-alist-lookup.
> 
> Ah, thanks.  So should I go ahead and make this change?  It looks pretty
> trivial, but I guess there could be interop problems with code that
> assumes the current odd behaviour.

What odd behavior is that?  I understood that we just display binary
garbage, something that no one should miss.

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Ah, thanks.  So should I go ahead and make this change?  It looks pretty
>> trivial, but I guess there could be interop problems with code that
>> assumes the current odd behaviour.
>
> What odd behavior is that?  I understood that we just display binary
> garbage, something that no one should miss.

We don't have any commands to yank HTML, so we don't display anything,
but I've got code like the following in one of my out-of-tree packages
(which will fail after the fix).  I'm with that, though, but I have no
idea how much other people would be impacted.

(defun ewp-yank-html ()

[...]

  (let ((data (loop for type in '(PRIMARY CLIPBOARD)
		    for data = (x-get-selection-internal type 'text/html)

[...]

      ;; Somehow the selection is UTF-16 when selecting text in
      ;; Firefox.
      (decode-coding-string data 'utf-16-le)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@IRO.UMontreal.CA,  31149@debbugs.gnu.org
> Date: Sun, 29 Sep 2019 12:02:42 +0200
> 
> (defun ewp-yank-html ()
> 
> [...]
> 
>   (let ((data (loop for type in '(PRIMARY CLIPBOARD)
> 		    for data = (x-get-selection-internal type 'text/html)
> 
> [...]
> 
>       ;; Somehow the selection is UTF-16 when selecting text in
>       ;; Firefox.
>       (decode-coding-string data 'utf-16-le)

And you do that for _every_ text you get from the clipboard?  Or do
you have some means of detecting those selected in Firefox?

In general, if the text is already decoded, I don't expect this extra
decoding to do anything bad, does it?

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>>       ;; Somehow the selection is UTF-16 when selecting text in
>>       ;; Firefox.
>>       (decode-coding-string data 'utf-16-le)
>
> And you do that for _every_ text you get from the clipboard?  Or do
> you have some means of detecting those selected in Firefox?

In this package, I'm assuming all the text comes from Firefox because
that's what I use.  :-)

> In general, if the text is already decoded, I don't expect this extra
> decoding to do anything bad, does it?

That's true, so perhaps this isn't a problem:

(decode-coding-string "fóo" 'utf-16-le)
=> "fóo"

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> (gui-get-selection nil 'text/html)
>
> returns utf-16 text when the primary selection is owned by Mozilla, but
> we decode it as latin-1 instead, so it looks like garbage.

This should now be fixed on the trunk, and hopefully I didn't regress
anything, but I have not regressed anything.  (I've only tested on
Debian and Macos.)  Let me know whether I broke something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This should now be fixed on the trunk, and hopefully I didn't regress
> anything, but I have not regressed anything. 

Urr.  I think it's getting a bit late in the day.  Substitute with
sentences that makes sense instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>> (gui-get-selection nil 'text/html)
>>
>> returns utf-16 text when the primary selection is owned by Mozilla, but
>> we decode it as latin-1 instead, so it looks like garbage.
>
> This should now be fixed on the trunk, and hopefully I didn't regress
> anything, but I have not regressed anything.  (I've only tested on
> Debian and Macos.)  Let me know whether I broke something.

It broke selection on Windows, so I've reverted and reopened this bug
report.

From the discussion on emacs-devel:

> > > Does reverting 5e66c75e0 fix the issue?
> > 
> > I've reverted it now and will have to reexamine the problem before
> > attempting a new fix.
> 
> Whatever you do, don't decode the selection text on MS-Windows.  It is
> already decoded (see w32-get-clipboard-data), and
> selection-coding-system is UTF-16 on MS-Windows, so decoding a decoded
> string by that will not do anything useful ;-)
> 
> The existing code carefully side-steps the decoding by looking at the
> foreign-selection property on the string, which the Windows code
> doesn't set.  But your changes removed that test, and thus caused the
> clipboard text to be decoded on Windows.

So I'll be re-exploring this issue once I get my Windows VMs back up and
can do some testing on Windows, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31149: 27.0.50; (gui-get-selection nil 'text/html) returns mis-decoded text

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It broke selection on Windows, so I've reverted and reopened this bug
> report.

I've now reapplied a tweaked version of the patch, and checked that it
doesn't break selection on Windows.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no