Face problems when yanking cyrillic text from other X applications

Face problems when yanking cyrillic text from other X applications

[Andrey]

When yanking text with cyrillic characters from other X applications
(for example Iceweasel aka Firefox, GEdit), the cyrillics are inserted
into buffer with incorrect face.

I have default face set to Terminus font, but those cyrillics inserted
from X clipboard are displayed with some definitely other font (the
glyphs are much wider; the font is probably Courier). The *latin*
characters are displayed just nicely. If I save and then revert
buffer, the face becomes correct.

My .emacs:

;; --- Customized ---
(custom-set-variables
  ;; custom-set-variables was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 '(TeX-PDF-mode t)
 '(TeX-output-view-style (quote (("^dvi$" ("^landscape$" "^pstricks$\\|^pst-\\|^psfrag$") "%(o?)dvips -t landscape %d -o && gv %f") ("^dvi$" "^pstricks$\\|^pst-\\|^psfrag$" "%(o?)dvips %d -o && gv %f") ("^dvi$" ("^a4\\(?:dutch\\|paper\\|wide\\)\\|sem-a4$" "^landscape$") "%(o?)xdvi %dS -paper a4r -s 0 %d") ("^dvi$" "^a4\\(?:dutch\\|paper\\|wide\\)\\|sem-a4$" "%(o?)xdvi %dS -paper a4 %d") ("^dvi$" ("^a5\\(?:comb\\|paper\\)$" "^landscape$") "%(o?)xdvi %dS -paper a5r -s 0 %d") ("^dvi$" "^a5\\(?:comb\\|paper\\)$" "%(o?)xdvi %dS -paper a5 %d") ("^dvi$" "^b5paper$" "%(o?)xdvi %dS -paper b5 %d") ("^dvi$" "^letterpaper$" "%(o?)xdvi %dS -paper us %d") ("^dvi$" "^legalpaper$" "%(o?)xdvi %dS -paper legal %d") ("^dvi$" "^executivepaper$" "%(o?)xdvi %dS -paper 7.25x10.5in %d") ("^dvi$" "." "%(o?)xdvi %d
 S %d") ("^pdf$" "." "evince %o") ("^html?$" "." "netscape %o"))))
 '(blink-cursor-mode nil)
 '(case-fold-search t)
 '(column-number-mode t)
 '(current-language-environment "UTF-8")
 '(debian-bug-From-address "Andrey <cmr.Pent@gmail.com>")
 '(debian-bug-use-From-address t)
 '(ediff-window-setup-function (quote ediff-setup-windows-plain))
 '(gnus-select-method (quote (nntp "news.gmane.org")))
 '(kill-whole-line t)
 '(mouse-wheel-mode t)
 '(pc-selection-mode t nil (pc-select))
 '(text-mode-hook (quote (turn-on-auto-fill text-mode-hook-identify)))
 '(transient-mark-mode t)
 '(truncate-lines t)
 '(use-dialog-box nil)
 '(w3m-coding-system (quote utf-8))
 '(w3m-default-coding-system (quote utf-8))
 '(w3m-file-coding-system (quote utf-8))
 '(w3m-file-name-coding-system (quote utf-8))
 '(w3m-mule-unicode-symbol (quote ("┼" "├" "┬" "┌" "┤" "│" "┐" "" "┴" "└" "─" "" "┘" "" "" "" "┼" " " " " " " " " " " " " "" " " " " " " "" " " "" "" "" "•" " " " " "○" "■" " " " " " " " " " " "○" " " " " "  ↑ ↓ ")))
 '(w3m-terminal-coding-system (quote utf-8))
 '(w3m-use-cookies t)
 '(x-select-enable-clipboard t))
(custom-set-faces
  ;; custom-set-faces was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 '(default ((t (:foreground "#4E4E4E"))))
 '(bold ((t (:foreground "black" :weight bold))))
 '(bold-italic ((t (:inherit (bold italic)))))
 '(custom-face-tag ((t (:inherit custom-variable-tag))))
 '(custom-group-tag ((((class color) (background light)) (:inherit custom-variable-tag :box (:line-width 1 :color "lightgray")))))
 '(custom-group-tag-face-1 ((t (:foreground "red" :weight bold :height 1.2))) t)
 '(custom-variable-tag-face ((t (:foreground "blue" :weight bold))) t)
 '(ediff-current-diff-A ((t (:background "lightgrey"))))
 '(ediff-current-diff-B ((((class color)) (:inherit ediff-current-diff-A))))
 '(ediff-current-diff-C ((((class color)) (:inherit ediff-current-diff-A))))
 '(ediff-even-diff-A ((((class color)) (:background "snow2"))))
 '(ediff-even-diff-B ((((class color)) (:inherit ediff-even-diff-A))))
 '(ediff-even-diff-C ((((class color)) (:inherit ediff-even-diff-A))))
 '(ediff-fine-diff-A ((t (:background "pale green" :foreground "firebrick"))))
 '(ediff-fine-diff-B ((((class color)) (:inherit ediff-fine-diff-A))))
 '(ediff-fine-diff-C ((((class color)) (:inherit ediff-fine-diff-A))))
 '(ediff-odd-diff-A ((((class color)) (:inherit ediff-even-diff-A))))
 '(ediff-odd-diff-B ((((class color)) (:inherit ediff-odd-diff-A))))
 '(ediff-odd-diff-C ((((class color)) (:inherit ediff-odd-diff-A))))
 '(fixed-pitch ((t nil)))
 '(italic ((t (:underline "lightgray"))))
 '(variable-pitch ((t nil))))
 
;; Scrolling
(setq scroll-conservatively 50)
(setq scroll-preserve-screen-position 't)
(setq scroll-margin 5)

;; --- Key bindings ---
(global-set-key [(control tab)] 'cyclebuffer-forward)
(global-set-key [(control shift iso-lefttab)] 'cyclebuffer-backward)

(global-set-key [(control x) (p)] 'svn-status)


In GNU Emacs 22.1.1 (i486-pc-linux-gnu, GTK+ Version 2.8.20)
 of 2007-07-07 on bauhaus, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.70101000
configured using `configure  '--build=i486-linux-gnu' '--host=i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs22:/etc/emacs:/usr/local/share/emacs/22.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/22.1/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/22.1/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ru_RU.UTF-8
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Re: Face problems when yanking cyrillic text from other X applications

[Eli Zaretskii]

> From: Andrey <pent@aparamon.msk.ru>
> Date: Sat, 03 Nov 2007 16:45:57 +0300
> 
> When yanking text with cyrillic characters from other X applications
> (for example Iceweasel aka Firefox, GEdit), the cyrillics are inserted
> into buffer with incorrect face.
> 
> I have default face set to Terminus font, but those cyrillics inserted
> from X clipboard are displayed with some definitely other font (the
> glyphs are much wider; the font is probably Courier).

Thank you for your report.

This might be a character encoding problem, not a face problem.  What
does Emacs display when type "C-u C-x =" with point positioned on one
of the cyrillic characters inserted from the clipboard?

Re: Face problems when yanking cyrillic text from other X applications

[cmr.Pent]

Yes, looks like it is an encoding problem:

  character: а (54225, #o151721, #xd3d1, U+0430)
    charset: japanese-jisx0208 (JISX0208.1983/1990 Japanese Kanji: ISO-
IR-87.)
 code point: #x27 #x51
     syntax: w 	which means: word
   category: Y:Cyrillic characters of 2-byte character sets j:Japanese
	     |:While filling, we can break a line at this character.
buffer code: #x92 #xA7 #xD1
  file code: #xD0 #xB0 (encoded by coding system mule-utf-8-unix)
    display: by this font (glyph code)
     -JIS-Fixed-Medium-R-Normal--16-150-75-75-C-160-JISX0208.1983-0
(#x2751)

There are text properties here:
  fontified            t

Andrey

Re: Face problems when yanking cyrillic text from other X applications

[Eli Zaretskii]

> From: "cmr.Pent@gmail.com" <cmr.Pent@gmail.com>
> Date: Mon, 05 Nov 2007 10:29:17 -0000
> 
> Yes, looks like it is an encoding problem:
> 
>   character: а (54225, #o151721, #xd3d1, U+0430)
>     charset: japanese-jisx0208 (JISX0208.1983/1990 Japanese Kanji: ISO-
> IR-87.)
>  code point: #x27 #x51
>      syntax: w 	which means: word
>    category: Y:Cyrillic characters of 2-byte character sets j:Japanese

That's what I guessed: these characters are double-width Cyrillic
characters used in Japanese scripts.

I think you need to find out why those characters came encoded like
that from whatever clipboard you were pasting them.

Re: Face problems when yanking cyrillic text from other X applications

[Jason Rumney]

cmr.Pent@gmail.com wrote:
> Yes, looks like it is an encoding problem:
>
>   character: а (54225, #o151721, #xd3d1, U+0430)
>     charset: japanese-jisx0208 (JISX0208.1983/1990 Japanese Kanji: ISO-
> IR-87.)
>  code point: #x27 #x51
>      syntax: w 	which means: word
>    category: Y:Cyrillic characters of 2-byte character sets j:Japanese
> 	     |:While filling, we can break a line at this character.
> buffer code: #x92 #xA7 #xD1
>   file code: #xD0 #xB0 (encoded by coding system mule-utf-8-unix)
>     display: by this font (glyph code)
>      -JIS-Fixed-Medium-R-Normal--16-150-75-75-C-160-JISX0208.1983-0
> (#x2751)
>
> There are text properties here:
>   fontified            t
>   

Have a look at the variable x-select-request-type. I think setting it to
'UTF8_STRING should give better results. I thought this was the default,
but reading the doc string, it appears that the default is to use some
sort of heuristic. I'm not sure why the heuristic prefers iso-2022-jp
encoded Cyrillic over UTF8 encoding in a ru_RU.UTF-8 locale.

Re: Face problems when yanking cyrillic text from other X applications

[Jason Rumney]

Jason Rumney wrote:
> but reading the doc string, it appears that the default is to use some
> sort of heuristic. I'm not sure why the heuristic prefers iso-2022-jp
> encoded Cyrillic over UTF8 encoding in a ru_RU.UTF-8 locale.
>   

The heuristics used are documented above the definition of
x-select-utf8-or-ctext.
I think we should change the second condition to match Cyrillic as well
as Latin characters.

    ;;   (2) Otherwise, if they are different at Nth character, and that
    ;;   of UTF8 is a Latin character and that of CTEXT belongs to a CJK
    ;;   character set, select UTF8.  Also select UTF8 if the Nth
    ;;   character of UTF8 is non-ASCII where as that of CTEXT is ASCII.
    ;;   This is because an X client may replace unsupported characters
    ;;   with some ASCII character (typically ` ' or `?') in CTEXT.

Re: Face problems when yanking cyrillic text from other X applications

[cmr.Pent]

On 6 нояб, 00:21, Jason Rumney <jas...@gnu.org> wrote:
> Have a look at the variable x-select-request-type. I think setting it to
> 'UTF8_STRING should give better results. I thought this was the default,
> but reading the doc string, it appears that the default is to use some
> sort of heuristic. I'm not sure why the heuristic prefers iso-2022-jp
> encoded Cyrillic over UTF8 encoding in a ru_RU.UTF-8 locale.

Thanks, adding

'(x-select-request-type 'UTF8_STRING)

to custom-set-variables section of my .emacs worked.

Andrey