bug#20010: 24.4; Issue with Unicode characters in package description

bug#20010: 24.4; Issue with Unicode characters in package description

[João Guerra]

[-- Attachment #1: Type: text/plain, Size: 4379 bytes --]

Packages with Unicode characters in their description mess with the
refreshing of the package archives. A popup buffer will always be shown
in such situations, asking for the proper coding system. This buffer
should be Unicode by default to avoid inconvenient situations like these.

For example, just by starting Emacs as normally but using the following
.emacs.d/init.el:

(require 'package)
(setq package-archives '(("melpa" . "http://melpa.org/packages/")
                         ("gnu" . "http://elpa.gnu.org/packages/")))
(package-refresh-contents)

A popup buffer will always be shown, stating:

These default coding systems were tried to encode text
in the buffer ` *temp*':
  (iso-latin-1-dos (3420 . 37326) (3421 . 40165))
However, each of them encountered characters it couldn't encode:
  iso-latin-1-dos cannot encode these: 野 鳥

Click on a character (or switch to this window by `C-x o'
and select the characters by RET) to jump to the place it appears,
where `C-u C-x =' will give information about it.

Select one of the safe coding systems listed below,
or cancel the writing with C-g and edit the buffer
   to remove or modify the problematic characters,
or specify any other coding system (and risk losing
   the problematic characters).

  utf-8 euc-jis-2004 euc-jp iso-2022-jp-2004 gb18030 utf-7 utf-16
  utf-16be-with-signature utf-16le-with-signature utf-16be utf-16le
  iso-2022-7bit utf-8-auto utf-8-with-signature eucjp-ms utf-8-hfs
  japanese-shift-jis-2004 japanese-iso-7bit-1978-irv utf-7-imap
  utf-8-emacs prefer-utf-8

Another popup windows pointing to the cause will also be shown:

(yatex . [(20150118 1600) nil "Yet Another tex-mode for emacs //野鳥//" tar
nil])


In GNU Emacs 24.4.1 (i686-pc-mingw32)
 of 2014-10-24 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr'

Important settings:
  value of $LANG: PTG
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <escape> <escape> <escape> <escape> <escape>
<escape> <escape> <escape> <escape> <escape> <escape>
C-x k C-x k <return> C-x k <return> C-x k <return>
C-x k <return> C-x k <return> M-x r e p o r t - e m
a c s - b u g <return>

Recent messages:

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail help-mode mm-archive message
format-spec rfc822 mml easymenu mml-sec mailabbrev gmm-utils mailheader
mm-decode mm-bodies mm-encode mail-utils network-stream starttls
url-http tls mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw
url-cache url-auth url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source eieio byte-opt bytecomp byte-compile cconv
eieio-core gnus-util mm-util help-fns mail-prsvr password-cache url-vars
epg package epg-config time-date tooltip electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns
disp-table w32-win w32-vars tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process w32notify w32 multi-tty emacs)

Memory information:
((conses 8 92381 16943)
 (symbols 32 22657 0)
 (miscs 32 40 202)
 (strings 16 21255 14970)
 (string-bytes 1 521369)
 (vectors 8 18291)
 (vector-slots 4 1126131 63824)
 (floats 8 71 297)
 (intervals 28 532 1923)
 (buffers 508 14))

[-- Attachment #2: Type: text/html, Size: 5865 bytes --]

bug#20010: 24.4; Issue with Unicode characters in package description

[Thierry Volpiatto]

João Guerra <joca.bt@gmail.com> writes:

> Packages with Unicode characters in their description mess with the
> refreshing of the package archives. A popup buffer will always be shown
> in such situations, asking for the proper coding system. This buffer
> should be Unicode by default to avoid inconvenient situations like these.
>
> For example, just by starting Emacs as normally but using the following
> .emacs.d/init.el:
>
> (require 'package)
> (setq package-archives '(("melpa" . "http://melpa.org/packages/")
>                          ("gnu" . "http://elpa.gnu.org/packages/")))
> (package-refresh-contents)
>
> A popup buffer will always be shown, stating:
>
> These default coding systems were tried to encode text
> in the buffer ` *temp*':
>   (iso-latin-1-dos (3420 . 37326) (3421 . 40165))
> However, each of them encountered characters it couldn't encode:
>   iso-latin-1-dos cannot encode these: 野 鳥

Easy to reproduce on Windows with just M-x list-packages.
let-binding `coding-system-for-write' to "utf-8" Fix the problem.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#20010: 24.4; Issue with Unicode characters in package description

[Stefan Monnier]

> (package-refresh-contents)
> A popup buffer will always be shown, stating:
> These default coding systems were tried to encode text
> in the buffer ` *temp*':

Indeed, we have a problem in package.el because we use the same routine
in two cases: one where we just want to transfer bytes (typically, to
download an package or the `archive-contents' file), and the other
where we want to see actual characters (when we fetch the *-readme.txt).

I installed the following patch which should at lease fix the immediate
problem you're seeing, tho there may still be corner cases where the bug
will appear.


        Stefan


diff --git a/lisp/url/url-handlers.el b/lisp/url/url-handlers.el
index 23006e1..ff3eafd 100644
--- a/lisp/url/url-handlers.el
+++ b/lisp/url/url-handlers.el
@@ -334,8 +334,13 @@ They count bytes from the beginning of the body."
         (unless (cadr size-and-charset)
           ;; If the headers don't specify any particular charset, use the
           ;; usual heuristic/rules that we apply to files.
-          (decode-coding-inserted-region start (point) url visit beg end replace))
-        (list url (car size-and-charset))))))
+          (decode-coding-inserted-region start (point) url
+                                         visit beg end replace))
+        (let ((inserted (car size-and-charset)))
+          (when (fboundp 'after-insert-file-set-coding)
+            (let ((insval (after-insert-file-set-coding inserted visit)))
+              (if insval (setq inserted insval))))
+          (list url inserted))))))
 
 (put 'insert-file-contents 'url-file-handlers 'url-insert-file-contents)