bug#19458: codepage

bug#19458: codepage

[igor denisov]

From:       @GOODY.i-did-not-set--mail-host-address--so-tickle-me
To: bug-gnu-emacs@gnu.org
Subject: 24.3; codepage
--text follows this line--
This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgment at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':
=====================================================
The following code decode cyrillic jast as it spposed to
(defun p (n)
  (print n))

(defvar n 'привет)

(p n)

When I issue
(netstat) only following numbers displayed

\200\252\342\250\242\255\353\245 and so on instead of Cyrillic letters


  TCP    0.0.0.0:135            goody:0                LISTENING
  TCP    0.0.0.0:445            goody:0                LISTENING
  TCP    0.0.0.0:554            goody:0                LISTENING
  TCP    0.0.0.0:623            goody:0                LISTENING
  TCP    0.0.0.0:2869           goody:0                LISTENING
  TCP    0.0.0.0:6646           goody:0                LISTENING

I checked the codepage

(w32-get-console-codepage)
866

(w32-get-codepage-charset 866)
nil ----- the codepage is not valid

On the bar where filename and mode is displayed there is such a thing as U\**
U was pressed with the following output

U -- utf-8-dos (alias: mule-utf-8-dos)

UTF-8 (no signature (BOM))
Type: utf-8 (UTF-8: Emacs internal multibyte form)
EOL type: CRLF
This coding system encodes the following charsets:

Character set: unicode

Unicode (ISO10646)

Number of contained characters: 256x256x17
Aliases: ucs
ASCII compatible.
Code space: [0 255 0 255 0 16 0 0]
Code offset: 0

=========================================================

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
d:/ProgramFiles/emacs-24.3/etc/DEBUG.


In GNU Emacs 24.3.1 (i386-mingw-nt6.2.9200)
 of 2013-03-17 on MARVIN
Windowing system distributor `Microsoft Corp.', version 6.2.9200
Configured using:
 `configure --with-gcc (4.7) --cflags
 -ID:/devel/emacs/libs/libXpm-3.5.8/include
 -ID:/devel/emacs/libs/libXpm-3.5.8/src
 -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
 -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
 -ID:/devel/emacs/libs/giflib-4.1.4-1/include
 -ID:/devel/emacs/libs/jpeg-6b-4/include
 -ID:/devel/emacs/libs/tiff-3.8.2-1/include
 -ID:/devel/emacs/libs/gnutls-3.0.9/include
 -ID:/devel/emacs/libs/libiconv-1.13.1-1-dev/include
 -ID:/devel/emacs/libs/libxml2-2.7.8/include/libxml2'

Important settings:
  value of $LANG: RUS
  locale-coding-system: cp1251
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <help-echo> <down-mouse-1> <mouse-1> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> M-x r e p o r t - e m a c s - b u g <r
eturn>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns disp-table w32-win
w32-vars tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32 multi-tty emacs)

bug#19458: codepage

[Eli Zaretskii]

> Date: Sun, 28 Dec 2014 13:04:21 +0400
> From: igor denisov <saufesma@gmail.com>
> 
> When I issue
> (netstat) only following numbers displayed
> 
> \200\252\342\250\242\255\353\245 and so on instead of Cyrillic letters

Can you please send (as an attachment) the output of the netstat
program if you invoke it from the cmd prompt on that system?

Also, does it help to do this:

 M-: (let ((coding-system-for-read cp866)) (netstat)) RET

?

> (w32-get-codepage-charset 866)
> nil ----- the codepage is not valid

I don't think this is related, Emacs doesn't call
w32-get-codepage-charset when you invoke 'netstat'.

bug#19458: codepage

[Eli Zaretskii]

> Date: Sun, 28 Dec 2014 20:01:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 19458@debbugs.gnu.org
> 
> > (w32-get-codepage-charset 866)
> > nil ----- the codepage is not valid
> 
> I don't think this is related, Emacs doesn't call
> w32-get-codepage-charset when you invoke 'netstat'.

Testing indicates that the API used by this function,
TranslateCharsetInfo, only works with ANSI codepages, while most
console codepages cause it to fail.  So I updated the doc string to
that effect.

Again, this has no bearing on your original problem with displaying
output from 'netstat'.

bug#19458: codepage

[Eli Zaretskii]

> Date: Sun, 28 Dec 2014 20:01:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 19458@debbugs.gnu.org
> 
> > Date: Sun, 28 Dec 2014 13:04:21 +0400
> > From: igor denisov <saufesma@gmail.com>
> > 
> > When I issue
> > (netstat) only following numbers displayed
> > 
> > \200\252\342\250\242\255\353\245 and so on instead of Cyrillic letters
> 
> Can you please send (as an attachment) the output of the netstat
> program if you invoke it from the cmd prompt on that system?
> 
> Also, does it help to do this:
> 
>  M-: (let ((coding-system-for-read cp866)) (netstat)) RET
> 
> ?

And how is (print n) related to the problems with 'netstat'?  Could
you please provide a full recipe, starting with "emacs -Q", that
reproduces the problem with displaying Cyrillic characters from
'netstat'?

bug#19458: codepage

[Eli Zaretskii]

> Date: Mon, 29 Dec 2014 12:02:21 +0400
> From: igor denisov <saufesma@gmail.com>
> 
> Hi Eli,

Please keep the bug address on the cc list.

> M-: (let ((coding-system-for-read cp866)) (netstat)) RET

Sorry, my bad.  Should be

   M-: (let ((coding-system-for-read 'cp866)) (netstat)) RET

(the quote in 'cp866 was missing).

> emacs.exe -Q
> 
> (netstat)
> 
> \200\252\342\250\242\255\353\245 \257\256\244\252\253\356\347\245\255\250\357
> 
> \210\254\357   \213\256\252\240\253\354\255\353\251  \240\244\340\245\341
> \202\255\245\350\255\250\251  \240\244\340\245\341
> \221\256\341\342\256\357\255\250\245

Please invoke "netstat" from the Windows shell prompt, redirecting its
output to a file, and send that file as a binary attachment.

> > And how is (print n) related to the problems with 'netstat'?

I'd still like the answer to this question, please.

bug#19458: codepage

[Eli Zaretskii]

> Date: Mon, 29 Dec 2014 21:31:06 +0400
> From: igor denisov <saufesma@gmail.com>
> 
> This works perfectly
> 
> M-: (let ((coding-system-for-read 'cp866)) (netstat)) RET

OK, thanks.  Now I know how to fix the command.

bug#19458: codepage

[Eli Zaretskii]

> Date: Thu, 8 Jan 2015 03:01:18 +0400
> From: igor denisov <saufesma@gmail.com>
> 
> >Then I thought if my system wide default codepage is
> > Unicode the output from netstat must be readable as well. For example
> > nestat in cmd.exe is just fine, and when I open it in emacs oops the
> > same thing.
> 
> Today I found that Unicode is not supported with Russian language

Not supported by what software?

And why do you need Unicode in this case?

bug#19458: codepage

[Eli Zaretskii]

> Date: Mon, 29 Dec 2014 19:38:22 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 19458@debbugs.gnu.org
> 
> > Date: Mon, 29 Dec 2014 21:31:06 +0400
> > From: igor denisov <saufesma@gmail.com>
> > 
> > This works perfectly
> > 
> > M-: (let ((coding-system-for-read 'cp866)) (netstat)) RET
> 
> OK, thanks.  Now I know how to fix the command.

OK, I fixed it now (commit 1f179ea on the emacs-24 branch).  The patch
appears below, if you want to start using it right away.

commit 1f179ea1bac65d911c616dafbf7147f0dbb39afd
Author: Eli Zaretskii <eliz@gnu.org>
Date:   Fri Jan 9 11:50:21 2015 +0200

    Fix encoding of I/O in net-utils.el for MS-Windows.  (Bug#19458)
    
     lisp/net/net-utils.el (net-utils-run-program, net-utils-run-simple):
     On MS-Windows, bind coding-system-for-read to the console output
     codepage.

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 2a6c960..8481dd1 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,9 @@
+2015-01-09  Eli Zaretskii  <eliz@gnu.org>
+
+	* net/net-utils.el (net-utils-run-program, net-utils-run-simple):
+	On MS-Windows, bind coding-system-for-read to the console output
+	codepage.  (Bug#19458)
+
 2015-01-08  Eli Zaretskii  <eliz@gnu.org>
 
 	* simple.el (line-move-visual): When converting X pixel coordinate
diff --git a/lisp/net/net-utils.el b/lisp/net/net-utils.el
index 28aa431..ebcbc71 100644
--- a/lisp/net/net-utils.el
+++ b/lisp/net/net-utils.el
@@ -328,7 +328,15 @@ (defun net-utils-remove-ctrl-m-filter (process output-string)
 
 (defun net-utils-run-program (name header program args)
   "Run a network information program."
-  (let ((buf (get-buffer-create (concat "*" name "*"))))
+  (let ((buf (get-buffer-create (concat "*" name "*")))
+	(coding-system-for-read
+	 ;; MS-Windows versions of network utilities output text
+	 ;; encoded in the console (a.k.a. "OEM") codepage, which is
+	 ;; different from the default system (a.k.a. "ANSI")
+	 ;; codepage.
+	 (if (eq system-type 'windows-nt)
+	     (intern (format "cp%d" (w32-get-console-output-codepage)))
+	   coding-system-for-read)))
     (set-buffer buf)
     (erase-buffer)
     (insert header "\n")
@@ -352,7 +360,15 @@ (defun net-utils-run-simple (buffer program-name args &optional nodisplay)
       (when proc
         (set-process-filter proc nil)
         (delete-process proc)))
-    (let ((inhibit-read-only t))
+    (let ((inhibit-read-only t)
+	(coding-system-for-read
+	 ;; MS-Windows versions of network utilities output text
+	 ;; encoded in the console (a.k.a. "OEM") codepage, which is
+	 ;; different from the default system (a.k.a. "ANSI")
+	 ;; codepage.
+	 (if (eq system-type 'windows-nt)
+	     (intern (format "cp%d" (w32-get-console-output-codepage)))
+	   coding-system-for-read)))
       (erase-buffer))
     (net-utils-mode)
     (setq-local net-utils--revert-cmd