From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: pjb@informatimago.com (Pascal J. Bourguignon) Newsgroups: gmane.emacs.help Subject: Re: How to determine encoding for file? Date: Sun, 24 Jan 2010 22:59:46 +0100 Organization: Informatimago Message-ID: <87y6jn2mgd.fsf@hubble.lan.informatimago.com> References: NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: ger.gmane.org 1264372853 18629 80.91.229.12 (24 Jan 2010 22:40:53 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 24 Jan 2010 22:40:53 +0000 (UTC) To: help-gnu-emacs@gnu.org Original-X-From: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane.org@gnu.org Sun Jan 24 23:40:45 2010 Return-path: Envelope-to: geh-help-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1NZB8O-00027Z-Ox for geh-help-gnu-emacs@m.gmane.org; Sun, 24 Jan 2010 23:40:45 +0100 Original-Received: from localhost ([127.0.0.1]:59593 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NZB8P-0005FJ-HI for geh-help-gnu-emacs@m.gmane.org; Sun, 24 Jan 2010 17:40:45 -0500 Original-Path: news.stanford.edu!usenet.stanford.edu!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail Original-Newsgroups: gnu.emacs.help Original-Lines: 53 Original-X-Trace: individual.net JEGVEx4qigCXaSe8TNNObQVrwX5rYMgTQsNhLa2COMhQJKAK5k Cancel-Lock: sha1:MWVlN2E4NTJlNjFiMTM2MDQwMTIxOWRmYWFkNDcxOWFhODFkZWY5Mw== sha1:hP+bnNrauOIKjwYTsSAgxWq+VEg= Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwAQMAAABtzGvEAAAABlBMVEUAAAD///+l2Z/dAAAA oElEQVR4nK3OsRHCMAwF0O8YQufUNIQRGIAja9CxSA55AxZgFO4coMgYrEDDQZWPIlNAjwq9 033pbOBPtbXuB6PKNBn5gZkhGa86Z4x2wE67O+06WxGD/HCOGR0deY3f9Ijwwt7rNGNf6Oac l/GuZTF1wFGKiYYHKSFAkjIo1b6sCYS1sVmFhhhahKQssRjRT90ITWUk6vvK3RsPGs+M1RuR mV+hO/VvFAAAAABJRU5ErkJggg== X-Accept-Language: fr, es, en X-Disabled: X-No-Archive: no User-Agent: Gnus/5.101 (Gnus v5.10.10) Emacs/22.3 (gnu/linux) Original-Xref: news.stanford.edu gnu.emacs.help:176358 X-BeenThere: help-gnu-emacs@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Users list for the GNU Emacs text editor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane.org@gnu.org Errors-To: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.help:71428 Archived-At: kj writes: > I've downloaded a large file that is supposed to contain a mixture > of Japanese and English (it's basically a learner's dictionary). > The English is displayed correctly, but not so for the Japanese. > > I've tried setting the buffer's coding system to utf-8, > japanese-shift-jis, japanese-shift-jis-mac, japanese-shift-jis-dos > (just guessing). None worked. > > In fact, I'm not even sure that any of these changes of the coding > system achieved *anything*, since the buffer's appearance remained > unchanged throughout all this mucking around. I used the command > set-buffer-file-coding-system to do this. Should I need to do > anything besides re-setting the coding system to see a change in > how the file is displayed? > > More importantly, is there a better way to determine a file's > correct coding system besides trial and error? No, there is no better way. >From the sequences of bytes you can find inside the files, you may eliminate some encodings (eg. some byte sequences are invalid in UTF-8), but there are several encodings that use all the byte values and where any sequence of byte is valid, so you cannot choose between these encoding without knowing the meaning of the file. For example, the following file: $ od -t x1 -c /tmp/a.txt 0000000 50 72 69 63 65 20 6f 66 20 74 68 65 20 69 74 65 P r i c e o f t h e i t e 0000020 6d 3a 20 31 34 20 a4 0a m : 1 4 244 \n 0000030 The encoding cannot be determined without knowing what item is referenced, and what price ranges are probable for this item. It could contain: Price of the item: 14 € or: Price of the item: 14 ¤ (It could also contain another character, but it would be less probable). -- __Pascal Bourguignon__ http://www.informatimago.com/