character encoding question

character encoding question

[Eric Abrahamsen]

I'm trying to get a better understanding of character encodings, as I
often have to deal with mis-encoded or mystery-encoded files. I've read
the Non-ASCII Characters section of the elisp manual, and have a fair
sense of what's going on, with a couple of remaining questions.

So the character 中 has a codepoint of #o47055 in octal notation.
Meanwhile:

(string-as-unibyte "中") --> \344\270\255

I understand that each of these three sections is a byte, also in octal.
What's the correspondence between these bytes and the multibyte
character's octal codepoint? Are there any functions that will get from
one to the other?

Second question: If emacs can't guess the encoding of a file, it gives
you an error message showing the bytes it can't decode, plus the
charsets it tried to use. How do I replicate that process manually?
Given a series of mystery bytes, can I test them against different
charsets, and see what gibberish emacs comes up with? I guess I'm
imagining something like "decode-char", except being able to feed it
bytes instead of a character...

Thanks!
Eric

Re: character encoding question

[Peter Dyballa]

Am 20.02.2013 um 07:34 schrieb Eric Abrahamsen:

> (string-as-unibyte "中") --> \344\270\255
> 
> I understand that each of these three sections is a byte, also in octal.
> What's the correspondence between these bytes and the multibyte
> character's octal codepoint? Are there any functions that will get from
> one to the other?

It's defined in Unicode by the Unicode consortium. The code points in Unicode can be represented by different systems: UTF-7, UTF-8, UTF-16 with least significant byte first or most significant byte first, UTF-32, maybe more. Wikipedia certainly is a good start.

In the example above some Unicode character (#o47055) is represented by a sequence of three bytes. Since the bytes are numerically all greater than 127 it must be saved in UTF-8 encoding. It's U+4E2D, some CJK Ideograph. 

> Second question: If emacs can't guess the encoding of a file, it gives
> you an error message showing the bytes it can't decode, plus the
> charsets it tried to use. How do I replicate that process manually?

C-x RET r – revert-buffer-with-coding-system. The function gives you the choice to select an encoding.

--
Greetings

  Pete

"Debugging? Klingons do not debug! Our software does not coddle the weak."

Re: character encoding question

[Stefan Monnier]

> So the character 中 has a codepoint of #o47055 in octal notation.

Internally, in your Emacs, yes.  This value actually depends on the
internal representation chosen by Emacs, which happens to be Unicode
since Emacs-23 (and was something else before).

> Meanwhile:
> (string-as-unibyte "中") --> \344\270\255

This again shows the internal byte representation of this char inside
a buffer, which is utf-8 since Emacs-23 and was something else before.
Strong recommendation: stay far away from string-as-* because that will
mess you up.
You want instead to use encode-coding-string.  E.g.

   (encode-coding-string "中" 'utf-8)  ==>  "\344\270\255"

> What's the correspondence between these bytes and the multibyte
> character's octal codepoint?

#o47055 is not "multibyte".  It's just its "name" aka "codepoint".
"\344\270\255" is one if its multibyte encodings.

> Are there any functions that will get from one to the other?

   (encode-coding-string (string #o47055) 'utf-8)  ==>  "\344\270\255"

> Given a series of mystery bytes, can I test them against different
> charsets, and see what gibberish Emacs comes up with?

   (decode-coding-string "\344\270\255" 'utf-8)  ==>   "中"


-- Stefan

Re: character encoding question

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 20 Feb 2013 09:42:49 -0500
> 
> > Given a series of mystery bytes, can I test them against different
> > charsets, and see what gibberish Emacs comes up with?
> 
>    (decode-coding-string "\344\270\255" 'utf-8)  ==>   "中"

I'd actually suggest decode-coding-region, because it doesn't require
copying the "mystery bytes" into a string, something that might change
the bytes.

Re: character encoding question

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Wed, 20 Feb 2013 09:42:49 -0500
>> 
>> > Given a series of mystery bytes, can I test them against different
>> > charsets, and see what gibberish Emacs comes up with?
>> 
>>    (decode-coding-string "\344\270\255" 'utf-8)  ==>   "中"
>
> I'd actually suggest decode-coding-region, because it doesn't require
> copying the "mystery bytes" into a string, something that might change
> the bytes.

Perfect, just what I was looking for. I was getting tired of repeated
revert-buffer-with-etc calls! The explanation about the octal notation
cleared things up nicely, as well.

Many thanks,
Eric