* character encoding question
@ 2013-02-20 6:34 Eric Abrahamsen
2013-02-20 10:44 ` Peter Dyballa
2013-02-20 14:42 ` Stefan Monnier
0 siblings, 2 replies; 5+ messages in thread
From: Eric Abrahamsen @ 2013-02-20 6:34 UTC (permalink / raw)
To: help-gnu-emacs
I'm trying to get a better understanding of character encodings, as I
often have to deal with mis-encoded or mystery-encoded files. I've read
the Non-ASCII Characters section of the elisp manual, and have a fair
sense of what's going on, with a couple of remaining questions.
So the character 中 has a codepoint of #o47055 in octal notation.
Meanwhile:
(string-as-unibyte "中") --> \344\270\255
I understand that each of these three sections is a byte, also in octal.
What's the correspondence between these bytes and the multibyte
character's octal codepoint? Are there any functions that will get from
one to the other?
Second question: If emacs can't guess the encoding of a file, it gives
you an error message showing the bytes it can't decode, plus the
charsets it tried to use. How do I replicate that process manually?
Given a series of mystery bytes, can I test them against different
charsets, and see what gibberish emacs comes up with? I guess I'm
imagining something like "decode-char", except being able to feed it
bytes instead of a character...
Thanks!
Eric
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: character encoding question
2013-02-20 6:34 character encoding question Eric Abrahamsen
@ 2013-02-20 10:44 ` Peter Dyballa
2013-02-20 14:42 ` Stefan Monnier
1 sibling, 0 replies; 5+ messages in thread
From: Peter Dyballa @ 2013-02-20 10:44 UTC (permalink / raw)
To: Eric Abrahamsen; +Cc: help-gnu-emacs
Am 20.02.2013 um 07:34 schrieb Eric Abrahamsen:
> (string-as-unibyte "中") --> \344\270\255
>
> I understand that each of these three sections is a byte, also in octal.
> What's the correspondence between these bytes and the multibyte
> character's octal codepoint? Are there any functions that will get from
> one to the other?
It's defined in Unicode by the Unicode consortium. The code points in Unicode can be represented by different systems: UTF-7, UTF-8, UTF-16 with least significant byte first or most significant byte first, UTF-32, maybe more. Wikipedia certainly is a good start.
In the example above some Unicode character (#o47055) is represented by a sequence of three bytes. Since the bytes are numerically all greater than 127 it must be saved in UTF-8 encoding. It's U+4E2D, some CJK Ideograph.
> Second question: If emacs can't guess the encoding of a file, it gives
> you an error message showing the bytes it can't decode, plus the
> charsets it tried to use. How do I replicate that process manually?
C-x RET r – revert-buffer-with-coding-system. The function gives you the choice to select an encoding.
--
Greetings
Pete
"Debugging? Klingons do not debug! Our software does not coddle the weak."
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: character encoding question
2013-02-20 6:34 character encoding question Eric Abrahamsen
2013-02-20 10:44 ` Peter Dyballa
@ 2013-02-20 14:42 ` Stefan Monnier
2013-02-20 18:48 ` Eli Zaretskii
1 sibling, 1 reply; 5+ messages in thread
From: Stefan Monnier @ 2013-02-20 14:42 UTC (permalink / raw)
To: help-gnu-emacs
> So the character 中 has a codepoint of #o47055 in octal notation.
Internally, in your Emacs, yes. This value actually depends on the
internal representation chosen by Emacs, which happens to be Unicode
since Emacs-23 (and was something else before).
> Meanwhile:
> (string-as-unibyte "中") --> \344\270\255
This again shows the internal byte representation of this char inside
a buffer, which is utf-8 since Emacs-23 and was something else before.
Strong recommendation: stay far away from string-as-* because that will
mess you up.
You want instead to use encode-coding-string. E.g.
(encode-coding-string "中" 'utf-8) ==> "\344\270\255"
> What's the correspondence between these bytes and the multibyte
> character's octal codepoint?
#o47055 is not "multibyte". It's just its "name" aka "codepoint".
"\344\270\255" is one if its multibyte encodings.
> Are there any functions that will get from one to the other?
(encode-coding-string (string #o47055) 'utf-8) ==> "\344\270\255"
> Given a series of mystery bytes, can I test them against different
> charsets, and see what gibberish Emacs comes up with?
(decode-coding-string "\344\270\255" 'utf-8) ==> "中"
-- Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: character encoding question
2013-02-20 14:42 ` Stefan Monnier
@ 2013-02-20 18:48 ` Eli Zaretskii
2013-02-21 2:42 ` Eric Abrahamsen
0 siblings, 1 reply; 5+ messages in thread
From: Eli Zaretskii @ 2013-02-20 18:48 UTC (permalink / raw)
To: help-gnu-emacs
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 20 Feb 2013 09:42:49 -0500
>
> > Given a series of mystery bytes, can I test them against different
> > charsets, and see what gibberish Emacs comes up with?
>
> (decode-coding-string "\344\270\255" 'utf-8) ==> "中"
I'd actually suggest decode-coding-region, because it doesn't require
copying the "mystery bytes" into a string, something that might change
the bytes.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: character encoding question
2013-02-20 18:48 ` Eli Zaretskii
@ 2013-02-21 2:42 ` Eric Abrahamsen
0 siblings, 0 replies; 5+ messages in thread
From: Eric Abrahamsen @ 2013-02-21 2:42 UTC (permalink / raw)
To: help-gnu-emacs
Eli Zaretskii <eliz@gnu.org> writes:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Wed, 20 Feb 2013 09:42:49 -0500
>>
>> > Given a series of mystery bytes, can I test them against different
>> > charsets, and see what gibberish Emacs comes up with?
>>
>> (decode-coding-string "\344\270\255" 'utf-8) ==> "中"
>
> I'd actually suggest decode-coding-region, because it doesn't require
> copying the "mystery bytes" into a string, something that might change
> the bytes.
Perfect, just what I was looking for. I was getting tired of repeated
revert-buffer-with-etc calls! The explanation about the octal notation
cleared things up nicely, as well.
Many thanks,
Eric
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-02-21 2:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-02-20 6:34 character encoding question Eric Abrahamsen
2013-02-20 10:44 ` Peter Dyballa
2013-02-20 14:42 ` Stefan Monnier
2013-02-20 18:48 ` Eli Zaretskii
2013-02-21 2:42 ` Eric Abrahamsen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).