From: "Stephen J. Turnbull" <stephen@xemacs.org>
To: Eli Zaretskii <eliz@gnu.org>
Cc: emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
Subject: Re: Emacs 23 character code space
Date: Thu, 27 Nov 2008 10:10:35 +0900 [thread overview]
Message-ID: <8763maq8hg.fsf@uwakimon.sk.tsukuba.ac.jp> (raw)
In-Reply-To: <uhc5ugrok.fsf@gnu.org>
Handa-san, konnichiwa. Chotto shitsurei shimasu. (Hi, Mr. Handa.
I'll be a little rude here in hope of saving you some time. :-)
Eli Zaretskii writes:
> > From: Kenichi Handa <handa@m17n.org>
> > CC: eliz@gnu.org, emacs-devel@gnu.org
> > Date: Wed, 26 Nov 2008 13:58:26 +0900
> >
> > I'll explain it a little bit more. To decode a character
> > sequence to a byte sequence, Emacs actually does two kinds
> > of decoding as below:
> >
> > (1) (2)
> > characters <-----> (charset code-point) pairs <-----> bytes
>
> Can you give a couple of examples, for some popular charsets, and how
> we decode bytes into characters thru these pairs of charsets and code
> points?
(As you point out later, this is normally denoted 'encoding' in Mule
functions like `encode-coding-region'. Encoding is actually the
harder case to explain because the sequence of bytes produced for a
given character is context-dependent, so let's look at the bytes -->
characters conversion.)
Let's begin at the beginning, with the number 1 represented by the
Japanese ideograph ichi in the iso-2022-jp coding system, in an HTML
header element alone on a line. As octets viewed as ASCII[1] it looks
like this in a file
< H 1 > ESC $ B 0 l ESC ( B < / H 1 > ^J
where spaces are represented by SPC (ie, there are no significant
spaces on the line).
(0) We decide to decode using the ISO-2022-JP, which allows multiple
charsets (at least ASCII, Japanese JIS X 0208 and JIS X 0212, and
explicitly extended to other registered charsets as ISO-2022-INT).
(1) We initialize current charset to ascii as mandated by the RFC
defining ISO-2022-JP.
(2) We collect the octet < and translate it to a pair (which is
trivial -- apply the identity to the octet, and pair it with
'ascii) giving `(ascii <)'
(3) We convert to a Mule character (which is an integer) with
"(csetidx(ascii) << 7) + ?<".
(4) We repeat (2) and (3) for H, 1, and >.
(5) We see the control sequence ESC $ B, and switch our current
charset to japanese-jisx0208.
(6) JIS is a dimension 2 charset, so we collect two octets 0 and l,
and form the pair (japanese-jisx0208 (0 l)).
(7) We convert to a Mule character (which is an integer) with
"(csetidx(japanese-jisx0208) << 14) + ?0 << 7 + ?l".
(8) We see the control sequence ESC ( B, and switch our current
charset to ascii.
(9) We repeat (2) and (3) for <, /, H, 1, >, and ^J.
(10) We are now done with the line, and we are ready to read more
ASCII *or an escape sequence*.
So the full process (1) -- (10) corresponds to the coding system
ISO-2022-JP, while the process (2) -- (3) corresponds to the charset
ascii, and the process (6) -- (7) to the charset japanese-jisx0208.
> Thanks. What confuses me is that, roughly, there's a charset in Emacs
> 23 for every coding-system, and they both have almost identical names.
No, not even close. It happens that people who use ISO 8859 coded
character sets generally only see such coding systems (ie, unibyte
uniscript), so you would get that impression, as the coding systems
for those encodings are trivial.
However, for Unicode (as Juanma points out) the charset is always
Unicode (I don't know how that is spelled in Emacs, and XEmacs
technically doesn't have such a charset currently), but various coding
systems such as UTF-8, UTF-16, and UTF-32, in big- and little-endian
are used to decode bytes to code points of Unicode. For the East
Asian coding systems and legacy ISO 2022 multiscript texts (ISO 2022
can handle not only Japanese, but also mixtures of, say, the Russian
ISO-8859-5 and Hebrew ISO-8859-8 repertoires), it's even more
complicated.
Note that because of the need to handle ASCII in programming, all of
the Asian coding systems must be multiscript, handling at least ASCII
and the national character set(s).
> For example, the code point of a-umlaut in the iso-8859-1 charset is
> exactly identical to the byte value produced by encoding that
> character with iso-8859-1 coding-system. So I wonder why we need
> both in Emacs. Why can't we, for example, decode bytes directly into
> Emacs characters?
Because each byte is overloaded. In the process above with the escape
sequences, we decode to a Mule string "<H1>[ichi]</H1>\n", while
without the escape sequences we would get "<H1>0l</H1>\n", also known
as "mojibake" (with a little bit of poetic license, the Japanese for
"monster characters").
Footnotes:
[1] This kind of ASCII-compatibility is deliberate in ISO 2022.
next prev parent reply other threads:[~2008-11-27 1:10 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-11-01 14:20 Emacs 23 character code space Eli Zaretskii
2008-11-01 16:46 ` Eli Zaretskii
2008-11-03 1:34 ` Kenichi Handa
2008-11-03 12:45 ` Kenichi Handa
2008-11-03 20:13 ` Eli Zaretskii
2008-11-04 7:35 ` Kenichi Handa
2008-11-04 20:19 ` Eli Zaretskii
2008-11-05 12:27 ` Kenichi Handa
2008-11-05 18:23 ` Eli Zaretskii
2008-11-22 18:25 ` Eli Zaretskii
2008-11-26 1:41 ` Kenichi Handa
2008-11-26 4:13 ` Eli Zaretskii
2008-11-26 4:24 ` Kenichi Handa
2008-11-26 4:58 ` Kenichi Handa
2008-11-26 20:26 ` Eli Zaretskii
2008-11-26 22:52 ` Juanma Barranquero
2008-11-27 1:10 ` Stephen J. Turnbull [this message]
2008-11-27 1:35 ` Kenichi Handa
2008-11-26 20:18 ` Eli Zaretskii
2008-11-27 1:29 ` Kenichi Handa
2008-11-29 17:12 ` Eli Zaretskii
2008-12-02 5:40 ` Kenichi Handa
2008-11-28 13:19 ` Eli Zaretskii
2008-12-02 5:44 ` Kenichi Handa
2008-12-02 19:40 ` Eli Zaretskii
2008-11-29 12:01 ` Eli Zaretskii
2008-11-22 16:28 ` Eli Zaretskii
2008-11-23 4:16 ` Stefan Monnier
2008-11-23 11:22 ` Eli Zaretskii
2008-11-26 1:51 ` Kenichi Handa
2008-11-23 8:29 ` Ulrich Mueller
2008-11-23 11:11 ` Eli Zaretskii
2008-11-23 11:55 ` Ulrich Mueller
2008-11-24 3:06 ` Stefan Monnier
2008-11-26 1:31 ` Kenichi Handa
2008-11-22 17:03 ` New function: what-file-line, used when writing gdb script richardeng
2008-11-07 7:21 ` Emacs 23 character code space Kenichi Handa
2008-11-07 10:27 ` Eli Zaretskii
2008-11-07 11:52 ` Kenichi Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8763maq8hg.fsf@uwakimon.sk.tsukuba.ac.jp \
--to=stephen@xemacs.org \
--cc=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
--cc=handa@m17n.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.