From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: "Stephen J. Turnbull" Newsgroups: gmane.emacs.devel Subject: Re: Emacs 23 character code space Date: Thu, 27 Nov 2008 10:10:35 +0900 Message-ID: <8763maq8hg.fsf@uwakimon.sk.tsukuba.ac.jp> References: NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1227748063 21913 80.91.229.12 (27 Nov 2008 01:07:43 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 27 Nov 2008 01:07:43 +0000 (UTC) Cc: emacs-devel@gnu.org, Kenichi Handa To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Nov 27 02:08:44 2008 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1L5VN6-0007cp-0L for ged-emacs-devel@m.gmane.org; Thu, 27 Nov 2008 02:08:44 +0100 Original-Received: from localhost ([127.0.0.1]:43047 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1L5VLw-0005nS-9n for ged-emacs-devel@m.gmane.org; Wed, 26 Nov 2008 20:07:32 -0500 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1L5VLp-0005jJ-M5 for emacs-devel@gnu.org; Wed, 26 Nov 2008 20:07:25 -0500 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1L5VLo-0005iB-O1 for emacs-devel@gnu.org; Wed, 26 Nov 2008 20:07:25 -0500 Original-Received: from [199.232.76.173] (port=53107 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1L5VLo-0005i4-K9 for emacs-devel@gnu.org; Wed, 26 Nov 2008 20:07:24 -0500 Original-Received: from mtps01.sk.tsukuba.ac.jp ([130.158.97.223]:50678) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1L5VLj-0008NJ-Tk; Wed, 26 Nov 2008 20:07:20 -0500 Original-Received: from uwakimon.sk.tsukuba.ac.jp (uwakimon.sk.tsukuba.ac.jp [130.158.99.156]) by mtps01.sk.tsukuba.ac.jp (Postfix) with ESMTP id 572221537B2; Thu, 27 Nov 2008 10:07:16 +0900 (JST) Original-Received: by uwakimon.sk.tsukuba.ac.jp (Postfix, from userid 1000) id D0EB61A2D03; Thu, 27 Nov 2008 10:10:35 +0900 (JST) In-Reply-To: X-Mailer: VM 8.0.12-devo-585 under 21.5 (beta28) "fuki" 83e35df20028+ XEmacs Lucid (x86_64-unknown-linux) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:106216 Archived-At: Handa-san, konnichiwa. Chotto shitsurei shimasu. (Hi, Mr. Handa. I'll be a little rude here in hope of saving you some time. :-) Eli Zaretskii writes: > > From: Kenichi Handa > > CC: eliz@gnu.org, emacs-devel@gnu.org > > Date: Wed, 26 Nov 2008 13:58:26 +0900 > > > > I'll explain it a little bit more. To decode a character > > sequence to a byte sequence, Emacs actually does two kinds > > of decoding as below: > > > > (1) (2) > > characters <-----> (charset code-point) pairs <-----> bytes > > Can you give a couple of examples, for some popular charsets, and how > we decode bytes into characters thru these pairs of charsets and code > points? (As you point out later, this is normally denoted 'encoding' in Mule functions like `encode-coding-region'. Encoding is actually the harder case to explain because the sequence of bytes produced for a given character is context-dependent, so let's look at the bytes --> characters conversion.) Let's begin at the beginning, with the number 1 represented by the Japanese ideograph ichi in the iso-2022-jp coding system, in an HTML header element alone on a line. As octets viewed as ASCII[1] it looks like this in a file < H 1 > ESC $ B 0 l ESC ( B < / H 1 > ^J where spaces are represented by SPC (ie, there are no significant spaces on the line). (0) We decide to decode using the ISO-2022-JP, which allows multiple charsets (at least ASCII, Japanese JIS X 0208 and JIS X 0212, and explicitly extended to other registered charsets as ISO-2022-INT). (1) We initialize current charset to ascii as mandated by the RFC defining ISO-2022-JP. (2) We collect the octet < and translate it to a pair (which is trivial -- apply the identity to the octet, and pair it with 'ascii) giving `(ascii <)' (3) We convert to a Mule character (which is an integer) with "(csetidx(ascii) << 7) + ?<". (4) We repeat (2) and (3) for H, 1, and >. (5) We see the control sequence ESC $ B, and switch our current charset to japanese-jisx0208. (6) JIS is a dimension 2 charset, so we collect two octets 0 and l, and form the pair (japanese-jisx0208 (0 l)). (7) We convert to a Mule character (which is an integer) with "(csetidx(japanese-jisx0208) << 14) + ?0 << 7 + ?l". (8) We see the control sequence ESC ( B, and switch our current charset to ascii. (9) We repeat (2) and (3) for <, /, H, 1, >, and ^J. (10) We are now done with the line, and we are ready to read more ASCII *or an escape sequence*. So the full process (1) -- (10) corresponds to the coding system ISO-2022-JP, while the process (2) -- (3) corresponds to the charset ascii, and the process (6) -- (7) to the charset japanese-jisx0208. > Thanks. What confuses me is that, roughly, there's a charset in Emacs > 23 for every coding-system, and they both have almost identical names. No, not even close. It happens that people who use ISO 8859 coded character sets generally only see such coding systems (ie, unibyte uniscript), so you would get that impression, as the coding systems for those encodings are trivial. However, for Unicode (as Juanma points out) the charset is always Unicode (I don't know how that is spelled in Emacs, and XEmacs technically doesn't have such a charset currently), but various coding systems such as UTF-8, UTF-16, and UTF-32, in big- and little-endian are used to decode bytes to code points of Unicode. For the East Asian coding systems and legacy ISO 2022 multiscript texts (ISO 2022 can handle not only Japanese, but also mixtures of, say, the Russian ISO-8859-5 and Hebrew ISO-8859-8 repertoires), it's even more complicated. Note that because of the need to handle ASCII in programming, all of the Asian coding systems must be multiscript, handling at least ASCII and the national character set(s). > For example, the code point of a-umlaut in the iso-8859-1 charset is > exactly identical to the byte value produced by encoding that > character with iso-8859-1 coding-system. So I wonder why we need > both in Emacs. Why can't we, for example, decode bytes directly into > Emacs characters? Because each byte is overloaded. In the process above with the escape sequences, we decode to a Mule string "

[ichi]

\n", while without the escape sequences we would get "

0l

\n", also known as "mojibake" (with a little bit of poetic license, the Japanese for "monster characters"). Footnotes: [1] This kind of ASCII-compatibility is deliberate in ISO 2022.