From mboxrd@z Thu Jan 1 00:00:00 1970 Path: main.gmane.org!not-for-mail From: Kenichi Handa Newsgroups: gmane.emacs.devel Subject: Re: unibyte<->multibyte conversion [Re: Emacs-diffs Digest, Vol 2, Issue 28] Date: Tue, 21 Jan 2003 17:04:37 +0900 (JST) Sender: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Message-ID: <200301210804.RAA03089@etlken.m17n.org> References: <3405-Sat18Jan2003154003+0200-eliz@is.elta.co.il> <200301200229.LAA16287@etlken.m17n.org> <6480-Mon20Jan2003214849+0200-eliz@is.elta.co.il> <200301210010.JAA17551@etlken.m17n.org> <200301210045.h0L0jS812745@rum.cs.yale.edu> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 (generated by SEMI 1.14.3 - "Ushinoya") Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Trace: main.gmane.org 1043136653 17465 80.91.224.249 (21 Jan 2003 08:10:53 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Tue, 21 Jan 2003 08:10:53 +0000 (UTC) Cc: emacs-devel@gnu.org Return-path: Original-Received: from quimby.gnus.org ([80.91.224.244]) by main.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 18atUh-0004XY-00 for ; Tue, 21 Jan 2003 09:10:51 +0100 Original-Received: from monty-python.gnu.org ([199.232.76.173]) by quimby.gnus.org with esmtp (Exim 3.12 #1 (Debian)) id 18atfx-0007r5-00 for ; Tue, 21 Jan 2003 09:22:30 +0100 Original-Received: from localhost ([127.0.0.1] helo=monty-python.gnu.org) by monty-python.gnu.org with esmtp (Exim 4.10.13) id 18atT8-0005TH-04 for emacs-devel@quimby.gnus.org; Tue, 21 Jan 2003 03:09:14 -0500 Original-Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.10.13) id 18atSJ-0005MZ-00 for emacs-devel@gnu.org; Tue, 21 Jan 2003 03:08:23 -0500 Original-Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.10.13) id 18atRO-0005Du-00 for emacs-devel@gnu.org; Tue, 21 Jan 2003 03:07:43 -0500 Original-Received: from tsukuba.m17n.org ([192.47.44.130]) by monty-python.gnu.org with esmtp (Exim 4.10.13) id 18atPY-0004sq-00; Tue, 21 Jan 2003 03:05:32 -0500 Original-Received: from fs.m17n.org (fs.m17n.org [192.47.44.2])h0L84ck07620; Tue, 21 Jan 2003 17:04:38 +0900 (JST) (envelope-from handa@m17n.org) Original-Received: from etlken.m17n.org (etlken.m17n.org [192.47.44.125]) h0L84cR09430; Tue, 21 Jan 2003 17:04:38 +0900 (JST) Original-Received: (from handa@localhost) by etlken.m17n.org (8.8.8+Sun/3.7W-2001040620) id RAA03089; Tue, 21 Jan 2003 17:04:37 +0900 (JST) Original-To: monnier+gnu/emacs@rum.cs.yale.edu In-reply-to: <200301210045.h0L0jS812745@rum.cs.yale.edu> (monnier+gnu/emacs@rum.cs.yale.edu) User-Agent: SEMI/1.14.3 (Ushinoya) FLIM/1.14.2 (Yagi-Nishiguchi) APEL/10.2 Emacs/21.2.92 (sparc-sun-solaris2.6) MULE/5.0 (SAKAKI) Original-cc: eliz@is.elta.co.il Original-cc: rms@gnu.org X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1b5 Precedence: list List-Id: Emacs development discussions. List-Help: List-Post: List-Subscribe: , List-Archive: List-Unsubscribe: , Errors-To: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Xref: main.gmane.org gmane.emacs.devel:10931 X-Report-Spam: http://spam.gmane.org/gmane.emacs.devel:10931 In article <200301210045.h0L0jS812745@rum.cs.yale.edu>, "Stefan Monnier" writes: >> unibyte sequence (hex): 81 81 C0 C0 >> result of conversion display in multbyte buf= fer >> string-as-multibyte: 9E A1 81 C0 C0 \201=C0\300 >> string-make-multibyte: 9E A1 9E A1 81 C0 81 C0 \201\201=C0=C0 >> string-to-multibyte: 9E A1 9E A1 C0 C0 \201\201\300\300 > I find the terminology and the concepts confusing. I agree that those names are not that intuitive, but the first two were there before I noticed it. :-p But, in what sense, the concepts are confusing? > On the other hand, I understand the concept of encoding and decoding. > The following equivalences almost hold: > (string-as-multibyte str) =3D=3D (decode-coding-string str 'internal) > (string-make-multibyte str) =3D=3D (decode-coding-string str 'default) > (string-to-multibyte str) =3D=3D (decode-coding-string str 'raw-text) > I said "almost" because: Please note that decode-coding-string also does eol conversion. Using 'internal-unix, 'default-unix, 'raw-text-unix will make them more equivalent. > 1 - there is no `internal' coding-system as of now. In Emacs-21 we'd > use `emacs-mule' but for Emacs-22 it would be `utf-8-emacs'. > I'm still not sure what such a thing is useful for, tho (see > my other email). Before we introduced eight-bit-XXXX, (insert (string-as-multibyte UNIBYTE-STRING)) was the only way to preserve the original byte sequence in a multibyte buffer. But, as we now have eight-bit-XXXX, I agree that string-as-multibyte is not that useful, string-to-multibyte is better. > 2 - there is no `default' coding-system either. Or maybe > locale-coding-system is this default: if your locale is > latin-1 then that's latin-1. If one does not do set-language-enviroment, locale-coding-system can be used as `default'. > For non-8-bit locales, I don't know what > string-make-multibyte does. In that case, it does latin-1 decoding, ... yes, not that good. > 3 - when called with a `raw-text' coding-system, decode-coding-string > returns a unibyte string, which is obviously not what we want here. > It might make sense for internal operations to return unibyte > strings for the `raw-text' case, but I was really surprised that > decode-coding-string would ever return a unibyte string. I tend to agree that it is better that decode-coding-string always return a multibyte string now. > I think avoiding string-FOO-multibyte and using decode-coding-string > instead would make things a lot more clear. I think string-FOO-multibyte (and also string-FOO-unibyte) are conceptually different from decoding (and encoding) operations. It's difficult for me to explain it clearly, but I'll try. Decoding and encoding are interface between Emacs and the outer world. Decoding is for converting an external byte sequence (i.e. belonging to a world out of Emacs) into Emacs' represenatation. Encoding is for converting Emacs' represenatation to a byte sequence that is used out of Emacs. But string-FOO-multi/unibyte are convesion within Emacs' world. And, if one wants to insert a result of encode-coding-string in a multibyte buffer (perhaps for some post-processing), what he should do? If we have string-to-multibyte, we can do this: (insert (string-to-multibyte (encode-coding-string MULTIBYTE-STRING CODING))) If we don't have it, and provided that decode-coding-string always returns a multibyte string, we must do: (insert (decode-coding-string (encode-coding-string MULTIBYTE-STRING CODING) 'raw-text-unix)) Isn't it very funny? By the way, I think the culprit of the current problem is this Emacs' doctrine: Do unibyte<->mutibyte conversion by "MAKE" by default. Although this doctrine surely works for handling unibyte and multibyte represenation transparently, it makes Elisp programmers very very confused. And it is useful only for people whose main charset is single-byte. I seriously considering changing it in emacs-unicode. --- Ken'ichi HANDA handa@m17n.org