Unibyte characters

Unibyte characters

[Eli Zaretskii]

The ELisp manual has (in node "Text Representation") this explanation
of what is a "unibyte character":

       In unibyte representation, each character occupies one byte and
    therefore the possible character codes range from 0 to 255.  Codes 0
    through 127 are ASCII characters; the codes from 128 through 255 are
    used for one non-ASCII character set [...]

But I think this is inaccurate and even misleading.  For starters,
unibyte buffers and strings can contain DBCS characters and UTF-8
encoded text, where a character certainly does not ``occupy one
byte''.

More generally, I think it is better to say that unibyte buffers and
strings hold raw 8-bit bytes, and that for 8859-x and single-byte
Windows codepages, each such byte represents a single character.

Am I missing something?

Re: Unibyte characters

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
> But I think this is inaccurate and even misleading.  For starters,
> unibyte buffers and strings can contain DBCS characters and UTF-8
> encoded text

It doesn't seem inaccurate or misleading -- it's talking about
characters as emacs (and the user, if the buffer is displayed) sees
them.  Text in a unibyte buffer is simply a bunch of binary characters
0-255; you can interpret them however you want, of course, but that's
not how emacs sees it.

-Miles

-- 
Road, n. A strip of land along which one may pass from where it is too
tiresome to be to where it is futile to go.

Re: Unibyte characters

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Fri, 31 Oct 2008 20:18:41 +0900
> Cc: emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
> 
> Text in a unibyte buffer is simply a bunch of binary characters
> 0-255

Here you are saying what I was saying: that these are just raw 8-bit
bytes.

> you can interpret them however you want, of course, but that's
> not how emacs sees it.

I don't mind saying that displaying such a buffer or string or
movement by characters _interprets_ each byte as a single character.
But interpretation and essence are two different things, and the
manual does not make a point of telling that what it describes is the
Emacs interpretation of such buffers, not what is actually held there.

Thanks for the feedback, I will try to rephrase that text to make this
distinction more clear.

Re: Unibyte characters

[Stefan Monnier]

>> Text in a unibyte buffer is simply a bunch of binary characters
>> 0-255

> Here you are saying what I was saying: that these are just raw 8-bit
> bytes.

>> you can interpret them however you want, of course, but that's
>> not how emacs sees it.

> I don't mind saying that displaying such a buffer or string or
> movement by characters _interprets_ each byte as a single character.
> But interpretation and essence are two different things, and the
> manual does not make a point of telling that what it describes is the
> Emacs interpretation of such buffers, not what is actually held there.

> Thanks for the feedback, I will try to rephrase that text to make this
> distinction more clear.

IIUC, this part of the manual dates back to the introduction of Mule,
when many people were using Emacs in unibyte mode.  Nowadays unibyte
mode is not recommended (I'd even be all happy to remove it altogether)
and unibyte buffers should only be used for binary, undecoded data
(i.e. for bytes, not for chars).

So I agree with Eli that we should update this text to insist that
a unibyte buffer only contains bytes, and then explain that if the
buffer is displayed, those bytes will be interpreted in
a particular way.

BTW IIRC the non-ascii part will just be displayed as \NNN nowadays,
rather than in some locale-dependent charset (such as latin-1).


        Stefan

Re: Unibyte characters

[Juanma Barranquero]

On Fri, Oct 31, 2008 at 15:41, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> BTW IIRC the non-ascii part will just be displayed as \NNN nowadays,
> rather than in some locale-dependent charset (such as latin-1).

Unless you've set unibyte-display-via-language-environment to t.

Which apparently works for everyone except me; I get #872/#1179, a
hard to pinpoint but easy to repeat crash in a random location upwards
(stackwise) of draw_glyphs...

  Juanma

Re: Unibyte characters

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Miles Bader <miles@gnu.org>,  handa@m17n.org,  emacs-devel@gnu.org
> Date: Fri, 31 Oct 2008 10:41:47 -0400
> 
> IIUC, this part of the manual dates back to the introduction of Mule,
> when many people were using Emacs in unibyte mode.

Yes.

> Nowadays unibyte mode is not recommended

I agree, but I was talking about the ELisp manual; Lisp programmers do
need to know about unibyte buffers and strings, even if users are
discouraged from using the unibyte mode.

Re: Unibyte characters

[Richard M. Stallman]

	   In unibyte representation, each character occupies one byte and
	therefore the possible character codes range from 0 to 255.  Codes 0
	through 127 are ASCII characters; the codes from 128 through 255 are
	used for one non-ASCII character set [...]

    But I think this is inaccurate and even misleading.  For starters,
    unibyte buffers and strings can contain DBCS characters and UTF-8
    encoded text, where a character certainly does not ``occupy one
    byte''.

As far as Emacs is concerned, that UTF-8 sequence is multiple
characters and each of those characters is one byte.  The fact that
one might interpret that byte sequence some other way in another
context is not a part of the Emacs text representation.

So the text is correct.  But it could be useful to add something
to explain how this unibyte text relates to other interpretations
of the same byte sequence.

Re: Unibyte characters

[Stefan Monnier]

>> IIUC, this part of the manual dates back to the introduction of Mule,
>> when many people were using Emacs in unibyte mode.
> Yes.
>> Nowadays unibyte mode is not recommended
> I agree, but I was talking about the ELisp manual; Lisp programmers do
> need to know about unibyte buffers and strings, even if users are
> discouraged from using the unibyte mode.

Right, but for Elisp, we should make it even more clear that unibyte
buffers contain only bytes and not chars and that those buffers should
basically never be displayed, other than for debugging ;-)


        Stefan


PS: Of course, editing binary files is also useful sometimes.

Re: Unibyte characters

[Stephen J. Turnbull]

Stefan Monnier writes:

 > Right, but for Elisp, we should make it even more clear that unibyte
 > buffers contain only bytes and not chars and that those buffers should
 > basically never be displayed, other than for debugging ;-)

There's no reason not to display when the coding system is iso-8859-1.

Re: Unibyte characters

[Stefan Monnier]

>> Right, but for Elisp, we should make it even more clear that unibyte
>> buffers contain only bytes and not chars and that those buffers should
>> basically never be displayed, other than for debugging ;-)

> There's no reason not to display when the coding system is iso-8859-1.

I have no idea what you're talking about.


        Stefan