Emacs 23 character code space

Emacs 23 character code space

[Eli Zaretskii]

This fragment from etc/NEWS:

    The character code space is now 0x0..0x3FFFFF with no gap.
    Characters of code 0x0..0x10FFFF are Unicode characters of the same code points.
    Characters of code 0x3FFF80..0x3FFFFF are raw 8-bit bytes.

seems to contradict itself: it says there's ``no gap'', but the codes
between 0x110000 and 0x3FFF7F do constitute a gap, don't they?

Re: Emacs 23 character code space

[Eli Zaretskii]

Another fragment from etc/NEWS that seems not entirely accurate:

    In buffers and strings, characters are represented by UTF-8 byte
    sequences in a multibyte buffer/string.

But UTF-8 defines 1- to 4-byte sequences to represent each Unicode
codepoint, whereas this comment from character.h:

    /* character code       1st byte   byte sequence
       --------------       --------   -------------
	    0-7F            00..7F     0xxxxxxx
	   80-7FF           C2..DF     110xxxxx 10xxxxxx
	  800-FFFF          E0..EF     1110xxxx 10xxxxxx 10xxxxxx
	10000-1FFFFF        F0..F7     11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
       200000-3FFF7F        F8         11111000 1000xxxx 10xxxxxx 10xxxxxx 10xxxxxx
       3FFF80-3FFFFF        C0..C1     1100000x 10xxxxxx (for eight-bit-char)
       400000-...           invalid

       invalid 1st byte     80..BF     10xxxxxx
			    F9..FF     11111xxx (xxx != 000)
    */

seems to tell that we use up to 5 bytes.

What am I missing?

Re: Emacs 23 character code space

[Kenichi Handa]

I'm now in Vietnam, and the Internet connection is very bad,
so here's a very short reply.

In article <u63n7wmri.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> This fragment from etc/NEWS:
>     The character code space is now 0x0..0x3FFFFF with no gap.
>     Characters of code 0x0..0x10FFFF are Unicode characters of the same code points.
>     Characters of code 0x3FFF80..0x3FFFFF are raw 8-bit bytes.

> seems to contradict itself: it says there's ``no gap'', but the codes
> between 0x110000 and 0x3FFF7F do constitute a gap, don't they?

Those are for character codes not unified with Unicode.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Kenichi Handa]

In article <E1KwoKX-0002Tk-Lp@etlken.m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> I'm now in Vietnam, and the Internet connection is very bad,
> so here's a very short reply.

I moved to another hotel, and the Internet connection is a
little bit better here. :-)

> In article <u63n7wmri.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > This fragment from etc/NEWS:
> >     The character code space is now 0x0..0x3FFFFF with no gap.
> >     Characters of code 0x0..0x10FFFF are Unicode characters of the same code points.
> >     Characters of code 0x3FFF80..0x3FFFFF are raw 8-bit bytes.

> > seems to contradict itself: it says there's ``no gap'', but the codes
> > between 0x110000 and 0x3FFF7F do constitute a gap, don't they?

> Those are for character codes not unified with Unicode.

I tried to rewrite nonascii.texi to clear the things.  I
finished upto the "Character Code" section as attached.
What do you think about it?

---
Kenichi Handa
handa@ni.aist.go.jp

@c -*-texinfo-*-
@c This is part of the GNU Emacs Lisp Reference Manual.
@c Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004,
@c   2005, 2006, 2007, 2008  Free Software Foundation, Inc.
@c See the file elisp.texi for copying conditions.
@setfilename ../../info/characters
@node Non-ASCII Characters, Searching and Matching, Text, Top
@chapter Non-@acronym{ASCII} Characters
@cindex multibyte characters
@cindex characters, multi-byte
@cindex non-@acronym{ASCII} characters

  This chapter covers the special issues relating to non-@acronym{ASCII}
characters and how they are stored in strings and buffers.

@menu
* Text Representations::    Unibyte and multibyte representations
* Converting Representations::  Converting unibyte to multibyte and vice versa.
* Selecting a Representation::  Treating a byte sequence as unibyte or multi.
* Character Codes::         How unibyte and multibyte relate to
                                codes of individual characters.
* Character Sets::          The space of possible character codes
                                is divided into various character sets.
* Chars and Bytes::         More information about multibyte encodings.
* Splitting Characters::    Converting a character to its byte sequence.
* Scanning Charsets::       Which character sets are used in a buffer?
* Translation of Characters::   Translation tables are used for conversion.
* Coding Systems::          Coding systems are conversions for saving files.
* Input Methods::           Input methods allow users to enter various
                                non-ASCII characters without special keyboards.
* Locales::                 Interacting with the POSIX locale.
@end menu

@node Text Representations
@section Text Representations
@cindex text representations

  Emacs has two @dfn{text representations}---two ways to represent
text in a string or buffer.  These are called @dfn{unibyte} and
@dfn{multibyte}.  Each string, and each buffer, uses one of these two
representations to store a sequence Emacs character.  Emacs classifies
characters into these three; @acronym{ASCII} characters,
non-@acronym{ASCII} charcters, and 8-bit charcters.  8-bit characters
correponds to raw bytes of 128 through 255.  For detail, @xref{Character Codes}.

@cindex unibyte text
@cindex unibyte character
  In unibyte representation, each character occupies one byte and
therefore the possible character codes range from 0 to 255.  Codes 0
through 127 are @acronym{ASCII} characters; the codes from 128 through 255
are 8-bit charactes.  Non-@acronym{ASCII} characters can not be stored
in unibyte text.  We call a character in unibyte text as unibyte
character.

@cindex leading code
@cindex multibyte text
@cindex multibyte character

  In multibyte representation, a character may occupy more than one
byte, and as a result, the full range of Emacs character codes
(#x0..#x3FFFFF) can be stored.  @acronym{ASCII} characters occupy one
byte, non-@acronym{ASCII} characters occupy two to five bytes (the
first byte is in the range #xC2 through #xF8, and the remaining bytes
are in the range #x80 through #xBF), and 8-bit characters occupy two
bytes (the first byte is #xC0 or $xC2, and the second byte is in the
range #x80 through #xBF).  Actually this representation is the same as
UTF-8 with extentions for non-Unicode characters and 8-bit characters.
It is assured that a byte sequence that doesn't fit above never appears
in this representation.

  In a buffer, the buffer-local value of the variable
@code{enable-multibyte-characters} specifies the representation used.
The representation for a string is determined and recorded in the string
when the string is constructed.

@defvar enable-multibyte-characters
This variable specifies the current buffer's text representation.
If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
it contains unibyte text.

You cannot set this variable directly; instead, use the function
@code{set-buffer-multibyte} to change a buffer's representation.
@end defvar

@defvar default-enable-multibyte-characters
This variable's value is entirely equivalent to @code{(default-value
'enable-multibyte-characters)}, and setting this variable changes that
default value.  Setting the local binding of
@code{enable-multibyte-characters} in a specific buffer is not allowed,
but changing the default value is supported, and it is a reasonable
thing to do, because it has no effect on existing buffers.

The @samp{--unibyte} command line option does its job by setting the
default value to @code{nil} early in startup.
@end defvar

@defun position-bytes position
Return the byte-position corresponding to buffer position
@var{position} in the current buffer.  This is 1 at the start of the
buffer, and counts upward in bytes.  If @var{position} is out of
range, the value is @code{nil}.
@end defun

@defun byte-to-position byte-position
Return the buffer position corresponding to byte-position
@var{byte-position} in the current buffer.  If @var{byte-position} is
out of range, the value is @code{nil}.  If @var{byte-position} is not
at a character boundary (in case of multibyte buffer), the value is
the buffer position of the character that occupies @var{byte-position}.
@end defun

@defun multibyte-string-p string
Return @code{t} if @var{string} is a multibyte string.
@end defun

@defun string-bytes string
@cindex string, number of bytes
This function returns the number of bytes in @var{string}.
If @var{string} is a multibyte string, this can be greater than
@code{(length @var{string})}.
@end defun

@node Converting Representations
@section Converting Text Representations

  Emacs can convert unibyte text to multibyte; it can also convert
multibyte text to unibyte provided that the multibyte text contains
only @acronym{ASCII} and 8-bit characters.  In
general these conversions happen when inserting text into a buffer, or
when putting text from several strings together in one string.  You can
also explicitly convert a string's contents to either representation.

  Emacs chooses the representation for a string based on the text that
it is constructed from.  The general rule is to convert unibyte text to
multibyte text when combining it with other multibyte text, because the
multibyte representation is more general and can hold whatever
characters the unibyte text has.

  When inserting text into a buffer, Emacs converts the text to the
buffer's representation, as specified by
@code{enable-multibyte-characters} in that buffer.  In particular, when
you insert multibyte text into a unibyte buffer, Emacs converts the text
to unibyte, even though this conversion cannot in general preserve all
the characters that might be in the multibyte text.  The other natural
alternative, to convert the buffer contents to multibyte, is not
acceptable because the buffer's representation is a choice made by the
user that cannot be overridden automatically.

  Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
unchanged, and converts 8-bit characters (codes 128 through 159) to
the corresponding representation for multibyte text.

  Converting multibyte text to unibyte is simpler: it discards all but
the low 8 bits of each character code.  It effectively converts all
@acronym{ASCII} and 8-bit characters to the corresponding unibyte
representation, but loose information for non-@acronym{ASCII}
characters.  Converting unibyte text to multibyte and back to unibyte
reproduces the original unibyte text.

The next three functions either return the argument @var{string}, or a
newly created string with no text properties.

@defun string-to-multibyte string
This function returns a multibyte string containing the same sequence
of characters as @var{string}.  If @var{string} is a multibyte string,
it is returned unchanged.
@end defun

@defun string-to-unibyte string
This function returns a unibyte string containing the same sequence of
characters as @var{string}.  It signals an error if @var{string}
contains a non-@acronym{ASCII} character.  If @var{string} is a
unibyte string, it is returned unchanged.
@end defun

@defun multibyte-char-to-unibyte char
This convert the multibyte character @var{char} to a unibyte
character.  If @var{char} is a non-@acronym{ASCII} character, the
value is -1.
@end defun

@defun unibyte-char-to-multibyte char
This convert the unibyte character @var{char} to a multibyte
character.
@end defun

@node Selecting a Representation
@section Selecting a Representation

  Sometimes it is useful to examine an existing buffer or string as
multibyte when it was unibyte, or vice versa.

@defun set-buffer-multibyte multibyte
Set the representation type of the current buffer.  If @var{multibyte}
is non-@code{nil}, the buffer becomes multibyte.  If @var{multibyte}
is @code{nil}, the buffer becomes unibyte.

This function leaves the buffer contents unchanged when viewed as a
sequence of bytes.  As a consequence, it can change the contents
viewed as characters; a sequence of three bytes which is treated as
one character in multibyte representation will count as three
characters in unibyte representation.  8-bit characters are an
exception.  They are represented by one byte in a unibyte buffer, but
when the buffer is set to multibyte, they are converted to two-byte
sequences, and vice versa.

This function sets @code{enable-multibyte-characters} to record which
representation is in use.  It also adjusts various data in the buffer
(including overlays, text properties and markers) so that they cover the
same text as they did before.

You cannot use @code{set-buffer-multibyte} on an indirect buffer,
because indirect buffers always inherit the representation of the
base buffer.
@end defun

@defun string-as-unibyte string
This function returns a string with the same bytes as @var{string} but
treating each byte as a character.  This means that the value may have
more characters than @var{string} has.  8-bit characters are an
exception.  Each of them is represented by two bytes in a multibyte
string, but is converted to one byte.

If @var{string} is already a unibyte string, then the value is
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.
@end defun

@defun string-as-multibyte string
This function returns a string with the same bytes as @var{string} but
treating each multibyte sequence as one character.  This means that the
value may have fewer characters than @var{string} has.  If a byte
sequence in @var{string} is invalid as a multibyte representation,
each byte in the sequence is converted to two-byte multibyte
representation of 8-bit characters.

If @var{string} is already a multibyte string, then the value is
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.
@end defun

@node Character Codes
@section Character Codes
@cindex character codes

  The unibyte and multibyte text representations use different
character codes.  The valid character codes for unibyte representation
range from 0 to 255---the values that can fit in one byte.  The valid
character codes for multibyte representation range from 0 to 4194303
(#x3FFFFF).  In this code space, codes 0 through 127 are for
@acronym{ASCII} charcters, codes 129 through 4194175 (#x3FFF7F) are
for non-@acronym{ASCII} characters (among them, codes 0 through
1114111 (#10FFFF) corresponds to Unicode characters of the same
codes), and codes 4194176 (#x3FFF80) through 4194303 (#x3FFFFF) are
for 8-bit characters.

@defun characterp charcode
This returns @code{t} if @var{charcode} is a valid character, and
@code{nil} otherwise.

@example
(characterp 65)
     @result{} t
(characterp 4194303)
     @result{} t
(characterp 4194304)
     @result{} nil
@end example
@end defun

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Mon, 03 Nov 2008 21:45:20 +0900
> 
> I tried to rewrite nonascii.texi to clear the things.  I
> finished upto the "Character Code" section as attached.
> What do you think about it?

Thanks, this definitely helps.  Unfortunately, you worked from a
non-current version of nonascii.texi; I already modified the first
section heavily.  Please take a look when you can: I intend to
downplay the unibyte stuff heavily, while the previous version gave
unibyte and multibyte almost equal coverage.

In any case, I will certainly use what you wrote.  Thanks!

> @acronym{ASCII} characters occupy one
> byte, non-@acronym{ASCII} characters occupy two to five bytes

So I guess you agree that NEWS is not entirely correct saying that we
use UTF-8 internally: UTF-8 uses only 1 to 4 bytes, not 1 to 5.
Should I fix NEWS in this regard, saying that the internal
representation is based on UTF-8, but extends it to handle additional
characters?

Re: Emacs 23 character code space

[Kenichi Handa]

In article <umyggva8e.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, this definitely helps.  Unfortunately, you worked from a
> non-current version of nonascii.texi; I already modified the first
> section heavily.  Please take a look when you can: I intend to
> downplay the unibyte stuff heavily, while the previous version gave
> unibyte and multibyte almost equal coverage.

Oops, I couldn't do "cvs update" until yesterday.  I've just
done it.

> In any case, I will certainly use what you wrote.  Thanks!

It seems that your last change is upto "@defun unibyte
string" (before @section Converting Text Representations).
May I leave the work of reflecting what I wrote to the
section before "Character Code" to you?  I'll continue to
fix the document after that section.

> > @acronym{ASCII} characters occupy one
> > byte, non-@acronym{ASCII} characters occupy two to five bytes

> So I guess you agree that NEWS is not entirely correct saying that we
> use UTF-8 internally: UTF-8 uses only 1 to 4 bytes, not 1 to 5.
> Should I fix NEWS in this regard, saying that the internal
> representation is based on UTF-8, but extends it to handle additional
> characters?

As far as I remember, "UTF-8" was, at first, a general
mechanism to serialize a 31-bit unsigned value into byte
stream (thus upto 6-byte sequence).  But, at some point, it
seems that it is restricted to the Unicode coverage (thus
upto 4-byte sequence).  I think it's a regression.  Anyway,
yes, it is better to mention that Emacs uses extended utf-8
(or original utf-8).

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Tue, 04 Nov 2008 16:35:10 +0900
> 
> It seems that your last change is upto "@defun unibyte
> string" (before @section Converting Text Representations).

Yes, I only had time to work on the first section for now.

> May I leave the work of reflecting what I wrote to the
> section before "Character Code" to you?

Sure.

> I'll continue to fix the document after that section.

Thanks.

> yes, it is better to mention that Emacs uses extended utf-8
> (or original utf-8).

That's what I did.  I will modify NEWS as well.

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uhc6nutv5.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > May I leave the work of reflecting what I wrote to the
> > section before "Character Code" to you?

> Sure.

Thank you.

> > I'll continue to fix the document after that section.

Attached is the remaining part.  Please reflect that to the
document.  I'm sorry for asking you this tiresome work.  I
still can't establish a reliable internet connection for
committing by myself.

---
Kenichi Handa
handa@ni.aist.go.jp

----------------------------------------------------------------------
@node Character Codes
@section Character Codes
@cindex character codes

  The unibyte and multibyte text representations use different
character codes.  The valid character codes for unibyte representation
range from 0 to 255---the values that can fit in one byte.  The valid
character codes for multibyte representation range from 0 to 4194303
(#x3FFFFF).  In this code space, codes 0 through 127 are for
@acronym{ASCII} charcters, codes 129 through 4194175 (#x3FFF7F) are
for non-@acronym{ASCII} characters (among them, codes 0 through
1114111 (#10FFFF) corresponds to Unicode characters of the same
codes), and codes 4194176 (#x3FFF80) through 4194303 (#x3FFFFF) are
for 8-bit characters.

@defun characterp charcode
This returns @code{t} if @var{charcode} is a valid character, and
@code{nil} otherwise.

@example
(characterp 65)
     @result{} t
(characterp 4194303)
     @result{} t
(characterp 4194304)
     @result{} nil
@end example
@end defun

@defun get-byte pos &optional string
This function returns a byte value of a character at poistion
@var{pos}.  If the current buffer is unibyte, byte values are the same
as character codes.  If it is multibyte, byte values of
@acronym{ASCII} characters are the same as chacter codes, whereas byte
values of 8-bit characters are the low 8-bit of character codes.  If
the character at @var{pos} is non-@acronym{ASCII}, an error is
signalled.

The optional argument @var{string} specifies a string from which to
get a byte value of a character.
@end defun

@node Character Sets
@section Character Sets
@cindex character sets

  A character set is a set of characters, and it assigns a unique code
point to each character belonging to the set.  Emacs decodes a
specific code point of a specific character set to an Emacs character.
For example, @code{iso-8859-1} is one character set, and it contains
the character A-grave by code-point #xC0.  Emacs decodes that
character into Emacs' character of code #xC0.  @code{iso-8859-2} is
another character set, and it contains the character R-acute by
code-point #xC0.  Emacs decodes that character into Emacs' character
of code #x154.  @code{unicode} is also a character set.  It contains
both A-grave and R-acute by code-point #xC0 and #x154 respectively.
Emacs decodes them to Emacs' character #xC0 and #x154.  Actually, all
characters in @code{unicode} are decoded to characters of the same
code values (in the range #x0 through #x10FFFF).

  Some character set contains characters not included in
@code{unicode} (for example, characters for private use).  Emacs
decodes such characters into a character code space uniquely kept for
each character set.  That code space is in the range #x110000 through
#x3FFF7F.

  Emacs has two special character set named @code{emacs} and
@code{eight-bit}.  The former contains all @acronym{ASCII} and
non-@acronym{ASCII} characters, and the latter contains all remaining
characters; i.e. all 8-bit characters.

  How to define a characte set is an arcane matter, and is not
documented here.

@defun charsetp object
Returns @code{t} if @var{object} is a symbol that names a character set,
@code{nil} otherwise.
@end defun

@defvar charset-list
The value is a list of all defined character set names.
@end defvar

@defun charset-list
This function returns the value of @code{charset-list}.  It is only
provided for backward compatibility.
@end defun

@defun charset-priority-list &optional highestp
This functions returns a list of character sets sorted by their
priorities.  If @var{highestp} is non-@code{nil}, returns a character
set of highest priority instread of returning a list.
@end defun

@defun char-charset character
This function returns the name of the character set of highest
priority that @var{character} belongs to.
@acronym{ASCII} characters are an exception.  For them, this function
always returns @code{ascii}.
@end defun

@defun charset-plist charset
This function returns the charset property list of the character set
@var{charset}.  Although @var{charset} is a symbol, this is not the same
as the property list of that symbol.  Charset properties are used for
special purposes within Emacs.
@end defun

@defun charset-dimension charset
This function returns the dimension of @var{charset}.  Here, dimension
means the number of bytes required to represent the highest code point
(not an Emacs character code) of a character.  For example, the
dimension of @code{iso-8859-1} is one, the dimension of
@code{japanese-jisx0208} is two, and the dimension of @code{unicode}
is three.
@end defun

@deffn Command list-charset-chars charset
This command displays a list of characters in the character set
@var{charset}.
@end deffn

@defun decode-char charset code-point
This function decodes a character of @var{charset} that has code point
@var{code-point} to Emacs' character, and return it.  If @var{charset}
doesn't contain a character of @var{code-point}, the value is @code{nil}.
If @var{code-point} doesnt't fit in Lisp integer, it can be specified
by a cons (HIGHER-16-BIT-VALUE . LOWER-16-BIT-VALUE).
@end defun

@defun encode-char char charset
This function encodes the character @var{char} to a code-point of
@var{charset}.  If @var{charset} doesn't contain @var{char}, the value
is @code{nil}.
@end defun

@defun split-char character
Return a list containing the name of the character set of highest
priority to which @var{character} belongs, followed by one to four
byte values (integers) representing a code point of @var{character}
within that character set.  The number of byte values is the character
set's dimension.  @acronym{ASCII} characters are an exception.  For
them, the first element of the list is always @code{ascii}.

It is only provided for backward compatibility.  You should use
@code{encode-char} instead.
@end defun

@defun make-char charset &optional code1 code2 code3 code4
This function returns the character in character set @var{charset} whose
code point is represented by the byte values @var{code1} through @var{code4}.
The number of byte values should be the dimesion of @var{charset}.
If the specified values are less than that dimension, 0 is assumed.

It is only provided for backward compatibility.  You should use
@code{decode-char} instead.
@end defun

@node Scanning Charsets
@section Scanning for Character Sets

  Sometimes it is useful to find out which character sets appear in a
part of a buffer or a string.  One use for this is in determining which
coding systems (@pxref{Coding Systems}) are capable of representing all
of the text in question.

@defun charset-after &optional pos
This function return the charset of highest priority containing a
character in the current buffer at position @var{pos}.  If @var{pos}
is omitted or @code{nil}, it defaults to the current value of point.
If @var{pos} is out of range, the value is @code{nil}.
@end defun

@defun find-charset-region beg end &optional translation
This function returns a list of the character sets of highest priority
that contain characters in the current buffer between positions
@var{beg} and @var{end}.

The optional argument @var{translation} specifies a translation table to
be used in scanning the text (@pxref{Translation of Characters}).  If it
is non-@code{nil}, then each character in the region is translated
through this table, and the value returned describes the translated
characters instead of the characters actually in the buffer.
@end defun

@defun find-charset-string string &optional translation
This function returns a list of the character sets of highest priority
that contain characters in the string @var{string}.  It is just like
@code{find-charset-region}, except that it applies to the contents of
@var{string} instead of part of the current buffer.
@end defun

@node Translation of Characters
@section Translation of Characters
@cindex character translation tables
@cindex translation tables

  A @dfn{translation table} is a char-table that specifies a mapping
of characters (or sequences of characters) into characters (or
sequences of characters).  These tables are used in encoding and
decoding, and for other purposes.  Some coding systems specify their
own particular translation tables; there are also default translation
tables which apply to all coding systems.

  A translation table has two extra slots; the first is @code{nil} or
a translation table that performs the reverse translation, and the
second is a maximum number of characters to look up for translation.

  In decoding, the translation table's translations are applied to the
characters that result from ordinary decoding.  If a coding system has
property @code{:deocde-translation-table}, that specifies the
translation table or the list of cascading translation table to use.
(This is a property of the coding system, as returned by
@code{coding-system-get}, not a property of the symbol that is the
coding system's name.  @xref{Coding System Basics,, Basic Concepts of
Coding Systems}.)  In the latter case, characters are translated by
each translation tables in the specified order.  And, at last, if
@code{standard-translation-table-for-decode} is non-@code{nil},
the resulting characters are translated by that table.

  In encoding, the translation table's translations are applied to the
characters in the buffer, and the result of translation is actually
encoded.  If a coding system has property
@code{:encode-translation-table}, that specifies the translation table
or the list of cascading translation tables to use.  In addition, if
the variable @code{standard-translation-table-for-encode} is
non-@code{nil}, it is used at last.

@defvar standard-translation-table-for-decode
This is the default translation table for decoding.  For coding
systems that specifies any other translation tables, those tables
translate charcters before this table. 
@end defvar

@defvar standard-translation-table-for-encode
This is the default translation table for encoding.  For
coding systems that specifies any other translation tables, those
tables translate characters before this table.
@end defvar

@defun make-translation-table &rest translations
This function returns a translation table based on the argument
@var{translations}.  Each element of @var{translations} should be a
list of elements of the form @code{(@var{from} . @var{to})}; this says
to translate the character @var{from} into @var{to}.

The arguments and the forms in each argument are processed in order,
and if a previous form already translates @var{to} to some other
character, say @var{to-alt}, @var{from} is also translated to
@var{to-alt}.

The first extra slot of the returned translation table is @code{nil};
i.e. this fucntion does not make a translation table for reverse
translation.
@end defun

@defun make-translation-table-from-vector vec
This function returns a translation table made from @var{vec} that is
an array of 256 elements to map byte values 0 through 255 to characters.
Elements may be nil for untranslated bytes.   The returned table has
a translation table for reverse mapping in the first extra slot.

This function provides an easy way to make a private coding system
that maps each byte to a specific character.  You can specify the
returned table and the reverse translation table in it to the
properties @code{:decode-translation-table} and
@code{:encode-translation-table} respectively in
@code{define-coding-system}.
@end defun

@defun make-translation-table-from-alist alist
This function is similar to @code{make-translation-table} but returns
a complex translation table rather than a simple one to one mapping.
@var{alist} is an alist, each element has the form @code{(@var{from}
. @var{to})}, where @var{from} and @var{to} are a character or a
vector of characters.  If @var{from} is a character, that character is
translated to @var{to} (i.e. to a character of a character sequence).
If @var{from} is a vector of characters, that sequence is translated
to @var{to}.  The returned table has a translation table for reverse
mapping in the first extra slot.
@end defun

@node Coding Systems
@section Coding Systems

@cindex coding system
  When Emacs reads or writes a file, and when Emacs sends text to a
subprocess or receives text from a subprocess, it normally performs
character code conversion and end-of-line conversion as specified
by a particular @dfn{coding system}.

  How to define a coding system is an arcane matter, and is not
documented here.

@menu
* Coding System Basics::        Basic concepts.
* Encoding and I/O::            How file I/O functions handle coding systems.
* Lisp and Coding Systems::     Functions to operate on coding system names.
* User-Chosen Coding Systems::  Asking the user to choose a coding system.
* Default Coding Systems::      Controlling the default choices.
* Specifying Coding Systems::   Requesting a particular coding system
                                    for a single file operation.
* Explicit Encoding::           Encoding or decoding text without doing I/O.
* Terminal I/O Encoding::       Use of encoding for terminal I/O.
* MS-DOS File Types::           How DOS "text" and "binary" files
                                    relate to coding systems.
@end menu

@node Coding System Basics
@subsection Basic Concepts of Coding Systems

@cindex character code conversion
  @dfn{Character code conversion} involves conversion between the encoding
used inside Emacs and some other encoding.  Emacs supports many
different encodings, in that it can convert to and from them.  For
example, it can convert text to or from encodings such as Latin 1, Latin
2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022.  In some
cases, Emacs supports several alternative encodings for the same
characters; for example, there are three coding systems for the Cyrillic
(Russian) alphabet: ISO, Alternativnyj, and KOI8.

FIXME: I don't know what this sentence means.
  Most coding systems specify a particular character code for
conversion, but some of them leave the choice unspecified---to be chosen
heuristically for each file, based on the data.

  In general, a coding system doesn't guarantee roundtrip identity:
decoding a byte sequence using coding system, then encoding the
resulting text in the same coding system, can produce a different byte
sequence.  For example, the following coding systems guarantee that the
byte sequence will be the same as what you originally decoded:

@quotation
iso-8859-1, utf-8, big5, shift_jis, euc-jp
@end quotation

but the following coding systems do not:

@quotation
iso-2022-jp, iso-2022-kr, utf-16
@end quotation

  Encoding buffer text and then decoding the result can also fail to
reproduce the original text.  For instance, if you encode a character
by a coding system not supporting that character, the result is
unpredictable, and thus decoding of it using the same coding
system may produce the different text.  Currently, Emacs can't report
the error of encoding unsupported characters.

@cindex EOL conversion
@cindex end-of-line conversion
@cindex line end conversion
  @dfn{End of line conversion} handles three different conventions used
on various systems for representing end of line in files.  The Unix
convention is to use the linefeed character (also called newline).  The
DOS convention is to use a carriage-return and a linefeed at the end of
a line.  The Mac convention is to use just carriage-return.

@cindex base coding system
@cindex variant coding system
  @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
conversion unspecified, to be chosen based on the data.  @dfn{Variant
coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
@code{latin-1-mac} specify the end-of-line conversion explicitly as
well.  Most base coding systems have three corresponding variants whose
names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.

  The coding system @code{raw-text} is special in that it prevents
character code conversion, and causes the buffer visited with that
coding system to be a unibyte buffer.  It does not specify the
end-of-line conversion, allowing that to be determined as usual by the
data, and has the usual three variants which specify the end-of-line
conversion.  @code{no-conversion} is equivalent to @code{raw-text-unix}:
it specifies no conversion of either character codes or end-of-line.

  The coding system @code{emacs-internal} specifies that the data is
represented in the internal Emacs encoding.  This is like
@code{raw-text} in that no code conversion happens, but different in
that the result is multibyte data.

@defun coding-system-get coding-system property
This function returns the specified property of the coding system
@var{coding-system}.  Most coding system properties exist for internal
purposes, but one that you might find useful is @code{:mime-charset}.
That property's value is the name used in MIME for the character coding
which this coding system can read and write.  Examples:

@example
(coding-system-get 'iso-latin-1 :mime-charset)
     @result{} iso-8859-1
(coding-system-get 'iso-2022-cn :mime-charset)
     @result{} iso-2022-cn
(coding-system-get 'cyrillic-koi8 :mime-charset)
     @result{} koi8-r
@end example

The value of the @code{:mime-charset} property is also defined
as an alias for the coding system.
@end defun

@node Encoding and I/O
@subsection Encoding and I/O

  The principal purpose of coding systems is for use in reading and
writing files.  The function @code{insert-file-contents} uses
a coding system for decoding the file data, and @code{write-region}
uses one to encode the buffer contents.

  You can specify the coding system to use either explicitly
(@pxref{Specifying Coding Systems}), or implicitly using a default
mechanism (@pxref{Default Coding Systems}).  But these methods may not
completely specify what to do.  For example, they may choose a coding
system such as @code{undefined} which leaves the character code
conversion to be determined from the data.  In these cases, the I/O
operation finishes the job of choosing a coding system.  Very often
you will want to find out afterwards which coding system was chosen.

@defvar buffer-file-coding-system
This buffer-local variable records the coding system used for saving the
buffer and for writing part of the buffer with @code{write-region}.  If
the text to be written cannot be safely encoded using the coding system
specified by this variable, these operations select an alternative
encoding by calling the function @code{select-safe-coding-system}
(@pxref{User-Chosen Coding Systems}).  If selecting a different encoding
requires to ask the user to specify a coding system,
@code{buffer-file-coding-system} is updated to the newly selected coding
system.

@code{buffer-file-coding-system} does @emph{not} affect sending text
to a subprocess.
@end defvar

@defvar save-buffer-coding-system
This variable specifies the coding system for saving the buffer (by
overriding @code{buffer-file-coding-system}).  Note that it is not used
for @code{write-region}.

When a command to save the buffer starts out to use
@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
and that coding system cannot handle
the actual text in the buffer, the command asks the user to choose
another coding system (by calling @code{select-safe-coding-system}).
After that happens, the command also updates
@code{buffer-file-coding-system} to represent the coding system that
the user specified.
@end defvar

@defvar last-coding-system-used
I/O operations for files and subprocesses set this variable to the
coding system name that was used.  The explicit encoding and decoding
functions (@pxref{Explicit Encoding}) set it too.

@strong{Warning:} Since receiving subprocess output sets this variable,
it can change whenever Emacs waits; therefore, you should copy the
value shortly after the function call that stores the value you are
interested in.
@end defvar

  The variable @code{selection-coding-system} specifies how to encode
selections for the window system.  @xref{Window System Selections}.

@defvar file-name-coding-system
The variable @code{file-name-coding-system} specifies the coding
system to use for encoding file names.  Emacs encodes file names using
that coding system for all file operations.  If
@code{file-name-coding-system} is @code{nil}, Emacs uses a default
coding system determined by the selected language environment.  In the
default language environment, any non-@acronym{ASCII} characters in
file names are not encoded specially; they appear in the file system
using the internal Emacs representation.
@end defvar

  @strong{Warning:} if you change @code{file-name-coding-system} (or
the language environment) in the middle of an Emacs session, problems
can result if you have already visited files whose names were encoded
using the earlier coding system and are handled differently under the
new coding system.  If you try to save one of these buffers under the
visited file name, saving may use the wrong file name, or it may get
an error.  If such a problem happens, use @kbd{C-x C-w} to specify a
new file name for that buffer.

@node Lisp and Coding Systems
@subsection Coding Systems in Lisp

  Here are the Lisp facilities for working with coding systems:

@defun coding-system-list &optional base-only
This function returns a list of all coding system names (symbols).  If
@var{base-only} is non-@code{nil}, the value includes only the
base coding systems.  Otherwise, it includes alias and variant coding
systems as well.
@end defun

@defun coding-system-p object
This function returns @code{t} if @var{object} is a coding system
name or @code{nil}.
@end defun

@defun check-coding-system coding-system
This function checks the validity of @var{coding-system}.
If that is valid or @code{nil}, it returns @var{coding-system}.
Otherwise it signals an error with condition @code{coding-system-error}.
@end defun

@defun coding-system-eol-type coding-system
This function returns the type of end-of-line (a.k.a.@: @dfn{eol})
conversion used by @var{coding-system}.  If @var{coding-system}
specifies a certain eol conversion, the return value is an integer 0,
1, or 2, standing for @code{unix}, @code{dos}, and @code{mac},
respectively.  If @var{coding-system} doesn't specify eol conversion
explicitly, the return value is a vector of coding systems, each one
with one of the possible eol conversion types, like this:

@lisp
(coding-system-eol-type 'latin-1)
     @result{} [latin-1-unix latin-1-dos latin-1-mac]
@end lisp

@noindent
If this function returns a vector, Emacs will decide, as part of the
text encoding or decoding process, what eol conversion to use.  For
decoding, the end-of-line format of the text is auto-detected, and the
eol conversion is set to match it (e.g., DOS-style CRLF format will
imply @code{dos} eol conversion).  For encoding, the eol conversion is
taken from the appropriate default coding system (e.g.,
@code{default-buffer-file-coding-system} for
@code{buffer-file-coding-system}), or from the default eol conversion
appropriate for the underlying platform.
@end defun

@defun coding-system-change-eol-conversion coding-system eol-type
This function returns a coding system which is like @var{coding-system}
except for its eol conversion, which is specified by @code{eol-type}.
@var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
@code{nil}.  If it is @code{nil}, the returned coding system determines
the end-of-line conversion from the data.

@var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
@code{dos} and @code{mac}, respectively.
@end defun

@defun coding-system-change-text-conversion eol-coding text-coding
This function returns a coding system which uses the end-of-line
conversion of @var{eol-coding}, and the text conversion of
@var{text-coding}.  If @var{text-coding} is @code{nil}, it returns
@code{undecided}, or one of its variants according to @var{eol-coding}.
@end defun

@defun find-coding-systems-region from to
This function returns a list of coding systems that could be used to
encode a text between @var{from} and @var{to}.  All coding systems in
the list can safely encode any multibyte characters in that portion of
the text.

If the text contains no multibyte characters, the function returns the
list @code{(undecided)}.
@end defun

@defun find-coding-systems-string string
This function returns a list of coding systems that could be used to
encode the text of @var{string}.  All coding systems in the list can
safely encode any multibyte characters in @var{string}.  If the text
contains no multibyte characters, this returns the list
@code{(undecided)}.
@end defun

@defun find-coding-systems-for-charsets charsets
This function returns a list of coding systems that could be used to
encode all the character sets in the list @var{charsets}.
@end defun

@defun detect-coding-region start end &optional highest
This function chooses a plausible coding system for decoding the text
from @var{start} to @var{end}.  This text should be a byte sequence;
i.e. @acronym{ASCII} or 8-bit characters (@pxref{Explicit Encoding}).

Normally this function returns a list of coding systems that could
handle decoding the text that was scanned.  They are listed in order of
decreasing priority.  But if @var{highest} is non-@code{nil}, then the
return value is just one coding system, the one that is highest in
priority.

If the region contains only @acronym{ASCII} characters except for such
ISO-2022 control characters ISO-2022 as @code{ESC}, the value is
@code{undecided} or @code{(undecided)}, or a variant specifying
end-of-line conversion, if that can be deduced from the text.
@end defun

@defun detect-coding-string string &optional highest
This function is like @code{detect-coding-region} except that it
operates on the contents of @var{string} instead of bytes in the buffer.
@end defun

  @xref{Coding systems for a subprocess,, Process Information}, in
particular the description of the functions
@code{process-coding-system} and @code{set-process-coding-system}, for
how to examine or set the coding systems used for I/O to a subprocess.

@node User-Chosen Coding Systems
@subsection User-Chosen Coding Systems

@cindex select safe coding system
@defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
This function selects a coding system for encoding specified text,
asking the user to choose if necessary.  Normally the specified text
is the text in the current buffer between @var{from} and @var{to}.  If
@var{from} is a string, the string specifies the text to encode, and
@var{to} is ignored.

If @var{default-coding-system} is non-@code{nil}, that is the first
coding system to try; if that can handle the text,
@code{select-safe-coding-system} returns that coding system.  It can
also be a list of coding systems; then the function tries each of them
one by one.  After trying all of them, it next tries the current
buffer's value of @code{buffer-file-coding-system} (if it is not
@code{undecided}), then the value of
@code{default-buffer-file-coding-system} and finally the user's most
preferred coding system, which the user can set using the command
@code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
Coding Systems, emacs, The GNU Emacs Manual}).

If one of those coding systems can safely encode all the specified
text, @code{select-safe-coding-system} chooses it and returns it.
Otherwise, it asks the user to choose from a list of coding systems
which can encode all the text, and returns the user's choice.

@var{default-coding-system} can also be a list whose first element is
t and whose other elements are coding systems.  Then, if no coding
system in the list can handle the text, @code{select-safe-coding-system}
queries the user immediately, without trying any of the three
alternatives described above.

The optional argument @var{accept-default-p}, if non-@code{nil},
should be a function to determine whether a coding system selected
without user interaction is acceptable. @code{select-safe-coding-system}
calls this function with one argument, the base coding system of the
selected coding system.  If @var{accept-default-p} returns @code{nil},
@code{select-safe-coding-system} rejects the silently selected coding
system, and asks the user to select a coding system from a list of
possible candidates.

@vindex select-safe-coding-system-accept-default-p
If the variable @code{select-safe-coding-system-accept-default-p} is
non-@code{nil}, its value overrides the value of
@var{accept-default-p}.

As a final step, before returning the chosen coding system,
@code{select-safe-coding-system} checks whether that coding system is
consistent with what would be selected if the contents of the region
were read from a file.  (If not, this could lead to data corruption in
a file subsequently re-visited and edited.)  Normally,
@code{select-safe-coding-system} uses @code{buffer-file-name} as the
file for this purpose, but if @var{file} is non-@code{nil}, it uses
that file instead (this can be relevant for @code{write-region} and
similar functions).  If it detects an apparent inconsistency,
@code{select-safe-coding-system} queries the user before selecting the
coding system.
@end defun

  Here are two functions you can use to let the user specify a coding
system, with completion.  @xref{Completion}.

@defun read-coding-system prompt &optional default
This function reads a coding system using the minibuffer, prompting with
string @var{prompt}, and returns the coding system name as a symbol.  If
the user enters null input, @var{default} specifies which coding system
to return.  It should be a symbol or a string.
@end defun

@defun read-non-nil-coding-system prompt
This function reads a coding system using the minibuffer, prompting with
string @var{prompt}, and returns the coding system name as a symbol.  If
the user tries to enter null input, it asks the user to try again.
@xref{Coding Systems}.
@end defun

@node Default Coding Systems
@subsection Default Coding Systems

  This section describes variables that specify the default coding
system for certain files or when running certain subprograms, and the
function that I/O operations use to access them.

  The idea of these variables is that you set them once and for all to the
defaults you want, and then do not change them again.  To specify a
particular coding system for a particular operation in a Lisp program,
don't change these variables; instead, override them using
@code{coding-system-for-read} and @code{coding-system-for-write}
(@pxref{Specifying Coding Systems}).

@defvar auto-coding-regexp-alist
This variable is an alist of text patterns and corresponding coding
systems. Each element has the form @code{(@var{regexp}
. @var{coding-system})}; a file whose first few kilobytes match
@var{regexp} is decoded with @var{coding-system} when its contents are
read into a buffer.  The settings in this alist take priority over
@code{coding:} tags in the files and the contents of
@code{file-coding-system-alist} (see below).  The default value is set
so that Emacs automatically recognizes mail files in Babyl format and
reads them with no code conversions.
@end defvar

@defvar file-coding-system-alist
This variable is an alist that specifies the coding systems to use for
reading and writing particular files.  Each element has the form
@code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
expression that matches certain file names.  The element applies to file
names that match @var{pattern}.

The @sc{cdr} of the element, @var{coding}, should be either a coding
system, a cons cell containing two coding systems, or a function name (a
symbol with a function definition).  If @var{coding} is a coding system,
that coding system is used for both reading the file and writing it.  If
@var{coding} is a cons cell containing two coding systems, its @sc{car}
specifies the coding system for decoding, and its @sc{cdr} specifies the
coding system for encoding.

If @var{coding} is a function name, the function should take one
argument, a list of all arguments passed to
@code{find-operation-coding-system}.  It must return a coding system
or a cons cell containing two coding systems.  This value has the same
meaning as described above.

If @var{coding} (or what returned by the above function) is
@code{undecided}, the normal code-detection is performed.
@end defvar

@defvar process-coding-system-alist
This variable is an alist specifying which coding systems to use for a
subprocess, depending on which program is running in the subprocess.  It
works like @code{file-coding-system-alist}, except that @var{pattern} is
matched against the program name used to start the subprocess.  The coding
system or systems specified in this alist are used to initialize the
coding systems used for I/O to the subprocess, but you can specify
other coding systems later using @code{set-process-coding-system}.
@end defvar

  @strong{Warning:} Coding systems such as @code{undecided}, which
determine the coding system from the data, do not work entirely reliably
with asynchronous subprocess output.  This is because Emacs handles
asynchronous subprocess output in batches, as it arrives.  If the coding
system leaves the character code conversion unspecified, or leaves the
end-of-line conversion unspecified, Emacs must try to detect the proper
conversion from one batch at a time, and this does not always work.

  Therefore, with an asynchronous subprocess, if at all possible, use a
coding system which determines both the character code conversion and
the end of line conversion---that is, one like @code{latin-1-unix},
rather than @code{undecided} or @code{latin-1}.

@defvar network-coding-system-alist
This variable is an alist that specifies the coding system to use for
network streams.  It works much like @code{file-coding-system-alist},
with the difference that the @var{pattern} in an element may be either a
port number or a regular expression.  If it is a regular expression, it
is matched against the network service name used to open the network
stream.
@end defvar

@defvar default-process-coding-system
This variable specifies the coding systems to use for subprocess (and
network stream) input and output, when nothing else specifies what to
do.

The value should be a cons cell of the form @code{(@var{input-coding}
. @var{output-coding})}.  Here @var{input-coding} applies to input from
the subprocess, and @var{output-coding} applies to output to it.
@end defvar

@defvar auto-coding-functions
This variable holds a list of functions that try to determine a
coding system for a file based on its undecoded contents.

Each function in this list should be written to look at text in the
current buffer, but should not modify it in any way.  The buffer will
contain undecoded text of parts of the file.  Each function should
take one argument, @var{size}, which tells it how many characters to
look at, starting from point.  If the function succeeds in determining
a coding system for the file, it should return that coding system.
Otherwise, it should return @code{nil}.

If a file has a @samp{coding:} tag, that takes precedence, so these
functions won't be called.
@end defvar

@defun find-operation-coding-system operation &rest arguments
This function returns the coding system to use (by default) for
performing @var{operation} with @var{arguments}.  The value has this
form:

@example
(@var{decoding-system} . @var{encoding-system})
@end example

The first element, @var{decoding-system}, is the coding system to use
for decoding (in case @var{operation} does decoding), and
@var{encoding-system} is the coding system for encoding (in case
@var{operation} does encoding).

The argument @var{operation} is a symbol, one of @code{write-region},
@code{start-process}, @code{call-process}, @code{call-process-region},
@code{insert-file-contents}, or @code{open-network-stream}.  These are
the names of the Emacs I/O primitives that can do character code and
eol conversion.

The remaining arguments should be the same arguments that might be given
to the corresponding I/O primitive.  Depending on the primitive, one
of those arguments is selected as the @dfn{target}.  For example, if
@var{operation} does file I/O, whichever argument specifies the file
name is the target.  For subprocess primitives, the process name is the
target.  For @code{open-network-stream}, the target is the service name
or port number.

Depending on @var{operation}, this function looks up the target in
@code{file-coding-system-alist}, @code{process-coding-system-alist},
or @code{network-coding-system-alist}.  If the target is found in the
alist, @code{find-operation-coding-system} returns its association in
the alist; otherwise it returns @code{nil}.

If @var{operation} is @code{insert-file-contents}, the argument
corresponding to the target may be a cons cell of the form
@code{(@var{filename} . @var{buffer})}).  In that case, @var{filename}
is a file name to look up in @code{file-coding-system-alist}, and
@var{buffer} is a buffer that contains the file's contents (not yet
decoded).  If @code{file-coding-system-alist} specifies a function to
call for this file, and that function needs to examine the file's
contents (as it usually does), it should examine the contents of
@var{buffer} instead of reading the file.
@end defun

@node Specifying Coding Systems
@subsection Specifying a Coding System for One Operation

  You can specify the coding system for a specific operation by binding
the variables @code{coding-system-for-read} and/or
@code{coding-system-for-write}.

@defvar coding-system-for-read
If this variable is non-@code{nil}, it specifies the coding system to
use for reading a file, or for input from a synchronous subprocess.

It also applies to any asynchronous subprocess or network stream, but in
a different way: the value of @code{coding-system-for-read} when you
start the subprocess or open the network stream specifies the input
decoding method for that subprocess or network stream.  It remains in
use for that subprocess or network stream unless and until overridden.

The right way to use this variable is to bind it with @code{let} for a
specific I/O operation.  Its global value is normally @code{nil}, and
you should not globally set it to any other value.  Here is an example
of the right way to use the variable:

@example
;; @r{Read the file with no character code conversion.}
;; @r{Assume @acronym{crlf} represents end-of-line.}
(let ((coding-system-for-read 'emacs-mule-dos))
  (insert-file-contents filename))
@end example

When its value is non-@code{nil}, this variable takes precedence over
all other methods of specifying a coding system to use for input,
including @code{file-coding-system-alist},
@code{process-coding-system-alist} and
@code{network-coding-system-alist}.
@end defvar

@defvar coding-system-for-write
This works much like @code{coding-system-for-read}, except that it
applies to output rather than input.  It affects writing to files,
as well as sending output to subprocesses and net connections.

When a single operation does both input and output, as do
@code{call-process-region} and @code{start-process}, both
@code{coding-system-for-read} and @code{coding-system-for-write}
affect it.
@end defvar

@defvar inhibit-eol-conversion
When this variable is non-@code{nil}, no end-of-line conversion is done,
no matter which coding system is specified.  This applies to all the
Emacs I/O and subprocess primitives, and to the explicit encoding and
decoding functions (@pxref{Explicit Encoding}).
@end defvar

@node Explicit Encoding
@subsection Explicit Encoding and Decoding
@cindex encoding in coding systems
@cindex decoding in coding systems

  All the operations that transfer text in and out of Emacs have the
ability to use a coding system to encode or decode the text.
You can also explicitly encode and decode text using the functions
in this section.

  The result of encoding, and the input to decoding, are not ordinary
text.  They logically consist of a series of byte values; that is, a
series of @acronym{ASCII} and 8-bit characters.  In a multibyte buffer
or string, 8-bit characters have character codes higher than 255, but
Emacs effectively get their byte values on those operation.

  The usual way to read a file into a buffer as a sequence of bytes, so
you can decode the contents explicitly, is with
@code{insert-file-contents-literally} (@pxref{Reading from Files});
alternatively, specify a non-@code{nil} @var{rawfile} argument when
visiting a file with @code{find-file-noselect}.  These methods result in
a unibyte buffer.

  The usual way to use the byte sequence that results from explicitly
encoding text is to copy it to a file or process---for example, to write
it with @code{write-region} (@pxref{Writing to Files}), and suppress
encoding by binding @code{coding-system-for-write} to
@code{no-conversion}.

  Here are the functions to perform explicit encoding or decoding.  The
encoding functions produce sequences of bytes; the decoding functions
are meant to operate on sequences of bytes.  All of these functions
discard text properties.

@deffn Command encode-coding-region start end coding-system &optional destination
This command encodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}.  The encoded text replaces the
original text in the buffer.  The result of encoding is logically a
sequence of bytes, but the buffer remains multibyte if it was multibyte
before.

This command returns the length of the encoded text.

However, you can get the encoded text in the different way by
specifying non-@code{nil} value to @var{destination}.  If it is a
buffer, the encoded text is inserted in that buffer after point (point
does not move).  If it is t, this function returns the encoded text as
a unibyte string.
@end deffn

@defun encode-coding-string string coding-system &optional nocopy buffer
This function encodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
encoded text, except when @var{nocopy} is non-@code{nil}, in which
case the function may return @var{string} itself if the encoding
operation is trivial.  The result of encoding is a unibyte string.

However, if @var{buffer} is a buffer, the encoded text is inserted in
that buffer after point (point does not move).  In this case, the
return value is the length of the encoded text.
@end defun

@deffn Command decode-coding-region start end coding-system destination
This command decodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}.  The decoded text replaces the
original text in the buffer.  To make explicit decoding useful, the text
before decoding ought to be a sequence of byte values, but both
multibyte and unibyte buffers are acceptable.

This command returns the length of the decoded text.

However, you can get the decoded text in the different way by
specifying non-@code{nil} value to @var{destination}.  If it is a
buffer, the decoded text is inserted in that buffer after point (point
does not move).  If it is t, this function returns the decoded text as
a multibyte string.
@end deffn

@defun decode-coding-string string coding-system &optional nocopy buffer
This function decodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
decoded text, except when @var{nocopy} is non-@code{nil}, in which
case the function may return @var{string} itself if the decoding
operation is trivial.  To make explicit decoding useful, the contents
of @var{string} ought to be a sequence of byte values, but a multibyte
string is acceptable.

However, if @var{buffer} is a buffer, the decoded text is inserted in
that buffer after point (point does not move).  In this case, the
return value is the length of the decoded text.
@end defun

@defun decode-coding-inserted-region from to filename &optional visit beg end replace
This function decodes the text from @var{from} to @var{to} as if
it were being read from file @var{filename} using @code{insert-file-contents}
using the rest of the arguments provided.

The normal way to use this function is after reading text from a file
without decoding, if you decide you would rather have decoded it.
Instead of deleting the text and reading it again, this time with
decoding, you can call this function.
@end defun

@node Terminal I/O Encoding
@subsection Terminal I/O Encoding

  Emacs can decode keyboard input using a coding system, and encode
terminal output.  This is useful for terminals that transmit or display
text using a particular encoding such as Latin-1.  Emacs does not set
@code{last-coding-system-used} for encoding or decoding for the
terminal.

@defun keyboard-coding-system
This function returns the coding system that is in use for decoding
keyboard input---or @code{nil} if no coding system is to be used.
@end defun

@deffn Command set-keyboard-coding-system coding-system
This command specifies @var{coding-system} as the coding system to
use for decoding keyboard input.  If @var{coding-system} is @code{nil},
that means do not decode keyboard input.
@end deffn

@defun terminal-coding-system
This function returns the coding system that is in use for encoding
terminal output---or @code{nil} for no encoding.
@end defun

@deffn Command set-terminal-coding-system coding-system
This command specifies @var{coding-system} as the coding system to use
for encoding terminal output.  If @var{coding-system} is @code{nil},
that means do not encode terminal output.
@end deffn

@node MS-DOS File Types
@subsection MS-DOS File Types
@cindex DOS file types
@cindex MS-DOS file types
@cindex Windows file types
@cindex file types on MS-DOS and Windows
@cindex text files and binary files
@cindex binary files and text files

  On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
end-of-line conversion for a file by looking at the file's name.  This
feature classifies files as @dfn{text files} and @dfn{binary files}.  By
``binary file'' we mean a file of literal byte values that are not
necessarily meant to be characters; Emacs does no end-of-line conversion
and no character code conversion for them.  On the other hand, the bytes
in a text file are intended to represent characters; when you create a
new file whose name implies that it is a text file, Emacs uses DOS
end-of-line conversion.

@defvar buffer-file-type
This variable, automatically buffer-local in each buffer, records the
file type of the buffer's visited file.  When a buffer does not specify
a coding system with @code{buffer-file-coding-system}, this variable is
used to determine which coding system to use when writing the contents
of the buffer.  It should be @code{nil} for text, @code{t} for binary.
If it is @code{t}, the coding system is @code{no-conversion}.
Otherwise, @code{undecided-dos} is used.

Normally this variable is set by visiting a file; it is set to
@code{nil} if the file was visited without any actual conversion.
@end defvar

@defopt file-name-buffer-file-type-alist
This variable holds an alist for recognizing text and binary files.
Each element has the form (@var{regexp} . @var{type}), where
@var{regexp} is matched against the file name, and @var{type} may be
@code{nil} for text, @code{t} for binary, or a function to call to
compute which.  If it is a function, then it is called with a single
argument (the file name) and should return @code{t} or @code{nil}.

When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
which coding system to use when reading a file.  For a text file,
@code{undecided-dos} is used.  For a binary file, @code{no-conversion}
is used.

If no element in this alist matches a given file name, then
@code{default-buffer-file-type} says how to treat the file.
@end defopt

@defopt default-buffer-file-type
This variable says how to handle files for which
@code{file-name-buffer-file-type-alist} says nothing about the type.

If this variable is non-@code{nil}, then these files are treated as
binary: the coding system @code{no-conversion} is used.  Otherwise,
nothing special is done for them---the coding system is deduced solely
from the file contents, in the usual Emacs fashion.
@end defopt

@node Input Methods
@section Input Methods
@cindex input methods

  @dfn{Input methods} provide convenient ways of entering non-@acronym{ASCII}
characters from the keyboard.  Unlike coding systems, which translate
non-@acronym{ASCII} characters to and from encodings meant to be read by
programs, input methods provide human-friendly commands.  (@xref{Input
Methods,,, emacs, The GNU Emacs Manual}, for information on how users
use input methods to enter text.)  How to define input methods is not
yet documented in this manual, but here we describe how to use them.

  Each input method has a name, which is currently a string;
in the future, symbols may also be usable as input method names.

@defvar current-input-method
This variable holds the name of the input method now active in the
current buffer.  (It automatically becomes local in each buffer when set
in any fashion.)  It is @code{nil} if no input method is active in the
buffer now.
@end defvar

@defopt default-input-method
This variable holds the default input method for commands that choose an
input method.  Unlike @code{current-input-method}, this variable is
normally global.
@end defopt

@deffn Command set-input-method input-method
This command activates input method @var{input-method} for the current
buffer.  It also sets @code{default-input-method} to @var{input-method}.
If @var{input-method} is @code{nil}, this command deactivates any input
method for the current buffer.
@end deffn

@defun read-input-method-name prompt &optional default inhibit-null
This function reads an input method name with the minibuffer, prompting
with @var{prompt}.  If @var{default} is non-@code{nil}, that is returned
by default, if the user enters empty input.  However, if
@var{inhibit-null} is non-@code{nil}, empty input signals an error.

The returned value is a string.
@end defun

@defvar input-method-alist
This variable defines all the supported input methods.
Each element defines one input method, and should have the form:

@example
(@var{input-method} @var{language-env} @var{activate-func}
 @var{title} @var{description} @var{args}...)
@end example

Here @var{input-method} is the input method name, a string;
@var{language-env} is another string, the name of the language
environment this input method is recommended for.  (That serves only for
documentation purposes.)

@var{activate-func} is a function to call to activate this method.  The
@var{args}, if any, are passed as arguments to @var{activate-func}.  All
told, the arguments to @var{activate-func} are @var{input-method} and
the @var{args}.

@var{title} is a string to display in the mode line while this method is
active.  @var{description} is a string describing this method and what
it is good for.
@end defvar

  The fundamental interface to input methods is through the
variable @code{input-method-function}.  @xref{Reading One Event},
and @ref{Invoking the Input Method}.

@node Locales
@section Locales
@cindex locale

  POSIX defines a concept of ``locales'' which control which language
to use in language-related features.  These Emacs variables control
how Emacs interacts with these features.

@defvar locale-coding-system
@cindex keyboard input decoding on X
This variable specifies the coding system to use for decoding system
error messages and---on X Window system only---keyboard input, for
encoding the format argument to @code{format-time-string}, and for
decoding the return value of @code{format-time-string}.
@end defvar

@defvar system-messages-locale
This variable specifies the locale to use for generating system error
messages.  Changing the locale can cause messages to come out in a
different language or in a different orthography.  If the variable is
@code{nil}, the locale is specified by environment variables in the
usual POSIX fashion.
@end defvar

@defvar system-time-locale
This variable specifies the locale to use for formatting time values.
Changing the locale can cause messages to appear according to the
conventions of a different language.  If the variable is @code{nil}, the
locale is specified by environment variables in the usual POSIX fashion.
@end defvar

@defun locale-info item
This function returns locale data @var{item} for the current POSIX
locale, if available.  @var{item} should be one of these symbols:

@table @code
@item codeset
Return the character set as a string (locale item @code{CODESET}).

@item days
Return a 7-element vector of day names (locale items
@code{DAY_1} through @code{DAY_7});

@item months
Return a 12-element vector of month names (locale items @code{MON_1}
through @code{MON_12}).

@item paper
Return a list @code{(@var{width} @var{height})} for the default paper
size measured in millimeters (locale items @code{PAPER_WIDTH} and
@code{PAPER_HEIGHT}).
@end table

If the system can't provide the requested information, or if
@var{item} is not one of those symbols, the value is @code{nil}.  All
strings in the return value are decoded using
@code{locale-coding-system}.  @xref{Locales,,, libc, The GNU Libc Manual},
for more information about locales and locale items.
@end defun

@ignore
   arch-tag: be705bf8-941b-4c35-84fc-ad7d20ddb7cb
@end ignore

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 05 Nov 2008 21:27:54 +0900
> 
> Attached is the remaining part.  Please reflect that to the
> document.

Thanks, will do.

> I'm sorry for asking you this tiresome work.  I
> still can't establish a reliable internet connection for
> committing by myself.

Don't worry about it.

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 05 Nov 2008 21:27:54 +0900
> 
> Attached is the remaining part.  Please reflect that to the
> document.

Thanks.  I have a few questions about this:

      A character set is a set of characters, and it assigns a unique code
    point to each character belonging to the set.  Emacs decodes a
    specific code point of a specific character set to an Emacs character.

Does this mean a character set is equivalent to a coding-system,
meaning that a coding-system is a mapping between a character set and
the Emacs internal codepoints?

    @defun charset-dimension charset
    This function returns the dimension of @var{charset}.  Here, dimension
    means the number of bytes required to represent the highest code point
    (not an Emacs character code) of a character.  For example, the
    dimension of @code{iso-8859-1} is one, the dimension of
    @code{japanese-jisx0208} is two, and the dimension of @code{unicode}
    is three.
    @end defun

I decided not to document this.  I think the concept of charset
dimension is too obscure to explain, and not really needed for Lisp
programs, unless they need to define a new charset, or display a
charset, and those are already done by Emacs infrastructure.  Do you
see any problems with not documenting this function?

      A translation table has two extra slots.  The first is either
    @code{nil} or a translation table that performs the reverse
    translation; the second is the maximum number of characters to look up
    for translation.

Could you please elaborate on the second extra slot: when and for what
purpose would there be a need to look up characters for translation?

TIA

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uiqqfipn3.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

>       A character set is a set of characters, and it assigns a unique code
>     point to each character belonging to the set.  Emacs decodes a
>     specific code point of a specific character set to an Emacs character.

> Does this mean a character set is equivalent to a coding-system,
> meaning that a coding-system is a mapping between a character set and
> the Emacs internal codepoints?

No.  A coding-system is a mapping between a sequence of
characters and a sequence of bytes.  The byte sequence
contains a byte not mapped to a character. For instance,
iso-2022 uses escape sequence, UTF-16 uses surrogate pairs.

>     @defun charset-dimension charset
>     This function returns the dimension of @var{charset}.  Here, dimension
>     means the number of bytes required to represent the highest code point
>     (not an Emacs character code) of a character.  For example, the
>     dimension of @code{iso-8859-1} is one, the dimension of
>     @code{japanese-jisx0208} is two, and the dimension of @code{unicode}
>     is three.
>     @end defun

> I decided not to document this.  I think the concept of charset
> dimension is too obscure to explain, and not really needed for Lisp
> programs, unless they need to define a new charset, or display a
> charset, and those are already done by Emacs infrastructure.  Do you
> see any problems with not documenting this function?

I think no.

>       A translation table has two extra slots.  The first is either
>     @code{nil} or a translation table that performs the reverse
>     translation; the second is the maximum number of characters to look up
>     for translation.

> Could you please elaborate on the second extra slot: when and for what
> purpose would there be a need to look up characters for translation?

To enable sequence-to-char translation.  See the description
of make-translation-table-from-alist.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 26 Nov 2008 10:41:01 +0900
> 
> In article <uiqqfipn3.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> >       A character set is a set of characters, and it assigns a unique code
> >     point to each character belonging to the set.  Emacs decodes a
> >     specific code point of a specific character set to an Emacs character.
> 
> > Does this mean a character set is equivalent to a coding-system,
> > meaning that a coding-system is a mapping between a character set and
> > the Emacs internal codepoints?
> 
> No.  A coding-system is a mapping between a sequence of
> characters and a sequence of bytes.

Then how to understand this sentence you wrote:

  Emacs decodes a specific code point of a specific character set to
  an Emacs character.

Under what circumstances does this decoding happen?  The only type of
``decoding'' I knew about until now was by decode-coding-region and
friends, which use a coding-system to decode a byte sequence into
Emacs characters.

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uljv7gm4j.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > No.  A coding-system is a mapping between a sequence of
> > characters and a sequence of bytes.

> Then how to understand this sentence you wrote:

>   Emacs decodes a specific code point of a specific character set to
>   an Emacs character.

Here, "decode" doesn't mean "decode by a coding system".

> Under what circumstances does this decoding happen?  The only type of
> ``decoding'' I knew about until now was by decode-coding-region and
> friends, which use a coding-system to decode a byte sequence into
> Emacs characters.

For instance, to get a glyph-code of X font, we decode a
character by a charset with that the font encodes glyph
codes.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Kenichi Handa]

In article <E1L5Bwf-0006ku-Ci@etlken.m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> In article <uljv7gm4j.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > > No.  A coding-system is a mapping between a sequence of
> > > characters and a sequence of bytes.

I'll explain it a little bit more.  To decode a character
sequence to a byte sequence, Emacs actually does two kinds
of decoding as below:

             (1)                                (2)
characters <-----> (charset code-point) pairs <-----> bytes

For the decoding of (1), Emacs uses infomaiton of coding
system to decide which charset to use, and then uses
informaiton of the selected charset to get a code point.

For the decoding of (2) Emacs uses only information of
coding system.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Wed, 26 Nov 2008 13:58:26 +0900
> 
> I'll explain it a little bit more.  To decode a character
> sequence to a byte sequence, Emacs actually does two kinds
> of decoding as below:
> 
>              (1)                                (2)
> characters <-----> (charset code-point) pairs <-----> bytes

Can you give a couple of examples, for some popular charsets, and how
we decode bytes into characters thru these pairs of charsets and code
points?

> For the decoding of (1), Emacs uses infomaiton of coding
> system to decide which charset to use, and then uses
> informaiton of the selected charset to get a code point.
> 
> For the decoding of (2) Emacs uses only information of
> coding system.

Thanks.  What confuses me is that, roughly, there's a charset in Emacs
23 for every coding-system, and they both have almost identical names.
For example, the code point of a-umlaut in the iso-8859-1 charset is
exactly identical to the byte value produced by encoding that
character with iso-8859-1 coding-system.  So I wonder why we need
both in Emacs.  Why can't we, for example, decode bytes directly into
Emacs characters?

Re: Emacs 23 character code space

[Juanma Barranquero]

On Wed, Nov 26, 2008 at 21:26, Eli Zaretskii <eliz@gnu.org> wrote:

> Thanks.  What confuses me is that, roughly, there's a charset in Emacs
> 23 for every coding-system, and they both have almost identical names.

That's true for many charsets / coding systems, but for example you
have coding systems utf-8, utf16le-with-signature, utf-7, utf-16le,
utf-16be, utf-16be-with-signature, utf-16, utf-8-with-signature,
utf-8-auto, etc., all of which encode the Unicode charset.

             Juanma

Re: Emacs 23 character code space

[Stephen J. Turnbull]

Handa-san, konnichiwa.  Chotto shitsurei shimasu.  (Hi, Mr. Handa.
I'll be a little rude here in hope of saving you some time. :-)

Eli Zaretskii writes:
 > > From: Kenichi Handa <handa@m17n.org>
 > > CC: eliz@gnu.org, emacs-devel@gnu.org
 > > Date: Wed, 26 Nov 2008 13:58:26 +0900
 > > 
 > > I'll explain it a little bit more.  To decode a character
 > > sequence to a byte sequence, Emacs actually does two kinds
 > > of decoding as below:
 > > 
 > >              (1)                                (2)
 > > characters <-----> (charset code-point) pairs <-----> bytes
 > 
 > Can you give a couple of examples, for some popular charsets, and how
 > we decode bytes into characters thru these pairs of charsets and code
 > points?

(As you point out later, this is normally denoted 'encoding' in Mule
functions like `encode-coding-region'.  Encoding is actually the
harder case to explain because the sequence of bytes produced for a
given character is context-dependent, so let's look at the bytes -->
characters conversion.)

Let's begin at the beginning, with the number 1 represented by the
Japanese ideograph ichi in the iso-2022-jp coding system, in an HTML
header element alone on a line.  As octets viewed as ASCII[1] it looks
like this in a file

< H 1 > ESC $ B 0 l ESC ( B < / H 1 > ^J

where spaces are represented by SPC (ie, there are no significant
spaces on the line).

(0) We decide to decode using the ISO-2022-JP, which allows multiple
    charsets (at least ASCII, Japanese JIS X 0208 and JIS X 0212, and
    explicitly extended to other registered charsets as ISO-2022-INT).
(1) We initialize current charset to ascii as mandated by the RFC
    defining ISO-2022-JP.
(2) We collect the octet < and translate it to a pair (which is
    trivial -- apply the identity to the octet, and pair it with
    'ascii) giving `(ascii <)'
(3) We convert to a Mule character (which is an integer) with
    "(csetidx(ascii) << 7) + ?<".
(4) We repeat (2) and (3) for H, 1, and >.
(5) We see the control sequence ESC $ B, and switch our current
    charset to japanese-jisx0208.
(6) JIS is a dimension 2 charset, so we collect two octets 0 and l,
    and form the pair (japanese-jisx0208 (0 l)).
(7) We convert to a Mule character (which is an integer) with
    "(csetidx(japanese-jisx0208) << 14) + ?0 << 7 + ?l".
(8) We see the control sequence ESC ( B, and switch our current
    charset to ascii.
(9) We repeat (2) and (3) for <, /, H, 1, >, and ^J.
(10) We are now done with the line, and we are ready to read more
     ASCII *or an escape sequence*.

So the full process (1) -- (10) corresponds to the coding system
ISO-2022-JP, while the process (2) -- (3) corresponds to the charset
ascii, and the process (6) -- (7) to the charset japanese-jisx0208.

 > Thanks.  What confuses me is that, roughly, there's a charset in Emacs
 > 23 for every coding-system, and they both have almost identical names.

No, not even close.  It happens that people who use ISO 8859 coded
character sets generally only see such coding systems (ie, unibyte
uniscript), so you would get that impression, as the coding systems
for those encodings are trivial.

However, for Unicode (as Juanma points out) the charset is always
Unicode (I don't know how that is spelled in Emacs, and XEmacs
technically doesn't have such a charset currently), but various coding
systems such as UTF-8, UTF-16, and UTF-32, in big- and little-endian
are used to decode bytes to code points of Unicode.  For the East
Asian coding systems and legacy ISO 2022 multiscript texts (ISO 2022
can handle not only Japanese, but also mixtures of, say, the Russian
ISO-8859-5 and Hebrew ISO-8859-8 repertoires), it's even more
complicated.

Note that because of the need to handle ASCII in programming, all of
the Asian coding systems must be multiscript, handling at least ASCII
and the national character set(s).

 > For example, the code point of a-umlaut in the iso-8859-1 charset is
 > exactly identical to the byte value produced by encoding that
 > character with iso-8859-1 coding-system.  So I wonder why we need
 > both in Emacs.  Why can't we, for example, decode bytes directly into
 > Emacs characters?

Because each byte is overloaded.  In the process above with the escape
sequences, we decode to a Mule string "<H1>[ichi]</H1>\n", while
without the escape sequences we would get "<H1>0l</H1>\n", also known
as "mojibake" (with a little bit of poetic license, the Japanese for
"monster characters").

Footnotes: 
[1]  This kind of ASCII-compatibility is deliberate in ISO 2022.

Re: Emacs 23 character code space

[Kenichi Handa]

In article <8763maq8hg.fsf@uwakimon.sk.tsukuba.ac.jp>, "Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Handa-san, konnichiwa.  Chotto shitsurei shimasu.  (Hi, Mr. Handa.
> I'll be a little rude here in hope of saving you some time. :-)

Hi, thank you for the detailed explanation.  I should have
read it before I replied to Eli.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 26 Nov 2008 13:24:09 +0900
> 
> >   Emacs decodes a specific code point of a specific character set to
> >   an Emacs character.
> 
> Here, "decode" doesn't mean "decode by a coding system".
> 
> > Under what circumstances does this decoding happen?  The only type of
> > ``decoding'' I knew about until now was by decode-coding-region and
> > friends, which use a coding-system to decode a byte sequence into
> > Emacs characters.
> 
> For instance, to get a glyph-code of X font, we decode a
> character by a charset with that the font encodes glyph
> codes.

But that's not really "decoding", is it?  By "decoding" we usually
mean conversion _to_ the Emacs internal representation, whereas in
your example, we convert _from_ the internal representation to some
other.

To avoid confusion, I suggest to talk about "conversion" of Emacs
characters to code points of a charset.  Do you agree?

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uiqqags1p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > For instance, to get a glyph-code of X font, we decode a
> > character by a charset with that the font encodes glyph
> > codes.

> But that's not really "decoding", is it?  By "decoding" we usually
> mean conversion _to_ the Emacs internal representation, whereas in
> your example, we convert _from_ the internal representation to some
> other.

Oops, sorry, I myself confused decoding and encoding.  Yes
the above is encoding.  And I did the same mistake in my
followup mail.

> To avoid confusion, I suggest to talk about "conversion" of Emacs
> characters to code points of a charset.  Do you agree?

As we have functions encode-char and decode-char, I think it
is better to keep using the words "encoding" and "decoding"
for both kind of conversions; i.e. character <->
(charset . code-point), and string/buffer <-> byte-sequence.

> > From: Kenichi Handa <handa@m17n.org>
[...]
> > I'll explain it a little bit more.  To decode a character
> > sequence to a byte sequence, Emacs actually does two kinds
> > of decoding as below:

As I wrote above, I made a mistake here.  So, I'll
paraphrase it as below.

To convert between a character sequence and a byte sequence,
Emacs actually does two steps of conversions as below.


characters --(1)-> (charset code-point) pairs --(3)-> bytes
           <-(2)--                            <-(4)--     

For the encoding of (1), Emacs uses infomaiton of coding
system to decide which charset to use, and then uses
informaiton of the selected charset to get a code point.
For the decoding of (2), Emacs uses informaiton of charset
to get character codes. 

For the encoding of (3) and the decoding of (4), Emacs uses
only information of coding system.

> Can you give a couple of examples, for some popular charsets, and how
> we decode bytes into characters thru these pairs of charsets and code
> points?

Ok.

Ex.1  utf-8

(1) and (2) are straight forward because charset is
`unicode' and Emacs character code and the code-point in
`unicode' are the same.  (3) decodes each (unicode
CODE-POINT) to utf-8 byte sequence, (4) does the reverse
conversion.

 "a\x3042x" -(1)-> (unicode #x61) (unicode #x3042) (unicode #x78)
            -(3)-> "#x61 #xE3 #x81 #x82 #x78"

Ex.2 iso-8859-2

(1) encodes each charater to code points of the charset
iso-8859-2 by the information of that charset, and (2) does
the reverse conversion.  (3) and (4) are straight forward
because the code-point sequence and the byte sequence are
the same.

Ex.3 iso-2022-jp (japanese)

(1) at first decides which charset (among what supported by
iso-2022-jp) to use for each character, and then encode the
charater to the correspoding (charset code-point) pair.  (2)
does the decoding using information of charset only.  (3)
generates a byte sequence from each code-point (one byte for
a charset of dimension 1, two bytes for a charset of
dimension 2), and also inserts a proper designation byte
sequence at charset boundary.
 "a\x3042x" -(1)-> (ascii #x61) (japanese-jisx0208 #x2422) (aciii #x78)
            -(3)-> "#x61 ESC $ B #x24 #x22 ESC ( B #x78"

Ex.4 gb2312 (chinese)

 "a\x3042x" -(1)-> (ascii #x61) (chinese-gb2312 #x2422) (aciii #x78)
            -(3)-> "#x61 #xA4 #xA2 #x78"

> Thanks.  What confuses me is that, roughly, there's a charset in Emacs
> 23 for every coding-system, and they both have almost identical names.

But there are coding-systems that have multiple charsets.
For instance, big5 coding-system support both ASCII and BIG5
charsets, iso-2022-7bit supports many many charsets.

> For example, the code point of a-umlaut in the iso-8859-1 charset is
> exactly identical to the byte value produced by encoding that
> character with iso-8859-1 coding-system.  So I wonder why we need
> both in Emacs.  Why can't we, for example, decode bytes directly into
> Emacs characters?

Getting a code point from byte sequence and getting a
character code from a code point are different generally
(the above example of iso-8859-1 is rather rare example).  I
hope you understand why by seeing the above examples.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Thu, 27 Nov 2008 10:29:50 +0900
> 
> To convert between a character sequence and a byte sequence,
> Emacs actually does two steps of conversions as below.
> 
> 
> characters --(1)-> (charset code-point) pairs --(3)-> bytes
>            <-(2)--                            <-(4)--     
> 
> For the encoding of (1), Emacs uses infomaiton of coding
> system to decide which charset to use, and then uses
> informaiton of the selected charset to get a code point.
> For the decoding of (2), Emacs uses informaiton of charset
> to get character codes. 
> 
> For the encoding of (3) and the decoding of (4), Emacs uses
> only information of coding system.

Thanks.

However, does this two-step procedure make a difference for a Lisp
programmer?  It sounds like this is an internal implementation detail
that happens transparently, so Lisp programmers shouldn't care about
it.

My original confusion was about when character sets are exposed to
Lisp application programs.  I understand that one situation is when
Emacs displays a character with some font.  Are there other
situations?

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uljv2foc1.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> My original confusion was about when character sets are exposed to
> Lisp application programs.  I understand that one situation is when
> Emacs displays a character with some font.  Are there other
> situations?

Arguments to these functions contains a charset or a list of
charsets:
  define-coding-system, map-charset-chars

In addition, I think "charset" is a very common concept for
users, and M-x list-charset-chars RET is a convenient
command.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 26 Nov 2008 10:41:01 +0900
> 
> >       A translation table has two extra slots.  The first is either
> >     @code{nil} or a translation table that performs the reverse
> >     translation; the second is the maximum number of characters to look up
> >     for translation.
> 
> > Could you please elaborate on the second extra slot: when and for what
> > purpose would there be a need to look up characters for translation?
> 
> To enable sequence-to-char translation.  See the description
> of make-translation-table-from-alist.

Thanks.  Should we also document define-translation-table?

Re: Emacs 23 character code space

[Kenichi Handa]

In article <u3ahcgf8h.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Kenichi Handa <handa@m17n.org>
> > CC: emacs-devel@gnu.org
> > Date: Wed, 26 Nov 2008 10:41:01 +0900
> > 
> > >       A translation table has two extra slots.  The first is either
> > >     @code{nil} or a translation table that performs the reverse
> > >     translation; the second is the maximum number of characters to look up
> > >     for translation.
> > 
> > > Could you please elaborate on the second extra slot: when and for what
> > > purpose would there be a need to look up characters for translation?
> > 
> > To enable sequence-to-char translation.  See the description
> > of make-translation-table-from-alist.

> Thanks.  Should we also document define-translation-table?

Perhaps not because that was necessary for CCL, but
nowadays, CCL itself is not that necessary..

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Tue, 02 Dec 2008 14:44:26 +0900
> 
> > Thanks.  Should we also document define-translation-table?
> 
> Perhaps not because that was necessary for CCL, but
> nowadays, CCL itself is not that necessary..

Yes, that's what I thought as well.

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 05 Nov 2008 21:27:54 +0900
> 
> > > I'll continue to fix the document after that section.
> 
> Attached is the remaining part.  Please reflect that to the
> document.  I'm sorry for asking you this tiresome work.  I
> still can't establish a reliable internet connection for
> committing by myself.

Thanks, I've now incorporated this into the CVS version.

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Mon, 03 Nov 2008 21:45:20 +0900
> 
> I tried to rewrite nonascii.texi to clear the things.  I
> finished upto the "Character Code" section as attached.
> What do you think about it?

Thanks!

I have a few questions:

      Emacs can convert unibyte text to multibyte; it can also convert
    multibyte text to unibyte provided that the multibyte text contains
    only @acronym{ASCII} and 8-bit characters.

What exactly is meant here by ``8-bit characters''?  Do you mean
eight-bit raw bytes, or do you mean Unicode characters whose
codepoints are below 256?

      Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
    unchanged, and converts 8-bit characters (codes 128 through 159) to
    the corresponding representation for multibyte text.

Again, by ``8-bit characters'' you mean raw 8-bit bytes here, right?

    @defun string-to-multibyte string
    This function returns a multibyte string containing the same sequence
    of characters as @var{string}.  If @var{string} is a multibyte string,
    it is returned unchanged.
    @end defun

I'm not sure I understand the effect of this function.  Does it decode
its argument, converting each byte to the corresponding internal
representation of the encoded single-byte character?  I think this is
not what it does, but then what does it do?

    @defun string-to-unibyte string
    This function returns a unibyte string containing the same sequence of
    characters as @var{string}.  It signals an error if @var{string}
    contains a non-@acronym{ASCII} character.  If @var{string} is a
    unibyte string, it is returned unchanged.
    @end defun

Since this function handles any non-ASCII characters lossily, when
would it be useful?

    @defun multibyte-char-to-unibyte char
    This convert the multibyte character @var{char} to a unibyte
    character.  If @var{char} is a non-@acronym{ASCII} character, the
    value is -1.
    @end defun

    @defun unibyte-char-to-multibyte char
    This convert the unibyte character @var{char} to a multibyte
    character.
    @end defun

Again, when are these functions useful?

Re: Emacs 23 character code space

[Stefan Monnier]

> What exactly is meant here by ``8-bit characters''?  Do you mean
> eight-bit raw bytes, or do you mean Unicode characters whose
> codepoints are below 256?

It should be eight-bit raw bytes.  In some cases it's difficult to tell
the difference, so Emacs may occasionally accept latin-1 chars as stand
ins for eight-bit raw bytes.

>     Converting unibyte text to multibyte text leaves @acronym{ASCII}
>     characters unchanged, and converts 8-bit characters (codes 128
>     through 159) to the corresponding representation for
>     multibyte text.
> Again, by ``8-bit characters'' you mean raw 8-bit bytes here, right?

Yes.

I think we should state somewhere that unibyte strings and buffers
contain bytes only.  And that multibyte strings and buffers contain
chars.  And that bytes are a subset of chars.

>     @defun string-to-multibyte string
>     This function returns a multibyte string containing the same sequence
>     of characters as @var{string}.  If @var{string} is a multibyte string,
>     it is returned unchanged.
>     @end defun

> I'm not sure I understand the effect of this function.

It returns a string containing the same bytes (in the sense of
ASCII+eight-bit, not in the sense of the underlying internal
representation, which we should as much as possible not mention
anywhere) but in a multibyte string instead.  I.e. the output is
a multibyte string of the same length whose chars are bytes.

>     @defun string-to-unibyte string
>     This function returns a unibyte string containing the same sequence of
>     characters as @var{string}.  It signals an error if @var{string}
>     contains a non-@acronym{ASCII} character.  If @var{string} is a
>     unibyte string, it is returned unchanged.
>     @end defun

> Since this function handles any non-ASCII characters lossily, when
> would it be useful?

I think the "non-ASCII" part is incorrect.  It probably should say
"non-byte char" instead.  It's useful when you have a multibyte string
which you (think you) know only holds bytes.

In 99% (actually 99.99999% for the `as' case) of the cases you shouldn't
use string-{as/make/to}-{uni/multi}byte.  Instead you should use
{en/de}code-coding-string.

>     @defun multibyte-char-to-unibyte char
>     This convert the multibyte character @var{char} to a unibyte
>     character.  If @var{char} is a non-@acronym{ASCII} character, the
>     value is -1.
>     @end defun

>     @defun unibyte-char-to-multibyte char
>     This convert the unibyte character @var{char} to a multibyte
>     character.
>     @end defun

> Again, when are these functions useful?

Rarely.


        Stefan

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 22 Nov 2008 23:16:49 -0500
> Cc: emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
> 
> I think we should state somewhere that unibyte strings and buffers
> contain bytes only.  And that multibyte strings and buffers contain
> chars.  And that bytes are a subset of chars.

Please take a look at the current version of nonascii.texi in CVS, I
already did state this.  Specific suggestions for improvement are
welcome, of course.

(The text I was quoting was the original one written by Handa-san, not
the one I put into the manual.)

> >     @defun string-to-multibyte string
> >     This function returns a multibyte string containing the same sequence
> >     of characters as @var{string}.  If @var{string} is a multibyte string,
> >     it is returned unchanged.
> >     @end defun
> 
> > I'm not sure I understand the effect of this function.
> 
> It returns a string containing the same bytes (in the sense of
> ASCII+eight-bit, not in the sense of the underlying internal
> representation, which we should as much as possible not mention
> anywhere) but in a multibyte string instead.  I.e. the output is
> a multibyte string of the same length whose chars are bytes.

So you are in effect saying that the effect of this function is only
well defined for a string that holds ASCII characters and raw 8-bit
bytes?

> >     @defun string-to-unibyte string
> >     This function returns a unibyte string containing the same sequence of
> >     characters as @var{string}.  It signals an error if @var{string}
> >     contains a non-@acronym{ASCII} character.  If @var{string} is a
> >     unibyte string, it is returned unchanged.
> >     @end defun
> 
> > Since this function handles any non-ASCII characters lossily, when
> > would it be useful?
> 
> I think the "non-ASCII" part is incorrect.  It probably should say
> "non-byte char" instead.

"Non-ASCII characters" here does not mean "anything but ASCII
characters", it means "any character except ASCII and raw 8-bit
bytes" (assuming I understand the text correctly).  I will make sure
this tricky distinction is clear in the manual.

> In 99% (actually 99.99999% for the `as' case) of the cases you shouldn't
> use string-{as/make/to}-{uni/multi}byte.  Instead you should use
> {en/de}code-coding-string.

This specific section is not about en/decoding text, it's about
converting between unibyte and multibyte.  Unless we want to remove
any mention of these capabilities (and leave Lisp programmers without
any documentation on how to handle binary data and/or byte streams of
undecoded text), I don't think we can remove the description of these
functions from the manual.

Re: Emacs 23 character code space

[Kenichi Handa]

In article <jwvbpw7dr2w.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > What exactly is meant here by ``8-bit characters''?  Do you mean
> > eight-bit raw bytes, or do you mean Unicode characters whose
> > codepoints are below 256?

> It should be eight-bit raw bytes.  In some cases it's difficult to tell
> the difference, so Emacs may occasionally accept latin-1 chars as stand
> ins for eight-bit raw bytes.

I classified characters into ASCII chars, non-ASCII chars,
and 8-bit chars at the beginning of "@node Text
Representations".  And, unibyte text can contain ASCII and
8-bit chars, multibyte text can contain all chars.

> I think we should state somewhere that unibyte strings and buffers
> contain bytes only.  And that multibyte strings and buffers contain
> chars.  And that bytes are a subset of chars.

I'm not sure that is good as far as the Lisp reader returns
a unibyte string from "abc".

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Ulrich Mueller]

>>>>> On Sat, 22 Nov 2008, Eli Zaretskii wrote:

>       Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
>     unchanged, and converts 8-bit characters (codes 128 through 159) to
>     the corresponding representation for multibyte text.

> Again, by ``8-bit characters'' you mean raw 8-bit bytes here, right?

Also, shouldn't it be "codes 128 through 255"? Or am I missing
something?

Ulrich

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@kph.uni-mainz.de>
> Date: Sun, 23 Nov 2008 09:29:57 +0100
> Cc: emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
> 
> >>>>> On Sat, 22 Nov 2008, Eli Zaretskii wrote:
> 
> >       Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
> >     unchanged, and converts 8-bit characters (codes 128 through 159) to
> >     the corresponding representation for multibyte text.
> 
> > Again, by ``8-bit characters'' you mean raw 8-bit bytes here, right?
> 
> Also, shouldn't it be "codes 128 through 255"? Or am I missing
> something?

Yes, you are missing the fact that 160-255 are valid 8859 code
points.  The text to which I referred is talking about 8-bit raw bytes
in the range 128-159, where there are no 8859 code points.

Re: Emacs 23 character code space

[Ulrich Mueller]

>>>>> On Sun, 23 Nov 2008, Eli Zaretskii wrote:

>> Also, shouldn't it be "codes 128 through 255"? Or am I missing
>> something?

> Yes, you are missing the fact that 160-255 are valid 8859 code
> points.  The text to which I referred is talking about 8-bit raw bytes
> in the range 128-159, where there are no 8859 code points.

Hm, strictly speaking these _are_ valid code points of ISO-8859-x
(aka ISO_8859-x:1987), only not assigned to displayable characters.

But of course you are right, I was missing the context of the cited
paragraph.

Ulrich

Re: Emacs 23 character code space

[Stefan Monnier]

> Also, shouldn't it be "codes 128 through 255"? Or am I missing
> something?

You're missing the fact that codes 128-255 can be one of two different
things, depending on context:
- for "unibyte chars", it's an "eight-bit" char (a byte)
- for "multibyte chars", it's a latin-1 char (not a byte).

See multibyte-char-to-unibyte and unibyte-char-to-multibyte.


        Stefan

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uk5aviv36.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

>       Emacs can convert unibyte text to multibyte; it can also convert
>     multibyte text to unibyte provided that the multibyte text contains
>     only @acronym{ASCII} and 8-bit characters.

> What exactly is meant here by ``8-bit characters''?  Do you mean
> eight-bit raw bytes, or do you mean Unicode characters whose
> codepoints are below 256?

The former; more precisely, characters representing
eight-bit raw bytes.  They have different character codes in
multibyte text (#x3FFF80..#x3FFFFF) and unibyte text
(#x80..#xFF).

>       Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
>     unchanged, and converts 8-bit characters (codes 128 through 159) to
>     the corresponding representation for multibyte text.

> Again, by ``8-bit characters'' you mean raw 8-bit bytes here, right?

Yes.

>     @defun string-to-multibyte string
>     This function returns a multibyte string containing the same sequence
>     of characters as @var{string}.  If @var{string} is a multibyte string,
>     it is returned unchanged.
>     @end defun

> I'm not sure I understand the effect of this function.  Does it decode
> its argument, converting each byte to the corresponding internal
> representation of the encoded single-byte character?  I think this is
> not what it does, but then what does it do?

No, all 8-bit characters (#x80..#xFF) in the source unibyte
string is converted to the multibyte representation of those
8-bit characters (#x3FFF80..#x3FFFFF).

>     @defun string-to-unibyte string
>     This function returns a unibyte string containing the same sequence of
>     characters as @var{string}.  It signals an error if @var{string}
>     contains a non-@acronym{ASCII} character.  If @var{string} is a
>     unibyte string, it is returned unchanged.
>     @end defun

> Since this function handles any non-ASCII characters lossily, when
> would it be useful?

If you know that a string containts only ASCII or 8-bit
characters, you can use it to get a unibyte string without
loosing information.

>     @defun multibyte-char-to-unibyte char
>     This convert the multibyte character @var{char} to a unibyte
>     character.  If @var{char} is a non-@acronym{ASCII} character, the
>     value is -1.
>     @end defun

>     @defun unibyte-char-to-multibyte char
>     This convert the unibyte character @var{char} to a multibyte
>     character.
>     @end defun

> Again, when are these functions useful?

Perhaps, we don't need them anymore.  We can use get-byte.
Anyway, the relationship of they and
string-to-unibyte/multibyte is this:

(defun string-to-unibyte (str)
  (let ((new (make-string (length str) 0)))
    (dotimes (i (length str))
      (let ((byte (multibyte-char-to-unibyte (aref str i))))
	(if (< byte 0)
	    (error))
	(aset new i byte)))
    new))

(defun string-to-multibyte (str)
  (let ((new (make-string (length str) 0)))
    (dotimes (i (length str))
      (aset new i (unibyte-char-to-multibyte (aref str i))))
    new))

---
Kenichi Handa
handa@ni.aist.go.jp

New function: what-file-line, used when writing gdb script

[richardeng]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

Hi all,

When I write gdb script, I need to set many breakpoints. such as:
b xxx_file.c:xxx_line_number

With following comamnd, I can do it more easy
-----
(defcustom what-file-line-separator ":"
  "Define the separator between file name and line number"
  :type 'string
  :group 'editing)

(defcustom what-file-line-yankp nil
  "Toggle on/off Yank to kill ring"
  :type 'boolean
  :group 'editing)

;; Maybe this variable is useless, user can copy what they want in mini-buffer                                  
(defcustom what-file-line-fullpath t
  "Toggle on/off file name fullpath"
  :type 'boolean
  :group 'editing)

(defun what-file-line ()
  "Print the current buffer's file name and line nubmer"
  (interactive)
  (let ((n (line-number-at-pos))
        (file (buffer-file-name))
        result)
    (setq result (concat file what-file-line-separator (number-to-string n)))
    (message "%s" result)
    (if what-file-line-yankp
        (kill-new result))))
-----
Q:
1. I don't know in which group the customized variables should be put?
2. The 3rd customized variable needn't, agree?
3. If this functionality is usefull, I want to implement a GUI version(as same as gdb-mouse-set-clear-breakpoint). So when user click the left margin when non-gdb-mode, user can get the filename-line pair)
 
What's your comment?


2008-11-23 



richardeng 

[-- Attachment #2: Type: text/html, Size: 5299 bytes --]

Re: Emacs 23 character code space

[Kenichi Handa]

I forgot to explain this.

In article <u63n7wmri.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> This fragment from etc/NEWS:
>     The character code space is now 0x0..0x3FFFFF with no gap.
>     Characters of code 0x0..0x10FFFF are Unicode characters of the same code points.
>     Characters of code 0x3FFF80..0x3FFFFF are raw 8-bit bytes.

> seems to contradict itself: it says there's ``no gap'', but the codes
> between 0x110000 and 0x3FFF7F do constitute a gap, don't they?

Any code between 0x110000 and 0x3FFF7F are valid as
character.  Some of them correspond to a certain character
set, some are not.  For the latter, only utf-emacs can
encode, and they can't be displayed with any font.  But
still they are valid as characters.  For instance, you can
set any syntax/category/char-code-property to them.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Emacs 23 character code space

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Fri, 07 Nov 2008 16:21:00 +0900
> 
> > This fragment from etc/NEWS:
> >     The character code space is now 0x0..0x3FFFFF with no gap.
> >     Characters of code 0x0..0x10FFFF are Unicode characters of the same code points.
> >     Characters of code 0x3FFF80..0x3FFFFF are raw 8-bit bytes.
> 
> > seems to contradict itself: it says there's ``no gap'', but the codes
> > between 0x110000 and 0x3FFF7F do constitute a gap, don't they?
> 
> Any code between 0x110000 and 0x3FFF7F are valid as
> character.  Some of them correspond to a certain character
> set, some are not.  For the latter, only utf-emacs can
> encode, and they can't be displayed with any font.  But
> still they are valid as characters.  For instance, you can
> set any syntax/category/char-code-property to them.

Thanks.  But I'm still a bit in the dark, as for how to describe this
correctly and concisely.  Do we actually use the range of codes
between 0x110000 and 0x3FFF7F?  If so, for what characters?  If we do
not use them now, are there some plans for using them in the future?

Re: Emacs 23 character code space

[Kenichi Handa]

In article <uy6zvsufb.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  But I'm still a bit in the dark, as for how to describe this
> correctly and concisely.  Do we actually use the range of codes
> between 0x110000 and 0x3FFF7F?  If so, for what characters?  If we do
> not use them now, are there some plans for using them in the future?

I wrote about the usage of that area in the thread "size of
emacs executable after unicode merge" as below.

> (0) At first, Emacs assigns a unique linear character code
>     space in upper Unicode area (#x110000-) to each big
>     character set (e.g. GB, JIS, KSC) (*see the note at the
>     tail).  The decoding of a character of a specific
>     charset into this area is quite fast (done just by a few
>     steps of arithmetic calculation).  Encoding is the same
>     too.
[...]
> *Note:
> 
> The reason Emacs assigns those linear area is because such
> big charsets tend to have their own private use area, and we
> must keep a unique characte code for them.  Those private
> characters are decoded and encoded without being mapped to
> Unicode are.

For example, the charset `japanese-jisx0208' is defined as
below.

(define-charset 'japanese-jisx0208
  "JISX0208.1983/1990 Japanese Kanji: ISO-IR-87"
  :short-name "JISX0208"
  :long-name "JISX0208.1983/1990 (Japanese): ISO-IR-87"
  :iso-final-char ?B
  :emacs-mule-id 146
  :code-space [33 126 33 126]
  :code-offset #x140000
  :unify-map "JISX0208")

So, for that charset, the code space for 8836 (=94x94)
characters are preserved in that upper area linearly from
#x140000.  Then, most of the characters are mapped into
Unicode area by the map "JISX0208".  The remaining
characters stay in this upper area.

---
Kenichi Handa
handa@ni.aist.go.jp