ucs-normalize and diacritics

ucs-normalize and diacritics

[Robert Pluim]

Hi,

in
<https://emacs.stackexchange.com/questions/43757/combining-diacritics-not-combining-visually>
someone noted that â (that is LATIN SMALL LETTER A followed by
COMBINING CIRCUMFLEX ACCENT) was not visually shown as â (LATIN SMALL
LETTER A WITH CIRCUMFLEX). I can play around with the various
ucs-normalize-* functions and get that to happen, but Iʼm just curious
why it doesnʼt happen automatically (assuming it should, of course. I
couldn't find any documentation on this kind of stuff).

Thanks

Robert

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Tue, 24 Jul 2018 20:17:00 +0200
> 
> in
> <https://emacs.stackexchange.com/questions/43757/combining-diacritics-not-combining-visually>
> someone noted that â (that is LATIN SMALL LETTER A followed by
> COMBINING CIRCUMFLEX ACCENT) was not visually shown as â (LATIN SMALL
> LETTER A WITH CIRCUMFLEX). I can play around with the various
> ucs-normalize-* functions and get that to happen, but Iʼm just curious
> why it doesnʼt happen automatically (assuming it should, of course.

It does display as â, but only if you use a font that supports both
'a' and '̂'.  Emacs can only compose characters that are displayed by
the same font.

> I couldn't find any documentation on this kind of stuff).

Not sure what documentation you were looking for, but the data
structure to make this happen is set up in composite.el:

  (when unicode-category-table
    (let ((elt `([,(purecopy "\\c.\\c^+") 1 compose-gstring-for-graphic]
		 [nil 0 compose-gstring-for-graphic])))
      (map-char-table
       #'(lambda (key val)
	   (if (memq val '(Mn Mc Me))
	       (set-char-table-range composition-function-table key elt)))
       unicode-category-table))
    ;; for dotted-circle
    (aset composition-function-table #x25CC
	  `([,(purecopy ".\\c^") 0 compose-gstring-for-dotted-circle])))

That "\\c.\\c^+" regexp causes any base character (\c.) followed by
one or more combining characters (\c^) to be composable using the
function compose-gstring-for-graphic (which see).

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> Date: Tue, 24 Jul 2018 22:07:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > I couldn't find any documentation on this kind of stuff).
> 
> Not sure what documentation you were looking for

Maybe what you were looking for is the documentation of
auto-composition-mode and composition-function-table which it
mentions.

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 24 Jul 2018 22:07:00 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > I couldn't find any documentation on this kind of stuff).
>> 
>> Not sure what documentation you were looking for
>
> Maybe what you were looking for is the documentation of
> auto-composition-mode and composition-function-table which it
> mentions.

Yes, that help. I'd searched for various combinations of 'combining'
and 'diacritic' but didnʼt find anything.

auto-composition-mode seems not to be documented anywhere other than
its doc string.

Thanks

Robert

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Date: Tue, 24 Jul 2018 20:17:00 +0200
>> 
>> in
>> <https://emacs.stackexchange.com/questions/43757/combining-diacritics-not-combining-visually>
>> someone noted that â (that is LATIN SMALL LETTER A followed by
>> COMBINING CIRCUMFLEX ACCENT) was not visually shown as â (LATIN SMALL
>> LETTER A WITH CIRCUMFLEX). I can play around with the various
>> ucs-normalize-* functions and get that to happen, but Iʼm just curious
>> why it doesnʼt happen automatically (assuming it should, of course.
>
> It does display as â, but only if you use a font that supports both
> 'a' and '̂'.  Emacs can only compose characters that are displayed by
> the same font.

That works fine for me if I set the font used for ̂ as my default
font. Would it be worth allowing emacs to change the font of the
preceding character to match the diacritic and (re-)attempt
composition?

Thanks

Robert

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 24 Jul 2018 22:48:50 +0200
> 
> auto-composition-mode seems not to be documented anywhere other than
> its doc string.

Character composition is one of the few areas in Emacs that are
notoriously under-documented.  Patches to document that are most
welcome (let me know if I can help by pointing out to its other
interesting aspects).

Thanks.

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 24 Jul 2018 22:54:46 +0200
> 
> > It does display as â, but only if you use a font that supports both
> > 'a' and '̂'.  Emacs can only compose characters that are displayed by
> > the same font.
> 
> That works fine for me if I set the font used for ̂ as my default
> font. Would it be worth allowing emacs to change the font of the
> preceding character to match the diacritic and (re-)attempt
> composition?

Theoretically, yes.  But it could mean a serious surgery in how
character composition is integrated into the display engine, and there
are also usability and UI aspects that need to be considered, because
the user might have good reasons to use the font she uses for the base
character.

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 24 Jul 2018 22:48:50 +0200
>> 
>> auto-composition-mode seems not to be documented anywhere other than
>> its doc string.
>
> Character composition is one of the few areas in Emacs that are
> notoriously under-documented.  Patches to document that are most
> welcome (let me know if I can help by pointing out to its other
> interesting aspects).

I think Iʼll start by putting pointers to auto-composition-mode in the
manual and lispref.

Regards

Robert

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> Date: Wed, 25 Jul 2018 05:36:52 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Would it be worth allowing emacs to change the font of the
> > preceding character to match the diacritic and (re-)attempt
> > composition?
> 
> Theoretically, yes.  But it could mean a serious surgery in how
> character composition is integrated into the display engine, and there
> are also usability and UI aspects that need to be considered, because
> the user might have good reasons to use the font she uses for the base
> character.

I've looked at the related code, and it least the first part doesn't
seem to be too hard: it just needs expanding some APIs.  The relevant
function is autocmp_chars, called from composition_reseat_it, which is
called by the CHAR_COMPOSED_P macro.  The only problem to solve is how
to determine what font to use if the composition includes more than 2
characters, and Emacs selects more than 2 fonts to display them (e.g.,
3-character composition, each character needs a different font).

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 25 Jul 2018 09:40:34 +0200
> 
> I think Iʼll start by putting pointers to auto-composition-mode in the
> manual and lispref.

Thanks in advance.

Re: ucs-normalize and diacritics

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Robert Pluim <rpluim@gmail.com>
>>> Cc: emacs-devel@gnu.org
>>> Date: Tue, 24 Jul 2018 22:48:50 +0200
>>> 
>>> auto-composition-mode seems not to be documented anywhere other than
>>> its doc string.
>>
>> Character composition is one of the few areas in Emacs that are
>> notoriously under-documented.  Patches to document that are most
>> welcome (let me know if I can help by pointing out to its other
>> interesting aspects).
>
> I think I¼ll start by putting pointers to auto-composition-mode in the
> manual and lispref.

(emacs)International Chars says:

       As a special case, if the character lies in the range 128 (0200
    octal) through 159 (0237 octal), it stands for a raw byte that does not
    correspond to any specific displayable character.  Such a character lies
    within the \x18eight-bit-control\x19 character set, and is displayed as an
    escaped octal character code.  In this case, \x18C-x =\x19 shows \x18part of
    display ...\x19 instead of \x18file\x19.

I can't get that to ever happen. I do

emacs -Q
C-x C-f /tmp/bin.txt
C-x 8 RET 80
C-b
C-x =

which gives

Char: € (128, #o200, #x80, file ...) point=1 of 1 (0%) column=0

If I save that buffer, and re-read it using 'raw-text, the display
looks like

\301\200

and C-u C-x =  on the \200 gives:

             position: 2 of 3 (33%), column: 4
            character: € (displayed as €) (codepoint 4194176, #o17777600, #x3fff80)
    preferred charset: tis620-2533 (TIS620.2533)
code point in charset: 0x80
               syntax: w 	which means: word
             category: L:Left-to-right (strong)
             to input: type "C-x 8 RET 3fff80"
          buffer code: #x80
            file code: #x80 (encoded by coding system raw-text-unix)
              display: no font available

Character code properties: customize what to show
  general-category: Cn (Other, Not Assigned)
  decomposition: (4194176) ('€')

I¼ve not been able to find any code that puts a 'display property on
the range 128 - 159 anywhere, so I wonder if the manual is out of
date?

Robert

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 25 Jul 2018 16:45:03 +0200
> 
>        As a special case, if the character lies in the range 128 (0200
>     octal) through 159 (0237 octal), it stands for a raw byte that does not
>     correspond to any specific displayable character.  Such a character lies
>     within the eight-bit-control character set, and is displayed as an
>     escaped octal character code.  In this case, C-x = shows part of
>     display ... instead of file.

This text is obsolete and inaccurate, it should be replaced/rewritten.

> emacs -Q
> C-x C-f /tmp/bin.txt
> C-x 8 RET 80
> C-b
> C-x =
> 
> which gives
> 
> Char: \200 (128, #o200, #x80, file ...) point=1 of 1 (0%) column=0

Try

  C-x 8 RET 3fff80 RET

> I wonder if the manual is out of date?

Yes.  This text is incorrect since Emacs 23.

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 25 Jul 2018 09:40:34 +0200
>> 
>> I think Iʼll start by putting pointers to auto-composition-mode in the
>> manual and lispref.
>
> Thanks in advance.

Hereʼs a first stab at it, intended for emacs-26. Probably the unicode
characters below will not survive intact.

diff --git i/doc/emacs/mule.texi w/doc/emacs/mule.texi
index 401c83dd49..925a582a4c 100644
--- i/doc/emacs/mule.texi
+++ w/doc/emacs/mule.texi
@@ -225,6 +225,51 @@ International Chars
   decomposition: (101 770) ('e' '^')
 @end smallexample
 
+@cindex diacritic
+@cindex composition
+  Sometimes Emacs will display a single character even when the buffer
+contains multiple characters, through a process known as @dfn{composition}.
+This is done via @code{auto-composition-mode}, which is enabled by default,
+and can only be done if the characters to be composed all exist within
+the same font.  The exact rules for which characters to compose are
+defined by the Unicode standard, but generally they concern
+diacritical marks such as accents.
+
+  For a successfully composed character, @kbd{C-u C-x =} displays
+details about the base character and the following character(s) it is
+composed with.  For example for @samp{e} composed with @samp{COMBINING
+CIRCUMFLEX ACCENT}, which visually would be very similar to the
+previous example, the output would look like:
+
+@smallexample
+             position: 146 of 147 (99%), column: 0
+            character: e (displayed as e) (codepoint 101, #o145, #x65)
+    preferred charset: ascii (ASCII (ISO646 IRV))
+code point in charset: 0x65
+               script: latin
+               syntax: w 	which means: word
+             category: .:Base, L:Left-to-right (strong), a:ASCII, l:Latin,
+                       r:Roman
+             to input: type "C-x 8 RET 65" or "C-x 8 RET LATIN SMALL LETTER E"
+          buffer code: #x65
+            file code: #x65 (encoded by coding system utf-8-unix)
+              display: composed to form "ê" (see below)
+
+Composed with the following character(s) "Ì‚" using this font:
+  xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-27-*-*-*-m-0-iso10646-1
+by these glyphs:
+  [0 1 101 72 16 2 15 21 0 nil]
+  [0 1 770 650 0 4 12 22 -17 [-16 0 0]]
+
+Character code properties: customize what to show
+  name: LATIN SMALL LETTER E
+  general-category: Ll (Letter, Lowercase)
+  decomposition: (101) ('e')
+
+There are text properties here:
+  fontified            t
+@end smallexample
+
 @node Language Environments
 @section Language Environments
 @cindex language environments
diff --git i/doc/lispref/nonascii.texi w/doc/lispref/nonascii.texi
index 4d75d6a1f1..c1943dda1c 100644
--- i/doc/lispref/nonascii.texi
+++ w/doc/lispref/nonascii.texi
@@ -500,10 +500,14 @@ Character Properties
 character's classification.  For unassigned codepoints, the value
 is @code{Cn}.
 
+@cindex diacritic
+@cindex composition
 @item canonical-combining-class
 Corresponds to the @code{Canonical_Combining_Class} Unicode property.
 The value is an integer.  For unassigned codepoints, the value
-is zero.
+is zero.  Emacs can use this to visually compose multiple characters,
+using @code{auto-composition-mode}, if all the characters concerned
+exist in the same font.
 
 @cindex bidirectional class of characters
 @item bidi-class

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 25 Jul 2018 05:36:52 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > Would it be worth allowing emacs to change the font of the
>> > preceding character to match the diacritic and (re-)attempt
>> > composition?
>> 
>> Theoretically, yes.  But it could mean a serious surgery in how
>> character composition is integrated into the display engine, and there
>> are also usability and UI aspects that need to be considered, because
>> the user might have good reasons to use the font she uses for the base
>> character.
>
> I've looked at the related code, and it least the first part doesn't
> seem to be too hard: it just needs expanding some APIs.  The relevant
> function is autocmp_chars, called from composition_reseat_it, which is
> called by the CHAR_COMPOSED_P macro.  The only problem to solve is how
> to determine what font to use if the composition includes more than 2
> characters, and Emacs selects more than 2 fonts to display them (e.g.,
> 3-character composition, each character needs a different font).

How common is 3-character composition likely to be? (for that matter,
how common is 2-character composition?  I explicitly use input methods
for this kind of stuff). I can envisage an algorithm that takes a
combining character, then scans backwards to see if the font used for
it will cover all previous characters, recursively. It does seem like
a lot of effort for a small return.

In any case, based on my quick testing with gedit, weʼre no better or
worse than it is, it also does not compose if the font of the base
character doesnʼt cover the combining character.

Robert

Re: ucs-normalize and diacritics

[Cesar Crusius]

[-- Attachment #1: Type: text/plain, Size: 1894 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> > Would it be worth allowing emacs to change the font of the
>>> > preceding character to match the diacritic and (re-)attempt
>>> > composition?
>>> 
>>> Theoretically, yes.  But it could mean a serious surgery in how
>>> character composition is integrated into the display engine, and there
>>> are also usability and UI aspects that need to be considered, because
>>> the user might have good reasons to use the font she uses for the base
>>> character.
>>
>> I've looked at the related code, and it least the first part doesn't
>> seem to be too hard: it just needs expanding some APIs.  The relevant
>> function is autocmp_chars, called from composition_reseat_it, which is
>> called by the CHAR_COMPOSED_P macro.  The only problem to solve is how
>> to determine what font to use if the composition includes more than 2
>> characters, and Emacs selects more than 2 fonts to display them (e.g.,
>> 3-character composition, each character needs a different font).
>
> How common is 3-character composition likely to be? (for that matter,
> how common is 2-character composition?  I explicitly use input methods
> for this kind of stuff). I can envisage an algorithm that takes a
> combining character, then scans backwards to see if the font used for
> it will cover all previous characters, recursively. It does seem like
> a lot of effort for a small return.

Recalling a recent discussion, they are unavoidable in polytonic Greek, because Unicode does not provide the pre-combined character. There's no other way to get an "rough breathing long alpha with acute accent," ᾱ̔́. (Which by the way Emacs handles nicely with the font I use, Iosevka.)

Granted, not many people will use this, but for those who do, they will be all over the place.

-- 
Cesar Crusius

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: ucs-normalize and diacritics

[Robert Pluim]

Cesar Crusius <cesar.crusius@gmail.com> writes:

>> How common is 3-character composition likely to be? (for that matter,
>> how common is 2-character composition?  I explicitly use input methods
>> for this kind of stuff). I can envisage an algorithm that takes a
>> combining character, then scans backwards to see if the font used for
>> it will cover all previous characters, recursively. It does seem like
>> a lot of effort for a small return.
>
> Recalling a recent discussion, they are unavoidable in polytonic
> Greek, because Unicode does not provide the pre-combined
> character. There's no other way to get an "rough breathing long alpha
> with acute accent," ᾱ̔́. (Which by the way Emacs handles nicely with the
> font I use, Iosevka.)

Even that is only a two character composition (unless Iʼve
misunderstood the what-cursor-position output), and itʼs rather
specialized, and you know what youʼre doing :-)

> Granted, not many people will use this, but for those who do, they
> will be all over the place.

I'm not sure I understand the comment. Current behaviour is what it
is, Iʼm not proposing anything that would make it worse.

Robert

Re: ucs-normalize and diacritics

[Yuri Khan]

On Thu, Jul 26, 2018 at 3:12 AM Robert Pluim <rpluim@gmail.com> wrote:

> how common is 2-character composition?

For Cyrillic letters and acute accent, there are no precomposed forms
at all, so when you need to explicitly call out the stressed syllable,
you’re going to use the composing acute accent. That need arises: (1)
in educational texts on every word with more than one syllable, as
training wheels of sort; (2) when you introduce an uncommon word or a
proper name that the reader is not expected to know how to stress; (3)
to disambiguate words that look the same but stressed differently.

There are precomposed forms for ё (Cyrillic {capital|small} letter io
= Cyrillic {capital|small} letter e + Combining diaeresis) and й
(Cyrillic {capital|small} letter short i = Cyrillic {capital|small}
letter i + Combining breve). Using composition for these would be
considered highly unusual.

Re: ucs-normalize and diacritics

[Cesar Crusius]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

> Cesar Crusius <cesar.crusius@gmail.com> writes:
>
>>> How common is 3-character composition likely to be? (for that matter,
>>> how common is 2-character composition?  I explicitly use input methods
>>> for this kind of stuff). I can envisage an algorithm that takes a
>>> combining character, then scans backwards to see if the font used for
>>> it will cover all previous characters, recursively. It does seem like
>>> a lot of effort for a small return.
>>
>> Recalling a recent discussion, they are unavoidable in polytonic
>> Greek, because Unicode does not provide the pre-combined
>> character. There's no other way to get an "rough breathing long alpha
>> with acute accent," ᾱ̔́. (Which by the way Emacs handles nicely with the
>> font I use, Iosevka.)
>
> Even that is only a two character composition (unless Iʼve
> misunderstood the what-cursor-position output), and itʼs rather
> specialized, and you know what youʼre doing :-)

There are three characters in total, ᾱ plus ̔plus ́.

>> Granted, not many people will use this, but for those who do, they
>> will be all over the place.
>
> I'm not sure I understand the comment. Current behaviour is what it
> is, Iʼm not proposing anything that would make it worse.

I'm just answering the "how common" question above, nothing else implied regarding the current discussion. For people writing polytonic Greek, compositions like the one above are not too uncommon, and two character compositions of the form "macron + combining accent" such as ᾱ́ are very common. FWIW.

Best,

-- 
Cesar Crusius

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: ucs-normalize and diacritics

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >        As a special case, if the character lies in the range 128 (0200
  > >     octal) through 159 (0237 octal), it stands for a raw byte that does not
  > >     correspond to any specific displayable character.  Such a character lies
  > >     within the eight-bit-control character set, and is displayed as an
  > >     escaped octal character code.  In this case, C-x = shows part of
  > >     display ... instead of file.

  > This text is obsolete and inaccurate, it should be replaced/rewritten.

Matt is just about to print the manual.  It is not too late to replace
this text!

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 25 Jul 2018 22:11:47 +0200
> 
> > I've looked at the related code, and it least the first part doesn't
> > seem to be too hard: it just needs expanding some APIs.  The relevant
> > function is autocmp_chars, called from composition_reseat_it, which is
> > called by the CHAR_COMPOSED_P macro.  The only problem to solve is how
> > to determine what font to use if the composition includes more than 2
> > characters, and Emacs selects more than 2 fonts to display them (e.g.,
> > 3-character composition, each character needs a different font).
> 
> How common is 3-character composition likely to be?

Among compositions, quite common, although not in the Latin scripts.

> (for that matter, how common is 2-character composition?  I
> explicitly use input methods for this kind of stuff).

There's no relation whatsoever between input methods and character
composition.

> In any case, based on my quick testing with gedit, weʼre no better or
> worse than it is, it also does not compose if the font of the base
> character doesnʼt cover the combining character.

Sure.

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Yuri Khan <yurivkhan@gmail.com>
> Date: Thu, 26 Jul 2018 04:01:25 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> There are precomposed forms for ё (Cyrillic {capital|small} letter io
> = Cyrillic {capital|small} letter e + Combining diaeresis) and й
> (Cyrillic {capital|small} letter short i = Cyrillic {capital|small}
> letter i + Combining breve). Using composition for these would be
> considered highly unusual.

I don't know what you meant "using composition" here, but Unicode
mandates that precomposed and decomposed representations should be
handled the same in text-processing applications.

Re: ucs-normalize and diacritics

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

Cesar Crusius <cesar.crusius@gmail.com> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> Cesar Crusius <cesar.crusius@gmail.com> writes:
>>
>>>> How common is 3-character composition likely to be? (for that matter,
>>>> how common is 2-character composition?  I explicitly use input methods
>>>> for this kind of stuff). I can envisage an algorithm that takes a
>>>> combining character, then scans backwards to see if the font used for
>>>> it will cover all previous characters, recursively. It does seem like
>>>> a lot of effort for a small return.
>>>
>>> Recalling a recent discussion, they are unavoidable in polytonic
>>> Greek, because Unicode does not provide the pre-combined
>>> character. There's no other way to get an "rough breathing long alpha
>>> with acute accent," ᾱ̔́. (Which by the way Emacs handles nicely with the
>>> font I use, Iosevka.)
>>
>> Even that is only a two character composition (unless Iʼve
>> misunderstood the what-cursor-position output), and itʼs rather
>> specialized, and you know what youʼre doing :-)
>
> There are three characters in total, ᾱ plus ̔plus ́.

Youʼre right, Iʼd missed the final combining accent.

Robert

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Date: Wed, 25 Jul 2018 16:45:03 +0200
>> 
>>        As a special case, if the character lies in the range 128 (0200
>>     octal) through 159 (0237 octal), it stands for a raw byte that does not
>>     correspond to any specific displayable character.  Such a character lies
>>     within the eight-bit-control character set, and is displayed as an
>>     escaped octal character code.  In this case, C-x = shows part of
>>     display ... instead of file.
>
> This text is obsolete and inaccurate, it should be replaced/rewritten.
>

How about something like:

  As a special case, if the character lies in the range #x3fff80
through #x3fff9a (128 through 159 decimal, with prefix #x3fff), it
stands for a raw byte that does not correspond to any specific
displayable character.  Such a character lies within the
@code{eight-bit-control} character set, and is displayed as an escaped
octal character code (0200 through 0237), or as an escaped hex
character code (x80 through x9a) if @code{display-raw-bytes-as-hex} is
non-@code{nil}.

Iʼm not sure the 'eight-bit-control' part is true, given the reference
to 'tis620-2533' in the what-cursor-position output.

>> emacs -Q
>> C-x C-f /tmp/bin.txt
>> C-x 8 RET 80
>> C-b
>> C-x =
>> 
>> which gives
>> 
>> Char: \200 (128, #o200, #x80, file ...) point=1 of 1 (0%) column=0
>
> Try
>
>   C-x 8 RET 3fff80 RET

Yes, that's better. So C-x 8 RET 80 results in emacs writing 2 bytes on
disk, but 3fff80 results in only one. The joys of multibyte :-)

Robert

Re: ucs-normalize and diacritics

[Matt Lavallee]

On 07/25/18 21:14, Richard Stallman wrote:

>   > >        As a special case, if the character lies in the range 128 (0200
>   > >     octal) through 159 (0237 octal), it stands for a raw byte that does not
>   > >     correspond to any specific displayable character.  Such a character lies
>   > >     within the eight-bit-control character set, and is displayed as an
>   > >     escaped octal character code.  In this case, C-x = shows part of
>   > >     display ... instead of file.
> 
>   > This text is obsolete and inaccurate, it should be replaced/rewritten.
> 
> Matt is just about to print the manual.  It is not too late to replace
> this text!

Hello all - I've just finished checking the publisher's proof for the
Emacs manual, but could do it again. I've CC'd FSF's John Hsieh to ask
if there's a prohibitive cost involved in obtaining another proof.

-- 
Matt Lavallee // Operations Assistant
Free Software Foundation
+1 617 542 5942

Join the FSF and help us
defend software freedom:
https://my.fsf.org/

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 26 Jul 2018 10:40:45 +0200
> 
> How about something like:
> 
>   As a special case, if the character lies in the range #x3fff80
> through #x3fff9a (128 through 159 decimal, with prefix #x3fff), it
> stands for a raw byte that does not correspond to any specific
> displayable character.  Such a character lies within the
> @code{eight-bit-control} character set, and is displayed as an escaped
> octal character code (0200 through 0237), or as an escaped hex
> character code (x80 through x9a) if @code{display-raw-bytes-as-hex} is
> non-@code{nil}.

Thanks, but the original text was wrong in more than one sense, and
needed a more thorough fix.  I pushed a fix, please see if the new
text is clear and accurate.

> Iʼm not sure the 'eight-bit-control' part is true, given the reference
> to 'tis620-2533' in the what-cursor-position output.

Nowadays we have a single character set 'eight-bit' that covers the
whole range #80 to #ff; the old partial charsets eight-bit-control and
eight-bit-graphic should not be mentioned in the user documentation.

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> Cc: rpluim@gmail.com, emacs-devel@gnu.org, mattlav@gnu.org,
>  John Hsieh <johnh@fsf.org>
> From: Matt Lavallee <mattlav@fsf.org>
> Date: Thu, 26 Jul 2018 10:15:17 -0400
> 
> > Matt is just about to print the manual.  It is not too late to replace
> > this text!
> 
> Hello all - I've just finished checking the publisher's proof for the
> Emacs manual, but could do it again. I've CC'd FSF's John Hsieh to ask
> if there's a prohibitive cost involved in obtaining another proof.

I only changed one paragraph, so perhaps you don't need to get a full
proof.

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 25 Jul 2018 21:59:00 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Robert Pluim <rpluim@gmail.com>
> >> Cc: emacs-devel@gnu.org
> >> Date: Wed, 25 Jul 2018 09:40:34 +0200
> >> 
> >> I think Iʼll start by putting pointers to auto-composition-mode in the
> >> manual and lispref.
> >
> > Thanks in advance.
> 
> Hereʼs a first stab at it, intended for emacs-26.

Thanks.

> Probably the unicode characters below will not survive intact.

They didn't, and their chance to display correctly in all the formats
we want to be able to produce the manual are slim.  My suggestion is
not to use lone diacritics at all, but instead use their Unicode
names, such as "COMBINING CIRCUMFLEX ACCENT" and maybe the codepoint,
as in "u+0302 COMBINING CIRCUMFLEX ACCENT".  Or maybe avoid showing
them entirely, see below.

A few more comments:

> +@cindex diacritic
> +@cindex composition

You are describing a command, auto-composition-mode, so there should
be an @findex entry for it here.

> +  Sometimes Emacs will display a single character even when the buffer
> +contains multiple characters, through a process known as @dfn{composition}.

"Display a single character" is in general inaccurate (although in the
case you give as example that is what happens).  Character composition
is not about characters, it's about "grapheme clusters".  But I think
it's too technical an issue to describe in the user manual, so I
suggest instead to say something simplified, like

  @cindex complex text layout
    Emacs supports @dfn{complex text layout} (abbreviated
  @acronym{CTL}), where several consecutive characters are displayed
  as a single unit, either a single font glyph or several glyphs whose
  shapes and relative positions are determined by the rules of the
  script to which the characters belong.  This happens automatically
  when @code{auto-composition-mode} is turned on (which is the default).

We could give a couple of examples before the last sentence, like
Arabic shaping and Latin characters with diacritics.

> +This is done via @code{auto-composition-mode}, which is enabled by default,
> +and can only be done if the characters to be composed all exist within
> +the same font.

Please try to avoid passive tense.  In this case:

  Emacs can only compose characters that have glyphs in the same font.

> +       The exact rules for which characters to compose are
> +defined by the Unicode standard, but generally they concern
> +diacritical marks such as accents.

Diacritics are a special case; character composition is a much more
general feature.  So I would lose the second part of this sentence, or
rephrase it as an example.

> +  For a successfully composed character, @kbd{C-u C-x =} displays
> +details about the base character and the following character(s) it is
> +composed with.  For example for @samp{e} composed with @samp{COMBINING
> +CIRCUMFLEX ACCENT}, which visually would be very similar to the
> +previous example, the output would look like:

I would suggest to avoid the detailed display (especially as it most
probably won't give good results in PDF and perhaps even HTML), but
instead just mention that the information about the composed
characters is displayed as part of "C-u C-x =".  The details are very
technical and IMO inappropriate for the user manual.  (OTOH, they
should be described in the ELisp manual, or at least we should explain
how to interpret them.)

>  @item canonical-combining-class
>  Corresponds to the @code{Canonical_Combining_Class} Unicode property.
>  The value is an integer.  For unassigned codepoints, the value
> -is zero.
> +is zero.  Emacs can use this to visually compose multiple characters,
> +using @code{auto-composition-mode}, if all the characters concerned
> +exist in the same font.

Again, this is just one special case of character compositions, and
not the most important one.  So I wonder what would be the value of
this text, unless we have a much more detailed and full description
elsewhere, and this text includes a cross-reference to those details.

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 26 Jul 2018 10:40:45 +0200
>> 
>> How about something like:
>> 
>>   As a special case, if the character lies in the range #x3fff80
>> through #x3fff9a (128 through 159 decimal, with prefix #x3fff), it
>> stands for a raw byte that does not correspond to any specific
>> displayable character.  Such a character lies within the
>> @code{eight-bit-control} character set, and is displayed as an escaped
>> octal character code (0200 through 0237), or as an escaped hex
>> character code (x80 through x9a) if @code{display-raw-bytes-as-hex} is
>> non-@code{nil}.
>
> Thanks, but the original text was wrong in more than one sense, and
> needed a more thorough fix.  I pushed a fix, please see if the new
> text is clear and accurate.

It¼s clear, but it¼s not 100% accurate as far as I can tell:

- C-x = shows 'raw-byte', not 'raw byte'
- It doesn¼t show this for the whole range 0200 to 0377, only for 0240
  to 0377, eg for 0200:

Char: € (4194176, #o17777600, #x3fff80, file #x80) point=1 of 256
(0%) column=0

C-u C-x = gives:

             position: 1 of 256 (0%), column: 0
            character: € (displayed as €) (codepoint 4194176, #o17777600, #x3fff80)
    preferred charset: tis620-2533 (TIS620.2533)
code point in charset: 0x80
               syntax: w 	which means: word
             category: L:Left-to-right (strong)
             to input: type "C-x 8 RET 3fff80"
          buffer code: #x80
            file code: #x80 (encoded by coding system raw-text-unix)
              display: no font available

Character code properties: customize what to show
  general-category: Cn (Other, Not Assigned)
  decomposition: (4194176) ('€')


Regards

Robert

Re: ucs-normalize and diacritics

[John Hsieh]

On 07/26/2018 02:42 PM, Eli Zaretskii wrote:
> I only changed one paragraph, so perhaps you don't need to get a full
> proof.

Ok, I am waiting to hear back from the printer. I will email this group
as soon as they respond.

John

Re: ucs-normalize and diacritics

[Stefan Monnier]

>> I only changed one paragraph, so perhaps you don't need to get a full
>> proof.
> Ok, I am waiting to hear back from the printer. I will email this group
> as soon as they respond.

FWIW, if it's too late it's really not a big deal IMO: it's not like the
manual is now 100% bug-free.


        Stefan

Re: ucs-normalize and diacritics

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Another possibility is to add an errata page to correct this point.

Of perhaps the text can be corrected with out a change of length,
and only one only a few pages would be changed.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 26 Jul 2018 22:40:04 +0200
> 
> It's clear, but it's not 100% accurate as far as I can tell:
> 
> - C-x = shows 'raw-byte', not 'raw byte'

Oops, fixed.

> - It doesn't show this for the whole range 0200 to 0377, only for 0240
>   to 0377, eg for 0200:

That's a bug, somewhere.  I will look into it when I have time.

Thanks.

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> Date: Fri, 27 Jul 2018 08:56:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > - It doesn't show this for the whole range 0200 to 0377, only for 0240
> >   to 0377, eg for 0200:
> 
> That's a bug, somewhere.  I will look into it when I have time.

Not a bug, an unintended consequence of how we set up charsets.  Now
fixed on the emacs-26 branch.

I wonder whether we can mark tis620-2533 a :supplementary-p charset.
Kenichi, do you envision any problems that could be caused by that?

Re: ucs-normalize and diacritics

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 27 Jul 2018 08:56:44 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > - It doesn't show this for the whole range 0200 to 0377, only for 0240
>> >   to 0377, eg for 0200:
>> 
>> That's a bug, somewhere.  I will look into it when I have time.
>
> Not a bug, an unintended consequence of how we set up charsets.  Now
> fixed on the emacs-26 branch.

Thanks, works for me.

Robert

Re: ucs-normalize and diacritics

[John Hsieh]

On 07/26/2018 11:21 PM, Richard Stallman wrote:
> Of perhaps the text can be corrected with out a change of length,
> and only one only a few pages would be changed.

I spoke to the printer, and they can correct a small change like this
without any issues. Matt did a full review of the proof, and I am
finishing up a second review. After that is done (by this coming
Monday), I will email Eli about sending us an updated file that we can
turn over to the printer to move forward.

John

Re: ucs-normalize and diacritics

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW, if it's too late it's really not a big deal IMO: it's not like the
  > manual is now 100% bug-free.

I agree.

However, if it is NOT too late, we may as well fix it before the printing.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ucs-normalize and diacritics

[John Hsieh]

Eli,

On 07/26/2018 05:01 PM, John Hsieh wrote:
> On 07/26/2018 02:42 PM, Eli Zaretskii wrote:
>> > I only changed one paragraph, so perhaps you don't need to get a full
>> > proof.
> Ok, I am waiting to hear back from the printer. I will email this group
> as soon as they respond.

Matt and I don't have other edits; can you send the updated file? I can
send an updated PDF to the printer, who can make the change without
additional cost. Thank you.

John

Re: ucs-normalize and diacritics

[K. Handa]

In article <83zhydrwoh.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > > - It doesn't show this for the whole range 0200 to 0377, only for 0240
> > >   to 0377, eg for 0200:
> > 
> > That's a bug, somewhere.  I will look into it when I have time.

> Not a bug, an unintended consequence of how we set up charsets.  Now
> fixed on the emacs-26 branch.

> I wonder whether we can mark tis620-2533 a :supplementary-p charset.
> Kenichi, do you envision any problems that could be caused by that?

If we put :supplementary-p t to tis620-2533, M-x list-character-sets
lists that charset under "Supplementary Character Sets" section, which
will surprise tis620-2533 users (I think there are still many).

And, anyway, that change won't change the behavior of char-charset used
by describe-char.  I have not yet read all mails in this thread, but if
the problem is that describe-char does not show eight-bit as a charset
of eight-bit characters, perhaps the best way is to modify describe-char
as this.

diff --git a/lisp/descr-text.el b/lisp/descr-text.el
index d8f8188eb1..aea090b907 100644
--- a/lisp/descr-text.el
+++ b/lisp/descr-text.el
@@ -403,6 +403,7 @@ describe-char
            (eight-bit-p (and (not enable-multibyte-characters) (>= char 128)))
            (charset (if eight-bit-p 'eight-bit
                       (or (get-text-property pos 'charset)
+                          (and (encode-char char 'eight-bit) 'eight-bit)
                           (char-charset char))))
            (composition (find-composition pos nil nil t))
            (component-chars nil)

---
K. Handa
handa@gnu.org

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: rpluim@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Jul 2018 21:37:40 +0900
> 
> > I wonder whether we can mark tis620-2533 a :supplementary-p charset.
> > Kenichi, do you envision any problems that could be caused by that?
> 
> If we put :supplementary-p t to tis620-2533, M-x list-character-sets
> lists that charset under "Supplementary Character Sets" section, which
> will surprise tis620-2533 users (I think there are still many).

I meanwhile removed tis620-2533 on master and made it an alias to
thai-iso8859-11.  It looked to me that a difference of one character
doesn't justify to have another charset with peculiar traits which in
addition causes unintuitive behavior.  Do you think that change is OK?

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: John Hsieh <johnh@fsf.org>
> Cc: rms@gnu.org, rpluim@gmail.com, emacs-devel@gnu.org, mattlav@gnu.org
> Date: Mon, 30 Jul 2018 21:09:24 -0400
> 
> Matt and I don't have other edits; can you send the updated file?

I don't have a PDF; can someone please help John with that?

Re: ucs-normalize and diacritics

[John Hsieh]

On 07/31/2018 11:22 AM, Eli Zaretskii wrote:
> I don't have a PDF; can someone please help John with that?

Oh, I meant an updated text/LaTeX file with the paragraph change; we can
make a PDF. JH

Re: ucs-normalize and diacritics

[John Hsieh]

On 07/31/2018 11:27 AM, John Hsieh wrote:
> On 07/31/2018 11:22 AM, Eli Zaretskii wrote:
>> > I don't have a PDF; can someone please help John with that?
> Oh, I meant an updated text/LaTeX file with the paragraph change; we can
> make a PDF. JH

Actually, if you can share a diff of the changes made to the repository
we can take it from there since we added the ISBN number. Thank you.

John

Re: ucs-normalize and diacritics

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

> Cc: mattlav@fsf.org, rms@gnu.org, rpluim@gmail.com, emacs-devel@gnu.org,
>  mattlav@gnu.org
> From: John Hsieh <johnh@fsf.org>
> Date: Tue, 31 Jul 2018 11:27:14 -0400
> 
> On 07/31/2018 11:22 AM, Eli Zaretskii wrote:
> > I don't have a PDF; can someone please help John with that?
> 
> Oh, I meant an updated text/LaTeX file with the paragraph change; we can
> make a PDF. JH

Sorry, attached below.


[-- Attachment #2: mule.texi --]
[-- Type: application/texinfo, Size: 84522 bytes --]

@c -*- coding: utf-8 -*-
@c This is part of the Emacs manual.
@c Copyright (C) 1997, 1999-2018 Free Software Foundation, Inc.
@c See file emacs.texi for copying conditions.
@node International
@chapter International Character Set Support
@c This node is referenced in the tutorial.  When renaming or deleting
@c it, the tutorial needs to be adjusted.  (TUTORIAL.de)
@cindex international scripts
@cindex multibyte characters
@cindex encoding of characters

@cindex Han
@cindex Hindi
@cindex Hangul
  Emacs supports a wide variety of international character sets,
including European and Vietnamese variants of the Latin alphabet, as
well as Arabic scripts, Brahmic scripts (for languages such as
Bengali, Hindi, and Thai), Cyrillic, Ethiopic, Georgian, Greek, Han
(for Chinese and Japanese), Hangul (for Korean), Hebrew and IPA@.
Emacs also supports various encodings of these characters that are used by
other internationalized software, such as word processors and mailers.

  Emacs allows editing text with international characters by supporting
all the related activities:

@itemize @bullet
@item
You can visit files with non-@acronym{ASCII} characters, save non-@acronym{ASCII} text, and
pass non-@acronym{ASCII} text between Emacs and programs it invokes (such as
compilers, spell-checkers, and mailers).  Setting your language
environment (@pxref{Language Environments}) takes care of setting up the
coding systems and other options for a specific language or culture.
Alternatively, you can specify how Emacs should encode or decode text
for each command; see @ref{Text Coding}.

@item
You can display non-@acronym{ASCII} characters encoded by the various
scripts.  This works by using appropriate fonts on graphics displays
(@pxref{Defining Fontsets}), and by sending special codes to text
displays (@pxref{Terminal Coding}).  If some characters are displayed
incorrectly, refer to @ref{Undisplayable Characters}, which describes
possible problems and explains how to solve them.

@item
Characters from scripts whose natural ordering of text is from right
to left are reordered for display (@pxref{Bidirectional Editing}).
These scripts include Arabic, Hebrew, Syriac, Thaana, and a few
others.

@item
You can insert non-@acronym{ASCII} characters or search for them.  To do that,
you can specify an input method (@pxref{Select Input Method}) suitable
for your language, or use the default input method set up when you choose
your language environment.  If
your keyboard can produce non-@acronym{ASCII} characters, you can select an
appropriate keyboard coding system (@pxref{Terminal Coding}), and Emacs
will accept those characters.  Latin-1 characters can also be input by
using the @kbd{C-x 8} prefix, see @ref{Unibyte Mode}.

With the X Window System, your locale should be set to an appropriate
value to make sure Emacs interprets keyboard input correctly; see
@ref{Language Environments, locales}.
@end itemize

  The rest of this chapter describes these issues in detail.

@menu
* International Chars::     Basic concepts of multibyte characters.
* Language Environments::   Setting things up for the language you use.
* Input Methods::           Entering text characters not on your keyboard.
* Select Input Method::     Specifying your choice of input methods.
* Coding Systems::          Character set conversion when you read and
                              write files, and so on.
* Recognize Coding::        How Emacs figures out which conversion to use.
* Specify Coding::          Specifying a file's coding system explicitly.
* Output Coding::           Choosing coding systems for output.
* Text Coding::             Choosing conversion to use for file text.
* Communication Coding::    Coding systems for interprocess communication.
* File Name Coding::        Coding systems for file @emph{names}.
* Terminal Coding::         Specifying coding systems for converting
                              terminal input and output.
* Fontsets::                Fontsets are collections of fonts
                              that cover the whole spectrum of characters.
* Defining Fontsets::       Defining a new fontset.
* Modifying Fontsets::      Modifying an existing fontset.
* Undisplayable Characters:: When characters don't display.
* Unibyte Mode::            You can pick one European character set
                              to use without multibyte characters.
* Charsets::                How Emacs groups its internal character codes.
* Bidirectional Editing::   Support for right-to-left scripts.
@end menu

@node International Chars
@section Introduction to International Character Sets

  The users of international character sets and scripts have
established many more-or-less standard coding systems for storing
files.  These coding systems are typically @dfn{multibyte}, meaning
that sequences of two or more bytes are used to represent individual
non-@acronym{ASCII} characters.

@cindex Unicode
  Internally, Emacs uses its own multibyte character encoding, which
is a superset of the @dfn{Unicode} standard.  This internal encoding
allows characters from almost every known script to be intermixed in a
single buffer or string.  Emacs translates between the multibyte
character encoding and various other coding systems when reading and
writing files, and when exchanging data with subprocesses.

@kindex C-h h
@findex view-hello-file
@cindex undisplayable characters
@cindex @samp{?} in display
  The command @kbd{C-h h} (@code{view-hello-file}) displays the file
@file{etc/HELLO}, which illustrates various scripts by showing
how to say ``hello'' in many languages.  If some characters can't be
displayed on your terminal, they appear as @samp{?} or as hollow boxes
(@pxref{Undisplayable Characters}).

  Keyboards, even in the countries where these character sets are
used, generally don't have keys for all the characters in them.  You
can insert characters that your keyboard does not support, using
@kbd{C-x 8 @key{RET}} (@code{insert-char}).  @xref{Inserting Text}.
Shorthands are available for some common characters; for example, you
can insert a left single quotation mark @t{‘} by typing @kbd{C-x 8
[}, or in Electric Quote mode, usually by simply typing @kbd{`}.
@xref{Quotation Marks}.  Emacs also supports
various @dfn{input methods}, typically one for each script or
language, which make it easier to type characters in the script.
@xref{Input Methods}.

@kindex C-x RET
  The prefix key @kbd{C-x @key{RET}} is used for commands that pertain
to multibyte characters, coding systems, and input methods.

@kindex C-x =@r{, and international characters}
@findex what-cursor-position@r{, and international characters}
  The command @kbd{C-x =} (@code{what-cursor-position}) shows
information about the character at point.  In addition to the
character position, which was described in @ref{Position Info}, this
command displays how the character is encoded.  For instance, it
displays the following line in the echo area for the character
@samp{c}:

@smallexample
Char: c (99, #o143, #x63) point=28062 of 36168 (78%) column=53
@end smallexample

  The four values after @samp{Char:} describe the character that
follows point, first by showing it and then by giving its character
code in decimal, octal and hex.  For a non-@acronym{ASCII} multibyte
character, these are followed by @samp{file} and the character's
representation, in hex, in the buffer's coding system, if that coding
system encodes the character safely and with a single byte
(@pxref{Coding Systems}).  If the character's encoding is longer than
one byte, Emacs shows @samp{file ...}.

@cindex eight-bit character set
@cindex raw bytes
  On rare occasions, Emacs encounters @dfn{raw bytes}: single bytes
whose values are in the range 128 (0200 octal) through 255 (0377
octal), which Emacs cannot interpret as part of a known encoding of
some non-ASCII character.  Such raw bytes are treated as if they
belonged to a special character set @code{eight-bit}; Emacs displays
them as escaped octal codes (this can be customized; @pxref{Display
Custom}).  In this case, @kbd{C-x =} shows @samp{raw-byte} instead of
@samp{file}.  In addition, @kbd{C-x =} shows the character codes of
raw bytes as if they were in the range @code{#x3FFF80..#x3FFFFF},
which is where Emacs maps them to distinguish them from Unicode
characters in the range @code{#x0080..#x00FF}.

@cindex character set of character at point
@cindex font of character at point
@cindex text properties at point
@cindex face at point
  With a prefix argument (@kbd{C-u C-x =}), this command displays a
detailed description of the character in a window:

@itemize @bullet
@item
The character set name, and the codes that identify the character
within that character set; @acronym{ASCII} characters are identified
as belonging to the @code{ascii} character set.

@item
The character's script, syntax and categories.

@item
What keys to type to input the character in the current input method
(if it supports the character).

@item
The character's encodings, both internally in the buffer, and externally
if you were to save the file.

@item
If you are running Emacs on a graphical display, the font name and
glyph code for the character.  If you are running Emacs on a text
terminal, the code(s) sent to the terminal.

@item
The character's text properties (@pxref{Text Properties,,,
elisp, the Emacs Lisp Reference Manual}), including any non-default
faces used to display the character, and any overlays containing it
(@pxref{Overlays,,, elisp, the same manual}).
@end itemize

  Here's an example, with some lines folded to fit into this manual:

@smallexample
             position: 1 of 1 (0%), column: 0
            character: ê (displayed as ê) (codepoint 234, #o352, #xea)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0xEA
               script: latin
               syntax: w        which means: word
             category: .:Base, L:Left-to-right (strong), c:Chinese,
                       j:Japanese, l:Latin, v:Viet
             to input: type "C-x 8 RET ea" or
                       "C-x 8 RET LATIN SMALL LETTER E WITH CIRCUMFLEX"
          buffer code: #xC3 #xAA
            file code: #xC3 #xAA (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-PfEd-DejaVu Sans Mono-normal-normal-
        normal-*-15-*-*-*-m-0-iso10646-1 (#xAC)

Character code properties: customize what to show
  name: LATIN SMALL LETTER E WITH CIRCUMFLEX
  old-name: LATIN SMALL LETTER E CIRCUMFLEX
  general-category: Ll (Letter, Lowercase)
  decomposition: (101 770) ('e' '^')
@end smallexample

@node Language Environments
@section Language Environments
@cindex language environments

  All supported character sets are supported in Emacs buffers whenever
multibyte characters are enabled; there is no need to select a
particular language in order to display its characters.
However, it is important to select a @dfn{language
environment} in order to set various defaults.  Roughly speaking, the
language environment represents a choice of preferred script rather
than a choice of language.

  The language environment controls which coding systems to recognize
when reading text (@pxref{Recognize Coding}).  This applies to files,
incoming mail, and any other text you read into Emacs.  It may also
specify the default coding system to use when you create a file.  Each
language environment also specifies a default input method.

@findex set-language-environment
@vindex current-language-environment
  To select a language environment, customize
@code{current-language-environment} or use the command @kbd{M-x
set-language-environment}.  It makes no difference which buffer is
current when you use this command, because the effects apply globally
to the Emacs session.  See the variable @code{language-info-alist} for
the list of supported language environments, and use the command
@kbd{C-h L @var{lang-env} @key{RET}} (@code{describe-language-environment})
for more information about the language environment @var{lang-env}.
Supported language environments include:

@c @cindex entries below are split between portions of the list to
@c make them more accurate, i.e., land on the line that mentions the
@c language.  However, makeinfo 4.x doesn't fill inside @quotation
@c lines that follow a @cindex entry and whose text has no whitespace.
@c To work around, we group the language environments together, so
@c that the blank that separates them triggers refill.
@quotation
@cindex ASCII (language environment)
@cindex Arabic
ASCII, Arabic,
@cindex Belarusian
@cindex Bengali
Belarusian, Bengali,
@cindex Brazilian Portuguese
@cindex Bulgarian
Brazilian Portuguese, Bulgarian,
@cindex Burmese
@cindex Cham
Burmese, Cham,
@cindex Chinese
Chinese-BIG5, Chinese-CNS, Chinese-EUC-TW, Chinese-GB,
Chinese-GB18030, Chinese-GBK,
@cindex Croatian
@cindex Cyrillic
Croatian, Cyrillic-ALT, Cyrillic-ISO, Cyrillic-KOI8,
@cindex Czech
@cindex Devanagari
Czech, Devanagari,
@cindex Dutch
@cindex English
Dutch, English,
@cindex Esperanto
@cindex Ethiopic
Esperanto, Ethiopic,
@cindex French
@cindex Georgian
French, Georgian,
@cindex German
@cindex Greek
@cindex Gujarati
German, Greek, Gujarati,
@cindex Hebrew
@cindex IPA
Hebrew, IPA,
@cindex Italian
Italian,
@cindex Japanese
@cindex Kannada
Japanese, Kannada,
@cindex Khmer
@cindex Korean
@cindex Lao
Khmer, Korean, Lao,
@cindex Latin
Latin-1, Latin-2, Latin-3, Latin-4, Latin-5, Latin-6, Latin-7,
Latin-8, Latin-9,
@cindex Latvian
@cindex Lithuanian
Latvian, Lithuanian,
@cindex Malayalam
@cindex Oriya
Malayalam, Oriya,
@cindex Persian
@cindex Polish
Persian, Polish,
@cindex Punjabi
@cindex Romanian
Punjabi, Romanian,
@cindex Russian
@cindex Sinhala
Russian, Sinhala,
@cindex Slovak
@cindex Slovenian
@cindex Spanish
Slovak, Slovenian, Spanish,
@cindex Swedish
@cindex TaiViet
Swedish, TaiViet,
@cindex Tajik
@cindex Tamil
Tajik, Tamil,
@cindex Telugu
@cindex Thai
Telugu, Thai,
@cindex Tibetan
@cindex Turkish
Tibetan, Turkish,
@cindex UTF-8
@cindex Ukrainian
UTF-8, Ukrainian,
@cindex Vietnamese
@cindex Welsh
Vietnamese, Welsh,
@cindex Windows-1255
and Windows-1255.
@end quotation

  To display the script(s) used by your language environment on a
graphical display, you need to have suitable fonts.
@xref{Fontsets}, for more details about setting up your fonts.

@findex set-locale-environment
@vindex locale-language-names
@vindex locale-charset-language-names
@cindex locales
  Some operating systems let you specify the character-set locale you
are using by setting the locale environment variables @env{LC_ALL},
@env{LC_CTYPE}, or @env{LANG}.  (If more than one of these is
set, the first one that is nonempty specifies your locale for this
purpose.)  During startup, Emacs looks up your character-set locale's
name in the system locale alias table, matches its canonical name
against entries in the value of the variables
@code{locale-charset-language-names} and @code{locale-language-names}
(the former overrides the latter),
and selects the corresponding language environment if a match is found.
It also adjusts the display
table and terminal coding system, the locale coding system, the
preferred coding system as needed for the locale, and---last but not
least---the way Emacs decodes non-@acronym{ASCII} characters sent by your keyboard.

@c This seems unlikely, doesn't it?
  If you modify the @env{LC_ALL}, @env{LC_CTYPE}, or @env{LANG}
environment variables while running Emacs (by using @kbd{M-x setenv}),
you may want to invoke the @code{set-locale-environment}
command afterwards to readjust the language environment from the new
locale.

@vindex locale-preferred-coding-systems
  The @code{set-locale-environment} function normally uses the preferred
coding system established by the language environment to decode system
messages.  But if your locale matches an entry in the variable
@code{locale-preferred-coding-systems}, Emacs uses the corresponding
coding system instead.  For example, if the locale @samp{ja_JP.PCK}
matches @code{japanese-shift-jis} in
@code{locale-preferred-coding-systems}, Emacs uses that encoding even
though it might normally use @code{japanese-iso-8bit}.

  You can override the language environment chosen at startup with
explicit use of the command @code{set-language-environment}, or with
customization of @code{current-language-environment} in your init
file.

@kindex C-h L
@findex describe-language-environment
  To display information about the effects of a certain language
environment @var{lang-env}, use the command @kbd{C-h L @var{lang-env}
@key{RET}} (@code{describe-language-environment}).  This tells you
which languages this language environment is useful for, and lists the
character sets, coding systems, and input methods that go with it.  It
also shows some sample text to illustrate scripts used in this
language environment.  If you give an empty input for @var{lang-env},
this command describes the chosen language environment.

@vindex set-language-environment-hook
  You can customize any language environment with the normal hook
@code{set-language-environment-hook}.  The command
@code{set-language-environment} runs that hook after setting up the new
language environment.  The hook functions can test for a specific
language environment by checking the variable
@code{current-language-environment}.  This hook is where you should
put non-default settings for specific language environments, such as
coding systems for keyboard input and terminal output, the default
input method, etc.

@vindex exit-language-environment-hook
  Before it starts to set up the new language environment,
@code{set-language-environment} first runs the hook
@code{exit-language-environment-hook}.  This hook is useful for undoing
customizations that were made with @code{set-language-environment-hook}.
For instance, if you set up a special key binding in a specific language
environment using @code{set-language-environment-hook}, you should set
up @code{exit-language-environment-hook} to restore the normal binding
for that key.

@node Input Methods
@section Input Methods

@cindex input methods
  An @dfn{input method} is a kind of character conversion designed
specifically for interactive input.  In Emacs, typically each language
has its own input method; sometimes several languages that use the same
characters can share one input method.  A few languages support several
input methods.

  The simplest kind of input method works by mapping @acronym{ASCII} letters
into another alphabet; this allows you to use one other alphabet
instead of @acronym{ASCII}.  The Greek and Russian input methods
work this way.

  A more powerful technique is composition: converting sequences of
characters into one letter.  Many European input methods use composition
to produce a single non-@acronym{ASCII} letter from a sequence that consists of a
letter followed by accent characters (or vice versa).  For example, some
methods convert the sequence @kbd{o ^} into a single accented letter.
These input methods have no special commands of their own; all they do
is compose sequences of printing characters.

  The input methods for syllabic scripts typically use mapping followed
by composition.  The input methods for Thai and Korean work this way.
First, letters are mapped into symbols for particular sounds or tone
marks; then, sequences of these that make up a whole syllable are
mapped into one syllable sign.

  Chinese and Japanese require more complex methods.  In Chinese input
methods, first you enter the phonetic spelling of a Chinese word (in
input method @code{chinese-py}, among others), or a sequence of
portions of the character (input methods @code{chinese-4corner} and
@code{chinese-sw}, and others).  One input sequence typically
corresponds to many possible Chinese characters.  You select the one
you mean using keys such as @kbd{C-f}, @kbd{C-b}, @kbd{C-n},
@kbd{C-p} (or the arrow keys), and digits, which have special meanings
in this situation.

  The possible characters are conceptually arranged in several rows,
with each row holding up to 10 alternatives.  Normally, Emacs displays
just one row at a time, in the echo area; @code{(@var{i}/@var{j})}
appears at the beginning, to indicate that this is the @var{i}th row
out of a total of @var{j} rows.  Type @kbd{C-n} or @kbd{C-p} to
display the next row or the previous row.

    Type @kbd{C-f} and @kbd{C-b} to move forward and backward among
the alternatives in the current row.  As you do this, Emacs highlights
the current alternative with a special color; type @kbd{C-@key{SPC}}
to select the current alternative and use it as input.  The
alternatives in the row are also numbered; the number appears before
the alternative.  Typing a number selects the associated alternative
of the current row and uses it as input.

  @key{TAB} in these Chinese input methods displays a buffer showing
all the possible characters at once; then clicking @kbd{mouse-2} on
one of them selects that alternative.  The keys @kbd{C-f}, @kbd{C-b},
@kbd{C-n}, @kbd{C-p}, and digits continue to work as usual, but they
do the highlighting in the buffer showing the possible characters,
rather than in the echo area.

  In Japanese input methods, first you input a whole word using
phonetic spelling; then, after the word is in the buffer, Emacs
converts it into one or more characters using a large dictionary.  One
phonetic spelling corresponds to a number of different Japanese words;
to select one of them, use @kbd{C-n} and @kbd{C-p} to cycle through
the alternatives.

  Sometimes it is useful to cut off input method processing so that the
characters you have just entered will not combine with subsequent
characters.  For example, in input method @code{latin-1-postfix}, the
sequence @kbd{o ^} combines to form an @samp{o} with an accent.  What if
you want to enter them as separate characters?

  One way is to type the accent twice; this is a special feature for
entering the separate letter and accent.  For example, @kbd{o ^ ^} gives
you the two characters @samp{o^}.  Another way is to type another letter
after the @kbd{o}---something that won't combine with that---and
immediately delete it.  For example, you could type @kbd{o o @key{DEL}
^} to get separate @samp{o} and @samp{^}.  Another method, more
general but not quite as easy to type, is to use @kbd{C-\ C-\} between
two characters to stop them from combining.  This is the command
@kbd{C-\} (@code{toggle-input-method}) used twice.
@ifnottex
@xref{Select Input Method}.
@end ifnottex

@cindex incremental search, input method interference
  @kbd{C-\ C-\} is especially useful inside an incremental search,
because it stops waiting for more characters to combine, and starts
searching for what you have already entered.

  To find out how to input the character after point using the current
input method, type @kbd{C-u C-x =}.  @xref{Position Info}.

@c TODO: document complex-only/default/t of
@c @code{input-method-verbose-flag}
@vindex input-method-verbose-flag
@vindex input-method-highlight-flag
  The variables @code{input-method-highlight-flag} and
@code{input-method-verbose-flag} control how input methods explain
what is happening.  If @code{input-method-highlight-flag} is
non-@code{nil}, the partial sequence is highlighted in the buffer (for
most input methods---some disable this feature).  If
@code{input-method-verbose-flag} is non-@code{nil}, the list of
possible characters to type next is displayed in the echo area (but
not when you are in the minibuffer).

@vindex quail-activate-hook
@findex quail-translation-keymap
  You can modify how an input method works by making your changes in a
function that you add to the hook variable @code{quail-activate-hook}.
@xref{Hooks}.  For example, you can redefine some of the input
method's keys by defining key bindings in the keymap returned by the
function @code{quail-translation-keymap}, using @code{define-key}.
@xref{Init Rebinding}.

  Another facility for typing characters not on your keyboard is by
using @kbd{C-x 8 @key{RET}} (@code{insert-char}) to insert a single
character based on its Unicode name or code-point; see @ref{Inserting
Text}.

@node Select Input Method
@section Selecting an Input Method

@table @kbd
@item C-\
Enable or disable use of the selected input method (@code{toggle-input-method}).

@item C-x @key{RET} C-\ @var{method} @key{RET}
Select a new input method for the current buffer (@code{set-input-method}).

@item C-h I @var{method} @key{RET}
@itemx C-h C-\ @var{method} @key{RET}
@findex describe-input-method
@kindex C-h I
@kindex C-h C-\
Describe the input method @var{method} (@code{describe-input-method}).
By default, it describes the current input method (if any).  This
description should give you the full details of how to use any
particular input method.

@item M-x list-input-methods
Display a list of all the supported input methods.
@end table

@findex set-input-method
@vindex current-input-method
@kindex C-x RET C-\
  To choose an input method for the current buffer, use @kbd{C-x
@key{RET} C-\} (@code{set-input-method}).  This command reads the
input method name from the minibuffer; the name normally starts with the
language environment that it is meant to be used with.  The variable
@code{current-input-method} records which input method is selected.

@findex toggle-input-method
@kindex C-\
  Input methods use various sequences of @acronym{ASCII} characters to
stand for non-@acronym{ASCII} characters.  Sometimes it is useful to
turn off the input method temporarily.  To do this, type @kbd{C-\}
(@code{toggle-input-method}).  To reenable the input method, type
@kbd{C-\} again.

  If you type @kbd{C-\} and you have not yet selected an input method,
it prompts you to specify one.  This has the same effect as using
@kbd{C-x @key{RET} C-\} to specify an input method.

  When invoked with a numeric argument, as in @kbd{C-u C-\},
@code{toggle-input-method} always prompts you for an input method,
suggesting the most recently selected one as the default.

@vindex default-input-method
  Selecting a language environment specifies a default input method for
use in various buffers.  When you have a default input method, you can
select it in the current buffer by typing @kbd{C-\}.  The variable
@code{default-input-method} specifies the default input method
(@code{nil} means there is none).

  In some language environments, which support several different input
methods, you might want to use an input method different from the
default chosen by @code{set-language-environment}.  You can instruct
Emacs to select a different default input method for a certain
language environment, if you wish, by using
@code{set-language-environment-hook} (@pxref{Language Environments,
set-language-environment-hook}).  For example:

@lisp
(defun my-chinese-setup ()
  "Set up my private Chinese environment."
  (if (equal current-language-environment "Chinese-GB")
      (setq default-input-method "chinese-tonepy")))
(add-hook 'set-language-environment-hook 'my-chinese-setup)
@end lisp

@noindent
This sets the default input method to be @code{chinese-tonepy}
whenever you choose a Chinese-GB language environment.

You can instruct Emacs to activate a certain input method
automatically.  For example:

@lisp
(add-hook 'text-mode-hook
  (lambda () (set-input-method "german-prefix")))
@end lisp

@noindent
This automatically activates the input method @code{german-prefix} in
Text mode.

@findex quail-set-keyboard-layout
  Some input methods for alphabetic scripts work by (in effect)
remapping the keyboard to emulate various keyboard layouts commonly used
for those scripts.  How to do this remapping properly depends on your
actual keyboard layout.  To specify which layout your keyboard has, use
the command @kbd{M-x quail-set-keyboard-layout}.

@findex quail-show-key
  You can use the command @kbd{M-x quail-show-key} to show what key (or
key sequence) to type in order to input the character following point,
using the selected keyboard layout.  The command @kbd{C-u C-x =} also
shows that information, in addition to other information about the
character.

@findex list-input-methods
  @kbd{M-x list-input-methods} displays a list of all the supported
input methods.  The list gives information about each input method,
including the string that stands for it in the mode line.

@node Coding Systems
@section Coding Systems
@cindex coding systems

  Users of various languages have established many more-or-less standard
coding systems for representing them.  Emacs does not use these coding
systems internally; instead, it converts from various coding systems to
its own system when reading data, and converts the internal coding
system to other coding systems when writing data.  Conversion is
possible in reading or writing files, in sending or receiving from the
terminal, and in exchanging data with subprocesses.

  Emacs assigns a name to each coding system.  Most coding systems are
used for one language, and the name of the coding system starts with
the language name.  Some coding systems are used for several
languages; their names usually start with @samp{iso}.  There are also
special coding systems, such as @code{no-conversion}, @code{raw-text},
and @code{emacs-internal}.

@cindex international files from DOS/Windows systems
  A special class of coding systems, collectively known as
@dfn{codepages}, is designed to support text encoded by MS-Windows and
MS-DOS software.  The names of these coding systems are
@code{cp@var{nnnn}}, where @var{nnnn} is a 3- or 4-digit number of the
codepage.  You can use these encodings just like any other coding
system; for example, to visit a file encoded in codepage 850, type
@kbd{C-x @key{RET} c cp850 @key{RET} C-x C-f @var{filename}
@key{RET}}.

  In addition to converting various representations of non-@acronym{ASCII}
characters, a coding system can perform end-of-line conversion.  Emacs
handles three different conventions for how to separate lines in a file:
newline (Unix), carriage return followed by linefeed (DOS), and just
carriage return (Mac).

@table @kbd
@item C-h C @var{coding} @key{RET}
Describe coding system @var{coding} (@code{describe-coding-system}).

@item C-h C @key{RET}
Describe the coding systems currently in use (@code{describe-coding-system}).

@item M-x list-coding-systems
Display a list of all the supported coding systems.
@end table

@kindex C-h C
@findex describe-coding-system
  The command @kbd{C-h C} (@code{describe-coding-system}) displays
information about particular coding systems, including the end-of-line
conversion specified by those coding systems.  You can specify a coding
system name as the argument; alternatively, with an empty argument, it
describes the coding systems currently selected for various purposes,
both in the current buffer and as the defaults, and the priority list
for recognizing coding systems (@pxref{Recognize Coding}).

@findex list-coding-systems
  To display a list of all the supported coding systems, type @kbd{M-x
list-coding-systems}.  The list gives information about each coding
system, including the letter that stands for it in the mode line
(@pxref{Mode Line}).

@cindex end-of-line conversion
@cindex line endings
@cindex MS-DOS end-of-line conversion
@cindex Macintosh end-of-line conversion
  Each of the coding systems that appear in this list---except for
@code{no-conversion}, which means no conversion of any kind---specifies
how and whether to convert printing characters, but leaves the choice of
end-of-line conversion to be decided based on the contents of each file.
For example, if the file appears to use the sequence carriage return
and linefeed to separate lines, DOS end-of-line conversion will be used.

  Each of the listed coding systems has three variants, which specify
exactly what to do for end-of-line conversion:

@table @code
@item @dots{}-unix
Don't do any end-of-line conversion; assume the file uses
newline to separate lines.  (This is the convention normally used
on Unix and GNU systems, and macOS.)

@item @dots{}-dos
Assume the file uses carriage return followed by linefeed to separate
lines, and do the appropriate conversion.  (This is the convention
normally used on Microsoft systems.@footnote{It is also specified for
MIME @samp{text/*} bodies and in other network transport contexts.  It
is different from the SGML reference syntax record-start/record-end
format, which Emacs doesn't support directly.})

@item @dots{}-mac
Assume the file uses carriage return to separate lines, and do the
appropriate conversion.  (This was the convention used in Classic Mac
OS.)
@end table

  These variant coding systems are omitted from the
@code{list-coding-systems} display for brevity, since they are entirely
predictable.  For example, the coding system @code{iso-latin-1} has
variants @code{iso-latin-1-unix}, @code{iso-latin-1-dos} and
@code{iso-latin-1-mac}.

@cindex @code{undecided}, coding system
  The coding systems @code{unix}, @code{dos}, and @code{mac} are
aliases for @code{undecided-unix}, @code{undecided-dos}, and
@code{undecided-mac}, respectively.  These coding systems specify only
the end-of-line conversion, and leave the character code conversion to
be deduced from the text itself.

@cindex @code{raw-text}, coding system
  The coding system @code{raw-text} is good for a file which is mainly
@acronym{ASCII} text, but may contain byte values above 127 that are
not meant to encode non-@acronym{ASCII} characters.  With
@code{raw-text}, Emacs copies those byte values unchanged, and sets
@code{enable-multibyte-characters} to @code{nil} in the current buffer
so that they will be interpreted properly.  @code{raw-text} handles
end-of-line conversion in the usual way, based on the data
encountered, and has the usual three variants to specify the kind of
end-of-line conversion to use.

@cindex @code{no-conversion}, coding system
  In contrast, the coding system @code{no-conversion} specifies no
character code conversion at all---none for non-@acronym{ASCII} byte values and
none for end of line.  This is useful for reading or writing binary
files, tar files, and other files that must be examined verbatim.  It,
too, sets @code{enable-multibyte-characters} to @code{nil}.

  The easiest way to edit a file with no conversion of any kind is with
the @kbd{M-x find-file-literally} command.  This uses
@code{no-conversion}, and also suppresses other Emacs features that
might convert the file contents before you see them.  @xref{Visiting}.

@cindex @code{emacs-internal}, coding system
  The coding system @code{emacs-internal} (or @code{utf-8-emacs},
which is equivalent) means that the file contains non-@acronym{ASCII}
characters stored with the internal Emacs encoding.  This coding
system handles end-of-line conversion based on the data encountered,
and has the usual three variants to specify the kind of end-of-line
conversion.

@node Recognize Coding
@section Recognizing Coding Systems

  Whenever Emacs reads a given piece of text, it tries to recognize
which coding system to use.  This applies to files being read, output
from subprocesses, text from X selections, etc.  Emacs can select the
right coding system automatically most of the time---once you have
specified your preferences.

  Some coding systems can be recognized or distinguished by which byte
sequences appear in the data.  However, there are coding systems that
cannot be distinguished, not even potentially.  For example, there is no
way to distinguish between Latin-1 and Latin-2; they use the same byte
values with different meanings.

  Emacs handles this situation by means of a priority list of coding
systems.  Whenever Emacs reads a file, if you do not specify the coding
system to use, Emacs checks the data against each coding system,
starting with the first in priority and working down the list, until it
finds a coding system that fits the data.  Then it converts the file
contents assuming that they are represented in this coding system.

  The priority list of coding systems depends on the selected language
environment (@pxref{Language Environments}).  For example, if you use
French, you probably want Emacs to prefer Latin-1 to Latin-2; if you use
Czech, you probably want Latin-2 to be preferred.  This is one of the
reasons to specify a language environment.

@findex prefer-coding-system
  However, you can alter the coding system priority list in detail
with the command @kbd{M-x prefer-coding-system}.  This command reads
the name of a coding system from the minibuffer, and adds it to the
front of the priority list, so that it is preferred to all others.  If
you use this command several times, each use adds one element to the
front of the priority list.

  If you use a coding system that specifies the end-of-line conversion
type, such as @code{iso-8859-1-dos}, what this means is that Emacs
should attempt to recognize @code{iso-8859-1} with priority, and should
use DOS end-of-line conversion when it does recognize @code{iso-8859-1}.

@vindex file-coding-system-alist
  Sometimes a file name indicates which coding system to use for the
file.  The variable @code{file-coding-system-alist} specifies this
correspondence.  There is a special function
@code{modify-coding-system-alist} for adding elements to this list.  For
example, to read and write all @samp{.txt} files using the coding system
@code{chinese-iso-8bit}, you can execute this Lisp expression:

@smallexample
(modify-coding-system-alist 'file "\\.txt\\'" 'chinese-iso-8bit)
@end smallexample

@noindent
The first argument should be @code{file}, the second argument should be
a regular expression that determines which files this applies to, and
the third argument says which coding system to use for these files.

@vindex inhibit-eol-conversion
@cindex DOS-style end-of-line display
  Emacs recognizes which kind of end-of-line conversion to use based on
the contents of the file: if it sees only carriage returns, or only
carriage return followed by linefeed sequences, then it chooses the
end-of-line conversion accordingly.  You can inhibit the automatic use
of end-of-line conversion by setting the variable
@code{inhibit-eol-conversion} to non-@code{nil}.  If you do that,
DOS-style files will be displayed with the @samp{^M} characters
visible in the buffer; some people prefer this to the more subtle
@samp{(DOS)} end-of-line type indication near the left edge of the
mode line (@pxref{Mode Line, eol-mnemonic}).

@vindex inhibit-iso-escape-detection
@cindex escape sequences in files
  By default, the automatic detection of the coding system is sensitive to
escape sequences.  If Emacs sees a sequence of characters that begin
with an escape character, and the sequence is valid as an ISO-2022
code, that tells Emacs to use one of the ISO-2022 encodings to decode
the file.

  However, there may be cases that you want to read escape sequences
in a file as is.  In such a case, you can set the variable
@code{inhibit-iso-escape-detection} to non-@code{nil}.  Then the code
detection ignores any escape sequences, and never uses an ISO-2022
encoding.  The result is that all escape sequences become visible in
the buffer.

  The default value of @code{inhibit-iso-escape-detection} is
@code{nil}.  We recommend that you not change it permanently, only for
one specific operation.  That's because some Emacs Lisp source files
in the Emacs distribution contain non-@acronym{ASCII} characters encoded in the
coding system @code{iso-2022-7bit}, and they won't be
decoded correctly when you visit those files if you suppress the
escape sequence detection.
@c I count a grand total of 3 such files, so is the above really true?

@vindex auto-coding-alist
@vindex auto-coding-regexp-alist
  The variables @code{auto-coding-alist} and
@code{auto-coding-regexp-alist} are
the strongest way to specify the coding system for certain patterns of
file names, or for files containing certain patterns, respectively.
These variables even override @samp{-*-coding:-*-} tags in the file
itself (@pxref{Specify Coding}).  For example, Emacs
uses @code{auto-coding-alist} for tar and archive files, to prevent it
from being confused by a @samp{-*-coding:-*-} tag in a member of the
archive and thinking it applies to the archive file as a whole.
@ignore
@c This describes old-style BABYL files, which are no longer relevant.
Likewise, Emacs uses @code{auto-coding-regexp-alist} to ensure that
RMAIL files, whose names in general don't match any particular
pattern, are decoded correctly.
@end ignore

@vindex auto-coding-functions
  Another way to specify a coding system is with the variable
@code{auto-coding-functions}.  For example, one of the builtin
@code{auto-coding-functions} detects the encoding for XML files.
Unlike the previous two, this variable does not override any
@samp{-*-coding:-*-} tag.

@node Specify Coding
@section Specifying a File's Coding System

  If Emacs recognizes the encoding of a file incorrectly, you can
reread the file using the correct coding system with @kbd{C-x
@key{RET} r} (@code{revert-buffer-with-coding-system}).  This command
prompts for the coding system to use.  To see what coding system Emacs
actually used to decode the file, look at the coding system mnemonic
letter near the left edge of the mode line (@pxref{Mode Line}), or
type @kbd{C-h C} (@code{describe-coding-system}).

@vindex coding
  You can specify the coding system for a particular file in the file
itself, using the @w{@samp{-*-@dots{}-*-}} construct at the beginning,
or a local variables list at the end (@pxref{File Variables}).  You do
this by defining a value for the ``variable'' named @code{coding}.
Emacs does not really have a variable @code{coding}; instead of
setting a variable, this uses the specified coding system for the
file.  For example, @w{@samp{-*-mode: C; coding: latin-1; -*-}} specifies
use of the Latin-1 coding system, as well as C mode.  When you specify
the coding explicitly in the file, that overrides
@code{file-coding-system-alist}.

@node Output Coding
@section Choosing Coding Systems for Output

@vindex buffer-file-coding-system
  Once Emacs has chosen a coding system for a buffer, it stores that
coding system in @code{buffer-file-coding-system}.  That makes it the
default for operations that write from this buffer into a file, such
as @code{save-buffer} and @code{write-region}.  You can specify a
different coding system for further file output from the buffer using
@code{set-buffer-file-coding-system} (@pxref{Text Coding}).

  You can insert any character Emacs supports into any Emacs buffer,
but most coding systems can only handle a subset of these characters.
Therefore, it's possible that the characters you insert cannot be
encoded with the coding system that will be used to save the buffer.
For example, you could visit a text file in Polish, encoded in
@code{iso-8859-2}, and add some Russian words to it.  When you save
that buffer, Emacs cannot use the current value of
@code{buffer-file-coding-system}, because the characters you added
cannot be encoded by that coding system.

  When that happens, Emacs tries the most-preferred coding system (set
by @kbd{M-x prefer-coding-system} or @kbd{M-x
set-language-environment}).  If that coding system can safely encode
all of the characters in the buffer, Emacs uses it, and stores its
value in @code{buffer-file-coding-system}.  Otherwise, Emacs displays
a list of coding systems suitable for encoding the buffer's contents,
and asks you to choose one of those coding systems.

  If you insert the unsuitable characters in a mail message, Emacs
behaves a bit differently.  It additionally checks whether the
@c What determines this?
most-preferred coding system is recommended for use in MIME messages;
if not, it informs you of this fact and prompts you for another coding
system.  This is so you won't inadvertently send a message encoded in
a way that your recipient's mail software will have difficulty
decoding.  (You can still use an unsuitable coding system if you enter
its name at the prompt.)

@c It seems that select-message-coding-system does this.
@c Both sendmail.el and smptmail.el call it; i.e., smtpmail.el still
@c obeys sendmail-coding-system.
@vindex sendmail-coding-system
  When you send a mail message (@pxref{Sending Mail}),
Emacs has four different ways to determine the coding system to use
for encoding the message text.  It first tries the buffer's own value of
@code{buffer-file-coding-system}, if that is non-@code{nil}.
Otherwise, it uses the value of @code{sendmail-coding-system}, if that
is non-@code{nil}.  Thirdly, it uses the value of
@code{default-sendmail-coding-system}.
If all of these three values are @code{nil}, Emacs encodes outgoing
mail using the default coding system for new files (i.e., the
default value of @code{buffer-file-coding-system}), which is
controlled by your choice of language environment.

@node Text Coding
@section Specifying a Coding System for File Text

  In cases where Emacs does not automatically choose the right coding
system for a file's contents, you can use these commands to specify
one:

@table @kbd
@item C-x @key{RET} f @var{coding} @key{RET}
Use coding system @var{coding} to save or revisit the file in
the current buffer (@code{set-buffer-file-coding-system}).

@item C-x @key{RET} c @var{coding} @key{RET}
Specify coding system @var{coding} for the immediately following
command (@code{universal-coding-system-argument}).

@item C-x @key{RET} r @var{coding} @key{RET}
Revisit the current file using the coding system @var{coding}
(@code{revert-buffer-with-coding-system}).

@item M-x recode-region @key{RET} @var{right} @key{RET} @var{wrong} @key{RET}
Convert a region that was decoded using coding system @var{wrong},
decoding it using coding system @var{right} instead.
@end table

@kindex C-x RET f
@findex set-buffer-file-coding-system
  The command @kbd{C-x @key{RET} f}
(@code{set-buffer-file-coding-system}) sets the file coding system for
the current buffer (i.e., the coding system to use when saving or
reverting the file).  You specify which coding system using the
minibuffer.  You can also invoke this command by clicking with
@kbd{mouse-3} on the coding system indicator in the mode line
(@pxref{Mode Line}).

  If you specify a coding system that cannot handle all the characters
in the buffer, Emacs will warn you about the troublesome characters,
and ask you to choose another coding system, when you try to save the
buffer (@pxref{Output Coding}).

@cindex specify end-of-line conversion
  You can also use this command to specify the end-of-line conversion
(@pxref{Coding Systems, end-of-line conversion}) for encoding the
current buffer.  For example, @kbd{C-x @key{RET} f dos @key{RET}} will
cause Emacs to save the current buffer's text with DOS-style
carriage return followed by linefeed line endings.

@kindex C-x RET c
@findex universal-coding-system-argument
  Another way to specify the coding system for a file is when you visit
the file.  First use the command @kbd{C-x @key{RET} c}
(@code{universal-coding-system-argument}); this command uses the
minibuffer to read a coding system name.  After you exit the minibuffer,
the specified coding system is used for @emph{the immediately following
command}.

  So if the immediately following command is @kbd{C-x C-f}, for example,
it reads the file using that coding system (and records the coding
system for when you later save the file).  Or if the immediately following
command is @kbd{C-x C-w}, it writes the file using that coding system.
When you specify the coding system for saving in this way, instead
of with @kbd{C-x @key{RET} f}, there is no warning if the buffer
contains characters that the coding system cannot handle.

  Other file commands affected by a specified coding system include
@kbd{C-x i} and @kbd{C-x C-v}, as well as the other-window variants
of @kbd{C-x C-f}.  @kbd{C-x @key{RET} c} also affects commands that
start subprocesses, including @kbd{M-x shell} (@pxref{Shell}).  If the
immediately following command does not use the coding system, then
@kbd{C-x @key{RET} c} ultimately has no effect.

  An easy way to visit a file with no conversion is with the @kbd{M-x
find-file-literally} command.  @xref{Visiting}.

  The default value of the variable @code{buffer-file-coding-system}
specifies the choice of coding system to use when you create a new file.
It applies when you find a new file, and when you create a buffer and
then save it in a file.  Selecting a language environment typically sets
this variable to a good choice of default coding system for that language
environment.

@kindex C-x RET r
@findex revert-buffer-with-coding-system
  If you visit a file with a wrong coding system, you can correct this
with @kbd{C-x @key{RET} r} (@code{revert-buffer-with-coding-system}).
This visits the current file again, using a coding system you specify.

@findex recode-region
  If a piece of text has already been inserted into a buffer using the
wrong coding system, you can redo the decoding of it using @kbd{M-x
recode-region}.  This prompts you for the proper coding system, then
for the wrong coding system that was actually used, and does the
conversion.  It first encodes the region using the wrong coding system,
then decodes it again using the proper coding system.

@node Communication Coding
@section Coding Systems for Interprocess Communication

  This section explains how to specify coding systems for use
in communication with other processes.

@table @kbd
@item C-x @key{RET} x @var{coding} @key{RET}
Use coding system @var{coding} for transferring selections to and from
other graphical applications (@code{set-selection-coding-system}).

@item C-x @key{RET} X @var{coding} @key{RET}
Use coding system @var{coding} for transferring @emph{one}
selection---the next one---to or from another graphical application
(@code{set-next-selection-coding-system}).

@item C-x @key{RET} p @var{input-coding} @key{RET} @var{output-coding} @key{RET}
Use coding systems @var{input-coding} and @var{output-coding} for
subprocess input and output in the current buffer
(@code{set-buffer-process-coding-system}).
@end table

@kindex C-x RET x
@kindex C-x RET X
@findex set-selection-coding-system
@findex set-next-selection-coding-system
  The command @kbd{C-x @key{RET} x} (@code{set-selection-coding-system})
specifies the coding system for sending selected text to other windowing
applications, and for receiving the text of selections made in other
applications.  This command applies to all subsequent selections, until
you override it by using the command again.  The command @kbd{C-x
@key{RET} X} (@code{set-next-selection-coding-system}) specifies the
coding system for the next selection made in Emacs or read by Emacs.

@vindex x-select-request-type
  The variable @code{x-select-request-type} specifies the data type to
request from the X Window System for receiving text selections from
other applications.  If the value is @code{nil} (the default), Emacs
tries @code{UTF8_STRING} and @code{COMPOUND_TEXT}, in this order, and
uses various heuristics to choose the more appropriate of the two
results; if none of these succeed, Emacs falls back on @code{STRING}.
If the value of @code{x-select-request-type} is one of the symbols
@code{COMPOUND_TEXT}, @code{UTF8_STRING}, @code{STRING}, or
@code{TEXT}, Emacs uses only that request type.  If the value is a
list of some of these symbols, Emacs tries only the request types in
the list, in order, until one of them succeeds, or until the list is
exhausted.

@kindex C-x RET p
@findex set-buffer-process-coding-system
  The command @kbd{C-x @key{RET} p} (@code{set-buffer-process-coding-system})
specifies the coding system for input and output to a subprocess.  This
command applies to the current buffer; normally, each subprocess has its
own buffer, and thus you can use this command to specify translation to
and from a particular subprocess by giving the command in the
corresponding buffer.

  You can also use @kbd{C-x @key{RET} c}
(@code{universal-coding-system-argument}) just before the command that
runs or starts a subprocess, to specify the coding system for
communicating with that subprocess.  @xref{Text Coding}.

  The default for translation of process input and output depends on the
current language environment.

@vindex locale-coding-system
@cindex decoding non-@acronym{ASCII} keyboard input on X
  The variable @code{locale-coding-system} specifies a coding system
to use when encoding and decoding system strings such as system error
messages and @code{format-time-string} formats and time stamps.  That
coding system is also used for decoding non-@acronym{ASCII} keyboard
input on the X Window System and for encoding text sent to the
standard output and error streams when in batch mode.  You should
choose a coding system that is compatible
with the underlying system's text representation, which is normally
specified by one of the environment variables @env{LC_ALL},
@env{LC_CTYPE}, and @env{LANG}.  (The first one, in the order
specified above, whose value is nonempty is the one that determines
the text representation.)

@node File Name Coding
@section Coding Systems for File Names

@table @kbd
@item C-x @key{RET} F @var{coding} @key{RET}
Use coding system @var{coding} for encoding and decoding file
names (@code{set-file-name-coding-system}).
@end table

@findex set-file-name-coding-system
@kindex C-x RET F
@cindex file names with non-@acronym{ASCII} characters
  The command @kbd{C-x @key{RET} F} (@code{set-file-name-coding-system})
specifies a coding system to use for encoding file @emph{names}.  It
has no effect on reading and writing the @emph{contents} of files.

@vindex file-name-coding-system
  In fact, all this command does is set the value of the variable
@code{file-name-coding-system}.  If you set the variable to a coding
system name (as a Lisp symbol or a string), Emacs encodes file names
using that coding system for all file operations.  This makes it
possible to use non-@acronym{ASCII} characters in file names---or, at
least, those non-@acronym{ASCII} characters that the specified coding
system can encode.

  If @code{file-name-coding-system} is @code{nil}, Emacs uses a
default coding system determined by the selected language environment,
and stored in the @code{default-file-name-coding-system} variable.
@c FIXME?  Is this correct?  What is the "default language environment"?
In the default language environment, non-@acronym{ASCII} characters in
file names are not encoded specially; they appear in the file system
using the internal Emacs representation.

@cindex file-name encoding, MS-Windows
@vindex w32-unicode-filenames
  When Emacs runs on MS-Windows versions that are descendants of the
NT family (Windows 2000, XP, and all the later versions), the value of
@code{file-name-coding-system} is largely ignored, as Emacs by default
uses APIs that allow passing Unicode file names directly.  By
contrast, on Windows 9X, file names are encoded using
@code{file-name-coding-system}, which should be set to the codepage
(@pxref{Coding Systems, codepage}) pertinent for the current system
locale.  The value of the variable @code{w32-unicode-filenames}
controls whether Emacs uses the Unicode APIs when it calls OS
functions that accept file names.  This variable is set by the startup
code to @code{nil} on Windows 9X, and to @code{t} on newer versions of
MS-Windows.

  @strong{Warning:} if you change @code{file-name-coding-system} (or the
language environment) in the middle of an Emacs session, problems can
result if you have already visited files whose names were encoded using
the earlier coding system and cannot be encoded (or are encoded
differently) under the new coding system.  If you try to save one of
these buffers under the visited file name, saving may use the wrong file
name, or it may encounter an error.  If such a problem happens, use @kbd{C-x
C-w} to specify a new file name for that buffer.

@findex recode-file-name
  If a mistake occurs when encoding a file name, use the command
@kbd{M-x recode-file-name} to change the file name's coding
system.  This prompts for an existing file name, its old coding
system, and the coding system to which you wish to convert.

@node Terminal Coding
@section Coding Systems for Terminal I/O

@table @kbd
@item C-x @key{RET} t @var{coding} @key{RET}
Use coding system @var{coding} for terminal output
(@code{set-terminal-coding-system}).

@item C-x @key{RET} k @var{coding} @key{RET}
Use coding system @var{coding} for keyboard input
(@code{set-keyboard-coding-system}).
@end table

@kindex C-x RET t
@findex set-terminal-coding-system
  The command @kbd{C-x @key{RET} t} (@code{set-terminal-coding-system})
specifies the coding system for terminal output.  If you specify a
character code for terminal output, all characters output to the
terminal are translated into that coding system.

  This feature is useful for certain character-only terminals built to
support specific languages or character sets---for example, European
terminals that support one of the ISO Latin character sets.  You need to
specify the terminal coding system when using multibyte text, so that
Emacs knows which characters the terminal can actually handle.

  By default, output to the terminal is not translated at all, unless
Emacs can deduce the proper coding system from your terminal type or
your locale specification (@pxref{Language Environments}).

@kindex C-x RET k
@findex set-keyboard-coding-system
@vindex keyboard-coding-system
  The command @kbd{C-x @key{RET} k} (@code{set-keyboard-coding-system}),
or the variable @code{keyboard-coding-system}, specifies the coding
system for keyboard input.  Character-code translation of keyboard
input is useful for terminals with keys that send non-@acronym{ASCII}
graphic characters---for example, some terminals designed for ISO
Latin-1 or subsets of it.

  By default, keyboard input is translated based on your system locale
setting.  If your terminal does not really support the encoding
implied by your locale (for example, if you find it inserts a
non-@acronym{ASCII} character if you type @kbd{M-i}), you will need to set
@code{keyboard-coding-system} to @code{nil} to turn off encoding.
You can do this by putting

@lisp
(set-keyboard-coding-system nil)
@end lisp

@noindent
in your init file.

  There is a similarity between using a coding system translation for
keyboard input, and using an input method: both define sequences of
keyboard input that translate into single characters.  However, input
methods are designed to be convenient for interactive use by humans, and
the sequences that are translated are typically sequences of @acronym{ASCII}
printing characters.  Coding systems typically translate sequences of
non-graphic characters.

@node Fontsets
@section Fontsets
@cindex fontsets

  A font typically defines shapes for a single alphabet or script.
Therefore, displaying the entire range of scripts that Emacs supports
requires a collection of many fonts.  In Emacs, such a collection is
called a @dfn{fontset}.  A fontset is defined by a list of font specifications,
each assigned to handle a range of character codes, and may fall back
on another fontset for characters that are not covered by the fonts
it specifies.

@cindex fonts for various scripts
@cindex Intlfonts package, installation
  Each fontset has a name, like a font.  However, while fonts are
stored in the system and the available font names are defined by the
system, fontsets are defined within Emacs itself.  Once you have
defined a fontset, you can use it within Emacs by specifying its name,
anywhere that you could use a single font.  Of course, Emacs fontsets
can use only the fonts that the system supports.  If some characters
appear on the screen as empty boxes or hex codes, this means that the
fontset in use for them has no font for those characters.  In this
case, or if the characters are shown, but not as well as you would
like, you may need to install extra fonts.  Your operating system may
have optional fonts that you can install; or you can install the GNU
Intlfonts package, which includes fonts for most supported
scripts.@footnote{If you run Emacs on X, you may need to inform the X
server about the location of the newly installed fonts with commands
such as:
@c FIXME?  I feel like this may be out of date.
@c E.g., the intlfonts tarfile is ~ 10 years old.

@example
 xset fp+ /usr/local/share/emacs/fonts
 xset fp rehash
@end example
}

  Emacs creates three fontsets automatically: the @dfn{standard
fontset}, the @dfn{startup fontset} and the @dfn{default fontset}.
@c FIXME?  The doc of *standard*-fontset-spec says:
@c "You have the biggest chance to display international characters
@c with correct glyphs by using the *standard* fontset." (my emphasis)
@c See https://lists.gnu.org/r/emacs-devel/2012-04/msg00430.html
The default fontset is most likely to have fonts for a wide variety of
non-@acronym{ASCII} characters, and is the default fallback for the
other two fontsets, and if you set a default font rather than fontset.
However, it does not specify font family names, so results can be
somewhat random if you use it directly.  You can specify a particular
fontset by starting Emacs with the @samp{-fn} option.  For example,

@example
emacs -fn fontset-standard
@end example

@noindent
You can also specify a fontset with the @samp{Font} resource (@pxref{X
Resources}).

  If no fontset is specified for use, then Emacs uses an
@acronym{ASCII} font, with @samp{fontset-default} as a fallback for
characters the font does not cover.  The standard fontset is only used if
explicitly requested, despite its name.

@findex describe-fontset
  To show the information about a specific fontset, use the
@w{@kbd{M-x describe-fontset}} command.  It prompts for a fontset
name, defaulting to the one used by the current frame, and then
displays all the subranges of characters and the fonts assigned to
them in that fontset.

  A fontset does not necessarily specify a font for every character
code.  If a fontset specifies no font for a certain character, or if
it specifies a font that does not exist on your system, then it cannot
display that character properly.  It will display that character as a
hex code or thin space or an empty box instead.  (@xref{Text Display, ,
glyphless characters}, for details.)

@node Defining Fontsets
@section Defining Fontsets

@vindex standard-fontset-spec
@vindex w32-standard-fontset-spec
@vindex ns-standard-fontset-spec
@cindex standard fontset
  When running on X, Emacs creates a standard fontset automatically according to the value
of @code{standard-fontset-spec}.  This fontset's name is

@example
-*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard
@end example

@noindent
or just @samp{fontset-standard} for short.

  On GNUstep and macOS, the standard fontset is created using the value of
@code{ns-standard-fontset-spec}, and on MS Windows it is
created using the value of @code{w32-standard-fontset-spec}.

@c FIXME?  How does one access these, or do anything with them?
@c Does it matter?
  Bold, italic, and bold-italic variants of the standard fontset are
created automatically.  Their names have @samp{bold} instead of
@samp{medium}, or @samp{i} instead of @samp{r}, or both.

@cindex startup fontset
  Emacs generates a fontset automatically, based on any default
@acronym{ASCII} font that you specify with the @samp{Font} resource or
the @samp{-fn} argument, or the default font that Emacs found when it
started.  This is the @dfn{startup fontset} and its name is
@code{fontset-startup}.  Emacs generates this fontset by replacing the
@var{charset_registry} field with @samp{fontset}, and replacing the
@var{charset_encoding} field with @samp{startup}, then using the
resulting string to specify a fontset.

  For instance, if you start Emacs with a font of this form,

@c FIXME?  I think this is a little misleading, because you cannot (?)
@c actually specify a font with wildcards, it has to be a complete spec.
@c Also, an X font specification of this form hasn't (?) been
@c mentioned before now, and is somewhat obsolete these days.
@c People are more likely to use a form like
@c emacs -fn "DejaVu Sans Mono-12"
@c How does any of this apply in that case?
@example
emacs -fn "*courier-medium-r-normal--14-140-*-iso8859-1"
@end example

@noindent
Emacs generates the following fontset and uses it for the initial X
window frame:

@example
-*-courier-medium-r-normal-*-14-140-*-*-*-*-fontset-startup
@end example

  The startup fontset will use the font that you specify, or a variant
with a different registry and encoding, for all the characters that
are supported by that font, and fallback on @samp{fontset-default} for
other characters.

  With the X resource @samp{Emacs.Font}, you can specify a fontset name
just like an actual font name.  But be careful not to specify a fontset
name in a wildcard resource like @samp{Emacs*Font}---that wildcard
specification matches various other resources, such as for menus, and
@c FIXME is this still true?
menus cannot handle fontsets.  @xref{X Resources}.

  You can specify additional fontsets using X resources named
@samp{Fontset-@var{n}}, where @var{n} is an integer starting from 0.
The resource value should have this form:

@smallexample
@var{fontpattern}, @r{[}@var{charset}:@var{font}@r{]@dots{}}
@end smallexample

@noindent
where @var{fontpattern} should have the form of a standard X font name
(see the previous fontset-startup example), except for the last two
fields.  They should have the form @samp{fontset-@var{alias}}.

  Each fontset has two names, one long and one short.  The long name
is  @var{fontpattern}.  The short name is @samp{fontset-@var{alias}},
the last 2 fields of the long name (e.g., @samp{fontset-startup} for
the  fontset automatically created at startup).  You can refer to the
fontset by either name.

  The construct @samp{@var{charset}:@var{font}} specifies which font to
use (in this fontset) for one particular character set.  Here,
@var{charset} is the name of a character set, and @var{font} is the
font to use for that character set.  You can use this construct any
number of times in defining one fontset.

  For the other character sets, Emacs chooses a font based on
@var{fontpattern}.  It replaces @samp{fontset-@var{alias}} with values
that describe the character set.  For the @acronym{ASCII} character font,
@samp{fontset-@var{alias}} is replaced with @samp{ISO8859-1}.

  In addition, when several consecutive fields are wildcards, Emacs
collapses them into a single wildcard.  This is to prevent use of
auto-scaled fonts.  Fonts made by scaling larger fonts are not usable
for editing, and scaling a smaller font is also not useful, because it is
better to use the smaller font in its own size, which is what Emacs
does.

  Thus if @var{fontpattern} is this,

@example
-*-fixed-medium-r-normal-*-24-*-*-*-*-*-fontset-24
@end example

@noindent
the font specification for @acronym{ASCII} characters would be this:

@example
-*-fixed-medium-r-normal-*-24-*-ISO8859-1
@end example

@noindent
and the font specification for Chinese GB2312 characters would be this:

@example
-*-fixed-medium-r-normal-*-24-*-gb2312*-*
@end example

  You may not have any Chinese font matching the above font
specification.  Most X distributions include only Chinese fonts that
have @samp{song ti} or @samp{fangsong ti} in the @var{family} field.  In
such a case, @samp{Fontset-@var{n}} can be specified as:

@smallexample
Emacs.Fontset-0: -*-fixed-medium-r-normal-*-24-*-*-*-*-*-fontset-24,\
        chinese-gb2312:-*-*-medium-r-normal-*-24-*-gb2312*-*
@end smallexample

@noindent
Then, the font specifications for all but Chinese GB2312 characters have
@samp{fixed} in the @var{family} field, and the font specification for
Chinese GB2312 characters has a wild card @samp{*} in the @var{family}
field.

@findex create-fontset-from-fontset-spec
  The function that processes the fontset resource value to create the
fontset is called @code{create-fontset-from-fontset-spec}.  You can also
call this function explicitly to create a fontset.

  @xref{Fonts}, for more information about font naming.

@node Modifying Fontsets
@section Modifying Fontsets
@cindex fontsets, modifying
@findex set-fontset-font

  Fontsets do not always have to be created from scratch.  If only
minor changes are required it may be easier to modify an existing
fontset.  Modifying @samp{fontset-default} will also affect other
fontsets that use it as a fallback, so can be an effective way of
fixing problems with the fonts that Emacs chooses for a particular
script.

Fontsets can be modified using the function @code{set-fontset-font},
specifying a character, a charset, a script, or a range of characters
to modify the font for, and a font specification for the font to be
used.  Some examples are:

@example
;; Use Liberation Mono for latin-3 charset.
(set-fontset-font "fontset-default" 'iso-8859-3
                  "Liberation Mono")

;; Prefer a big5 font for han characters.
(set-fontset-font "fontset-default"
                  'han (font-spec :registry "big5")
                  nil 'prepend)

;; Use DejaVu Sans Mono as a fallback in fontset-startup
;; before resorting to fontset-default.
(set-fontset-font "fontset-startup" nil "DejaVu Sans Mono"
                  nil 'append)

;; Use MyPrivateFont for the Unicode private use area.
(set-fontset-font "fontset-default"  '(#xe000 . #xf8ff)
                  "MyPrivateFont")

@end example

@cindex ignore font
@cindex fonts, how to ignore
@vindex face-ignored-fonts
  Some fonts installed on your system might be broken, or produce
unpleasant results for characters for which they are used, and you may
wish to instruct Emacs to completely ignore them while searching for a
suitable font required to display a character.  You can do that by
adding the offending fonts to the value of the variable
@code{face-ignored-fonts}, which is a list.  Here's an example to put
in your @file{~/.emacs}:

@example
(add-to-list 'face-ignored-fonts "Some Bad Font")
@end example

@node Undisplayable Characters
@section Undisplayable Characters

  There may be some non-@acronym{ASCII} characters that your
terminal cannot display.  Most text terminals support just a single
character set (use the variable @code{default-terminal-coding-system}
to tell Emacs which one, @ref{Terminal Coding}); characters that
can't be encoded in that coding system are displayed as @samp{?} by
default.

  Graphical displays can display a broader range of characters, but
you may not have fonts installed for all of them; characters that have
no font appear as a hollow box.

  If you use Latin-1 characters but your terminal can't display
Latin-1, you can arrange to display mnemonic @acronym{ASCII} sequences
instead, e.g., @samp{"o} for o-umlaut.  Load the library
@file{iso-ascii} to do this.

@vindex latin1-display
  If your terminal can display Latin-1, you can display characters
from other European character sets using a mixture of equivalent
Latin-1 characters and @acronym{ASCII} mnemonics.  Customize the variable
@code{latin1-display} to enable this.  The mnemonic @acronym{ASCII}
sequences mostly correspond to those of the prefix input methods.

@node Unibyte Mode
@section Unibyte Editing Mode

@cindex European character sets
@cindex accented characters
@cindex ISO Latin character sets
@cindex Unibyte operation
  The ISO 8859 Latin-@var{n} character sets define character codes in
the range 0240 to 0377 octal (160 to 255 decimal) to handle the
accented letters and punctuation needed by various European languages
(and some non-European ones).  Note that Emacs considers bytes with
codes in this range as raw bytes, not as characters, even in a unibyte
buffer, i.e., if you disable multibyte characters.  However, Emacs can
still handle these character codes as if they belonged to @emph{one}
of the single-byte character sets at a time.  To specify @emph{which}
of these codes to use, invoke @kbd{M-x set-language-environment} and
specify a suitable language environment such as @samp{Latin-@var{n}}.
@xref{Disabling Multibyte, , Disabling Multibyte Characters, elisp,
GNU Emacs Lisp Reference Manual}.

@vindex unibyte-display-via-language-environment
  Emacs can also display bytes in the range 160 to 255 as readable
characters, provided the terminal or font in use supports them.  This
works automatically.  On a graphical display, Emacs can also display
single-byte characters through fontsets, in effect by displaying the
equivalent multibyte characters according to the current language
environment.  To request this, set the variable
@code{unibyte-display-via-language-environment} to a non-@code{nil}
value.  Note that setting this only affects how these bytes are
displayed, but does not change the fundamental fact that Emacs treats
them as raw bytes, not as characters.

@cindex @code{iso-ascii} library
  If your terminal does not support display of the Latin-1 character
set, Emacs can display these characters as @acronym{ASCII} sequences which at
least give you a clear idea of what the characters are.  To do this,
load the library @code{iso-ascii}.  Similar libraries for other
Latin-@var{n} character sets could be implemented, but have not been
so far.

@findex standard-display-8bit
@cindex 8-bit display
  Normally non-ISO-8859 characters (decimal codes between 128 and 159
inclusive) are displayed as octal escapes.  You can change this for
non-standard extended versions of ISO-8859 character sets by using the
function @code{standard-display-8bit} in the @code{disp-table} library.

  There are two ways to input single-byte non-@acronym{ASCII}
characters:

@itemize @bullet
@cindex 8-bit input
@item
You can use an input method for the selected language environment.
@xref{Input Methods}.  When you use an input method in a unibyte
buffer, the non-@acronym{ASCII} character you specify with it is
converted to unibyte.

@item
If your keyboard can generate character codes 128 (decimal) and up,
representing non-@acronym{ASCII} characters, you can type those
character codes directly.

On a graphical display, you should not need to do anything special to
use these keys; they should simply work.  On a text terminal, you
should use the command @kbd{M-x set-keyboard-coding-system} or
customize the variable @code{keyboard-coding-system} to specify which
coding system your keyboard uses (@pxref{Terminal Coding}).  Enabling
this feature will probably require you to use @key{ESC} to type Meta
characters; however, on a console terminal or a terminal emulator such
as @code{xterm}, you can arrange for Meta to be converted to @key{ESC}
and still be able to type 8-bit characters present directly on the
keyboard or using @key{Compose} or @key{AltGr} keys.  @xref{User Input}.

@cindex @code{iso-transl} library
@cindex compose character
@cindex dead character
@item
You can use the key @kbd{C-x 8} as a compose-character prefix for
entry of non-@acronym{ASCII} Latin-1 and a few other printing
characters.  @kbd{C-x 8} is good for insertion (in the minibuffer as
well as other buffers), for searching, and in any other context where
a key sequence is allowed.

@kbd{C-x 8} works by loading the @code{iso-transl} library.  Once that
library is loaded, the @key{Alt} modifier key, if the keyboard has
one, serves the same purpose as @kbd{C-x 8}: use @key{Alt} together
with an accent character to modify the following letter.  In addition,
if the keyboard has keys for the Latin-1 dead accent characters,
they too are defined to compose with the following character, once
@code{iso-transl} is loaded.

Use @kbd{C-x 8 C-h} to list all the available @kbd{C-x 8} translations.
@end itemize

@node Charsets
@section Charsets
@cindex charsets

  In Emacs, @dfn{charset} is short for ``character set''.  Emacs
supports most popular charsets (such as @code{ascii},
@code{iso-8859-1}, @code{cp1250}, @code{big5}, and @code{unicode}), in
addition to some charsets of its own (such as @code{emacs},
@code{unicode-bmp}, and @code{eight-bit}).  All supported characters
belong to one or more charsets.

  Emacs normally does the right thing with respect to charsets, so
that you don't have to worry about them.  However, it is sometimes
helpful to know some of the underlying details about charsets.

  One example is font selection (@pxref{Fonts}).  Each language
environment (@pxref{Language Environments}) defines a priority
list for the various charsets.  When searching for a font, Emacs
initially attempts to find one that can display the highest-priority
charsets.  For instance, in the Japanese language environment, the
charset @code{japanese-jisx0208} has the highest priority, so Emacs
tries to use a font whose @code{registry} property is
@samp{JISX0208.1983-0}.

@findex list-charset-chars
@cindex characters in a certain charset
@findex describe-character-set
  There are two commands that can be used to obtain information about
charsets.  The command @kbd{M-x list-charset-chars} prompts for a
charset name, and displays all the characters in that character set.
The command @kbd{M-x describe-character-set} prompts for a charset
name, and displays information about that charset, including its
internal representation within Emacs.

@findex list-character-sets
  @kbd{M-x list-character-sets} displays a list of all supported
charsets.  The list gives the names of charsets and additional
information to identity each charset; for more details, see the
@url{https://www.itscj.ipsj.or.jp/itscj_english/iso-ir/ISO-IR.pdf,
ISO International Register of Coded Character Sets to be Used with
Escape Sequences (ISO-IR)} maintained by
the @url{https://www.itscj.ipsj.or.jp/itscj_english/,
Information Processing Society of Japan/Information Technology
Standards Commission of Japan (IPSJ/ITSCJ)}.  In this list,
charsets are divided into two categories: @dfn{normal charsets} are
listed first, followed by @dfn{supplementary charsets}.  A
supplementary charset is one that is used to define another charset
(as a parent or a subset), or to provide backward-compatibility for
older Emacs versions.

  To find out which charset a character in the buffer belongs to, put
point before it and type @kbd{C-u C-x =} (@pxref{International
Chars}).

@node Bidirectional Editing
@section Bidirectional Editing
@cindex bidirectional editing
@cindex right-to-left text

  Emacs supports editing text written in scripts, such as Arabic,
Farsi, and Hebrew, whose natural ordering of horizontal text for
display is from right to left.  However, digits and Latin text
embedded in these scripts are still displayed left to right.  It is
also not uncommon to have small portions of text in Arabic or Hebrew
embedded in an otherwise Latin document; e.g., as comments and strings
in a program source file.  For these reasons, text that uses these
scripts is actually @dfn{bidirectional}: a mixture of runs of
left-to-right and right-to-left characters.

  This section describes the facilities and options provided by Emacs
for editing bidirectional text.

@cindex logical order
@cindex visual order
  Emacs stores right-to-left and bidirectional text in the so-called
@dfn{logical} (or @dfn{reading}) order: the buffer or string position
of the first character you read precedes that of the next character.
Reordering of bidirectional text into the @dfn{visual} order happens
at display time.  As a result, character positions no longer increase
monotonically with their positions on display.  Emacs implements the
Unicode Bidirectional Algorithm (UBA) described in the
@uref{http://unicode.org/reports/tr9/, Unicode Standard Annex #9}, for
reordering of bidirectional text for display.
It deviates from the UBA only in how continuation lines are displayed
when text direction is opposite to the base paragraph direction,
e.g., when a long line of English text appears in a right-to-left
paragraph.

@vindex bidi-display-reordering
  The buffer-local variable @code{bidi-display-reordering} controls
whether text in the buffer is reordered for display.  If its value is
non-@code{nil}, Emacs reorders characters that have right-to-left
directionality when they are displayed.  The default value is
@code{t}.

@cindex base direction of paragraphs
@cindex paragraph, base direction
@vindex bidi-paragraph-start-re
@vindex bidi-paragraph-separate-re
  Each paragraph of bidirectional text can have its own @dfn{base
direction}, either right-to-left or left-to-right.  Text in
left-to-right paragraphs begins on the screen at the left margin of
the window and is truncated or continued when it reaches the right
margin.  By contrast, text in right-to-left paragraphs is displayed
starting at the right margin and is continued or truncated at the left
margin.  By default, paragraph boundaries are empty lines, i.e., lines
consisting entirely of whitespace characters.  To change that, you can
customize the two variables @code{bidi-paragraph-start-re} and
@code{bidi-paragraph-separate-re}, whose values should be regular
expressions (strings); e.g., to have a single newline start a new
paragraph, set both of these variables to @code{"^"}.  These two
variables are buffer-local (@pxref{Locals}).

@vindex bidi-paragraph-direction
  Emacs determines the base direction of each paragraph dynamically,
based on the text at the beginning of the paragraph.  However,
sometimes a buffer may need to force a certain base direction for its
paragraphs.  The variable @code{bidi-paragraph-direction}, if
non-@code{nil}, disables the dynamic determination of the base
direction, and instead forces all paragraphs in the buffer to have the
direction specified by its buffer-local value.  The value can be either
@code{right-to-left} or @code{left-to-right}.  Any other value is
interpreted as @code{nil}.

@cindex LRM
@cindex RLM
  Alternatively, you can control the base direction of a paragraph by
inserting special formatting characters in front of the paragraph.
The special character @code{RIGHT-TO-LEFT MARK}, or @sc{rlm}, forces
the right-to-left direction on the following paragraph, while
@code{LEFT-TO-RIGHT MARK}, or @sc{lrm} forces the left-to-right
direction.  (You can use @kbd{C-x 8 @key{RET}} to insert these characters.)
In a GUI session, the @sc{lrm} and @sc{rlm} characters display as very
thin blank characters; on text terminals they display as blanks.

  Because characters are reordered for display, Emacs commands that
operate in the logical order or on stretches of buffer positions may
produce unusual effects.  For example, the commands @kbd{C-f} and
@kbd{C-b} move point in the logical order, so the cursor will
sometimes jump when point traverses reordered bidirectional text.
Similarly, a highlighted region covering a contiguous range of
character positions may look discontinuous if the region spans
reordered text.  This is normal and similar to the behavior of other
programs that support bidirectional text.

@kindex RIGHT@r{, and bidirectional text}
@kindex LEFT@r{, and bidirectional text}
@findex right-char@r{, and bidirectional text}
@findex left-char@r{, and bidirectional text}
  Cursor motion commands bound to arrow keys, such as @key{LEFT} and
@kbd{C-@key{RIGHT}}, are sensitive to the base direction of the
current paragraph.  In a left-to-right paragraph, commands bound to
@key{RIGHT} with or without modifiers move @emph{forward} through
buffer text, but in a right-to-left paragraph they move
@emph{backward} instead.  This reflects the fact that in a
right-to-left paragraph buffer positions predominantly increase when
moving to the left on display.

  When you move out of a paragraph, the meaning of the arrow keys
might change if the base direction of the preceding or the following
paragraph is different from the paragraph out of which you moved.
When that happens, you need to adjust the arrow key you press to the
new base direction.

@vindex visual-order-cursor-movement
@cindex cursor, visual-order motion
  By default, @key{LEFT} and @key{RIGHT} move in the logical order,
but if @code{visual-order-cursor-movement} is non-@code{nil}, these
commands move to the character that is, correspondingly, to the left
or right of the current screen position, moving to the next or
previous screen line as appropriate.  Note that this might potentially
move point many buffer positions away, depending on the surrounding
bidirectional context.

Re: ucs-normalize and diacritics

[John Hsieh]

On 07/31/2018 12:28 PM, Eli Zaretskii wrote:
> Sorry, attached below.

This works, thank you.

John

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> Cc: mattlav@fsf.org, rms@gnu.org, rpluim@gmail.com, emacs-devel@gnu.org,
>  mattlav@gnu.org
> From: John Hsieh <johnh@fsf.org>
> Date: Tue, 31 Jul 2018 11:45:36 -0400
> 
> Actually, if you can share a diff of the changes made to the repository
> we can take it from there since we added the ISBN number. Thank you.

The diffs are below:

diff --git a/doc/emacs/mule.texi b/doc/emacs/mule.texi
index 401c83d..6c0c5b2 100644
--- a/doc/emacs/mule.texi
+++ b/doc/emacs/mule.texi
@@ -156,12 +156,19 @@ International Chars
 (@pxref{Coding Systems}).  If the character's encoding is longer than
 one byte, Emacs shows @samp{file ...}.
 
-  As a special case, if the character lies in the range 128 (0200
-octal) through 159 (0237 octal), it stands for a raw byte that
-does not correspond to any specific displayable character.  Such a
-character lies within the @code{eight-bit-control} character set,
-and is displayed as an escaped octal character code.  In this case,
-@kbd{C-x =} shows @samp{part of display ...} instead of @samp{file}.
+@cindex eight-bit character set
+@cindex raw bytes
+  On rare occasions, Emacs encounters @dfn{raw bytes}: single bytes
+whose values are in the range 128 (0200 octal) through 255 (0377
+octal), which Emacs cannot interpret as part of a known encoding of
+some non-ASCII character.  Such raw bytes are treated as if they
+belonged to a special character set @code{eight-bit}; Emacs displays
+them as escaped octal codes (this can be customized; @pxref{Display
+Custom}).  In this case, @kbd{C-x =} shows @samp{raw-byte} instead of
+@samp{file}.  In addition, @kbd{C-x =} shows the character codes of
+raw bytes as if they were in the range @code{#x3FFF80..#x3FFFFF},
+which is where Emacs maps them to distinguish them from Unicode
+characters in the range @code{#x0080..#x00FF}.
 
 @cindex character set of character at point
 @cindex font of character at point

Re: ucs-normalize and diacritics

[K. Handa]

In article <838t5rph0r.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> I meanwhile removed tis620-2533 on master and made it an alias to
> thai-iso8859-11.  It looked to me that a difference of one character
> doesn't justify to have another charset with peculiar traits which in
> addition causes unintuitive behavior.  Do you think that change is OK?

I think it is not the right way.  Currently,

(encode-char #xa0 'tis620-2533) => nil
(encode-char #xa0 'thai-iso8859-11) => 32

But, your change results in:

(encode-char #xa0 'tis620-2533) => 32

which is simply wrong.

That unintuitive behavior you mentioned is just Emacs' fault, and can be
fixed, for instance, by my previous patch.

---
K. Handa
handa@gnu.org

Re: ucs-normalize and diacritics

[Andreas Schwab]

On Aug 03 2018, handa@gnu.org (K. Handa) wrote:

> That unintuitive behavior you mentioned is just Emacs' fault, and can be
> fixed, for instance, by my previous patch.

Why was tis620-2533 made a superset of eight-bit-control in the first
place (and mule-lao as well)?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: rpluim@gmail.com, emacs-devel@gnu.org
> Date: Fri, 03 Aug 2018 23:21:54 +0900
> 
> In article <838t5rph0r.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I meanwhile removed tis620-2533 on master and made it an alias to
> > thai-iso8859-11.  It looked to me that a difference of one character
> > doesn't justify to have another charset with peculiar traits which in
> > addition causes unintuitive behavior.  Do you think that change is OK?
> 
> I think it is not the right way.  Currently,
> 
> (encode-char #xa0 'tis620-2533) => nil
> (encode-char #xa0 'thai-iso8859-11) => 32
> 
> But, your change results in:
> 
> (encode-char #xa0 'tis620-2533) => 32
> 
> which is simply wrong.

How important is that discrepancy?  After all, tis620-2533 doesn't
have the #xa0 character, so the situation above should never happen,
right?  OTOH, it is strange to have two charsets that are identical
except for one character.

> That unintuitive behavior you mentioned is just Emacs' fault, and can be
> fixed, for instance, by my previous patch.

Yes, it could be fixed, but why do we need to jump through hoops to
keep one more charset, when we already have a charset that is
identical to it except for one character?

Re: ucs-normalize and diacritics

[K. Handa]

In article <87r2jfcxwu.fsf@igel.home>, Andreas Schwab <schwab@linux-m68k.org> writes:

> Why was tis620-2533 made a superset of eight-bit-control in the first
> place (and mule-lao as well)?

Sorry, but I don't remember well.  :-(
The addtion of that charset was before the mule-related codes were
merged to Emacs.  Most likely some document I got from Thai researcher
said so.  I've just consulted wikipedia
https://en.wikipedia.org/wiki/Thai_Industrial_Standard_620-2533
and it says "Code values 00-1F, 7F, 80-9F, A0, DB-DE and FC-FF are not
assigned to characters by TIS-620."!
So, including eight-bit-control in tis620-2533 may have been a mistake.
In that case, the current problem can be fixed by simply removing
eight-bit-control from the superset list of tis620-2533.

---
K. Handa
handa@gnu.org

Re: ucs-normalize and diacritics

[K. Handa]

In article <83r2jftjsu.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > (encode-char #xa0 'tis620-2533) => 32
> > 
> > which is simply wrong.

> How important is that discrepancy?  After all, tis620-2533 doesn't
> have the #xa0 character, so the situation above should never happen,
> right?

?? It happend when I wrote the code above.  How can you tell that
no one try the same thing?

> OTOH, it is strange to have two charsets that are identical
> except for one character.

However strange it is, it is the fact that those two chararcter sets are
different at that point.  Please see this page:
https://en.wikipedia.org/wiki/Thai_Industrial_Standard_620-2533

> Yes, it could be fixed, but why do we need to jump through hoops to
> keep one more charset, when we already have a charset that is
> identical to it except for one character?

The reason of having tis620-2533 was that it was added before
iso-8859-11.  And, there have been no positive reason of deleting
tis620-2533 even after we added iso-8859-11.  It is better to completely
drop the support of tis620-2533 (i.e. do not keep it as an alias for
iso-8859-11) than make Emacs tell a lie, but I think it is much better
to modify describe-char.

---
K. Handa
handa@gnu.org

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: eliz@gnu.org, rpluim@gmail.com, emacs-devel@gnu.org
> Date: Sun, 05 Aug 2018 22:22:51 +0900
> 
> So, including eight-bit-control in tis620-2533 may have been a mistake.
> In that case, the current problem can be fixed by simply removing
> eight-bit-control from the superset list of tis620-2533.

So you are saying that replacing this:

  :superset '(ascii eight-bit-control (thai-tis620 . 128)))

with this:

  :superset '(ascii (thai-tis620 . 128)))

in the definition of tis620-2533 is all that is needed?

Re: ucs-normalize and diacritics

[K. Handa]

In article <83tvo8suhd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > So, including eight-bit-control in tis620-2533 may have been a mistake.
> > In that case, the current problem can be fixed by simply removing
> > eight-bit-control from the superset list of tis620-2533.

> So you are saying that replacing this:

>   :superset '(ascii eight-bit-control (thai-tis620 . 128)))

> with this:

>   :superset '(ascii (thai-tis620 . 128)))

> in the definition of tis620-2533 is all that is needed?

Yes if it is ok believe the description of wikipedia.

---
K. Handa
handa@gnu.org

Re: ucs-normalize and diacritics

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: schwab@linux-m68k.org, rpluim@gmail.com, emacs-devel@gnu.org
> Date: Tue, 07 Aug 2018 07:49:44 +0900
> 
> In article <83tvo8suhd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > So, including eight-bit-control in tis620-2533 may have been a mistake.
> > > In that case, the current problem can be fixed by simply removing
> > > eight-bit-control from the superset list of tis620-2533.
> 
> > So you are saying that replacing this:
> 
> >   :superset '(ascii eight-bit-control (thai-tis620 . 128)))
> 
> > with this:
> 
> >   :superset '(ascii (thai-tis620 . 128)))
> 
> > in the definition of tis620-2533 is all that is needed?
> 
> Yes if it is ok believe the description of wikipedia.

OK, I've now done so.  Thanks for the feedback.