Re: [harder@ifa.au.dk: `set-locale-environment' bug]

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Kenichi Handa]

In article <E1AEH21-00079f-Je@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

> can u work on this?
> ------- Start of forwarded message -------
> To: emacs-pretest-bug@gnu.org
> From: Jesper Harder <harder@ifa.au.dk>
> Date: Mon, 27 Oct 2003 04:31:50 +0100
> Subject: `set-locale-environment' bug
[...]
> 1. Start Emacs in a en_US.iso88591 locale:

>    LANG=en_US.iso88591 emacs -q --no-site-file

> 2. Evaluate:

>    (setq standard-display-table nil)

> 3. Evaluating this expression:

>    (set-locale-environment)

> results in the following backtrace:

> Debugger entered--Lisp error: (wrong-type-argument arrayp nil)
>   aset(nil 160 [160])
>   (if (and (>= l 32) (< l 127)) (aset standard-display-table l nil) (aset standard-display-table l (vector l)))
>   (while (<= l h) (if (and ... ...) (aset standard-display-table l nil) (aset standard-display-table l ...)) (setq l (1+ l)))
>   standard-display-8bit(160 255)

Several functions in disp-table.el
(e.g. standard-display-8bit, standard-display-default)
assume that standard-display-table is already a dispaly-table.

I've just added:
    (or standard-display-table
        (setq standard-display-table (make-display-table)))
to all such functions that modify standard-display-table.

> By the way, the node "(elisp)Active Display Table" states that the
> default value for `standard-display-table' is nil.

> This isn't correct since the default value depends on the locale -- it
> is non-nil in the above locale.

Right.  How about writing the info as below?

This variable's value is the default display table, used
whenever a window has no display table and neither does the
buffer displayed in that window.  This variable is usually
@code{nil}, but set to a proper display table when Emacs is
started in such locales that use a single byte character set
(e.g. en_US.iso88591).

---
Ken'ichi HANDA
handa@m17n.org

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Jesper Harder]

Kenichi Handa <handa@m17n.org> writes:

> I've just added:
>     (or standard-display-table
>         (setq standard-display-table (make-display-table)))
> to all such functions that modify standard-display-table.

Thanks.

>> By the way, the node "(elisp)Active Display Table" states that the
>> default value for `standard-display-table' is nil.
>
> Right.  How about writing the info as below?
>
> This variable's value is the default display table, used whenever a
> window has no display table and neither does the buffer displayed in
> that window.  This variable is usually @code{nil}, but set to a
> proper display table when Emacs is started in such locales that use
> a single byte character set (e.g. en_US.iso88591).

I'm still a bit confused about _why_ the display table should be
non-nil in a single byte locale.

8bit characters like æøå display just fine for me if
`standard-display-table' is nil (both in X and on a console).

-- 
Cheers,
Jesper Harder

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Eli Zaretskii]

> From: Jesper Harder <harder@ifa.au.dk>
> Date: Tue, 28 Oct 2003 15:21:59 +0100
> 
> I'm still a bit confused about _why_ the display table should be
> non-nil in a single byte locale.

Because Emacs sets up a few entries in that display table, to match
what most users expect.  For details, see the function
standard-display-european-internal defined on mule-cmds.el.

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Jesper Harder]

"Eli Zaretskii" <eliz@elta.co.il> writes:

>> From: Jesper Harder <harder@ifa.au.dk>
>
>> I'm still a bit confused about _why_ the display table should be
>> non-nil in a single byte locale.
>
> Because Emacs sets up a few entries in that display table, to match
> what most users expect.  For details, see the function
> standard-display-european-internal defined on mule-cmds.el.

Thank you.  Now that I've looked at it, I know that I definitely want
to set `standard-display-table' to nil :-(

Most of what `standard-display-european-internal' does is wrong for my
use, viz.:

 (standard-display-8bit (if (eq window-system 'pc) 128 160) 255)

This controls how eight-bit-control and eight-bit-graphics is
displayed, right?  The presence of those charsets is usually a bug, so
I'd rather be alerted than hiding their presence.

 (aset standard-display-table 146 [39])

I think this is a bit evil.  Lisp, TeX, C, etc. do not consider 146
and 39 to be the same character.  Making them indistinguishable makes
it hard to locate errors caused by the presence of \222 rather than '
in source code.

 (aset standard-display-table ?' [?’])
 (aset standard-display-table ?` [?‘])

This is nice.  But it might do some people a disservice by making them
believe that `foo' actually looks good, while it'll look wrong and
ugly for 99% of all computer users.

-- 
Cheers,
Jesper Harder

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Eli Zaretskii]

> From: Jesper Harder <harder@ifa.au.dk>
> Date: Wed, 29 Oct 2003 00:52:36 +0100
> 
>  (standard-display-8bit (if (eq window-system 'pc) 128 160) 255)
> 
> This controls how eight-bit-control and eight-bit-graphics is
> displayed, right?

No.  window-system's value is `pc' only for the MS-DOS port of
Emacs.  On other systems, this sets up the display of characters
whose 8-bit codepoints are [160..255], i.e., the normal 8-bit region
of Latin-1 characters.

In other words, for a typical Unix or GNU system in the dk locale,
this line instructs Emacs to send the 8-bit codes of Latin-1
characters directly to the terminal, which is what I think you'd want,
since terminals in your locale generally support display of Latin-1
characters.

>  (aset standard-display-table 146 [39])
> 
> I think this is a bit evil.  Lisp, TeX, C, etc. do not consider 146
> and 39 to be the same character.  Making them indistinguishable makes
> it hard to locate errors caused by the presence of \222 rather than '
> in source code.

Then undo that line in your ~/.emacs.  I think most users do want
that, though; the code is based on user experience, IIRC.  In
particular, many mail messages sadly include that character, so
displaying it as \222 would make problems elsewhere.

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Jesper Harder]

Eli Zaretskii <eliz@elta.co.il> writes:

>> From: Jesper Harder <harder@ifa.au.dk>
>
>>  (standard-display-8bit (if (eq window-system 'pc) 128 160) 255)
>> 
>> This controls how eight-bit-control and eight-bit-graphics is
>> displayed, right?
>
> this sets up the display of characters whose 8-bit codepoints are
> [160..255], i.e., the normal 8-bit region of Latin-1 characters.
>
> In other words, for a typical Unix or GNU system in the dk locale,
> this line instructs Emacs to send the 8-bit codes of Latin-1
> characters directly to the terminal, which is what I think you'd
> want, since terminals in your locale generally support display of
> Latin-1 characters.

I don't understand.  As far as I can tell it makes no difference for
the display of Latin-1 characters.

It does _not_ change the dislay of `M-x list-charset-chars
latin-iso8859-1' in an xterm, the console or in X.

It does, however, change the display of the eight-bit-graphic
"charset".

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Eli Zaretskii]

> From: Jesper Harder <harder@ifa.au.dk>
> Date: Wed, 29 Oct 2003 17:49:13 +0100
> 
> It does, however, change the display of the eight-bit-graphic
> "charset".

Then perhaps I didn't remember that code correctly.  Sorry.

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Richard Stallman]

    > This isn't correct since the default value depends on the locale -- it
    > is non-nil in the above locale.

    Right.  How about writing the info as below?

    This variable's value is the default display table, used
    whenever a window has no display table and neither does the
    buffer displayed in that window.  This variable is usually
    @code{nil}, but set to a proper display table when Emacs is
    started in such locales that use a single byte character set
    (e.g. en_US.iso88591).

ok.  thanks.

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Kenichi Handa]

I'm sorry for this late response on this thread.

In article <m3he1tzblk.fsf@defun.localdomain>, Jesper Harder <harder@ifa.au.dk> writes:
>>>  By the way, the node "(elisp)Active Display Table" states that the
>>>  default value for `standard-display-table' is nil.
>> 
>>  Right.  How about writing the info as below?
>> 
>>  This variable's value is the default display table, used whenever a
>>  window has no display table and neither does the buffer displayed in
>>  that window.  This variable is usually @code{nil}, but set to a
>>  proper display table when Emacs is started in such locales that use
>>  a single byte character set (e.g. en_US.iso88591).

> I'm still a bit confused about _why_ the display table should be
> non-nil in a single byte locale.

> 8bit characters like æøå display just fine for me if
> `standard-display-table' is nil (both in X and on a console).

Actually, the relevant codes were written by someone while I
was too busy to work on Emacs.  So, I myself don't really
understand the underlining idea of those codes.

We must consider these parameters for displaying:
  * multibyte mode or unibyte mode
  * on terminal or on a windowing system
  * on Unix-like system, on MS-DOS, or on MS-Windows.
This yields 8 cases (because some conbinations are invalid).

But, in multibyte-mode, it seems that only MS-DOS case (on
terminal) really requires standard-display-table to be
setup.  That is because, in that case, characters in the
current codepage that are not supported by Emacs are decoded
into eight-bit-control/graphic, and they must be displayed
by the font of the current codepage as is.

Eli, is that right?

---
Ken'ichi HANDA
handa@m17n.org

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Eli Zaretskii]

> Date: Mon, 10 Nov 2003 11:36:00 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> But, in multibyte-mode, it seems that only MS-DOS case (on
> terminal) really requires standard-display-table to be
> setup.  That is because, in that case, characters in the
> current codepage that are not supported by Emacs are decoded
> into eight-bit-control/graphic, and they must be displayed
> by the font of the current codepage as is.
> 
> Eli, is that right?

Yes and no.

No, because unsupported characters are not displayed as the current
codepage's glyphs, they are displayed as a special glyph whose value
is in `dos-unsupported-char-glyph'.

Yes, because term/internal.el, the MS-DOS ``internal terminal''
emulator, sets up standard-display-table to display non-ASCII
characters beyond the current codepage, such as Latin-2 for a cp-850
locale, as strings of ASCII characters, see IT-display-table-setup.

I think in the past standard-display-table was used on multibyte
sessions on Unix as well, to map some characters originating from
Microsoft's codepages to printable ASCII chars, but I'm not sure
whether I remember correctly and if so, why was that removed.

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Kenichi Handa]

In article <9743-Mon10Nov2003073149+0200-eliz@elta.co.il>, "Eli Zaretskii" <eliz@elta.co.il> writes:
>>  But, in multibyte-mode, it seems that only MS-DOS case (on
>>  terminal) really requires standard-display-table to be
>>  setup.  That is because, in that case, characters in the
>>  current codepage that are not supported by Emacs are decoded
>>  into eight-bit-control/graphic, and they must be displayed
>>  by the font of the current codepage as is.
>>  
>>  Eli, is that right?

> Yes and no.

> No, because unsupported characters are not displayed as the current
> codepage's glyphs, they are displayed as a special glyph whose value
> is in `dos-unsupported-char-glyph'.

> Yes, because term/internal.el, the MS-DOS ``internal terminal''
> emulator, sets up standard-display-table to display non-ASCII
> characters beyond the current codepage, such as Latin-2 for a cp-850
> locale, as strings of ASCII characters, see IT-display-table-setup.

Thank you for the explanation.  It seems that all settings
about display table and terminal coding system is done by
dos-codepage-setup via term-setup-hook.  So, mule-cmds.el
should do nothing for them on DOS, right?

> I think in the past standard-display-table was used on multibyte
> sessions on Unix as well, to map some characters originating from
> Microsoft's codepages to printable ASCII chars, but I'm not sure
> whether I remember correctly and if so, why was that removed.

I think we must think over making the code in mule-cmds
clearer now.  The current code is quite confusing and it
seems that there are bugs.

For instance, at starup time,
set-display-table-and-terminal-coding-system is called in
this way now:

command-line ->
  set-local-environment ->
    set-language-environment ->
      set-display-table-and-terminal-coding-system (in unibyte mode only) ->
    set-display-table-and-terminal-coding-system (in multibyte mode only)

And set-display-table-and-terminal-coding-system calls
standard-display-european-internal in a lang. env. that has
`unibyte-display' setting, and resets standard-display-table
for 8-bit chars otherwise.

So, for instance, if we start in some European locale in
multibyte mode, standard-display-european-internal is
called, but when we switch to Japanese lang. env., the
standard-display-table is not reset.  On the other hand, if
we start in Japanese locale in multibyte mode, even if we
switch to some European locale,
standard-display-european-internal is not called.

But I need more time to study the current code to give a
concrete proposal for solving it.

---
Ken'ichi HANDA
handa@m17n.org

Re: [harder@ifa.au.dk: `set-locale-environment' bug]

[Eli Zaretskii]

> Date: Wed, 12 Nov 2003 11:40:15 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> It seems that all settings about display table and terminal coding
> system is done by dos-codepage-setup via term-setup-hook.

Yup.

> So, mule-cmds.el should do nothing for them on DOS, right?

I'm not sure.  First, dos-cpNNN-setup still calls
set-language-environment (and the user could theoretically set some
language environment manually).  standard-display-european-internal
could also be called in some unexpected way, even on DOS.  So
mule-cmds.el shouldn't do anything that's wrong for the DOS port; this
the DOS-specific code in standard-display-european-internal and in
set-language-environment.  We could, of course, move the DOS-specific
parts to term/internal.el so that mule-cmds.el is cleaner.

> I think we must think over making the code in mule-cmds
> clearer now.  The current code is quite confusing and it
> seems that there are bugs.

I wouldn't be surprised, given the amount of changes that went under
the bridge since that code was written.

> So, for instance, if we start in some European locale in
> multibyte mode, standard-display-european-internal is
> called, but when we switch to Japanese lang. env., the
> standard-display-table is not reset.  On the other hand, if
> we start in Japanese locale in multibyte mode, even if we
> switch to some European locale,
> standard-display-european-internal is not called.

I see how this happens (set-display-table-and-terminal-coding-system
is called in the multibyte mode from set-locale-environment, not from
set-language-environment), but I'm not 100% sure this is a bug.  I
think we need to discuss the meaning of changing the language
environment without switching the locale.  When would a user want to
do that, and what should Emacs do to adapt itself to such a situation?