display table for eight-bit-graphic

display table for eight-bit-graphic

[Dave Love]

I just got reminded about this.  I think the standard display table
should display eight-bit-graphic characters as octal in multibyte
mode.  Displaying them as graphics can be misleading.

I've long been using

(when default-enable-multibyte-characters
  (dotimes (i 128)
    (aset standard-display-table (+ 128 i) nil)))

Re: display table for eight-bit-graphic

[Kenichi Handa]

In article <rzqbs263bxm.fsf@albion.dl.ac.uk>, Dave Love <d.love@dl.ac.uk> writes:

> I just got reminded about this.  I think the standard display table
> should display eight-bit-graphic characters as octal in multibyte
> mode.  Displaying them as graphics can be misleading.

I agree, and I vaguely remember that we discussed it looong
ago, and people agreed on displaying them as octal.

But, I've just found that standard-display-table is setup
when we start Emacs with any locale of single byte charset
(e.g. iso-8859-1).  It seems that it is done intentionally
by set-locale-environment as below.

	  ;; If default-enable-multibyte-characters is nil,
	  ;; we are using single-byte characters,
	  ;; so the display table and terminal coding system are irrelevant.
	  (when default-enable-multibyte-characters
	    (set-display-table-and-terminal-coding-system language-name))

I don't know the reason why it does this.  Does anyone
remember that?

---
Ken'ichi HANDA
handa@m17n.org

Re: display table for eight-bit-graphic

[Kai Großjohann]

Kenichi Handa <handa@m17n.org> writes:

> In article <rzqbs263bxm.fsf@albion.dl.ac.uk>, Dave Love <d.love@dl.ac.uk> writes:
>
>> I just got reminded about this.  I think the standard display table
>> should display eight-bit-graphic characters as octal in multibyte
>> mode.  Displaying them as graphics can be misleading.
>
> I agree, and I vaguely remember that we discussed it looong
> ago, and people agreed on displaying them as octal.

I thought it was discussed and it was agreed to display them as
graphic?  But I agree that it is misleading.

I think the idea was that people who use eight-bit locales expect
things to be displayed using that charset -- so if a random byte
comes along, and it happens to match the code for ä in that locale,
then display that byte as ä.  Regardless of whether the random byte
is really known to be in iso-8859-1 encoding.  So Emacs would match
the behavior of /bin/cat in this respect.

Hm.  I think that Erik Naggum has argued in this direction before.
His stuff is sometimes difficult to read (because you have to ignore
the rant) but I guess his arguments are more coherent and
comprehensible than mine.
-- 
Ambibibentists unite!

Re: display table for eight-bit-graphic

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, 25 Jan 2003 11:17:50 +0100, kai.grossjohann@uni-duisburg.de (Kai =?iso-8859-1?q?Gro=DFjohann?=) wrote:
>
> I thought it was discussed and it was agreed to display them as
> graphic?  But I agree that it is misleading.
>
> I think the idea was that people who use eight-bit locales expect
> things to be displayed using that charset -- so if a random byte
> comes along, and it happens to match the code for ה in that locale,
> then display that byte as ה.  Regardless of whether the random byte
> is really known to be in iso-8859-1 encoding.  So Emacs would match
> the behavior of /bin/cat in this respect.

I agree. For persons in the ISO-8859-x languages, the 8 bit graphics
is much better than the octal representation. It's like the emails in
quoted-printable are better than base64 - You can read the former as
text even if some characters are non-readable, the latter must be
decoded (correctly!) before it can be read.

Ehud.


- --
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 mailto:ehud@unix.mvs.co.il                  Better  Safe  Than  Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE+Mw6hLFvTvpjqOY0RAl08AKCC2sNxgI0JuRP9PtT6lm5Qta7IaQCdGUaC
flQw3yrqD5sqxM7ktrVgnjg=
=eRWT
-----END PGP SIGNATURE-----

Re: display table for eight-bit-graphic

[Kai Großjohann]

"Ehud Karni" <ehud@unix.mvs.co.il> writes:

> I agree. For persons in the ISO-8859-x languages, the 8 bit graphics
> is much better than the octal representation.

I do not agree that displaying the graphics is better.

It might seem so at first sight, but there are problems later on: for
instance, you can't search for the 8bit graphics characters by typing
Latin-1 characters, and people will surely be *very* surprised that
they can't find their characters!

Maybe it would be useful to highlight the graphics characters in some
way so that it is clear that they aren't normal characters.
-- 
Ambibibentists unite!

Re: display table for eight-bit-graphic

[Richard Stallman]

    But, I've just found that standard-display-table is setup
    when we start Emacs with any locale of single byte charset
    (e.g. iso-8859-1).  It seems that it is done intentionally
    by set-locale-environment as below.

The reason for this is probably for the sake of unibyte buffers.
This way, people who don't like MULE and use Emacs in unibyte
mode with European character sets get the same behavior as before.

In the past, this code only affected unibyte buffers because those
character codes 128-255 normally only appeared in them.  But nowadays,
all those codes are normal in multibyte buffers too.  The display
table treats each code the same way regardless of whether it comes
from a unibyte buffer or a multibyte buffer.

Re: display table for eight-bit-graphic

[Kenichi Handa]

In article <E18coqR-0001T9-00@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     But, I've just found that standard-display-table is setup
>     when we start Emacs with any locale of single byte charset
>     (e.g. iso-8859-1).  It seems that it is done intentionally
>     by set-locale-environment as below.

> The reason for this is probably for the sake of unibyte buffers.
> This way, people who don't like MULE and use Emacs in unibyte
> mode with European character sets get the same behavior as before.

I don't think so.  See this comment again:

	  ;; If default-enable-multibyte-characters is nil,
	  ;; we are using single-byte characters,
	  ;; so the display table and terminal coding system are irrelevant.
	  (when default-enable-multibyte-characters
	    (set-display-table-and-terminal-coding-system language-name))

It seesm that the intention is to use the display table for
multibyte buffers.

> In the past, this code only affected unibyte buffers because those
> character codes 128-255 normally only appeared in them.  But nowadays,
> all those codes are normal in multibyte buffers too.  The display
> table treats each code the same way regardless of whether it comes
> from a unibyte buffer or a multibyte buffer.

Yes.  I don't object to the currently behaviour of display
table itself.  If one really wants to see
eight-bit-control/graphic chars in a multibyte buffer by
some glyph, it's ok to use display table as he wishes.

What I object is to setting up the display table as now by default.

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
> "Ehud Karni" <ehud@unix.mvs.co.il> writes:

>>  I agree. For persons in the ISO-8859-x languages, the 8 bit graphics
>>  is much better than the octal representation.

> I do not agree that displaying the graphics is better.

> It might seem so at first sight, but there are problems later on: for
> instance, you can't search for the 8bit graphics characters by typing
> Latin-1 characters, and people will surely be *very* surprised that
> they can't find their characters!

I agree with that.  Showing different characters by the same
glyph is the source of confusion at least for novice users.
First of all, 8-bit characters should not appear in a
multibyte buffer usually.  It it does, mainly it's because
of a bug of some program, on in a case that it should be
treated as raw bytes, not as characters.  In both cases, it
is better that they are not displayed as graphics.

> Maybe it would be useful to highlight the graphics characters in some
> way so that it is clear that they aren't normal characters.

I'm not sure that is a good idea.  Highlighting means many
things.  Octal displaying is far better to indicate that
they aren't normal characters.

---
Ken'ichi HANDA
handa@m17n.org

Re: display table for eight-bit-graphic

[Richard Stallman]

    > This way, people who don't like MULE and use Emacs in unibyte
    > mode with European character sets get the same behavior as before.

    I don't think so.  See this comment again:

	      ;; If default-enable-multibyte-characters is nil,
	      ;; we are using single-byte characters,
	      ;; so the display table and terminal coding system are irrelevant.

Where is that comment?

	      (when default-enable-multibyte-characters
		(set-display-table-and-terminal-coding-system language-name))

    It seesm that the intention is to use the display table for
    multibyte buffers.

It is definitely intended to use the display table for multibyte
buffers--but I think that's not actually the issue you are talking
about.

Anyway, I don't think this actually contradicts anything I said.
The display table is used for unibyte buffers, too.  If we
want unibyte buffers to display the international graphics,
we have to set up the display table for codes 128-255.

Re: display table for eight-bit-graphic

[Kenichi Handa]

In article <E18dfi5-0003Mg-00@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>>  This way, people who don't like MULE and use Emacs in unibyte
>>  mode with European character sets get the same behavior as before.

>     I don't think so.  See this comment again:

> 	      ;; If default-enable-multibyte-characters is nil,
> 	      ;; we are using single-byte characters,
> 	      ;; so the display table and terminal coding system are irrelevant.

> Where is that comment?

It's in the function set-locale-environment (in mule-cmds.el).

> 	      (when default-enable-multibyte-characters
> 		(set-display-table-and-terminal-coding-system language-name))

>     It seesm that the intention is to use the display table for
>     multibyte buffers.

> It is definitely intended to use the display table for multibyte
> buffers--but I think that's not actually the issue you are talking
> about.

I'm talking that using the display table in multibyte
session is not good.

> Anyway, I don't think this actually contradicts anything I said.
> The display table is used for unibyte buffers, too.  If we
> want unibyte buffers to display the international graphics,
> we have to set up the display table for codes 128-255.

We don't have to use the display table even for unibyte
buffers.  Even for a unibyte buffer,
get_next_display_element in xdisp.c doesn't generate a octal
form "\XXX" for a code in the range 128-255 if the code can
be converted to a multibyte character by
unibyte_char_to_multibyte.

So, such a code is given to x_produce_glyph as is.  Then, in
x_produce_glyph, if unibyte-display-via-language-environment
is nil, that code is used as is (thus is displayed by the
default font), otherwise, that code is converted to a
multibyte char (thus is displayed by a font selected from
the current fontset).

In short, both in multibyte and unibyte sesseion, there's no
need of setting the display table now.

---
Ken'ichi HANDA
handa@m17n.org

Re: display table for eight-bit-graphic

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> The display table is used for unibyte buffers, too.  If we
> want unibyte buffers to display the international graphics,
> we have to set up the display table for codes 128-255.

I don't think this is required for unibyte _buffers_ only unibyte
_sessions_ if people insist on using them.

Re: display table for eight-bit-graphic

[Dave Love]

Incidentally, I think the display table should not be set for char 146
-- see comments around where it's done.  To sort that out, you should
[dr]ecode the text as windows-1252 if that's what it actually is.  If
it isn't actually windows-1252 or similar, the display table entry is
misleading.

Re: display table for eight-bit-graphic

[Dave Love]

Kenichi Handa <handa@m17n.org> writes:

> What I object is to setting up the display table as now by default.

Likewise.

[The original of these never reached me:]

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
> > "Ehud Karni" <ehud@unix.mvs.co.il> writes:
> 
> >>  I agree. For persons in the ISO-8859-x languages, the 8 bit graphics
> >>  is much better than the octal representation.

Displaying them as graphics is actually worse for people in locales
where the graphics might make sense, but are actually misleading.  Do
you really want undecodeable utf-8 byte sequences, or GBK &c, to
display as Hebrew characters?

> > It might seem so at first sight, but there are problems later on: for
> > instance, you can't search for the 8bit graphics characters by typing
> > Latin-1 characters, and people will surely be *very* surprised that
> > they can't find their characters!

The point is that these aren't actually well-defined characters --
they're random bytes.  If you were sure they represented, say, Latin-1
code points, you could treat them similarly to equivalent characters
from other charsets using translation-table-for-input, but then you
should have decoded the text correctly.

> I agree with that.  Showing different characters by the same
> glyph is the source of confusion at least for novice users.

That sort of thing is also a possible security-type risk.  There are
potential exploits from conning people into thinking the text is
different from what it actually is.

> > Maybe it would be useful to highlight the graphics characters in some
> > way so that it is clear that they aren't normal characters.
> 
> I'm not sure that is a good idea.  Highlighting means many
> things.  Octal displaying is far better to indicate that
> they aren't normal characters.

Exactly.  The only reasonable alternative would be to display them
with single-width glyphs or suitably-propertized text that displayed
the binary values somehow.  I think Yudit does that, but I wouldn't
suggest following it, especially as the glyph contents would typically
be too small.

[It is, in principle, a good idea to flag possibly confusing
characters with faces, such as latin1-disp now allows.  That doesn't
actually work well in practice, since the face information allowed in
the display table doesn't get merged by redisplay.  Thus if you
underline substitutions, they don't inherit the foreground/background
of the adjacent text.  I don't know whether or not this is actually a
bug.]

Re: display table for eight-bit-graphic

[Richard Stallman]

    > It is definitely intended to use the display table for multibyte
    > buffers--but I think that's not actually the issue you are talking
    > about.

    I'm talking that using the display table in multibyte
    session is not good.

    > Anyway, I don't think this actually contradicts anything I said.
    > The display table is used for unibyte buffers, too.  If we
    > want unibyte buffers to display the international graphics,
    > we have to set up the display table for codes 128-255.

    We don't have to use the display table even for unibyte
    buffers.  Even for a unibyte buffer,
    get_next_display_element in xdisp.c doesn't generate a octal
    form "\XXX" for a code in the range 128-255 if the code can
    be converted to a multibyte character by
    unibyte_char_to_multibyte.

It sounds like you're suggesting that we eliminate the display
table feature.

Perhaps the display table is obsolete as a means of displaying
non-ASCII characters, assuming we set up for proper conversion
to unibyte according to the language in use.  But aren't display
tables used for other things too?

Re: display table for eight-bit-graphic

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 03 Mar 2003 13:59:49 -0500
> 
> It sounds like you're suggesting that we eliminate the display
> table feature.
> 
> Perhaps the display table is obsolete as a means of displaying
> non-ASCII characters, assuming we set up for proper conversion
> to unibyte according to the language in use.  But aren't display
> tables used for other things too?

FWIW, the DOS port (see term/internal.el) and Dave's latin-disp.el use
the display tables to display non-ASCII characters on character
terminals that don't support such characters.

So I think this is an important feature.

Re: display table for eight-bit-graphic

[Kenichi Handa]

Sorry for the late reply on this matter.

In article <E18pvAD-0004ru-00@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>>  It is definitely intended to use the display table for multibyte
>>  buffers--but I think that's not actually the issue you are talking
>>  about.

>     I'm talking that using the display table in multibyte
>     session is not good.

>>  Anyway, I don't think this actually contradicts anything I said.
>>  The display table is used for unibyte buffers, too.  If we
>>  want unibyte buffers to display the international graphics,
>>  we have to set up the display table for codes 128-255.

>     We don't have to use the display table even for unibyte
>     buffers.  Even for a unibyte buffer,
>     get_next_display_element in xdisp.c doesn't generate a octal
>     form "\XXX" for a code in the range 128-255 if the code can
>     be converted to a multibyte character by
>     unibyte_char_to_multibyte.

> It sounds like you're suggesting that we eliminate the display
> table feature.

No.  The display table is useful even in a multibyte buffer
if Emacs is running on a terminal like the case of MSDOS.
For instance, if one is using a terminal that can display
only ASCII, he may want to setup the display table so that,
for instance, `À' (A-grave) is displayed as "A`".

> Perhaps the display table is obsolete as a means of displaying
> non-ASCII characters, assuming we set up for proper conversion
> to unibyte according to the language in use.  

I think the display table is obsolete only as a means of
displaying 8-bit characaters in unibyte buffer and
eight-bit-control/graphic characters.  At least in the
latter case, they should be displayed by octal-form by
default.

---
Ken'ichi HANDA
handa@m17n.org

Re: display table for eight-bit-graphic

[Richard Stallman]

    I think the display table is obsolete only as a means of
    displaying 8-bit characaters in unibyte buffer and
    eight-bit-control/graphic characters.

That seems like a surprising assertion to me.  Is it because
you think people would use a terminal coding system instead?

Re: display table for eight-bit-graphic

[Kenichi Handa]

In article <E18vZFq-0006v0-00@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     I think the display table is obsolete only as a means of
>     displaying 8-bit characaters in unibyte buffer and
>     eight-bit-control/graphic characters.

> That seems like a surprising assertion to me.  Is it because
> you think people would use a terminal coding system instead?

No.

In unibyte buffer, by setting
unibyte-display-via-language-environment to non-nil, 8-bit
characters are converted to multibyte characters by
unibyte-char-to-multibyte on displaying.  The merit of this
is that when a user changes the language environment, 8-bit
characters are displayed correctly based on the current
language environment.  The demerit may be that unibyte users
don't want such a nifty feature, but want to stick to the
font he specified.

---
Ken'ichi HANDA
handa@m17n.org

Re: display table for eight-bit-graphic

[Richard Stallman]

    In unibyte buffer, by setting
    unibyte-display-via-language-environment to non-nil, 8-bit
    characters are converted to multibyte characters by
    unibyte-char-to-multibyte on displaying.  The merit of this
    is that when a user changes the language environment, 8-bit
    characters are displayed correctly based on the current
    language environment.

That is a sensible feature, but it isn't an exact replacement
for the current features that use the display table,
such as set-language-environment in unibyte mode.  It uses
set-display-table-and-terminal-coding-system.

It's possible that the unibyte-display-via-language-environment feature
does mostly the same thing as set-display-table-and-terminal-coding-system,
but that also sets up the display table for code 146.  If it calls
standard-display-european-internal, that also sets the display
table for code 160.

If you can make unibyte-display-via-language-environment handle
codes 146 and 160 in these ways, then perhaps it would be
an exact replacement for the current feature.  I am not sure.
I am also unsure whether that change would have some other drawback.