Text mode Emacs in multibyte terminal

Text mode Emacs in multibyte terminal

[Andreas Schwab]

When running a text mode Emacs in a multibyte environment (eg. utf8 xterm
with LANG=de_DE.UTF-8), and visiting a binary file (like /bin/true) you
get a garbled screen. The problem is that all non-control characters are
written verbatim to the terminal as if it were single-byte encoded.  Even
setting the buffer to multibyte and the terminal coding system to utf-8
doesn't help.  The only way to get a correct display is to load the file
with a forced coding system of utf-8 or some single-byte coding system.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Text mode Emacs in multibyte terminal

[Stefan Monnier]

> The problem is that all non-control characters are
> written verbatim to the terminal as if it were single-byte encoded.

You mean all the eight-bit-* chars?  I'd argue that such chars should always
be written as \NNN except for the special legacy case of unibyte sessions.


        Stefan

Re: Text mode Emacs in multibyte terminal

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The problem is that all non-control characters are
>> written verbatim to the terminal as if it were single-byte encoded.
>
> You mean all the eight-bit-* chars?

Only eight-bit-graphic.

> I'd argue that such chars should always be written as \NNN except for
> the special legacy case of unibyte sessions.

Looking closer the problem is the standard-display-table.  It is set up to
display all characters codes 0xa1-0xff as itself.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Text mode Emacs in multibyte terminal

[Kenichi Handa]

Sorry for the late response.

In article <87d5l0gs9n.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  The problem is that all non-control characters are
>>  written verbatim to the terminal as if it were single-byte encoded.

> You mean all the eight-bit-* chars?  I'd argue that such chars should always
> be written as \NNN except for the special legacy case of unibyte sessions.

This was the case that a binary file is read into a unibyte
buffer, and \240..\377 are displayed as is; i.e. those exact
bytes are sent to the terminal.  That's happened because
Emacs setup standard-display-table for Latin-1 even if the
locale is UTF-8.  I'v just installed a fix not to setup
standard-display-table in such a case.

But, it seems that you are proposing not to setup
standard-display-table even if the locale is Latin-1, right?

---
Kenichi Handa
handa@m17n.org

Re: Text mode Emacs in multibyte terminal

[Stefan Monnier]

>>> The problem is that all non-control characters are
>>> written verbatim to the terminal as if it were single-byte encoded.

>> You mean all the eight-bit-* chars?  I'd argue that such chars should always
>> be written as \NNN except for the special legacy case of unibyte sessions.

> This was the case that a binary file is read into a unibyte
> buffer, and \240..\377 are displayed as is; i.e. those exact
> bytes are sent to the terminal.  That's happened because
> Emacs setup standard-display-table for Latin-1 even if the
> locale is UTF-8.  I'v just installed a fix not to setup
> standard-display-table in such a case.

> But, it seems that you are proposing not to setup
> standard-display-table even if the locale is Latin-1, right?

Yes, that's indeed what I'm proposing.


        Stefan

Re: Text mode Emacs in multibyte terminal

[Kenichi Handa]

In article <87vey13w7y.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  But, it seems that you are proposing not to setup
>>  standard-display-table even if the locale is Latin-1, right?

> Yes, that's indeed what I'm proposing.

As I'm not a Latin-X user, I'm not sure about it, but I've
thought that a Latin-1 user expects to see \240..\377 of a
unibyte buffer (for instance the case of visiting a binary
file) by Latin-1 characters.  If that's not true anymore, I
agree with such a change, but of course, after the release.

---
Kenichi Handa
handa@m17n.org

Re: Text mode Emacs in multibyte terminal

[Stefan Monnier]

>>> But, it seems that you are proposing not to setup
>>> standard-display-table even if the locale is Latin-1, right?
>> Yes, that's indeed what I'm proposing.
> As I'm not a Latin-X user, I'm not sure about it, but I've
> thought that a Latin-1 user expects to see \240..\377 of a
> unibyte buffer (for instance the case of visiting a binary
> file) by Latin-1 characters.  If that's not true anymore, I
> agree with such a change, but of course, after the release.

It's never been true for me.  When visiting a binary file, I don't expect
to see much of anything, really, other than hopefully some recognizable
ASCII somewhere.

But I don't claim to be representative of all latin-1 users.


        Stefan

Re: Text mode Emacs in multibyte terminal

[Richard M. Stallman]

    But, it seems that you are proposing not to setup
    standard-display-table even if the locale is Latin-1, right?

This is not a bug fix, it is a proposal to change the features.
Now is not the time to consider such a change, so please
let's drop the idea.