Is it possible to use new Emacsen with non-ASCII?

Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

I spent (again!) a lot of time trying to teach Emacs to show 8-bit data
(in a TTY).  After a lot of experiments, I see that

  emacs -q -nw --unibyte
	   (standard-display-8bit 128 254)
           (set-input-mode (car (current-input-mode))
                           (nth 1 (current-input-mode))
                           '8bit)

allows one to edit the files.  Moreover, after I know the answer, I
can see that old Emacs manuals were at least mentioning these
commands.  However, newer Emacsen replaced this info by something
absolutely uncomprehensible...

Additionally, I have no idea how to make this (or something else) work
without --unibyte.  Even adding something like
(set-language-environment "Cyrillic-KOI8") all that is shown is
question marks...

Puzzled,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Jason Rumney]

On May 15, 1:37 am, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:

> Additionally, I have no idea how to make this (or something else) work
> without --unibyte.  Even adding something like
> (set-language-environment "Cyrillic-KOI8") all that is shown is
> question marks...

What is terminal-coding-system set to?

If your LANG or LC_ALL environment variables are set appropriately,
Emacs should automatically use the correct terminal-coding-system,
language-environment etc.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Jason Rumney 
<jasonrumney@gmail.com>], who wrote in article <39eea3f0-65d9-4a9f-aeda-ffeb7e7ef52d@t54g2000hsg.googlegroups.com>:
> On May 15, 1:37 am, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:
> 
> > Additionally, I have no idea how to make this (or something else) work
> > without --unibyte.  Even adding something like
> > (set-language-environment "Cyrillic-KOI8") all that is shown is
> > question marks...
> 
> What is (terminal-coding-system) set to?

When?  After (set-language-environment "Cyrillic-KOI8")?  nil

> If your LANG or LC_ALL environment variables are set appropriately,
> Emacs should automatically use the correct terminal-coding-system,
> language-environment etc.

Irrelevant.  I can't set LANG.

Thanks anyway,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Thu, 15 May 2008 18:02:44 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > What is (terminal-coding-system) set to?
> 
> When?  After (set-language-environment "Cyrillic-KOI8")?  nil

Then set it to whatever your terminal supports (koi8-r, I presume, but
that's a guess).  "M-x set-terminal-coding-system RET koi8-r RET".

Alternatively, try "(set-locale-environment koi8-r)", and see if that
does what you want.  This will also set your keyboard encoding to
koi8-r, and I'm not sure you want that.  (Maybe you should consider
describing more about what you want: what files do you want to edit,
how would you like your keyboard to be set, and what characters can
your terminal display.  There are probably a couple of lines you need
to add to your .emacs to get what you want, but it's hard to give
advice without knowing the conditions.)

> > If your LANG or LC_ALL environment variables are set appropriately,
> > Emacs should automatically use the correct terminal-coding-system,
> > language-environment etc.
> 
> Irrelevant.  I can't set LANG.

That's quite understandable, but set-locale-environment from within
Emacs has the same effect on Emacs, and doesn't affect anything but
Emacs.

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> Date: Thu, 15 May 2008 22:06:48 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Alternatively, try "(set-locale-environment koi8-r)", and see if that

Sorry, that should have been

   (set-locale-environment "koi8-r")

of course (i.e. the argument is a string).

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> Date: Thu, 15 May 2008 22:18:59 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Thu, 15 May 2008 22:06:48 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > Alternatively, try "(set-locale-environment koi8-r)", and see if that
> 
> Sorry, that should have been
> 
>    (set-locale-environment "koi8-r")
> 
> of course (i.e. the argument is a string).

Oh, and of course remove that --unibyte from the Emacs invocation
command line, otherwise all the encode/decode wizardry will never work
correctly.  (Sorry, I have difficulties even thinking about using that
option.)

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11649.1210880089.18990.help-gnu-emacs@gnu.org>:
> Oh, and of course remove that --unibyte from the Emacs invocation
> command line, otherwise all the encode/decode wizardry will never work
> correctly.  (Sorry, I have difficulties even thinking about using that
> option.)

It looks like you think there are cases it works correctly.  Did you
read what I wrote?

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 16 May 2008 07:29:17 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > Oh, and of course remove that --unibyte from the Emacs invocation
> > command line, otherwise all the encode/decode wizardry will never work
> > correctly.  (Sorry, I have difficulties even thinking about using that
> > option.)
> 
> It looks like you think there are cases it works correctly.  Did you
> read what I wrote?

Yes.

I'm guessing you have some non-standard system setup that causes Emacs
not to work out of the box.  Once you describe that setup (I sent
detailed questions elsewhere in this thread), I'm sure a solution will
be found very easily that doesn't involve --unibyte.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11665.1210928327.18990.help-gnu-emacs@gnu.org>:

> not to work out of the box.  Once you describe that setup (I sent
> detailed questions elsewhere in this thread), I'm sure a solution will
> be found very easily that doesn't involve --unibyte.

Sorry, I looked through the thread 3 times, and still I see no
questions...  (Unless you ask about input/output encodings - and I
discussed it elsewhere in the thread.)

Yours,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11643.1210878436.18990.help-gnu-emacs@gnu.org>:
> > > What is (terminal-coding-system) set to?
> > 
> > When?  After (set-language-environment "Cyrillic-KOI8")?  nil
> 
> Then set it to whatever your terminal supports (koi8-r, I presume, but
> that's a guess).  "M-x set-terminal-coding-system RET koi8-r RET".

And why set-language-environment did not do it?  

> Alternatively, try "(set-locale-environment koi8-r)", and see if that
> does what you want.  This will also set your keyboard encoding to
> koi8-r, and I'm not sure you want that.  (Maybe you should consider
> describing more about what you want: what files do you want to edit,
> how would you like your keyboard to be set, and what characters can
> your terminal display.  There are probably a couple of lines you need
> to add to your .emacs to get what you want, but it's hard to give
> advice without knowing the conditions.)

I did describe what I want (but maybe in a different place in the
thread): Be as smart as cat is.  Allow characters which are input from
keyboard (TTY), and allow them to be written a file.

> > > If your LANG or LC_ALL environment variables are set appropriately,
> > > Emacs should automatically use the correct terminal-coding-system,
> > > language-environment etc.
> > 
> > Irrelevant.  I can't set LANG.

> That's quite understandable, but set-locale-environment from within
> Emacs has the same effect on Emacs, and doesn't affect anything but
> Emacs.

I do not follow.  Do you say that set-locale-environment is going to
be useless anyway if the locale is not supported by the CRTL?  Then
what should I do?

[Suppose for a moment that I do not care which 8-bit chars are word
 chars, which are whitespace etc.  All I need Emacs is to read bytes
 from TTY, and write them down to a file.]

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 16 May 2008 07:24:49 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > > When?  After (set-language-environment "Cyrillic-KOI8")?  nil
> > 
> > Then set it to whatever your terminal supports (koi8-r, I presume, but
> > that's a guess).  "M-x set-terminal-coding-system RET koi8-r RET".
> 
> And why set-language-environment did not do it?  

It isn't supposed to.  set-language-environment sets things up for
using a given language inside Emacs, but it does not assume your
terminal and keyboard necessarily support that language.  So it leaves
the job of specifying the terminal's and keyboard's encoding to you.

> > Alternatively, try "(set-locale-environment koi8-r)", and see if that
> > does what you want.  This will also set your keyboard encoding to
> > koi8-r, and I'm not sure you want that.  (Maybe you should consider
> > describing more about what you want: what files do you want to edit,
> > how would you like your keyboard to be set, and what characters can
> > your terminal display.  There are probably a couple of lines you need
> > to add to your .emacs to get what you want, but it's hard to give
> > advice without knowing the conditions.)
> 
> I did describe what I want (but maybe in a different place in the
> thread): Be as smart as cat is.  Allow characters which are input from
> keyboard (TTY), and allow them to be written a file.

Well, that is impossible, as I wrote elsewhere in this thread, not
with Emacs, anyway.

But what you say here is a solution, not the problem.  By ``what you
want'' I meant the description of your system setup as far as
non-ASCII support by its keyboard and display devices is considered.
I asked a few specific questions (see the citation above).  I'm quite
sure that if you describe that, an easy solution can be found in no
time.  I'm talking from experience: I'm using Cyrillic scripts all the
time in an environment that has nothing to do with Cyrillic languages
and doesn't support them natively.

> > > > If your LANG or LC_ALL environment variables are set appropriately,
> > > > Emacs should automatically use the correct terminal-coding-system,
> > > > language-environment etc.
> > > 
> > > Irrelevant.  I can't set LANG.
> 
> > That's quite understandable, but set-locale-environment from within
> > Emacs has the same effect on Emacs, and doesn't affect anything but
> > Emacs.
> 
> I do not follow.  Do you say that set-locale-environment is going to
> be useless anyway if the locale is not supported by the CRTL?

No, I'm saying that set-locale-environment does the same for Emacs as
setting LANG outside Emacs, with the crucial difference that the
former does not affect anything outside Emacs.  I assumed, perhaps
mistakenly, that you cannot set LANG because that would disrupt
operation of other programs on that machine.  If this assumption is
wrong, then please tell why you cannot set LANG; perhaps I
misunderstood the nature of your situation.

> [Suppose for a moment that I do not care which 8-bit chars are word
>  chars, which are whitespace etc.  All I need Emacs is to read bytes
>  from TTY, and write them down to a file.]

This cannot be done, not with Emacs.  Its current design simply
doesn't support such a paradigm, because the design assumes that
editing multilingual text is an important first-class goal of the
user.  It might be possible to tweak Emacs into behaving like what you
want, but I personally wouldn't bother to waste my time looking for
such a tweak, not the least because I don't need it: Emacs works well
for me, on text-mode and GUI terminals alike, with scripts that are
utterly alien to the underlying locale.  I'm quite sure it will work
well for you as well, once the nature of your system setup is
understood and accounted for in your .emacs.

Re: Is it possible to use new Emacsen with non-ASCII?

[Peter Dyballa]

Am 15.05.2008 um 02:37 schrieb Ilya Zakharevich:

> I spent (again!) a lot of time trying to teach Emacs to show 8-bit  
> data
> (in a TTY).


You should use a font in this terminal emulation (or real hardware?)  
that has these glyphs and it's best you have in the shell's  
environment LANG or better LC_CTYPE set to a value that arranges all  
you need. From LC_CTYPE GNU Emacs derives its "language-environment"  
automatically.

--
Greetings

   Pete

To most people solutions mean finding the answers. But to chemists  
solutions
are things that are still all mixed up.

Re: Is it possible to use new Emacsen with non-ASCII?

[Joel J. Adamson]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> I spent (again!) a lot of time trying to teach Emacs to show 8-bit data
> (in a TTY).  After a lot of experiments, I see that

Are you referring to a console, e.g. Linux console, or a terminal
emulator (such as xterm)?  You may need a utf-8 console, available in
newer versions of some distros (e.g., Slackware 12.1 that I just
installed last weekend); and you may need a capable font, many of which
are available.

Joel

-- 
Joel J. Adamson
Biostatistician
Pediatric Psychopharmacology Research Unit
Massachusetts General Hospital
Boston, MA  02114
(617) 643-1432
(303) 880-3109
Public key: http://pgp.mit.edu
http://www.unc.edu/~adamsonj

Ineffective security measure:

The information transmitted in this electronic communication is intended only
for the person or entity to whom it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or other
use of or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you received this
information in error, please contact the Compliance HelpLine at 800-856-1983 and
properly dispose of this information.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Peter Dyballa 
<Peter_Dyballa@Web.DE>], who wrote in article <mailman.11617.1210850110.18990.help-gnu-emacs@gnu.org>:
> 
> Am 15.05.2008 um 02:37 schrieb Ilya Zakharevich:
> 
> > I spent (again!) a lot of time trying to teach Emacs to show 8-bit  
> > data
> > (in a TTY).

> You should use a font in this terminal emulation (or real hardware?)  
> that has these glyphs

TTY works fine.  Thanks.

> and it's best you have in the shell's  
> environment LANG or better LC_CTYPE set to a value that arranges all  
> you need. From LC_CTYPE GNU Emacs derives its "language-environment"  
> automatically.

Impossible.  locale() does not support the environment of the TTY
(even if I *know* what is this environment, and why should I?).

Yours,
Ilya

P.S.  Let me remind what is an 8-bit-safe program: I come to a TTY.
      The keyboard allows me to enter some characters (encoded as
      8-bit).  The screen is able to show glyphs for these characters.

      Given this, I can use

	    cat >> file

      to "edit" a file.  Note that I do not care in any way what is
      the encoding of these chars...  Emacs should be at least as
      "smart" as cat.

      How should one convince Emacs to be as smart as cat, and why
      this is not documented?

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Thu, 15 May 2008 18:10:12 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> P.S.  Let me remind what is an 8-bit-safe program: I come to a TTY.
>       The keyboard allows me to enter some characters (encoded as
>       8-bit).  The screen is able to show glyphs for these characters.
> 
>       Given this, I can use
> 
> 	    cat >> file
> 
>       to "edit" a file.  Note that I do not care in any way what is
>       the encoding of these chars...  Emacs should be at least as
>       "smart" as cat.
> 
>       How should one convince Emacs to be as smart as cat, and why
>       this is not documented?

You need to decide whether you want Emacs or cat.  Emacs is not cat,
and cannot be convinced to work like cat.  Trying that will only
complicate your life more than you'd like, and won't give you what you
want.  In fact, using --unibyte you already complicate your life more
than necessary.

Let Emacs work like it was designed to: decode characters when you
visit a file and encode them back when it displays them on the screen.
Try it, it really works.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11644.1210878709.18990.help-gnu-emacs@gnu.org>:
> You need to decide whether you want Emacs or cat.  Emacs is not cat,
> and cannot be convinced to work like cat.

Well, with --unibyte it does act as cat.  What was it exactly which
you tried to convey?

> Let Emacs work like it was designed to: decode characters when you
> visit a file and encode them back when it displays them on the screen.
> Try it, it really works.

Thanks.  I did.  It did not.

Yours,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 16 May 2008 07:28:04 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > You need to decide whether you want Emacs or cat.  Emacs is not cat,
> > and cannot be convinced to work like cat.
> 
> Well, with --unibyte it does act as cat.

No, it does not.  What it does is jumping through the hoops to try
very hard to make you _think_ it does.  There's no way to force the
current versions of Emacs not to transform the bytestream it read from
a file on their way to the display.  And since --unibyte is a bastard
option as far as Emacs developers are concerned, it gets almost no
attention in testing and maintenance, and therefore tends to be buggy.
I recommend to stay away of it.

> > Let Emacs work like it was designed to: decode characters when you
> > visit a file and encode them back when it displays them on the screen.
> > Try it, it really works.
> 
> Thanks.  I did.  It did not.

Well, let's hear what you did to customize Emacs, and what problems
you saw.  But again, to give you efficient advice, I need to know what
scripts can your display and keyboard support.  For example, I assume
that you are using a text-mode terminal that can display Cyrillic
letters if it receives them KOI8-R encoded -- is that right?  What
about the keyboard -- how do you type Cyrillic characters, and what
8-bit codes does your keyboard setup produce? are they also KOI8-R
encoded?

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> Date: Fri, 16 May 2008 11:55:48 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Well, let's hear what you did to customize Emacs, and what problems
> you saw.  But again, to give you efficient advice, I need to know what
> scripts can your display and keyboard support.  For example, I assume
> that you are using a text-mode terminal that can display Cyrillic
> letters if it receives them KOI8-R encoded -- is that right?  What
> about the keyboard -- how do you type Cyrillic characters, and what
> 8-bit codes does your keyboard setup produce? are they also KOI8-R
> encoded?

Oh, and btw: what Emacs version is that? is that 22.2 or something
else?  (Apologies if you already told that and I missed it.)

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11667.1210930077.18990.help-gnu-emacs@gnu.org>:
> > Date: Fri, 16 May 2008 11:55:48 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > Well, let's hear what you did to customize Emacs, and what problems
> > you saw.  But again, to give you efficient advice, I need to know what
> > scripts can your display and keyboard support.  For example, I assume
> > that you are using a text-mode terminal that can display Cyrillic
> > letters if it receives them KOI8-R encoded -- is that right?  What
> > about the keyboard -- how do you type Cyrillic characters, and what
> > 8-bit codes does your keyboard setup produce? are they also KOI8-R
> > encoded?
> 
> Oh, and btw: what Emacs version is that? is that 22.2 or something
> else?  (Apologies if you already told that and I missed it.)

Sorry, I did not.  It is GNU Emacs 21.2.1

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 16 May 2008 11:16:29 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> It is GNU Emacs 21.2.1

I'd recommend to upgrade to 22.2, but the recipe I sent should work
with Emacs 21.x as well, except that -Q is not available; use -q
 --no-site-file instead.

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> Date: Fri, 16 May 2008 14:56:26 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> > Date: Fri, 16 May 2008 11:16:29 +0000 (UTC)
> > Bcc: ilya@gnu.org
> > Originator: ilya@powdermilk.math.berkeley.edu
> > 
> > It is GNU Emacs 21.2.1
> 
> I'd recommend to upgrade to 22.2, but the recipe I sent should work
> with Emacs 21.x as well

Sorry, I spoke too soon (I guess I underestimated how long ago I
switched from Emacs 21.x to 22.x).  Here's the equivalent recipe for
Emacs 21:

  emacs -q --no-site-file
  M-x set-language-environment RET Cyrillic-KOI8 RET
  C-x RET t Cyrillic-KOI8 RET
  C-x RET k Cyrillic-KOI8 RET

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11687.1210939428.18990.help-gnu-emacs@gnu.org>:
>   emacs -q --no-site-file
>   M-x set-language-environment RET Cyrillic-KOI8 RET
>   C-x RET t Cyrillic-KOI8 RET
>   C-x RET k Cyrillic-KOI8 RET

This works.  Now, why this is not documented?

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Sat, 17 May 2008 00:24:45 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> >   emacs -q --no-site-file
> >   M-x set-language-environment RET Cyrillic-KOI8 RET
> >   C-x RET t Cyrillic-KOI8 RET
> >   C-x RET k Cyrillic-KOI8 RET
> 
> This works.

Good.

> Now, why this is not documented?

Can you tell me how would you like it to be documented, and with what
index entries?  Emacs 21 is no longer maintained, but I could check in
the current development sources whether the documentation need some
work in this area.  The 3 individual commands I used are all
documented, so I need to understand what kind of text did you expect
to see that describes them all together, in order to see what needs to
be added to the manual.

Re: Is it possible to use new Emacsen with non-ASCII?

[Chris McMahan]

A 'thank you' would be nice for the help you received. If you want it
documented, write something up yourself and make a suggestion where to
place your it.

- Chris

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Eli Zaretskii 
> <eliz@gnu.org>], who wrote in article <mailman.11687.1210939428.18990.help-gnu-emacs@gnu.org>:
>>   emacs -q --no-site-file
>>   M-x set-language-environment RET Cyrillic-KOI8 RET
>>   C-x RET t Cyrillic-KOI8 RET
>>   C-x RET k Cyrillic-KOI8 RET
>
> This works.  Now, why this is not documented?
>
> Thanks,
> Ilya

-- 
     (.   .)
  =ooO=(_)=Ooo=====================================
  Chris McMahan | first_initiallastname@one.dot.net
  =================================================

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Chris McMahan <first_initiallastname@one.dot.net>
> Date: Mon, 19 May 2008 09:40:24 -0400
> 
> 
> If you want it documented, write something up yourself and make a
> suggestion where to place your it.

That would of course be nice, but it's not required.  I'm willing to
try to write it myself (if it's not already there), if only I knew
what Ilya wanted to find in the manual.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11853.1211223527.18990.help-gnu-emacs@gnu.org>:
> That would of course be nice, but it's not required.  I'm willing to
> try to write it myself (if it's not already there), if only I knew
> what Ilya wanted to find in the manual.

That part is very easy.  (And I explained it many times already.)

The hoops the user should go through to make Emacs 8-bit-clean in an
8-bit-clean TTY should be completely documented, in one section of the
manual.  (It would be yet greater if this were available via an
environment variable, such as

 env EMACS_8BIT_CLEAN=1 emacs -nw

).

Yours,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11687.1210939428.18990.help-gnu-emacs@gnu.org>:
> Sorry, I spoke too soon (I guess I underestimated how long ago I
> switched from Emacs 21.x to 22.x).  Here's the equivalent recipe for
> Emacs 21:
> 
>   emacs -q --no-site-file
>   M-x set-language-environment RET Cyrillic-KOI8 RET
>   C-x RET t Cyrillic-KOI8 RET
>   C-x RET k Cyrillic-KOI8 RET

I spoke too soon.  Typed literally, it works.  But the "equivalent"
programmatic code does not:

(let ((v (getenv "EMACS_LANGENV")))
  (and v (progn
           (set-language-environment v)
           (set-terminal-coding-system (intern v))
           (set-keyboard-coding-system (intern v))
           (set-input-mode (car (current-input-mode))
                           (nth 1 (current-input-mode))
                           '8bit)
           (run-with-idle-timer
            5 nil 'message              ; Once, after 5 sec of inactivity
            "Switch Cyr/Lat by Ctrl-\\, Cyr-Yawerty/Cyr-GOST by f9 \\"))))

I get

    Debugger entered--Lisp error: (coding-system-error Cyrillic-KOI8)
      set-terminal-coding-system-internal(Cyrillic-KOI8)
      set-terminal-coding-system(Cyrillic-KOI8)

What did I do wrong?

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[David Kastrup]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Eli Zaretskii 
> <eliz@gnu.org>], who wrote in article <mailman.11687.1210939428.18990.help-gnu-emacs@gnu.org>:
>> Sorry, I spoke too soon (I guess I underestimated how long ago I
>> switched from Emacs 21.x to 22.x).  Here's the equivalent recipe for
>> Emacs 21:
>> 
>>   emacs -q --no-site-file
>>   M-x set-language-environment RET Cyrillic-KOI8 RET
>>   C-x RET t Cyrillic-KOI8 RET
>>   C-x RET k Cyrillic-KOI8 RET
>
> I spoke too soon.  Typed literally, it works.  But the "equivalent"
> programmatic code does not:
>
> (let ((v (getenv "EMACS_LANGENV")))
>   (and v (progn
>            (set-language-environment v)
>            (set-terminal-coding-system (intern v))
>            (set-keyboard-coding-system (intern v))
>            (set-input-mode (car (current-input-mode))
>                            (nth 1 (current-input-mode))
>                            '8bit)
>            (run-with-idle-timer
>             5 nil 'message              ; Once, after 5 sec of inactivity
>             "Switch Cyr/Lat by Ctrl-\\, Cyr-Yawerty/Cyr-GOST by f9 \\"))))
>
> I get
>
>     Debugger entered--Lisp error: (coding-system-error Cyrillic-KOI8)
>       set-terminal-coding-system-internal(Cyrillic-KOI8)
>       set-terminal-coding-system(Cyrillic-KOI8)
>
> What did I do wrong?

'cyrillic-koi8 and 'Cyrillic-KOI8 are different symbols.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Is it possible to use new Emacsen with non-ASCII?

[Jason Rumney]

On May 22, 9:26 pm, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:

> >   emacs -q --no-site-file
> >   M-x set-language-environment RET Cyrillic-KOI8 RET
> >   C-x RET t Cyrillic-KOI8 RET
> >   C-x RET k Cyrillic-KOI8 RET
>
> I spoke too soon.  Typed literally, it works.  But the "equivalent"
> programmatic code does not:
>
> (let ((v (getenv "EMACS_LANGENV")))
>   (and v (progn
>            (set-language-environment v)
>            (set-terminal-coding-system (intern v))
>            (set-keyboard-coding-system (intern v))
>            (set-input-mode (car (current-input-mode))
>                            (nth 1 (current-input-mode))
>                            '8bit)
>            (run-with-idle-timer
>             5 nil 'message              ; Once, after 5 sec of inactivity
>             "Switch Cyr/Lat by Ctrl-\\, Cyr-Yawerty/Cyr-GOST by f9 \\"))))

Coding systems all have names that are lower-case. Interactively the
completion code will take care of the case conversion on input, but
programatically you'll have to downcase v before interning.

This will work for the Cyrillic-KOI8 language environment, and for a
few others where the language environment name matches the prefered
coding-system or an alias, but if you are looking for a more general
mechanism, I'd recommend using (car (get-language-info v 'coding-
priority)) as a more robust way to get the coding system from the
language environment. The downcase method will fail for language
environments like "Tajik", which specifes koi8-t as its prefered
coding-system (also "French", "German" etc which specify iso-latin-1).

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Jason Rumney 
<jasonrumney@gmail.com>], who wrote in article <bc82e254-8f58-4f79-9184-aecc6f65eabb@e53g2000hsa.googlegroups.com>:
> Coding systems all have names that are lower-case. Interactively the
> completion code will take care of the case conversion on input, but
> programatically you'll have to downcase v before interning.
> 
> This will work for the Cyrillic-KOI8 language environment, and for a
> few others where the language environment name matches the prefered
> coding-system or an alias, but if you are looking for a more general
> mechanism, I'd recommend using (car (get-language-info v 'coding-
> priority)) as a more robust way to get the coding system from the
> language environment. The downcase method will fail for language
> environments like "Tajik", which specifes koi8-t as its prefered
> coding-system (also "French", "German" etc which specify iso-latin-1).

Confirmed to work (both with unibyte mode, and without).  The current
code is

(let (vv (v (getenv "EMACS_LANGENV")))
  (and v (progn
           (set-language-environment v)
           (setq vv (car (get-language-info v 'coding-priority)))
           (set-terminal-coding-system vv)
           (set-keyboard-coding-system vv)
           (set-input-mode (car (current-input-mode))
                           (nth 1 (current-input-mode))
                           '8bit)
           (run-with-idle-timer
            5 nil 'message              ; Once, after 5 sec of inactivity
            "Switch Cyr/Lat by Ctrl-\\, Cyr-Yawerty/Cyr-GOST by f9 \\"))))

However, it become harder and harder to document as "a solution"...

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Jason Rumney]

On May 23, 10:55 am, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:

> However, it become harder and harder to document as "a solution"...

A solution to what? It looks like a kludge that is useful to you as a
workaround for a misconfigured system that does not support the locale
that is appropriate for your terminal. Most users in such a position
would be happy with the solution documented in the manual with hard-
coded values for language and coding-system:

(set-language-environment "Cyrillic-KOI8") ; From the "International"
node in the manual
(set-terminal-coding-system 'cyrillic-koi8)   ; From the "Terminal
Coding" node in the manual
(set-keyboard-coding-system 'cyrillic-koi8) ; likewise

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Jason Rumney 
<jasonrumney@gmail.com>], who wrote in article <bf47fc0f-3589-4da6-8a24-9d3f0f442f8b@2g2000hsn.googlegroups.com>:
> On May 23, 10:55 am, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:
> 
> > However, it become harder and harder to document as "a solution"...
> 
> A solution to what?

To work around the fact that by default Emacs is only 7-bit-clean.

> It looks like a kludge that is useful to you as a
> workaround for a misconfigured system

First of all, I never in my life saw a system which was not
misconfigured (and I saw thousands).

> that does not support the locale that is appropriate for your
> terminal.

Second, no system will ever support all the possible terminal
configurations.

> Most users in such a position
> would be happy with the solution documented in the manual with hard-
> coded values for language and coding-system:

> (set-language-environment "Cyrillic-KOI8") ; From the "International"
> node in the manual
> (set-terminal-coding-system 'cyrillic-koi8)   ; From the "Terminal
> Coding" node in the manual
> (set-keyboard-coding-system 'cyrillic-koi8) ; likewise

This looks to make Emacs 8-bit-clean.  If this solution were
documented in one particular place which is easy to find, I would be
quite happy.

Unfortunately, as you saw, even Emacs developers were not able to
provide a "quite working" solution on the first (and second) try.  I
do not think this indicates an acceptable state of the docs...

So "Most users" won't be "quite happy" with Emacs in 8-bit environments...

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Sat, 24 May 2008 06:58:20 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > On May 23, 10:55 am, Ilya Zakharevich <nospam-ab...@ilyaz.org> wrote:
> > 
> > > However, it become harder and harder to document as "a solution"...
> > 
> > A solution to what?
> 
> To work around the fact that by default Emacs is only 7-bit-clean.

Actually, Emacs is not even 7-bit-clean: if the file includes ESC
characters followed by what looks like ISO-2022 encoded non-ASCII
files, Emacs will try to decode those sequences.

I already told numerous times in this thread that being what you call
``7- or 8-bit clean'' is not an Emacs design goal.  Emacs design goal
is to DTRT with text files that include encoded non-ASCII characters.
In the absence of meta-data that would tell Emacs how a given file was
encoded and what encodings are supported by the terminal and the
keyboard, Emacs tries to consult the locale info, and failing that it
tries some guesswork.

When a system is set up such that none of this works, you have to
tinker with Emacs to make it DTRT.  That is your case.  Leaving alone
the philosophical issue of whether a system can be misconfigured or
not, your setup is certainly not within the mainstream of Unix system
setup model.  So it's a small wonder that it requires tinkering.

> Unfortunately, as you saw, even Emacs developers were not able to
> provide a "quite working" solution on the first (and second) try.

Emacs developers are not magicians.  Faced with your strangely
stubborn refusal to describe your system setup in detail, it's a small
wonder that it took many iterations to understand what kind of problem
you have that needs to be solved.  Once this was even remotely clear,
no thanks to you, a working solution was immediately posted for 2
different versions of Emacs, even though you then continued to demand
hand-holding with trivial problems which I'd expect a veteran Emacs
hacker such as yourself to be able to overcome in a matter of seconds.

All in all, it almost sounds like your urge to denigrate Emacs more
often than not overwhelms your own need to get help for solving your
problems.  So much so that you risk leaving your problems unsolved, as
long as you can kick Emacs in the butt.  Good luck with this attitude.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11686.1210938990.18990.help-gnu-emacs@gnu.org>:
> > It is GNU Emacs 21.2.1
> 
> I'd recommend to upgrade to 22.2,

If it were my computers, I would install the last version of
Emacs which supports internationalization, v19.

> but the recipe I sent should work
> with Emacs 21.x as well, except that -Q is not available; use -q
>  --no-site-file instead.

Well, it does not.

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Sat, 17 May 2008 00:21:54 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > I'd recommend to upgrade to 22.2,
> 
> If it were my computers, I would install the last version of
> Emacs which supports internationalization, v19.

Well, then just remember that whenever they decide to upgrade to 22,
you should switch to `(set-locale-environment "koi8-r")', although
what worked with 21.x should also continue working.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11744.1211008926.18990.help-gnu-emacs@gnu.org>:
> Well, then just remember that whenever they decide to upgrade to 22,
> you should switch to `(set-locale-environment "koi8-r")', although
> what worked with 21.x should also continue working.

Now I'm completely confused...  Would the invocation which works with
21 work also with 22?

If so, why would I want to switch from code which works with two
versions of Emacs to code which works only with one?

Puzzled,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Thu, 22 May 2008 20:23:25 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > Well, then just remember that whenever they decide to upgrade to 22,
> > you should switch to `(set-locale-environment "koi8-r")', although
> > what worked with 21.x should also continue working.
> 
> Now I'm completely confused...  Would the invocation which works with
> 21 work also with 22?

I just said (see above) that it should also work in 22.x.  I just
didn't try it, so I don't know for sure.

> If so, why would I want to switch from code which works with two
> versions of Emacs to code which works only with one?

Because (1) it's shorter, and (2) the 22.x method is The Right Way,
whereas the 21.x method is a kludgy approximation.

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.12090.1211533232.18990.help-gnu-emacs@gnu.org>:
> > If so, why would I want to switch from code which works with two
> > versions of Emacs to code which works only with one?
> 
> Because (1) it's shorter, and (2) the 22.x method is The Right Way,

(1) I do not care if it is short if does not work with many versions;

(2) I've heard such statements many times.  I will believe it when I
    can test it...

> whereas the 21.x method is a kludgy approximation.

Each iteration of this discussion makes "kludgy" look more and more
like an understatement...

Thanks,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11664.1210928215.18990.help-gnu-emacs@gnu.org>:
> > > You need to decide whether you want Emacs or cat.  Emacs is not cat,
> > > and cannot be convinced to work like cat.

> > Well, with --unibyte it does act as cat.

> No, it does not.  What it does is jumping through the hoops to try
> very hard to make you _think_ it does.

That's good enough for me.  So: how I convince Emacs to do the same
without --unibyte?

> > > Let Emacs work like it was designed to: decode characters when you
> > > visit a file and encode them back when it displays them on the screen.
> > > Try it, it really works.
> > 
> > Thanks.  I did.  It did not.
> 
> Well, let's hear what you did to customize Emacs, and what problems
> you saw.

I explained it in my first message, did not I?  What more do you want
to hear?  (LANG is undefined.)

For simplicity, you may assume that the user has no idea what is the
encoding of TTY input and output - but the user knows that they match
perfectly (and would remain the same tomorrow).

All user wants to do is to take notes (so they are visible in scratch
buffer), save them to a file, and get they intact next day (using the
same TTY).

> But again, to give you efficient advice, I need to know what
> scripts can your display and keyboard support.  For example, I assume
> that you are using a text-mode terminal that can display Cyrillic
> letters if it receives them KOI8-R encoded -- is that right?

OK, if this is the simplest case for you to cover: assume that the
user knows that the TTY works in KOI8-r.

> What about the keyboard -- how do you type Cyrillic characters,

This is a TTY.  You have no idea what I do to make it send certain
bytes - and you should not care.

> and what 8-bit codes does your keyboard setup produce?

All of them.  [Assume that 0..0x1f are reserved for standard control chars]

> are they also KOI8-R encoded?

As I said, input and output encodings match.

Hope this helps,
Ilya

Re: Is it possible to use new Emacsen with non-ASCII?

[Eli Zaretskii]

> From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
> Date: Fri, 16 May 2008 11:15:12 +0000 (UTC)
> Bcc: ilya@gnu.org
> Originator: ilya@powdermilk.math.berkeley.edu
> 
> > > Well, with --unibyte it does act as cat.
> 
> > No, it does not.  What it does is jumping through the hoops to try
> > very hard to make you _think_ it does.
> 
> That's good enough for me.  So: how I convince Emacs to do the same
> without --unibyte?

See below.

> OK, if this is the simplest case for you to cover: assume that the
> user knows that the TTY works in KOI8-r.
> 
> > What about the keyboard -- how do you type Cyrillic characters,
> 
> This is a TTY.  You have no idea what I do to make it send certain
> bytes - and you should not care.
> 
> > and what 8-bit codes does your keyboard setup produce?
> 
> All of them.  [Assume that 0..0x1f are reserved for standard control chars]
> 
> > are they also KOI8-R encoded?
> 
> As I said, input and output encodings match.

Then please try this:

  emacs -nw -Q
  M-: (set-locale-environment "koi8-r") RET

Does this do what you want if you then type Cyrillic text or visit
files with Cyrillic text encoded in KOI8-R?

Re: Is it possible to use new Emacsen with non-ASCII?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Eli Zaretskii 
<eliz@gnu.org>], who wrote in article <mailman.11681.1210938659.18990.help-gnu-emacs@gnu.org>:
>   emacs -nw -Q
>   M-: (set-locale-environment "koi8-r") RET
> 
> Does this do what you want if you then type Cyrillic text or visit
> files with Cyrillic text encoded in KOI8-R?

In no way.  Not even using C-\ (2-char sequences shown for each
keypress); 8-bit stuff entered from keyboard are interpreted as Meta.

Thanks,
Ilya