Re: windows-1251 language environment

Re: windows-1251 language environment

[Kenichi Handa]

At first, I have installed a mechanism of auto-loading a
coding system.  As Dave suggested long ago, I added a code
to auto-load a coding system in the function
Fcheck_coding_system.  See the change in code-pages.el which
adds one autoload cookie for iso-8859-11 as an example.

I have not yet tested it fully.  We may have to add GCPROs
in several places.

In article <200310060013.52049.pogonyshev@gmx.net>, Paul Pogonyshev <pogonyshev@gmx.net> writes:
>>  [Not even utf-8?]  That's not the point, though.  You're arguing
>>  essentially for an explosion of language environments crossing the
>>  existing ones with windows-125N &c (not just Cyrillic ones).  We think
>>  that isn't the right way to approach the problem and no-one else seems
>>  likely to work on this.

I agree that we should avoid creating many many predefined
language environments.

> This basically ends the discussion.  However, i don't agree that it's the
> most intuitive interface.

Right.  I think what we need for language environment is a
mechanism similar to face; i.e. creating a new one easily
while allowing inheriting, and customizing an existing one
easily.

> Let's imagine how it happens.  People look at language/coding support and
> see language environment.  They think: "Aha, i set a language environment
> and everything works".  But then they suddenly discover that setting
> "Russian" (or some else) language environment doesn't suffice.  They look
> for another, similar environment, but with different codepage.  And don't
> found it.

For instance, in such a case, we can allow people to create
a new lang. env. by inheriting, for instance, Russian, and
modifying coding-system to windows-1251.

C-h L (describe-language-environment) should also have
these clickable lines:
	You can customize this language environment.
and
	You can create a new language environment that inherits
	this language environment.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Anton Zinoviev]

On  7.X.2003 at 11:54 Kenichi Handa wrote:
> 
> For instance, in such a case, we can allow people to create
> a new lang. env. by inheriting, for instance, Russian, and
> modifying coding-system to windows-1251.

That is a good feature, it will be usefull say for Bulgarian as
Bulgarians use two completely different keyboard layouts.  The example
given here however is very bad.

CP1251 is not similar to the others CP???? coding systems because it
is used extensively for several languages in the GNU systems.
Unfortunately the name of this coding system often confuses people.
This coding system is standard for Belarusian and Bulgarian languages
and it is used very often also for other languages.  It supports and
_is_ often used for all non-Asian Cyrillic languages.

The generic language environment for CP1251 should have a generic
input method (rather than Russian).  The TUTORIAL should not be
presented in Russian as many people outside the former SU do not know
Russian.  The mails sent in this environment should not be encoded in
koi8-r.

Anton Zinoviev

P.S. On 26 September I sent here a message regarding the Cyrillic
     support of Emacs.  

Re: windows-1251 language environment

[Dave Love]

Anton Zinoviev <anton@lml.bas.bg> writes:

> Bulgarians use two completely different keyboard layouts.

> CP1251 is not similar to the others CP???? coding systems because it
> is used extensively for several languages in the GNU systems.

I don't understand this.  As far as I know, the only significant
difference between that and the other 125N charsets is that 1251 is
used by the two GNU locales.  Presumably the others are widely used
for the languages they support.

> The generic language environment for CP1251 should have a generic
> input method (rather than Russian).

I'm still unconvinced that there should be such a _language_
environment.

> The TUTORIAL should not be presented in Russian as many people
> outside the former SU do not know Russian.

Actually, the environments named after a charset don't have tutorials.
I think the tutorial is currently the only thing language-specific in
language environments unless you count input methods which only make
sense in specific languages.

> The mails sent in this environment should not be encoded in koi8-r.

I'm surprised that might happen if you prefer a different coding
system, but I only know how Gnus Message mode works in detail.  [I
don't know why there are a separate Message and Sendmail modes...]

Actually, how mail should be encoded isn't always clear cut.  For
instance, if you are replying to koi8-r mail, you could argue the
reply should be in koi8-r even if you have preferred a different
Cyrillic charset generally.

By the way, I think the Bulgarian tutorial should be encoded in
windows-1251.  Only non-Bulgarians commented (negatively) on that as
far as I know.

Re: windows-1251 language environment

[Anton Zinoviev]

On 12.X.2003 at 16:45 Dave Love wrote:
> 
> > CP1251 is not similar to the others CP???? coding systems because it
> > is used extensively for several languages in the GNU systems.
> 
> I don't understand this. 

What is the difference between CP1251 and some ISO 8859 coding system?
The only difference that I can see is the name.

> As far as I know, the only significant difference between that and
> the other 125N charsets is that 1251 is used by the two GNU locales.
> Presumably the others are widely used for the languages they
> support.

I'am pretty sure that the other 125N coding systems are not as widely
used in GNU systems as 1251 is.  The other 125N coding systems have
corresponding ISO 8859.  You don't have to use CP1252 because you can
use ISO 8859-1.  The correspondence between CP1250 and ISO8859-2 is
not as close, but again -- if your encoding is ISO8859-2 you can read
text coded with CP1250.  However if you have text coded with CP1251
then you want be able to read anything unless your system uses exactly
CP1251.  Thats one reason why CP1251 became standard for the
Belarusian and Bulgarian Glibc locales and thats why there are some
GNU/Linux distributions that make CP1251 standard also for Russian and
Ukrainian.  Another reason is that among plenty of Cyrillic coding
systems CP1251 is the only that supports all Cyrillic Slavic
languages.  It supports also several non-slavic Cyrillic languages.

It is interesting to know which CP125N encodings XFree86 supports.
That are CP1251, CP1255 and CP1256.  The first is Cyrillic, the second
is Hebrew and the third is Arabic.  This is another demonstration of
the problems that non-Latin users have.

Anton Zinoviev

Re: windows-1251 language environment

[Terje Rosten]

* Anton Zinoviev
| 
| What is the difference between CP1251 and some ISO 8859 coding system?

>From Unicode Demystified, Richard Gillam, ISBN 0-201-70052-2, page 41:
  
  Windows code page 1252, the Western European encoding for Windows,
  for example, is a superset of ISO 8859-1. It includes all of the
  Latin-1 characters, but then fills the C1 space with additional
  printing characters.

I believe the same is true for code page 1251.

The C1 space in ISO 8859 is reserved for control characters.


 - Terje

Re: windows-1251 language environment

[Anton Zinoviev]

On 17.X.2003 at 15:22 Terje Rosten wrote:
> * Anton Zinoviev
> | 
> | What is the difference between CP1251 and some ISO 8859 coding system?
> 
> >From Unicode Demystified, Richard Gillam, ISBN 0-201-70052-2, page 41:
>   
>   Windows code page 1252, the Western European encoding for Windows,
>   for example, is a superset of ISO 8859-1. It includes all of the
>   Latin-1 characters, but then fills the C1 space with additional
>   printing characters.
> 
> I believe the same is true for code page 1251.

No, that is not true.

I'd like to stress that cp1251 is a non-latin coding system.  Even if
you terminal supports only ASCII, then you probably can read French or
German or Spanish, etc, as most of the characters are basic latin.
But try to use the command `iconv -tebcdic-us' in order to recode some
English text to EBCDIC.  Then you will understand better the
difference between the Latin and the non-Latin cp125N coding systems.

Anton Zinoviev

Re: windows-1251 language environment

[Terje Rosten]

* Anton Zinoviev
| 
| No, that is not true.
| 
| I'd like to stress that cp1251 is a non-latin coding system.

Sorry, I was a bit quick. Overlook the Latin part, it seems to me that
code page 1251 is filled with printing characters in the C1 space
while all ISO 8859 encodings have control characters here. Hence it is
a difference between code page 1251 and any ISO 8859 encoding.


 - Terje

Re: windows-1251 language environment

[Dave Love]

Anton Zinoviev <anton@lml.bas.bg> writes:

> You don't have to use CP1252 because you can use ISO 8859-1.

Please don't tell me what charsets I need.  I've provided support for
the ones you need -- with considerable opposition -- and I've fought
for people to be able to use which charsets they want/need.

> This is another demonstration of the problems that non-Latin users
> have.

I'm sorry, I don't understand the problem, and particularly why you're
saying this to me.  It sounds as though you're confusing me with the
Russian-speaking maintainer who has long opposed providing the proper
windows-125[12] repertoire.  There is rather extensive support for
Cyrillic and other non-Latin scripts in Emacs.  The multilingual
features actually come from Japan.  Obviously there is room for
improvement, but not by considering particular scripts or character
sets as special cases.

Re: windows-1251 language environment

[Richard Stallman]

      Obviously there is room for
    improvement, but not by considering particular scripts or character
    sets as special cases.

There is no aprioii principle against considering them as special cases.
there have to be valid special reasons for doing do.

Re: windows-1251 language environment

[Anton Zinoviev]

On 21.X.2003 at 23:38 Dave Love wrote:
> Anton Zinoviev <anton@lml.bas.bg> writes:
> 
> > You don't have to use CP1252 because you can use ISO 8859-1.
> 
> Please don't tell me what charsets I need.

If my message seams offensive to you -- I apologise.

When I wrote "You don't have to use CP1252 because you can use ISO
8859-1" I wanted to write "People don't have to use CP1252 because
they can use ISO 8859-1".

> I've provided support for the ones you need -- with considerable
> opposition -- and I've fought for people to be able to use which
> charsets they want/need.

I know this -- thank you.

Anton Zinoviev

Re: windows-1251 language environment

[Richard Stallman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Inheritance of language environments could be a useful idea,
but I am not sure how much it would save.  Consider this one:

    (set-language-info-alist
     "French" '((tutorial . "TUTORIAL.fr")
		(charset ascii latin-iso8859-1)
		(coding-system iso-latin-1 iso-latin-9)
		(coding-priority iso-latin-1)
		(nonascii-translation . latin-iso8859-1)
		(unibyte-syntax . "latin-1")
		(unibyte-display . iso-latin-1)
		(input-method . "latin-1-prefix")
		(sample-text . "French (Français)	Bonjour, Salut")
		(documentation . "\
    This language environment is almost the same as Latin-1,
    but it selects the French tutorial and input method."))
     '("European"))

We could make it shorter using inheritance, but it is already pretty
short; is it really worth while making it any shorter?

Re: windows-1251 language environment

[Kenichi Handa]

In article <E1A76I5-0002tC-Rk@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

> Inheritance of language environments could be a useful idea,
> but I am not sure how much it would save.  Consider this one:
[...]
> We could make it shorter using inheritance, but it is already pretty
> short; is it really worth while making it any shorter?

The purpose of inheritance is not only to make it shorter.
With inheritance, users can make a new lang. env. easily by
inheriting the one that matches best to his purpose.

Another way is to make a new lang. env. by copying an
existing one.

With inheritance, the change to parent affects lang. envs
inheritting it.  With copying, no.  So, purhaps both method
is useful (as well as the case of face).

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Richard Stallman]

    With inheritance, the change to parent affects lang. envs
    inheritting it.  With copying, no.  So, purhaps both method
    is useful (as well as the case of face).

Is it useful enough to warrant spending time writing it and debugging
it, and spend time and paper documenting it?  We could do other things
instead.

Re: windows-1251 language environment

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> Is it useful enough to warrant spending time writing it and debugging
> it, and spend time and paper documenting it?  We could do other things
> instead.

Apart from making `language' settings interact well with the locale
charset, another thing that needs to be done in this area is
customizing fontsets conveniently and making the default font usage
sensible for the locale.  E.g. users presumably want to favour a
windows-1251 font for all characters it can encode when the locale
uses windows-1251 (although in that case it isn't a standard X
encoding, and I don't know if any X fonts use it).

I ran into problems related to normal face customization when I worked
on it, and I don't think they got fixed.

Re: windows-1251 language environment

[Kenichi Handa]

In article <E1A7c1w-00079A-Bu@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     With inheritance, the change to parent affects lang. envs
>     inheritting it.  With copying, no.  So, purhaps both method
>     is useful (as well as the case of face).

> Is it useful enough to warrant spending time writing it and debugging
> it, and spend time and paper documenting it?  We could do other things
> instead.

This issue is not that urgent, but I'm sure we must do
something to allow users to customize a language environment
and to avoid the explosion of numbers of language
environments.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Richard Stallman]

    This issue is not that urgent, but I'm sure we must do
    something to allow users to customize a language environment

Why aren't the existing facilities enough for that?

    and to avoid the explosion of numbers of language
    environments.

Asking users to define their own language environments, instead of
defining them in Emacs, can help with that.  I don't see how
inheritance would help.

Re: windows-1251 language environment

[Kenichi Handa]

In article <E1A9Usz-0006PI-SL@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     This issue is not that urgent, but I'm sure we must do
>     something to allow users to customize a language environment

> Why aren't the existing facilities enough for that?

Existing facility is to write Emacs Lisp code in .emacs
which is not for normal users.

>     and to avoid the explosion of numbers of language
>     environments.

> Asking users to define their own language environments, instead of
> defining them in Emacs, can help with that.  I don't see how
> inheritance would help.

Currently, a user must define his own language environment
from scratch.  But, usually what they want is to prefer the
diffent default input method, prefer the different coding
system than the existing one.  In such a case, what he have
to do is, currently, see source code of
lisp/language/XXX.el, copy

   (set-language-info-alist ....)

into his .emacs, and modify some part of it.  If we allow
inheritance, he can do something like:

(define-language-environment Japanese-utf-8
  '((inherit . "Japanese")
    (coding-sytem utf-8 iso-2022-jp euc-jp)))

>     My intention is to have something like "new" under this menu:
> 	    <menu-bar> <options> <mule> <set-language-environment>
>     When "new" is selected, Emacs put a user in a buffer in
>     which he can create his own language environment
>     interactively.  Typing C-c C-c should ask if he want to use
>     that environment in the future session.

> That seems like a good feature.  It could be implemented with
> or without define-derived-language.

Yes, but in the above buffer, it seems that "inherit" and
"copy" button is helpful.  For instance, clicking "inherit"
and selecting LANG will show the default values for each
slots.  Clicking "copy" and selecting LANG fills all slots
by values of LANG.

I don't intend to have the actual functions
define-derived-language.  I think just adding a new key
"inherit" in language-info-alist is enough.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Stefan Monnier]

> Currently, a user must define his own language environment
> from scratch.  But, usually what they want is to prefer the
> diffent default input method, prefer the different coding
> system than the existing one.  In such a case, what he have
> to do is, currently, see source code of
> lisp/language/XXX.el, copy

>    (set-language-info-alist ....)

> into his .emacs, and modify some part of it.  If we allow
> inheritance, he can do something like:

> (define-language-environment Japanese-utf-8
>   '((inherit . "Japanese")
>     (coding-sytem utf-8 iso-2022-jp euc-jp)))

I've never seen anyone do that.  Instead, people do:

   (set-language-environment "foo")
   (prefer-coding-system 'bar)
   (setq default-input-method "baz")

That doesn't look particularly bad to me, except for the fact that
some things are set via `setq' others via `set-foo-bar', etc...
And of course the fact that it can only be done in elisp but
not in `custom'.

> Yes, but in the above buffer, it seems that "inherit" and
> "copy" button is helpful.  For instance, clicking "inherit"
> and selecting LANG will show the default values for each
> slots.  Clicking "copy" and selecting LANG fills all slots
> by values of LANG.

I don't think users who are unable to use elisp will be able to make any
use of the subtle distinction between `inherit' and `copy'.  I think only
`inherit' makes sense in a user-interface.


        Stefan

Re: windows-1251 language environment

[Stephen J. Turnbull]

>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

    Stefan> I've never seen anyone do that.  Instead, people do:

    Stefan>    (set-language-environment "foo")
    Stefan>    (prefer-coding-system 'bar)
    Stefan>    (setq default-input-method "baz")

On my Windows box I have three Japanese environments.  One is for
TRAMPed files from my Linux host where the coding system is EUC, the
second is native and Shift-JIS, and the third can be used on either
with UTF-8.  I also fiddle the coding priority list more generally,
although I'm not sure that is really needed.  Pretty soon I hope to
add Korean to the list of needed environments (Japanese serves well
enough for English and Lisp).

It would be nice to be able to do all the changes with one
set-language-environment (usually invoked from the menubar) call.


-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: windows-1251 language environment

[Stefan Monnier]

> On my Windows box I have three Japanese environments.  One is for
> TRAMPed files from my Linux host where the coding system is EUC, the
> second is native and Shift-JIS, and the third can be used on either
> with UTF-8.  I also fiddle the coding priority list more generally,
> although I'm not sure that is really needed.  Pretty soon I hope to
> add Korean to the list of needed environments (Japanese serves well
> enough for English and Lisp).

Interesting.
How does this use of multiple language environments work in practice?
I mean, the language environment setting is global, so how does it
interact with the various pre-existing buffers when you switch?

What is changed between those environments other than the preferred coding
system ?

Is this arrangement natural, convenient, and intuitive, or is it more
like a work around for lack of other forms of configurations (such as a way
to specify preferred coding system based on mount-points, for example) ?


        Stefan

Re: windows-1251 language environment

[Stephen J. Turnbull]

>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

    Stefan> How does this use of multiple language environments work
    Stefan> in practice?  I mean, the language environment setting is
    Stefan> global, so how does it interact with the various
    Stefan> pre-existing buffers when you switch?

I have a C-c binding to switch language environments, works like C-x
b.  Emacs doesn't know a buffer<->language environment mapping.

I suspect there may be underlying problems, for example with XIM,
because my usage is quite restricted and I just may not be seeing
them.  (I don't use XIM because my preferred IM is kinput2, which eats
meta characters in its Xt filter and thus conflicts badly with Emacs.)

    Stefan> What is changed between those environments other than the
    Stefan> preferred coding system ?

(1) The whole coding priority list.  On Unix I have lots of files in
ISO-2022-JP, and I need to bump the priority of little-endian UTF-16
too, so that I can (sometimes) get autorecognition of .doc files (I
don't know of a better doc2txt utility than Emacs for _Japanese_ .doc
at this time).  I don't do mail from Windows, and I have Word, so I
don't do those tweaks on Windows.

(2) If I were capable of handling other languages, the default input
method would change.  (This would probably have to tweak C-\'s notion
of current input method and its state of activation, too, my language
switch function does that when I switch to or from English.)

(3) If these hypothetical other languages were European, I'd probably
want my ispell dictionary to change, too.  (Yah, that's not part of
an Emacs "language environment", but maybe it should be?)

    Stefan> Is this arrangement natural, convenient, and intuitive,

Natural, well, we're talking about a perverse environment to start with.

I tend to work on one system, multiple tasks, for an extended amount
of time.  So I don't need to switch very often.  I think the "natural"
thing to do (in XEmacs) would be to make language environment be a
specifier variable (XEmacs's generalization of buffer/frame locals).

Actually, what I'd _really_ like is to automatically be in Japanese
language environment (with input method invoked) in regions of
japanese-jisx*, and in English environment elsewhere.  The whole POSIX
concept of locale (as a global variable that "gets set") gives me an
itch.  It makes sense in the POSIX environment, at least a little bit,
but Emacs should be able to do a lot better.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: windows-1251 language environment

[Dave Love]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

[About something that hasn't reached me.]

> Is this arrangement natural, convenient, and intuitive,

It doesn't look that way to me.  If there's some problem (in Emacs)
with using the multilingual features buffer-locally, it probably
should be fixed.

> or is it more
> like a work around for lack of other forms of configurations (such as a way
> to specify preferred coding system based on mount-points, for example) ?

You can do that in Emacs.

Re: windows-1251 language environment

[Dave Love]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> And of course the fact that it can only be done in elisp but
> not in `custom'.

At least the language environment and the default input method are
customized.

Re: windows-1251 language environment

[Richard Stallman]

    Currently, a user must define his own language environment
    from scratch.  But, usually what they want is to prefer the
    diffent default input method, prefer the different coding
    system than the existing one.

Other people pointed out that it's easy to do this by first specifying
the standard language environment, and second overriding other
specific data.

A customization UI could generate the Lisp code in .emacs to specify
the information in exactly that way.

Re: windows-1251 language environment

[Kenichi Handa]

Richard Stallman <rms@gnu.org> writes:
>     Currently, a user must define his own language environment
>     from scratch.  But, usually what they want is to prefer the
>     diffent default input method, prefer the different coding
>     system than the existing one.

> Other people pointed out that it's easy to do this by first specifying
> the standard language environment, and second overriding other
> specific data.

> A customization UI could generate the Lisp code in .emacs to specify
> the information in exactly that way.

That makes the UI code unnecessarily complicated.  In
addition, in that way, C-h L (describe-language-environment)
doesn't work.

I think it is simpler to make the UI allows a user to
customize the variable `language-info-alist' by modifying
the existing slot or by adding a new slot to
language-info-alist.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Dave Love]

Kenichi Handa <handa@m17n.org> writes:

> At first, I have installed a mechanism of auto-loading a
> coding system.

Is it really worth it for the 8-bit sets?  I'm not sure whether to
think this is significant or not:

let ((stats (garbage-collect)))
  (load "code-pages")
  (pp (list stats (garbage-collect))))
 => (((50679 . 9813)
      (10196 . 1)
      (587 . 129)
      127582 175645
      (55 . 76)
      (48 . 22)
      (8025 . 1488))
     ((54148 . 6344)
      (10538 . 39)
      (588 . 128)
      134356 312873
      (55 . 76)
      (48 . 22)
      (8443 . 1070)))

> I have not yet tested it fully.  We may have to add GCPROs
> in several places.

<URL:ftp://dlpx1.dl.ac.uk/fx/emacs/Mule/autoload-coding-systems.diff>
was reasonably well tested, probably also on a system where gcpro was
relevant then (though I don't know if I tested it by forcing GC during
loading).

> Right.  I think what we need for language environment is a
> mechanism similar to face; i.e. creating a new one easily
> while allowing inheriting, and customizing an existing one
> easily.

I did something about customizing the language alist some time ago.
As far as I remember, it naturally used the mechanism for overriding
elements of the standard value in customized lists, and that interface
was vetoed for reasons I didn't understand.  Presumably inheritance
would use a similar mechanism, but I'm not sure how useful it is.

As it is, a language environment has to cover several things that
should be orthogonal:

 * The language (which might influence input methods as well as the
   default Ispell dictionary, at least);

 * The charset/coding system preferences (which might also influence
   input methods, though the hooks now in Quail make that less of an
   issue);

 * Other things that currently aren't dealt with properly, like paper
   size (for ps-print), calendar holidays &c (which may depend on the
   locale territory, not just the language).

> For instance, in such a case, we can allow people to create
> a new lang. env. by inheriting, for instance, Russian, and
> modifying coding-system to windows-1251.

Is this actually better than allowing them to specify (the equivalent
of) the locale ru_RU.windows-1251?

Re: windows-1251 language environment

[Stefan Monnier]

>> For instance, in such a case, we can allow people to create
>> a new lang. env. by inheriting, for instance, Russian, and
>> modifying coding-system to windows-1251.
> Is this actually better than allowing them to specify (the equivalent
> of) the locale ru_RU.windows-1251?

Completely agreed.  Locales are pretty limited, but language environments
even more so.  And locales are more likely to be already properly setup
(for other applications).

Language environments might make sense internally, but I think we should
encourage people to configure their Emacs by doing:
- 1st and foremost, set your locale properly, or call
  `set-locale-environment' in your .emacs.
- If that's not quite right, fix the parts you don't like directly
  by specifying preferred coding systems, preferred language for tutorial,
  preferred input systems, ...

Also I wonder what would be the point of doing:

   (define-derived-language "foo" "bar"
     bla bla bla)
   (set-language-environment "foo")

rather than

   (set-language-environment "bar")
   bla bla bla

Or do you expect define-derived-language to be used elsewhere than in
a .emacs ?


        Stefan

Re: windows-1251 language environment

[Richard Stallman]

    Language environments might make sense internally, but I think we should
    encourage people to configure their Emacs by doing:
    - 1st and foremost, set your locale properly, or call
      `set-locale-environment' in your .emacs.

This might make sense if these locale names are effectively standard.
They are not standardized by POSIX.  Is there a de-facto standard?
How widely is it used?

In general, I would prefer to have some Emacs-native way to do set
these things, rather than rely on an external data type such as a
locale name which is not defined naturally in Emacs terms.

Re: windows-1251 language environment

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> This might make sense if these locale names are effectively standard.
> They are not standardized by POSIX.  Is there a de-facto standard?

See `locale-language-names' and comments in it.  The major problem is
interpreting the codeset part in terms of a coding system since the
names are quite variable.

I don't know how the equivalent is supposed to work in MS Windows &
al.

> In general, I would prefer to have some Emacs-native way to do set
> these things, rather than rely on an external data type such as a
> locale name which is not defined naturally in Emacs terms.

If you don't try to interpret it, Emacs probably won't behave
consistently with other programs.  Anyhow, you can set the coding
priority and input method trivially, and they are the only relevant
things at present.

Re: windows-1251 language environment

[Richard Stallman]

    See `locale-language-names' and comments in it.  The major problem is
    interpreting the codeset part in terms of a coding system since the
    names are quite variable.

    I don't know how the equivalent is supposed to work in MS Windows &
    al.

It sounds like you are saying that there's a more-or-less standard
for GNU and Unix systems, but that Windows is different.  Is that right?

    > In general, I would prefer to have some Emacs-native way to do set
    > these things, rather than rely on an external data type such as a
    > locale name which is not defined naturally in Emacs terms.

    If you don't try to interpret it, Emacs probably won't behave
    consistently with other programs.

We are miscommunicating.  Of course Emacs should pay attention to the
locale, as well as it reliably can do so.  What I said is that Emacs
should not *rely on* an external mechanism such as locales for
customization.  It should provide an Emacs-natural customize mechanism
*also*.

Re: windows-1251 language environment

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> It sounds like you are saying that there's a more-or-less standard
> for GNU and Unix systems, but that Windows is different.  Is that
> right?

Actually, it looks as if I mis-interpreted eggert's comments in
mule-cmds.el.  They appear to say that the components of the name are
standardized, but the Single Unix Spec v3 (now equivalent to POSIX?)
actually says they are implementation-defined:

  If the locale value has the form:
                 
  language[_territory][.codeset]
  
     it refers to an implementation-provided locale, where settings of
     language, territory, and codeset are implementation-defined.
          
     LC_COLLATE , LC_CTYPE , LC_MESSAGES , LC_MONETARY , LC_NUMERIC , and
     LC_TIME are defined to accept an additional field @ modifier, which
     allows the user to select a specific instance of localization data
     within a single category (for example, for selecting the dictionary as
     opposed to the character ordering of data). The syntax for these
     environment variables is thus defined as:
            
  [language[_territory][.codeset][@modifier]]

            
Of the systems I can check, current Solaris, Tru64 and Irix do use the
ISO language and territory codes.

> What I said is that Emacs
> should not *rely on* an external mechanism such as locales for
> customization.  It should provide an Emacs-natural customize mechanism
> *also*.

I don't think customization based on language, territory and codeset
is really unnatural.  (Which isn't to say that locales are perfect.)
The relevant parameters can already be set individually in Lisp,
obviously.

Re: windows-1251 language environment

[Richard Stallman]

      They appear to say that the components of the name are
    standardized, but the Single Unix Spec v3 (now equivalent to POSIX?)
    actually says they are implementation-defined:

POSIX has never tried to specify the locales.  But if there is a
broadly accepted de-facto standard, that's just as good for us to use
as an official standard.

    > What I said is that Emacs
    > should not *rely on* an external mechanism such as locales for
    > customization.  It should provide an Emacs-natural customize mechanism
    > *also*.

    I don't think customization based on language, territory and codeset
    is really unnatural.

1. You specify the locale with an envvar.  Emacs must provide
a way to do it with commands or user options.

2. It is ok to have a command or user option whose argument or value
is a locale name.  But there should also be a command or user option
where you specify just the language, and other things default from
that to the extent possible.  And that's what should be in the menus.

Re: windows-1251 language environment

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> 1. You specify the locale with an envvar.  Emacs must provide
> a way to do it with commands or user options.

I'm not disagreeing.  (I actually thought `set-locale-environment' was
interactive, but I see it isn't.)

> 2. It is ok to have a command or user option whose argument or value
> is a locale name.  But there should also be a command or user option
> where you specify just the language, and other things default from
> that to the extent possible.  And that's what should be in the menus.

The trouble with `language' environments is that they have little to
do with the language per se.  If you're not going by system defaults
(locales), I think you want to provide orthogonal customization of
language, codeset, and other features.  If you lump them all together,
you'll have probably four for each Western European language: latin-1,
latin-9, windows-1252 and utf-8.  Then multiply by two or three for
the possible input methods.  Then the calendar is different in
territories with the same language...

Re: windows-1251 language environment

[Alex Schroeder]

Dave Love <d.love@dl.ac.uk> writes:

> The trouble with `language' environments is that they have little to
> do with the language per se.  If you're not going by system defaults
> (locales), I think you want to provide orthogonal customization of
> language, codeset, and other features.  If you lump them all together,
> you'll have probably four for each Western European language: latin-1,
> latin-9, windows-1252 and utf-8.  Then multiply by two or three for
> the possible input methods.  Then the calendar is different in
> territories with the same language...

I agree with Dave Love.  Take Switzerland as an example, and let us
only talk about German and French, excluding Italian and
Rhaeto-Romance.  We speak German, but we use a different ispell
dictionary because instead of a sharp s we use double-s, and we use
our own keyboard layout.  We use any of the four coding systems Dave
lists.  We use a different date format (last time I checked).  The
French speaking Swiss use the French ispell dictionary, but they, too,
have a different keyboard layout.

The language per se is not enough information.

Alex.
-- 
http://www.emacswiki.org/alex/
There is no substitute for experience.

Re: windows-1251 language environment

[Kenichi Handa]

In article <87ekx0dne0.fsf@emacswiki.org>, Alex Schroeder <alex@emacswiki.org> writes:

> Dave Love <d.love@dl.ac.uk> writes:
>>  The trouble with `language' environments is that they have little to
>>  do with the language per se.  If you're not going by system defaults
>>  (locales), I think you want to provide orthogonal customization of
>>  language, codeset, and other features.  If you lump them all together,
>>  you'll have probably four for each Western European language: latin-1,
>>  latin-9, windows-1252 and utf-8.  Then multiply by two or three for
>>  the possible input methods.  Then the calendar is different in
>>  territories with the same language...

> I agree with Dave Love.  Take Switzerland as an example, and let us
> only talk about German and French, excluding Italian and
> Rhaeto-Romance.  We speak German, but we use a different ispell
> dictionary because instead of a sharp s we use double-s, and we use
> our own keyboard layout.  We use any of the four coding systems Dave
> lists.  We use a different date format (last time I checked).  The
> French speaking Swiss use the French ispell dictionary, but they, too,
> have a different keyboard layout.

> The language per se is not enough information.

That's why I didn't simply use the term "language" but used
"language environment".  I agree that it's ridiculous to
provide all possible combinations in advance.  But, I think
there should be a way to name a set of various orthogonal
customizations so that a user can easily switch from one set
to another by specifying a name.  I think we should treat a
"language environment" as something like a `theme' which is
a collection of customizations of faces.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Richard Stallman]

    The trouble with `language' environments is that they have little to
    do with the language per se.  If you're not going by system defaults
    (locales), I think you want to provide orthogonal customization of
    language, codeset, and other features.  If you lump them all together,
    you'll have probably four for each Western European language: latin-1,
    latin-9, windows-1252 and utf-8.

that wd be tru if thatwere the one and only way to spec all of them.
but we already have other ways in emacs.

Re: windows-1251 language environment

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

>     The trouble with `language' environments is that they have little to
>     do with the language per se.  If you're not going by system defaults
>     (locales), I think you want to provide orthogonal customization of
>     language, codeset, and other features.  If you lump them all together,
>     you'll have probably four for each Western European language: latin-1,
>     latin-9, windows-1252 and utf-8.
>
> that wd be tru if thatwere the one and only way to spec all of them.
> but we already have other ways in emacs.

I was responding to:

  But there should also be a command or user option
  where you specify just the language, and other things default from
  that to the extent possible.  And that's what should be in the menus.

I can't see how simply `language' is the appropriate way to deal with
this, and a one-dimensional specification of the parameters seems
quite impractical, as above.  Also I can't see why customizing
language, territory and character set (with defaults for the latter
two) isn't a reasonable thing to offer users.  I have given this some
thought and some implementation.

If you must consider only language, why isn't this an appropriate use
of Custom themes, anyway?

Re: windows-1251 language environment

[Richard Stallman]

with broken arm, i dont have time to keep on arguing w you
about this, so youll just have to accept the decision ive stated.

Re: windows-1251 language environment

[Jason Rumney]

Dave Love <d.love@dl.ac.uk> writes:

> See `locale-language-names' and comments in it.  The major problem is
> interpreting the codeset part in terms of a coding system since the
> names are quite variable.
> 
> I don't know how the equivalent is supposed to work in MS Windows &
> al.

MS Windows uses three letter codes for languages instead of the two
letter ISO codes that POSIX uses. In most cases the first two letters
are the same, so most of the regexps in locale-language-names
work. Japanese and Chinese are exceptions, which are listed in the
non-standard section at the bottom of locale-language-names. 

Re: windows-1251 language environment

[Kevin Rodgers]

Jason Rumney wrote:

> MS Windows uses three letter codes for languages instead of the two
> letter ISO codes that POSIX uses. In most cases the first two letters
> are the same, so most of the regexps in locale-language-names
> work. Japanese and Chinese are exceptions, which are listed in the
> non-standard section at the bottom of locale-language-names. 

ISO 639 (Code for the Representation of the Names of Languages) defines

2-letter codes, whereas ISO 639-2 (... Part 2: Alpha-3 Code) defines 3-
letter codes.  Is that what Microsoft uses?

-- 
Kevin Rodgers

Re: windows-1251 language environment

[Jason Rumney]

Kevin Rodgers wrote:

> ISO 639 (Code for the Representation of the Names of Languages) defines
> 
> 2-letter codes, whereas ISO 639-2 (... Part 2: Alpha-3 Code) defines 3-
> letter codes.  Is that what Microsoft uses?

I'm pretty sure CHS and CHT (for Simplified and Traditional Chinese) do 
not come from any official list. But it appears that might be where JPN 
for Japanese comes from. All the other languages that Windows supports 
seem to be the ISO 639 two letter code plus an extra letter, I don't 
have time right now to go through and see if they are the same as 639-2.

Re: windows-1251 language environment

[Kenichi Handa]

In article <jwv8yntgfly.fsf-monnier+emacs/devel@vor.iro.umontreal.ca>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> Or do you expect define-derived-language to be used elsewhere than in
> a .emacs ?

My intention is to have something like "new" under this menu:
	<menu-bar> <options> <mule> <set-language-environment>
When "new" is selected, Emacs put a user in a buffer in
which he can create his own language environment
interactively.  Typing C-c C-c should ask if he want to use
that environment in the future session.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Richard Stallman]

    My intention is to have something like "new" under this menu:
	    <menu-bar> <options> <mule> <set-language-environment>
    When "new" is selected, Emacs put a user in a buffer in
    which he can create his own language environment
    interactively.  Typing C-c C-c should ask if he want to use
    that environment in the future session.

That seems like a good feature.  It could be implemented with
or without define-derived-language.

Re: windows-1251 language environment

[Dave Love]

I installed some stuff I had hanging around which should DTRT if, for
instance, you set LANG to `ru_RU.cp1251'.  Of course, setlocale in
this environment will typically fail (on a GNU system, at least).
That may cause trouble by provoking messages from subprocesses, for
instance.

This also isn't the way I think locale processing should be done
properly, especially as it doesn't deal with settings for LC_TIME and
others, which should be matched on the territory.  (Note that in Emacs
22 we stand a better chance of doing something sensible with the
LC_COLLATE part of the locale if we can use facilities from libc.)

Re: windows-1251 language environment

[Dave Love]

I wrote:

> I did something about customizing the language alist some time ago.

I realize I actually installed something in the Emacs 22 branch,
though it isn't what I meant above.  See `language-info-custom-alist'.
It's not terribly satisfactory, particularly because it's not
undoable.

The fact that I'd forgotten about it shows how long that stuff has
been hanging around filing to see the light of day.

Re: windows-1251 language environment

[Kenichi Handa]

In article <rzqismxvwz1.fsf@albion.dl.ac.uk>, Dave Love <d.love@dl.ac.uk> writes:

> Kenichi Handa <handa@m17n.org> writes:
>>  At first, I have installed a mechanism of auto-loading a
>>  coding system.

> Is it really worth it for the 8-bit sets?  I'm not sure whether to
> think this is significant or not:

> let ((stats (garbage-collect)))
>   (load "code-pages")
>   (pp (list stats (garbage-collect))))

It's not easy to see the effect of preloading code-pages
from this output.   But, when I put this in loadup.el
	(load "international/code-pages")
the resulting Emacs is 585K-byte bigger.   I think we can't
ignore this size.

>>  I have not yet tested it fully.  We may have to add GCPROs
>>  in several places.

> <URL:ftp://dlpx1.dl.ac.uk/fx/emacs/Mule/autoload-coding-systems.diff>
> was reasonably well tested, probably also on a system where gcpro was
> relevant then (though I don't know if I tested it by forcing GC during
> loading).

Stefan also showed me his implementation.  I reached my code
because I didn't want to change anything other than the
place related to coding-system.  That is because this
mechanism is obsolte in emacs-unicode in which defining a
coding system is so cheap that there's no need of this
mechanism.

>>  Right.  I think what we need for language environment is a
>>  mechanism similar to face; i.e. creating a new one easily
>>  while allowing inheriting, and customizing an existing one
>>  easily.

> I did something about customizing the language alist some time ago.
> As far as I remember, it naturally used the mechanism for overriding
> elements of the standard value in customized lists, and that interface
> was vetoed for reasons I didn't understand.  Presumably inheritance
> would use a similar mechanism, but I'm not sure how useful it is.

> As it is, a language environment has to cover several things that
> should be orthogonal:

>  * The language (which might influence input methods as well as the
>    default Ispell dictionary, at least);

>  * The charset/coding system preferences (which might also influence
>    input methods, though the hooks now in Quail make that less of an
>    issue);

>  * Other things that currently aren't dealt with properly, like paper
>    size (for ps-print), calendar holidays &c (which may depend on the
>    locale territory, not just the language).

>>  For instance, in such a case, we can allow people to create
>>  a new lang. env. by inheriting, for instance, Russian, and
>>  modifying coding-system to windows-1251.

> Is this actually better than allowing them to specify (the equivalent
> of) the locale ru_RU.windows-1251?

If that locale is available to a user, that is ok.
Otherwise, there should be a way to provide the same
environment to him.

---
Ken'ichi HANDA
handa@m17n.org

Re: windows-1251 language environment

[Dave Love]

Kenichi Handa <handa@m17n.org> writes:

> It's not easy to see the effect of preloading code-pages
> from this output.   But, when I put this in loadup.el
> 	(load "international/code-pages")
> the resulting Emacs is 585K-byte bigger.   I think we can't
> ignore this size.

Yes, sorry, I mis-read the GC data.  They do show the space used,
though I haven't checked they add up.  However, the allocated data
should mostly be able to go in purespace if they are dumped.

> Stefan also showed me his implementation.

We seem to have the usual duplication of effort :-(.

> If that locale is available to a user, that is ok.

It probably isn't on the system, but I think a locale name is a good
way to specify the thing.  The components are all well-defined as long
as the codeset is in the IANA list.  You could even specify the input
method as a modifier part for Emacs's purposes.

> Otherwise, there should be a way to provide the same
> environment to him.

Isn't it just this?

(set-language-environment "Russian")
(prefer-coding-system 'windows-1251)
(setq default-input-method ...)

The preferred coding system isn't currently customized, but I don't
think there's a good reason for that.