Setting file-name/process coding system from LANG

Setting file-name/process coding system from LANG

[Miles Bader]

I use "Japanese" language environment, and my LANG environment variable is
set to "ja_JP.UTF-8".

It seems to me that LANG should be enough to force filenames and process
I/O coding to use utf-8, regardless of the Emacs language-environment
setting -- after all, the language-environment is more of a broad hint how
to operate, whereas LANG can say specifically what coding system to use,
and will cause subprocesses to do their I/O in that coding system -- but
this doesn't seem to happen.

If I start emacs with -q, it will set things up correctly from LANG (so
for instance, starting a shell and do a ls will show utf-8 encoded
filenames correctly).

However, as soon as I do `M-x set-language-environment RET Japanese RET',
then default-file-name-coding-system gets set to `japanese-iso-8bit', and
the default coding system for process I/O is also `japanese-iso-8bit',
which is wrong.

I use the following code in my .emacs:

   (when (and (getenv "LANG") (string-match "\\<utf-8$" (getenv "LANG")))
     (setq file-name-coding-system 'utf-8)
     (setq process-coding-system-alist '(("" . utf-8-unix))))

which forces them to use utf-8 instead, and this works, but I'd think
this shouldn't be necessary if I have LANG set.

[Note I'm currently using Emacs 22 (unicode branch), but the same thing
happens with CVS trunk Emacs.]

Thanks,

-Miles
-- 
A zen-buddhist walked into a pizza shop and
said, "Make me one with everything."

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    It seems to me that LANG should be enough to force filenames and process
    I/O coding to use utf-8, regardless of the Emacs language-environment
    setting 

Does LANG have this effect on other programs?

LANG itself should not control anything in Emacs directly,
but maybe it should be used toinitialize someting else
that would control this.

Re: Setting file-name/process coding system from LANG

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
>     It seems to me that LANG should be enough to force filenames and process
>     I/O coding to use utf-8, regardless of the Emacs language-environment
>     setting 
>
> Does LANG have this effect on other programs?

In general yes, it seems too.  Most importantly, the LANG setting
determines how sub-programs invoked from Emacs encode their error messages,
so Emacs should be using the same encoding for process I/O.

> LANG itself should not control anything in Emacs directly, but maybe
> it should be used toinitialize someting else that would control this.

Yes -- and in fact it _already_ does this in Emacs.

The problem is that setting the Emacs language-environment (with
`set-language-environment') _overrides_ the initial settings made based on
LANG.  That's wrong in this case, I think.

BTW, another possibly related issue is that if I start `emacs -q' with
LANG=ja_JP, then the language-environment is automatically set to
"Japanese" -- great!  But if I start `emacs -q' with LANG=ja_JP.utf8, then
the language environment is set to "utf-8", and that doesn't make much
sense to me -- indeed the whole concept of a "utf-8 language environment"
seems kind of weird.  In this case I think it should instead (1) set the
language-environt to "Japanese", and then (2) set the various I/O encodings
to be `utf-8'.

-Miles
-- 
We have met the enemy, and he is us.  -- Pogo

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    Yes -- and in fact it _already_ does this in Emacs.

    The problem is that setting the Emacs language-environment (with
    `set-language-environment') _overrides_ the initial settings made based on
    LANG.  That's wrong in this case, I think.

So you're suggesting that set-language-environment simply not change
these settings?  That might be ok; however, in other cases
we might want different behavior.

What precisely are these settings?

    BTW, another possibly related issue is that if I start `emacs -q' with
    LANG=ja_JP, then the language-environment is automatically set to
    "Japanese" -- great!  But if I start `emacs -q' with LANG=ja_JP.utf8, then
    the language environment is set to "utf-8", and that doesn't make much
    sense to me -- indeed the whole concept of a "utf-8 language environment"
    seems kind of weird.  In this case I think it should instead (1) set the
    language-environt to "Japanese", and then (2) set the various I/O encodings
    to be `utf-8'.

Handa, what do you think?

Re: Setting file-name/process coding system from LANG

[Miles Bader]

On Sat, Oct 16, 2004 at 09:53:01AM -0400, Richard Stallman wrote:
>     The problem is that setting the Emacs language-environment (with
>     `set-language-environment') _overrides_ the initial settings made based on
>     LANG.  That's wrong in this case, I think.
> 
> So you're suggesting that set-language-environment simply not change
> these settings?  That might be ok; however, in other cases
> we might want different behavior.

I think the values for these settings that set-language-environment uses
might provide useful _defaults_ (if there is no more specific setting in
force).

Currently set-language-environment sets `default-file-name-coding-system',
which is overridden by `file-name-coding-system'.  Startup also apparently
sets `default-file-name-coding-system' using LANG.  If startup set
`file-name-coding-system' from LANG instead, then I think that would provide
the proper semantics in this case.

> What precisely are these settings?

I'm thinking about file-name-coding-system (and
default-file-name-coding-system); I don't know where the process I/O coding
system is stored.

-Miles
-- 
"Whatever you do will be insignificant, but it is very important that
 you do it."  Mahatma Ghandi

Re: Setting file-name/process coding system from LANG

[Kenichi Handa]

In article <E1CIozV-00037e-K4@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     BTW, another possibly related issue is that if I start `emacs -q' with
>     LANG=ja_JP, then the language-environment is automatically set to
>     "Japanese" -- great!  But if I start `emacs -q' with LANG=ja_JP.utf8, then
>     the language environment is set to "utf-8", and that doesn't make much
>     sense to me -- indeed the whole concept of a "utf-8 language environment"
>     seems kind of weird.  In this case I think it should instead (1) set the
>     language-environt to "Japanese", and then (2) set the various I/O encodings
>     to be `utf-8'.

> Handa, what do you think?

A while ago, I proposed an idea about make language
environment more hierarchical and make it possible to
generate one on demand from locale, but you refused such a
change.

In the above case, we should have a language environment
Japanese-UTF-8.  But, preparing all such lang. envs. in
advance is a waste of memory (we surely need many XXX-UTF-8
lang. envs.).  So, my idea was to generate that lang env by
inheriting Japanese lang. env. and overriding several
settings (values for keys coding-system, coding-priority,
nonascii-translation of language-info-alist) for UTF-8 when
Emacs runs on ja_JP.utf8 locale and Japanese-UTF-8 doesn't
exist.

Then a user can customize (and store in .emacs) that
lang. env. if he wish.

Of course, such a change requires carefull reconstructure of
language-info-alist, and thus we may have to postpone it
until 21.4 release.

---
Ken'ichi HANDA
handa@m17n.org

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    A while ago, I proposed an idea about make language
    environment more hierarchical and make it possible to
    generate one on demand from locale, but you refused such a
    change.

I disliked this because it involves a lot of additional complexity.
I think we don't need all that complexity.

When Emacs sees LANG=ja_JP.utf8, surely it can select the Japanese
language environment, then do something else to cause use of UTF8.
Surely it can do this without having an explicit language environment
for the combination of the two.

Isn't there an easy way to do that?

Re: Setting file-name/process coding system from LANG

[Kenichi Handa]

In article <E1CJnFJ-00006C-7e@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     A while ago, I proposed an idea about make language
>     environment more hierarchical and make it possible to
>     generate one on demand from locale, but you refused such a
>     change.

> I disliked this because it involves a lot of additional complexity.
> I think we don't need all that complexity.

> When Emacs sees LANG=ja_JP.utf8, surely it can select the Japanese
> language environment, then do something else to cause use of UTF8.
> Surely it can do this without having an explicit language environment
> for the combination of the two.

Then a user can't recover that original setting when he once
switch to, e.g., Latin-1.  Just switching back to Japanese
can't setup coding systems for utf8.

Another problem is the case of LANG=zh_CN.utf8.  Currently
we don't have Chinese lang. env.  We only have Chinese-GB,
Chinese-EUC-TW, Chinese-BIG5, Chinese-CNS.  Which one to
use?

---
Ken'ichi HANDA
handa@m17n.org

Re: Setting file-name/process coding system from LANG

[Stefan Monnier]

> Then a user can't recover that original setting when he once
> switch to, e.g., Latin-1.  Just switching back to Japanese
> can't setup coding systems for utf8.

M-x set-locale-environment RET

In my experience set-locale-environment has always been closer to what
I wanted to do than set-language-environment.
It also has the advantage of having a meaning outside of Emacs and of
conveniently separate the issue of language from the issue of coding-system.


        Stefan

Re: Setting file-name/process coding system from LANG

[Miles Bader]

On Tue, Oct 19, 2004 at 10:13:53PM +0900, Kenichi Handa wrote:
> > When Emacs sees LANG=ja_JP.utf8, surely it can select the Japanese
> > language environment, then do something else to cause use of UTF8.
> > Surely it can do this without having an explicit language environment
> > for the combination of the two.
> 
> Then a user can't recover that original setting when he once
> switch to, e.g., Latin-1.  Just switching back to Japanese
> can't setup coding systems for utf8.

Why can't we do what I suggested earlier, and set `file-name-coding-system'
(and presumably some analogue for process I/O and terminal-coding-system)
instead of `default-file-name-coding-system' from the "utf8" portion of LANG?

Then `set-language-environment' could set `default-file-name-coding-system'
but it wouldn't actually have any effect -- which at least in this
circumstance seems correct.  Other language-environment settings (like the
tutorial language etc) would have the proper effect.

> Another problem is the case of LANG=zh_CN.utf8.  Currently
> we don't have Chinese lang. env.  We only have Chinese-GB,
> Chinese-EUC-TW, Chinese-BIG5, Chinese-CNS.  Which one to
> use?

Presumably whichever one best matches the "country" portion of LANG.

-Miles
-- 
.Numeric stability is probably not all that important when you're guessing.

Re: Setting file-name/process coding system from LANG

[Kenichi Handa]

In article <jwvr7nuhlb3.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  Then a user can't recover that original setting when he once
>>  switch to, e.g., Latin-1.  Just switching back to Japanese
>>  can't setup coding systems for utf8.

> M-x set-locale-environment RET

> In my experience set-locale-environment has always been closer to what
> I wanted to do than set-language-environment.
> It also has the advantage of having a meaning outside of Emacs and of
> conveniently separate the issue of language from the issue of coding-system.

Yes, asking users to use set-locale-environment instead of
set-language-environment is one way.  But, I've thought that
set-locale-environment is not a command we usually ask a
user to use, but a function that runs at startup time.  Do
you mean to recommend users to use set-locale-environment
always?

---
Ken'ichi HANDA
handa@m17n.org

Re: Setting file-name/process coding system from LANG

[Kenichi Handa]

In article <20041019215111.GA9395@fencepost>, Miles Bader <miles@gnu.org> writes:

> On Tue, Oct 19, 2004 at 10:13:53PM +0900, Kenichi Handa wrote:
>>  > When Emacs sees LANG=ja_JP.utf8, surely it can select the Japanese
>>  > language environment, then do something else to cause use of UTF8.
>>  > Surely it can do this without having an explicit language environment
>>  > for the combination of the two.
>>  
>>  Then a user can't recover that original setting when he once
>>  switch to, e.g., Latin-1.  Just switching back to Japanese
>>  can't setup coding systems for utf8.

> Why can't we do what I suggested earlier, and set `file-name-coding-system'
> (and presumably some analogue for process I/O and terminal-coding-system)
> instead of `default-file-name-coding-system' from the "utf8" portion of LANG?

If it always sets file-name-coding-system, there's no need
of having default-file-name-coding-system.  And, that means
that even if one changes to latin-1 lang. env.,
file-name-coding-system is still utf-8.  Is it ok?  Am I
misunderstanding something?

> Then `set-language-environment' could set `default-file-name-coding-system'
> but it wouldn't actually have any effect -- which at least in this
> circumstance seems correct.  Other language-environment settings (like the
> tutorial language etc) would have the proper effect.

>>  Another problem is the case of LANG=zh_CN.utf8.  Currently
>>  we don't have Chinese lang. env.  We only have Chinese-GB,
>>  Chinese-EUC-TW, Chinese-BIG5, Chinese-CNS.  Which one to
>>  use?

> Presumably whichever one best matches the "country" portion of LANG.

It seems to be difficult.

% locale -a|grep zh
shows these:

zh_CN
zh_CN.gb18030
zh_CN.gb2312
zh_CN.gbk
zh_CN.utf8
zh_HK
zh_HK.big5hkscs
zh_HK.utf8
zh_SG
zh_SG.gb2312
zh_SG.gbk
zh_TW
zh_TW.big5
zh_TW.euctw
zh_TW.utf8

---
Ken'ichi HANDA
handa@m17n.org

Re: Setting file-name/process coding system from LANG

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
>>>  Then a user can't recover that original setting when he once
>>>  switch to, e.g., Latin-1.  Just switching back to Japanese
>>>  can't setup coding systems for utf8.
>
>> Why can't we do what I suggested earlier, and set `file-name-coding-system'
>> (and presumably some analogue for process I/O and terminal-coding-system)
>> instead of `default-file-name-coding-system' from the "utf8" portion of LANG?
>
> If it always sets file-name-coding-system, there's no need
> of having default-file-name-coding-system.

Sure there is:  it gives set-language-environment something to set.

In other words, the setting established by set-language-environment is a
_default_, but the setting established by LANG (or by the user
explicitly, if uses `C-x C-m whatever') is `real'.

> And, that means that even if one changes to latin-1 lang. env.,
> file-name-coding-system is still utf-8.  Is it ok?

I'm not sure; it could be.  LANG primarily seems to describe the way
programs should interact with other programs and the user, but
`language-environment' seems a bit more vague.

A user might very well change the language-environment to get various
customizations for his own typical usage -- LEIM input system, tutorial
language, file-coding priorities -- but he's still using the same
system, his terminal hasn't changed, etc.

If we assume that LANG was set correctly for this system (it describes
the way typical tools work), then it seems reasonable for emacs to
continue interacting with the system in the same way even if the user
changes the language-environment.

However if the user uses `set-language-environment' because LANG wasn't
set, or because it was set incorrectly, he probably wants it to override
everything.

Which of these scenarios is more common?  I'm not sure, but it doesn't
seem unreasonable to assume that _if_ LANG contains an explicit coding
system, that it's "correct".

[BTW, another thing: if set-language-environment _does_ change the
process-related coding systems, it probably also ought to change the
value of LANG that subprocesses get, so at least it's _consistent_...]

>>>  Another problem is the case of LANG=zh_CN.utf8.  Currently
>>>  we don't have Chinese lang. env.  We only have Chinese-GB,
>>>  Chinese-EUC-TW, Chinese-BIG5, Chinese-CNS.  Which one to
>>>  use?
>
>> Presumably whichever one best matches the "country" portion of LANG.
>
> % locale -a|grep zh
> shows these:
>
> zh_CN
> zh_CN.gb18030
> zh_CN.gb2312
> zh_CN.gbk
> zh_CN.utf8
> zh_HK
> zh_HK.big5hkscs
> zh_HK.utf8
> zh_SG
> zh_SG.gb2312
> zh_SG.gbk
> zh_TW
> zh_TW.big5
> zh_TW.euctw
> zh_TW.utf8

Well you could just choose whichever is most popular in each country...

[perhaps have an alist do mapping from "chinese country" to "chinese
lang. env." -- by default it would be sorted by popularity, but a user
could change it if he didn't like that... or just set the language
environment explicitly.]

-Miles
-- 
Yo mama's so fat when she gets on an elevator it HAS to go down.

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    Another problem is the case of LANG=zh_CN.utf8.  Currently
    we don't have Chinese lang. env.  We only have Chinese-GB,
    Chinese-EUC-TW, Chinese-BIG5, Chinese-CNS.  Which one to
    use?

I don't know the answer, but dynamically generated language
environments are not the answer.  They won't make it any easier to
decide which one to use.

You could pick one as a default.  You could make it ask the user.

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    M-x set-locale-environment RET

Perhaps we should add a single new language environment
named `locale' or `according-to-locale' which has the effect
of doing set-locale-environment.

Re: Setting file-name/process coding system from LANG

[Stefan Monnier]

>> And, that means that even if one changes to latin-1 lang. env.,
>> file-name-coding-system is still utf-8.  Is it ok?

There is no latin-1 language, so the "latin-1 lang env" is an ill-defined
concept anyway.  Thus we can make it behave any way we want.

> Yes, asking users to use set-locale-environment instead of
> set-language-environment is one way.  But, I've thought that
> set-locale-environment is not a command we usually ask a
> user to use, but a function that runs at startup time.  Do
> you mean to recommend users to use set-locale-environment
> always?

Yes, kind of.  I'm not sure exactly what I want, but I just know that the
structure in locales (mostly the distinction between language and encoding)
is very helpful.  I think we should move set-language-environment in
this direction, or make set-locale-environment easier to use interactively
and then recommend that.


        Stefan

Re: Setting file-name/process coding system from LANG

[Richard Stallman]

    Currently set-language-environment sets `default-file-name-coding-system',
    which is overridden by `file-name-coding-system'.  Startup also apparently
    sets `default-file-name-coding-system' using LANG.  If startup set
    `file-name-coding-system' from LANG instead, then I think that would provide
    the proper semantics in this case.

What do people think of this suggestion?

    Which of these scenarios is more common?  I'm not sure, but it doesn't
    seem unreasonable to assume that _if_ LANG contains an explicit coding
    system, that it's "correct".

As long as it is not hard for users to override this setting
explicitly, when they want to, I agree.

There was also the suggestion of advising users to run
set-locale-environment in order to get back to the settings
specified by the locale.

And I had this idea:

    Perhaps we should add a single new language environment
    named `locale' or `according-to-locale' which has the effect
    of doing set-locale-environment.

Another idea is for set-language-environment to allow a language
environment name followed by a coding system name.  This would appear
to the user rather similar to Handa's suggestion, but it is much
easier to implement.  It would not construct a new language
environment for the combination, it would only set the language
environment and then set the coding system.


The previous discussion did not consider all of these alternatives
side by side.  Could people start a new discussion doing so?  Note
that the issue that was raised about how to choose the default coding
system for Chinese language environments is not really relevant here;
it is a real issue, but it is orthogonal to this one.