L10N & LC variables

L10N & LC variables

[Jean-Christophe Helary]

The first think we need for l10n, is have emacs be aware of the linguistic environment in which it is running (or eventually the language environment that it uses internally, which is much easier). I am not using gnu/linux so I am not sure how to do that there (although it does look relatively straightforward).

On Mac, the locale variables don't seem to be defined the way they are on gnu/linux. Here I only have "LC_CTYPE=UTF-8" and no other LC variable. I've checked that yesterday and it looks like there is a system "AppleLocale" variable that supersedes the LC variables and that returns the language selected in the global system preferences. There is also an "AppleLanguages" that returns *all* the languages added by the user in that same global system preference.

Here is what I get for my system:

> env | grep LC_
  LC_CTYPE=UTF-8

> defaults read -g AppleLocale
  en_JP

> defaults read -g AppleLanguages
  (
    "en-JP",
    "fr-JP",
    "ja-JP"
  )


I've checked whether that variable was used in Emacs and I found reference to it in Changelog.12.

> 2005-04-14  YAMAMOTO Mitsuharu  <mituharu@math.s.chiba-u.ac.jp>
> 
> 	* international/mule-cmds.el (set-locale-environment): On Mac OS,
> 	use preferences AppleLocale and AppleLanguages, and variable
> 	mac-system-locale for default locale.  On Mac OS Classic, use
> 	mac-system-coding-system for default coding systems.

But there is no further references to mac-system-locale in the other changelogs and it looks like mule-cmds.el does not define it and there does not seem to be any place in the Emacs code that refers to AppleLocale or AppleLanguages anymore.

So, is that something that's been removed?

Jean-Christophe 

Re: L10N & LC variables

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Fri, 26 May 2017 23:41:17 +0900
> 
> The first think we need for l10n, is have emacs be aware of the linguistic environment in which it is running (or eventually the language environment that it uses internally, which is much easier). I am not using gnu/linux so I am not sure how to do that there (although it does look relatively straightforward).
> 
> On Mac, the locale variables don't seem to be defined the way they are on gnu/linux. Here I only have "LC_CTYPE=UTF-8" and no other LC variable. I've checked that yesterday and it looks like there is a system "AppleLocale" variable that supersedes the LC variables and that returns the language selected in the global system preferences. There is also an "AppleLanguages" that returns *all* the languages added by the user in that same global system preference.
> 
> Here is what I get for my system:
> 
> > env | grep LC_
>   LC_CTYPE=UTF-8
> 
> > defaults read -g AppleLocale
>   en_JP
> 
> > defaults read -g AppleLanguages
>   (
>     "en-JP",
>     "fr-JP",
>     "ja-JP"
>   )
> 
> 
> I've checked whether that variable was used in Emacs and I found reference to it in Changelog.12.
> 
> > 2005-04-14  YAMAMOTO Mitsuharu  <mituharu@math.s.chiba-u.ac.jp>
> > 
> > 	* international/mule-cmds.el (set-locale-environment): On Mac OS,
> > 	use preferences AppleLocale and AppleLanguages, and variable
> > 	mac-system-locale for default locale.  On Mac OS Classic, use
> > 	mac-system-coding-system for default coding systems.
> 
> But there is no further references to mac-system-locale in the other changelogs and it looks like mule-cmds.el does not define it and there does not seem to be any place in the Emacs code that refers to AppleLocale or AppleLanguages anymore.
> 
> So, is that something that's been removed?

I don't know, but doesn't current-language-environment work on macOS
and give you what you want?  If not, why not?

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 26, 2017, at 23:53, Eli Zaretskii <eliz@gnu.org> wrote:

>>> env | grep LC_
>>  LC_CTYPE=UTF-8
>> 
>>> defaults read -g AppleLocale
>>  en_JP
>> 
>>> defaults read -g AppleLanguages
>>  (
>>    "en-JP",
>>    "fr-JP",
>>    "ja-JP"
>>  )

> 
> I don't know, but doesn't current-language-environment work on macOS
> and give you what you want?  If not, why not?

It gives me "Japanese", If I change it to "French" in the Multilingual Environment settings, it gives me "French". It defaults to "English", as you know, but there is no indication that it reads LC_* or AppleL* variables to gets it value.

Jean-Christophe 

Re: L10N & LC variables

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Sat, 27 May 2017 00:10:51 +0900
> Cc: emacs-devel@gnu.org
> 
> > I don't know, but doesn't current-language-environment work on macOS
> > and give you what you want?  If not, why not?
> 
> It gives me "Japanese"

And is that wrong?

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 27, 2017, at 0:20, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
>> Date: Sat, 27 May 2017 00:10:51 +0900
>> Cc: emacs-devel@gnu.org
>> 
>>> I don't know, but doesn't current-language-environment work on macOS
>>> and give you what you want?  If not, why not?
>> 
>> It gives me "Japanese"
> 
> And is that wrong?

Yes, sorry I thought that was obvious considering what I had written before.

My OS language is EN and since my physical region is Japan Mac codes it as en_jp.

So I would expect to see current-language-environment give me English.

Whatever I set on the OS side is reflected in AppleLocale but not in current-language-environment.

Jean-Christophe

Re: L10N & LC variables

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Sat, 27 May 2017 00:54:47 +0900
> 
> >> It gives me "Japanese"
> > 
> > And is that wrong?
> 
> Yes, sorry I thought that was obvious considering what I had written before.
> 
> My OS language is EN and since my physical region is Japan Mac codes it as en_jp.
> 
> So I would expect to see current-language-environment give me English.

Then "M-x set-language-environment RET English RET" should do what you
want.

Re: L10N & LC variables

[Paul Eggert]

On 05/26/2017 07:41 AM, Jean-Christophe Helary wrote:
> 2005-04-14  YAMAMOTO Mitsuharu<mituharu@math.s.chiba-u.ac.jp>
>
> 	* international/mule-cmds.el (set-locale-environment): On Mac OS,
> 	use preferences AppleLocale and AppleLanguages, and variable
> 	mac-system-locale for default locale.  On Mac OS Classic, use
> 	mac-system-coding-system for default coding systems.
> But there is no further references to mac-system-locale in the other changelogs

That code was removed here:

http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9e2a2647758db83b490e2993aa31cd4607305a82

in a commit dated 2008-07-27 that removed support for Carbon. From what 
you say, that part of the 2008 change is making life for macOS users 
more difficult, in that they sometimes need to manually set their 
language environment instead of having it be deduced automatically from 
system settings. Presumably there is a non-Carbon way to do the 
equivalent of the old code, and if you or some other macOS expert would 
write it, that would remove this minor glitch in the default Emacs 
configuration on macOS.

Here's a tip to help you research similar issues in the future: to find 
when the string "AppleLocale" was removed from the file 
lisp/international/mule-cmds.el, you can use this shell command:

git log -S AppleLocale lisp/international/mule-cmds.el

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 27, 2017, at 3:42, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> On 05/26/2017 07:41 AM, Jean-Christophe Helary wrote:
>> 2005-04-14  YAMAMOTO Mitsuharu<mituharu@math.s.chiba-u.ac.jp>
>> 
>> 	* international/mule-cmds.el (set-locale-environment): On Mac OS,
>> 	use preferences AppleLocale and AppleLanguages, and variable
>> 	mac-system-locale for default locale.  On Mac OS Classic, use
>> 	mac-system-coding-system for default coding systems.
>> But there is no further references to mac-system-locale in the other changelogs
> 
> That code was removed here:
> 
> http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9e2a2647758db83b490e2993aa31cd4607305a82
> 
> in a commit dated 2008-07-27 that removed support for Carbon. From what you say, that part of the 2008 change is making life for macOS users more difficult, in that they sometimes need to manually set their language environment instead of having it be deduced automatically from system settings.

Sorry if I got my wording wrong. macOS does not seem to use the LC_ system to provide locale information to the applications. It uses 2 variables that are:

AppleLanguages: a string list of locale names the user has in her global preferences (usually just one item, usually a "standard" (language)_(region) pair. But she can add other locales and set priorities to provide for applications that offer a localization but not others.

AppleLocale: the string that is the first element of the list.

The two variables are set from the GUI based "System Preferences" application but they probably can be modified from the command line as well if necessary.

> Presumably there is a non-Carbon way to do the equivalent of the old code, and if you or some other macOS expert would write it, that would remove this minor glitch in the default Emacs configuration on macOS.

That would be great. I'll see if I can grasp the code.

> Here's a tip to help you research similar issues in the future: to find when the string "AppleLocale" was removed from the file lisp/international/mule-cmds.el, you can use this shell command:
> 
> git log -S AppleLocale lisp/international/mule-cmds.el

That's extremely helpful. Thank you *very* much.

Jean-Christophe 

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 27, 2017, at 2:39, Eli Zaretskii <eliz@gnu.org> wrote:

>> My OS language is EN and since my physical region is Japan Mac codes it as en_jp.
>> 
>> So I would expect to see current-language-environment give me English.
> 
> Then "M-x set-language-environment RET English RET" should do what you
> want.

Yes, and it does, but what I want is Emacs to recognize the OS locale information and provide me that by default first, before allowing me to set an arbitrary language environment that is independent of the system provided locale.

I'll first check what Paul found (the removal of Carbon support international/mule-cmds.el)

Jean-Christophe 

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 27, 2017, at 3:42, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> On 05/26/2017 07:41 AM, Jean-Christophe Helary wrote:
>> 2005-04-14  YAMAMOTO Mitsuharu<mituharu@math.s.chiba-u.ac.jp>
>> 
>> 	* international/mule-cmds.el (set-locale-environment): On Mac OS,
>> 	use preferences AppleLocale and AppleLanguages, and variable
>> 	mac-system-locale for default locale.  On Mac OS Classic, use
>> 	mac-system-coding-system for default coding systems.
>> But there is no further references to mac-system-locale in the other changelogs
> 
> That code was removed here:
> 
> http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9e2a2647758db83b490e2993aa31cd4607305a82
> 
> in a commit dated 2008-07-27 that removed support for Carbon. From what you say, that part of the 2008 change is making life for macOS users more difficult,

Interesting. It looks like the whole macOS preference settings acquisition code was taken from Emacs in that commit, including a lisp function written in C: mac-get-preference. That was quite a radical commit...

> Presumably there is a non-Carbon way to do the equivalent of the old code, and if you or some other macOS expert would write it, that would remove this minor glitch in the default Emacs configuration on macOS.

(shell-command "defaults read -g AppleLocale") seems to do the job well, although it may not be considered super elegant.

(side note: le Elisp Reference index does not seem to have an entry for "shell-command" although "read-shell-command" that is also defined in simple.el is there.)

Jean-Christophe 

Re: L10N & LC variables

[Alan Third]

On Sat, May 27, 2017 at 08:13:44AM +0900, Jean-Christophe Helary wrote:
> 
> Sorry if I got my wording wrong. macOS does not seem to use the LC_
> system to provide locale information to the applications.

I’m not sure if it’s relevant to what you’re looking at, but
ns_init_locale in nsterm.m sets the LANG environment variable from the
macOS locale in an attempt to emulate the setup you get from running
Emacs from the terminal.

I guess that what you’re talking about implementing would make that
code obsolete.
-- 
Alan Third

Re: L10N & LC variables

[Jean-Christophe Helary]

> On May 27, 2017, at 9:38, Alan Third <alan@idiocy.org> wrote:
> 
>> Sorry if I got my wording wrong. macOS does not seem to use the LC_
>> system to provide locale information to the applications.
> 
> I’m not sure if it’s relevant to what you’re looking at, but
> ns_init_locale in nsterm.m sets the LANG environment variable from the
> macOS locale in an attempt to emulate the setup you get from running
> Emacs from the terminal.

Thank you Alan.

Looking at the definition, it seems to be what I'm looking for. Why is that variable not used in other parts of the code ? It does not seem to be accessed at all...

Jean-Christophe 

Re: L10N & LC variables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1173 bytes --]

>
> > 2005-04-14  YAMAMOTO Mitsuharu  <mituharu@math.s.chiba-u.ac.jp>
> >
> >       * international/mule-cmds.el (set-locale-environment): On Mac OS,
> >       use preferences AppleLocale and AppleLanguages, and variable
> >       mac-system-locale for default locale.  On Mac OS Classic, use
> >       mac-system-coding-system for default coding systems.
>
> But there is no further references to mac-system-locale in the other
> changelogs and it looks like mule-cmds.el does not define it and there does
> not seem to be any place in the Emacs code that refers to AppleLocale or
> AppleLanguages anymore.
>
> So, is that something that's been removed?
>

There are two complete implementations of Emacs on macOS. One originate
from the NextSTEP world and is called the "NS" port, this is the official
version today. The second originate from Mac OS classic and use the prefix
"mac-". Up to Emacs 22, this was the official version of Emacs, but was
removed in 2008. It is still being maintained by Yamamoto Mitsuharu san.

Apparently, the "mac" port supports AppleLocale and AppleLanguage whereas
the "NS" port doesn't. However, contributions are welcome!

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1614 bytes --]

Re: L10N & LC variables

[Alan Third]

On Sat, May 27, 2017 at 09:54:58AM +0900, Jean-Christophe Helary wrote:
> > On May 27, 2017, at 9:38, Alan Third <alan@idiocy.org> wrote:
> > 
> >> Sorry if I got my wording wrong. macOS does not seem to use the LC_
> >> system to provide locale information to the applications.
> > 
> > I’m not sure if it’s relevant to what you’re looking at, but
> > ns_init_locale in nsterm.m sets the LANG environment variable from the
> > macOS locale in an attempt to emulate the setup you get from running
> > Emacs from the terminal.
> 
> Thank you Alan.
> 
> Looking at the definition, it seems to be what I'm looking for. Why
> is that variable not used in other parts of the code ? It does not
> seem to be accessed at all...

LANG is used in set-locale-environment in mule-cmds.el, but I don’t
know what is set using it.

I know almost nothing about locales in emacs. The main motivation for
setting LANG on macOS was so that external commands would be given the
correct locale, eg. spell checkers.

Would it be worth creating a function available from lisp that returns
system locale information?
-- 
Alan Third

Re: L10N & LC variables

[Eli Zaretskii]

> Date: Sat, 27 May 2017 08:33:59 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> LANG is used in set-locale-environment in mule-cmds.el, but I don’t
> know what is set using it.

The doc string of set-locale-environment says quite a lot about that:

  This sets the language environment, the coding system priority,
  the default input method and sometimes other things.

> I know almost nothing about locales in emacs.

They are mainly used for setting up the defaults for encoding and
decoding incoming/outgoing text outside of Emacs.

> The main motivation for setting LANG on macOS was so that external
> commands would be given the correct locale, eg. spell checkers.

Spell-checking uses this to tell the speller what dictionary to use.

> Would it be worth creating a function available from lisp that returns
> system locale information?

What information would that function return?  Does "C-h L" do what you
want?  If not, what is missing?

Anyway, I think we should separate the issues related to translation
infrastructure from issues related to setting the language for
fetching strings translated into that language.  They are two
orthogonal problems; in particular, I can easily envision an Emacs
user wishing to set a language that has nothing to do with the user's
locale, for example to test some translations.  Also, the notion of
"current language" has implications elsewhere in Emacs, like in case
conversions.

We should handle these two classes of issues separately.

Re: L10N & LC variables

[Jean-Christophe Helary]

> On 27 mai 2017, at 16:59, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> What information would that function return?

Its definition looks like if returns the user set locale. I don't know how to check what it returns.

>  Does "C-h L" do what you want?

No. C-h L returns what is set within Emacs (English in my case), which as I mentioned earlier is disconnected from what the OS provides (currently French as I test this call).

>  If not, what is missing?

And Emacs lisp function that returns the value of what ns_init_locale returns.

> Anyway, I think we should separate the issues related to translation
> infrastructure from issues related to setting the language for
> fetching strings translated into that language.

Absolutely. I started the thread because I wanted to set language strings based on the user set language preferences but noticed that they were not accessed by Emacs on a Mac. Once I find a solution for that, I think we can move on since I guess it is trivial to access such data on gnu/linux (and I have no way to check that on Windows).

> They are two
> orthogonal problems; in particular, I can easily envision an Emacs
> user wishing to set a language that has nothing to do with the user's
> locale, for example to test some translations.

That's also a possibility I consider. We need to make sure that the Emacs hard coded default value is overridden by the OS provided value at first install, and that the user can easily set a language of her choice. But as you say it is not directly related to l10n.

Jean-Christophe