Re: master e4e1e268c8e 2/2: Set a default locale on Android

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> branch: master
> commit e4e1e268c8e9f7de6fe5d4b05beb595a1781c02c
> Author: Po Lu <luangruo@yahoo.com>
> Commit: Po Lu <luangruo@yahoo.com>
> 
>     Set a default locale on Android
>     
>     * doc/emacs/android.texi (Android Environment): Revise for
>     change.
>     
>     * src/android.c (initEmacs): Set LANG to either en_US.utf8 or C,
>     subject to whether the system is newer than Android
>     5.0 (inclusive).

Is this the best we can do on Android?  AFAIU, the issue here is that
the locales available on Android are different from those recognized
by the C library.  But cannot we solve this by implementing a mapping
from Android locales to libc locales?  If not, why not?

I think arbitrarily using en_US.utf8 could surprise users and worse,
could produce effects that break UX.  For example, LANG defines the
default spell-checking dictionary, so the above basically forces users
of non-English locales to manually change the dictionary each time,
or customize their Emacs to do that, something they don't have to do
on other platforms.

We solved a similar problem on MS-Windows, so why cannot we solve it
on Android?

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Is this the best we can do on Android?  AFAIU, the issue here is that
> the locales available on Android are different from those recognized
> by the C library.  But cannot we solve this by implementing a mapping
> from Android locales to libc locales?  If not, why not?
>
> I think arbitrarily using en_US.utf8 could surprise users and worse,
> could produce effects that break UX.  For example, LANG defines the
> default spell-checking dictionary, so the above basically forces users
> of non-English locales to manually change the dictionary each time,
> or customize their Emacs to do that, something they don't have to do
> on other platforms.
>
> We solved a similar problem on MS-Windows, so why cannot we solve it
> on Android?

This is the behavior of other Android programs placed in similar
situations, and is only designed to enable programs to output UTF-8
error messages and the like, not for these programs to print messages in
the language the user has configured, most of which the Android C
library doesn't support anyway.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Dec 2023 08:11:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is this the best we can do on Android?  AFAIU, the issue here is that
> > the locales available on Android are different from those recognized
> > by the C library.  But cannot we solve this by implementing a mapping
> > from Android locales to libc locales?  If not, why not?
> >
> > I think arbitrarily using en_US.utf8 could surprise users and worse,
> > could produce effects that break UX.  For example, LANG defines the
> > default spell-checking dictionary, so the above basically forces users
> > of non-English locales to manually change the dictionary each time,
> > or customize their Emacs to do that, something they don't have to do
> > on other platforms.
> 
> This is the behavior of other Android programs placed in similar
> situations, and is only designed to enable programs to output UTF-8
> error messages and the like, not for these programs to print messages in
> the language the user has configured, most of which the Android C
> library doesn't support anyway.

I wasn't talking about error messages, I was talking about other uses
of LANG and locale information in Emacs.  E.g., we define the default
coding-systems based on the locale, and detect-coding-* guesswork uses
it.  How does the solution you installed affect all that when, for
example, Emacs needs to visit a file or show an email or use
sub-process output encoded in Latin-1 with no charset information
available from other sources?

As for other programs' behavior: since when we consider that a
dictate?  We take the platform-specific behavior into consideration,
but make our decisions based on additional aspects as well.  And I
still don't see why in this case we should use such a simplistic
solution.  Is there any problem in having a mapping of the kind I
suggested?  If that is possible, it sounds to me like a simple-enough
solution that will make the Android build behave like Emacs users
expect, with minimal costs and with predictable success.

In any case, if there are Android-specific aspects of this, they
should be prominently mentioned in the "Language Environments" section
of the user manual.  I don't think they are, as of now, unless I
missed something.

Thanks.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I wasn't talking about error messages, I was talking about other uses
> of LANG and locale information in Emacs.  E.g., we define the default
> coding-systems based on the locale, and detect-coding-* guesswork uses
> it.  How does the solution you installed affect all that when, for
> example, Emacs needs to visit a file or show an email or use
> sub-process output encoded in Latin-1 with no charset information
> available from other sources?

Android subprocesses won't output information in Latin-1, as the C
library doesn't support these locales.  Furthermore, there are no OS
locales that don't use UTF-8 under Android.

See here:

  https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#51

> As for other programs' behavior: since when we consider that a
> dictate?  We take the platform-specific behavior into consideration,
> but make our decisions based on additional aspects as well.  And I
> still don't see why in this case we should use such a simplistic
> solution.  Is there any problem in having a mapping of the kind I
> suggested?  If that is possible, it sounds to me like a simple-enough
> solution that will make the Android build behave like Emacs users
> expect, with minimal costs and with predictable success.

It is not possible, since en_US.utf8 and C are the only locales
supported under Android.  System languages besides that are implemented
in the GUI toolkit, which does not take LANG into consideration.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Dec 2023 14:49:35 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I wasn't talking about error messages, I was talking about other uses
> > of LANG and locale information in Emacs.  E.g., we define the default
> > coding-systems based on the locale, and detect-coding-* guesswork uses
> > it.  How does the solution you installed affect all that when, for
> > example, Emacs needs to visit a file or show an email or use
> > sub-process output encoded in Latin-1 with no charset information
> > available from other sources?
> 
> Android subprocesses won't output information in Latin-1, as the C
> library doesn't support these locales.  Furthermore, there are no OS
> locales that don't use UTF-8 under Android.
> 
> See here:
> 
>   https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#51

So a program on Android cannot output an arbitrary byte stream, which
just happens to be text encoded in non-UTF-8 encoding?

And anyway, what about the other examples I gave, with visiting files
and receiving email? what about them?  How can Android prevent such
files or such email messages from getting into Emacs?

> > As for other programs' behavior: since when we consider that a
> > dictate?  We take the platform-specific behavior into consideration,
> > but make our decisions based on additional aspects as well.  And I
> > still don't see why in this case we should use such a simplistic
> > solution.  Is there any problem in having a mapping of the kind I
> > suggested?  If that is possible, it sounds to me like a simple-enough
> > solution that will make the Android build behave like Emacs users
> > expect, with minimal costs and with predictable success.
> 
> It is not possible, since en_US.utf8 and C are the only locales
> supported under Android.  System languages besides that are implemented
> in the GUI toolkit, which does not take LANG into consideration.

We are mis-communicating.  I wasn't talking about using the locale in
the libc calls, I was talking about the places and the functionalities
where Emacs uses the locale to determine some of its defaults, which
are unrelated to what libc does.  For example, we don't use libc
functions for decoding and encoding text, we do that in our own code,
and that code has its defaults set according to the locale.  So even
if libc must think we are in en_US.utf8, the Emacs code doesn't have
to, and it should still tailor its behavior wrt i18n to the current
locale.

I presented several specific examples where we use the locale
information for setting up defaults in Emacs, which are unrelated to
libc functions.  I don't see your responses to those examples.  So let
me ask specific questions about the behavior of the current code on
master:

  . what will be the default coding-systems in Emacs? will they always
    be UTF-8?
  . what will be the default dictionary for spell-checking? will it
    always be English?
  . what will be the default language-environment in Emacs? will it
    always be UTF-8? if so, what would be the default input method to
    be turned on when the user types 'C-\' ?

These are just a few examples of locale-dependent behavior that we
have in Emacs; there are likely others.  We need to discuss how to
give users of Emacs on Android the behavior they expect in these
functionalities, regardless of what the underlying libc supports.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> So a program on Android cannot output an arbitrary byte stream, which
> just happens to be text encoded in non-UTF-8 encoding?

Such situations are possible under any system, as programs which don't
observe the C library's configuration can always output text in a coding
system at variance with it.

> And anyway, what about the other examples I gave, with visiting files
> and receiving email? what about them?  How can Android prevent such
> files or such email messages from getting into Emacs?

Emacs should detect the coding system from a coding: cookie or the
file's content, just as it does under other systems where utf-8-unix is
the locale-coding-system.

> We are mis-communicating.  I wasn't talking about using the locale in
> the libc calls, I was talking about the places and the functionalities
> where Emacs uses the locale to determine some of its defaults, which
> are unrelated to what libc does.  For example, we don't use libc
> functions for decoding and encoding text, we do that in our own code,
> and that code has its defaults set according to the locale.  So even
> if libc must think we are in en_US.utf8, the Emacs code doesn't have
> to, and it should still tailor its behavior wrt i18n to the current
> locale.

The coding system is quite immaterial here, as none of the Android
system languages prescribe any other than utf-8-unix.

> I presented several specific examples where we use the locale
> information for setting up defaults in Emacs, which are unrelated to
> libc functions.  I don't see your responses to those examples.  So let
> me ask specific questions about the behavior of the current code on
> master:
>
>   . what will be the default coding-systems in Emacs? will they always
>     be UTF-8?

Yes.

>   . what will be the default dictionary for spell-checking? will it
>     always be English?

Yes, unless the system's spell checker is being used.

>   . what will be the default language-environment in Emacs? will it
>     always be UTF-8? if so, what would be the default input method to
>     be turned on when the user types 'C-\' ?

It'll always be UTF-8, and typing C-\ should prompt the user for the
input method to enable.

> These are just a few examples of locale-dependent behavior that we
> have in Emacs; there are likely others.  We need to discuss how to
> give users of Emacs on Android the behavior they expect in these
> functionalities, regardless of what the underlying libc supports.

These are all meaningful considerations, but they don't really pertain
to locale-coding-system or LANG, as no circumstances under Android can
arise that warrant setting it to a value besides utf-8-unix.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Dec 2023 15:24:02 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So a program on Android cannot output an arbitrary byte stream, which
> > just happens to be text encoded in non-UTF-8 encoding?
> 
> Such situations are possible under any system, as programs which don't
> observe the C library's configuration can always output text in a coding
> system at variance with it.

If that encoding is consistent with the current system locale, Emacs
is expected to be able to decode it.

> > And anyway, what about the other examples I gave, with visiting files
> > and receiving email? what about them?  How can Android prevent such
> > files or such email messages from getting into Emacs?
> 
> Emacs should detect the coding system from a coding: cookie or the
> file's content, just as it does under other systems where utf-8-unix is
> the locale-coding-system.

Again, if the current locale is not UTF-8, Emacs should be able to
decode text even in the absence of any cookies.  When the file's
contents could be decoded with more than a single coding-system, Emacs
uses the locale's defaults to break the tie.

> > We are mis-communicating.  I wasn't talking about using the locale in
> > the libc calls, I was talking about the places and the functionalities
> > where Emacs uses the locale to determine some of its defaults, which
> > are unrelated to what libc does.  For example, we don't use libc
> > functions for decoding and encoding text, we do that in our own code,
> > and that code has its defaults set according to the locale.  So even
> > if libc must think we are in en_US.utf8, the Emacs code doesn't have
> > to, and it should still tailor its behavior wrt i18n to the current
> > locale.
> 
> The coding system is quite immaterial here, as none of the Android
> system languages prescribe any other than utf-8-unix.

Once again: Emacs is expected to support more than that, it is
expected to behave according to the current system locale.

> >   . what will be the default coding-systems in Emacs? will they always
> >     be UTF-8?
> 
> Yes.
> 
> >   . what will be the default dictionary for spell-checking? will it
> >     always be English?
> 
> Yes, unless the system's spell checker is being used.
> 
> >   . what will be the default language-environment in Emacs? will it
> >     always be UTF-8? if so, what would be the default input method to
> >     be turned on when the user types 'C-\' ?
> 
> It'll always be UTF-8, and typing C-\ should prompt the user for the
> input method to enable.

Such behaviors are unacceptable.  We should fix them so that Emacs on
Android behaves like it does on other platforms, or else we cannot
release such an Android port.  Please work on fixing these glaring
omissions.

> > These are just a few examples of locale-dependent behavior that we
> > have in Emacs; there are likely others.  We need to discuss how to
> > give users of Emacs on Android the behavior they expect in these
> > functionalities, regardless of what the underlying libc supports.
> 
> These are all meaningful considerations, but they don't really pertain
> to locale-coding-system or LANG, as no circumstances under Android can
> arise that warrant setting it to a value besides utf-8-unix.

I disagree, and the above examples confirm my worries.  This must be
fixed.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> If that encoding is consistent with the current system locale, Emacs
> is expected to be able to decode it.

The system's locale is _always_ UTF-8.

> Again, if the current locale is not UTF-8, Emacs should be able to
> decode text even in the absence of any cookies.  When the file's
> contents could be decoded with more than a single coding-system, Emacs
> uses the locale's defaults to break the tie.

We are mis-communicating: there are no non-UTF-8 locales under Android,
which is a guarantee from the Android developers.  There are different
languages, but they are uniformly encoded as UTF-8.

> Once again: Emacs is expected to support more than that, it is
> expected to behave according to the current system locale.

There is _NO_ system locale besides UTF-8.  Irrespective of the language
selected, the locale installed amounts to en_US.utf-8, with the
intention being that such programs as spell checkers will detect and
apply the user's language preferences themselves.

>> >   . what will be the default coding-systems in Emacs? will they always
>> >     be UTF-8?
>> 
>> Yes.
>> 
>> >   . what will be the default dictionary for spell-checking? will it
>> >     always be English?
>> 
>> Yes, unless the system's spell checker is being used.
>> 
>> >   . what will be the default language-environment in Emacs? will it
>> >     always be UTF-8? if so, what would be the default input method to
>> >     be turned on when the user types 'C-\' ?
>> 
>> It'll always be UTF-8, and typing C-\ should prompt the user for the
>> input method to enable.
>
> Such behaviors are unacceptable.  We should fix them so that Emacs on
> Android behaves like it does on other platforms, or else we cannot
> release such an Android port.  Please work on fixing these glaring
> omissions.

How can it be fixed, when there are only two locales (C and UTF-8) in
existence?  Android spell checkers are meant to provide their own means
for detecting and applying the system's language preferences, because
the system does not on their behalf.

Thanks.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Dec 2023 18:30:51 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If that encoding is consistent with the current system locale, Emacs
> > is expected to be able to decode it.
> 
> The system's locale is _always_ UTF-8.

Now I'm confused: if the above is true, then what is the below about?

    Emacs does not respect the locale configured for user applications
  in the system, for the selection of locales available there does not
  match that supplied by the C library.  When Emacs starts on Android
  5.0 or newer, the @code{LANG} environment variable is set to
  @code{en_US.utf8}, which induces subprocesses linked against the
  Android C library to print output sensibly.

My interpretation of this was that Android does support locales
"configured for user applications", but Emacs doesn't respect that.
If that interpretation is incorrect, then how should the above text be
interpreted?  And what does "locale configured for user application"
mean then?

> >> >   . what will be the default coding-systems in Emacs? will they always
> >> >     be UTF-8?
> >> 
> >> Yes.
> >> 
> >> >   . what will be the default dictionary for spell-checking? will it
> >> >     always be English?
> >> 
> >> Yes, unless the system's spell checker is being used.
> >> 
> >> >   . what will be the default language-environment in Emacs? will it
> >> >     always be UTF-8? if so, what would be the default input method to
> >> >     be turned on when the user types 'C-\' ?
> >> 
> >> It'll always be UTF-8, and typing C-\ should prompt the user for the
> >> input method to enable.
> >
> > Such behaviors are unacceptable.  We should fix them so that Emacs on
> > Android behaves like it does on other platforms, or else we cannot
> > release such an Android port.  Please work on fixing these glaring
> > omissions.
> 
> How can it be fixed, when there are only two locales (C and UTF-8) in
> existence?

Let's defer this question until we are on the same page wrt locales
and language settings on Android.  As an active user of Android
(though not of Emacs on Android), I know that there _are_
configuration settings for system language and for each application,
they are under Settings->General Management->Language and
Settings->General Management->App Languages.  There are also settings
for keyboard, which are conceptually part of the language-environment
in Emacs.  Some Android smartphones also have Regional Preferences.

> Android spell checkers are meant to provide their own means
> for detecting and applying the system's language preferences, because
> the system does not on their behalf.

How will that work with ispell.el in Emacs?

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 08 Dec 2023 18:30:51 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > If that encoding is consistent with the current system locale, Emacs
>> > is expected to be able to decode it.
>> 
>> The system's locale is _always_ UTF-8.
>
> Now I'm confused: if the above is true, then what is the below about?
>
>     Emacs does not respect the locale configured for user applications
>   in the system, for the selection of locales available there does not
>   match that supplied by the C library.  When Emacs starts on Android
>   5.0 or newer, the @code{LANG} environment variable is set to
>   @code{en_US.utf8}, which induces subprocesses linked against the
>   Android C library to print output sensibly.
>
> My interpretation of this was that Android does support locales
> "configured for user applications", but Emacs doesn't respect that.
> If that interpretation is incorrect, then how should the above text be
> interpreted?  And what does "locale configured for user application"
> mean then?

I could've chosen better nomenclature.  That was meant as a reference to
the system's configured language, in Settings.

> Let's defer this question until we are on the same page wrt locales
> and language settings on Android.  As an active user of Android
> (though not of Emacs on Android), I know that there _are_
> configuration settings for system language and for each application,
> they are under Settings->General Management->Language and
> Settings->General Management->App Languages.  There are also settings
> for keyboard, which are conceptually part of the language-environment
> in Emacs.  Some Android smartphones also have Regional Preferences.

Yes, but these don't affect the value of the LANG variable, and each of
the languages within uses the UTF-8 coding system.  FTR, this is System
-> Languages & Input -> Languages for me.

> How will that work with ispell.el in Emacs?

Probably the copy of aspell must be modified; but the spell checker I
personally use on Android systems is jinx from ELPA, which is much
faster, and requires the default language be set by hand.

Thanks.

Re: master e4e1e268c8e 2/2: Set a default locale on Android

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 Dec 2023 21:40:42 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The system's locale is _always_ UTF-8.
> >
> > Now I'm confused: if the above is true, then what is the below about?
> >
> >     Emacs does not respect the locale configured for user applications
> >   in the system, for the selection of locales available there does not
> >   match that supplied by the C library.  When Emacs starts on Android
> >   5.0 or newer, the @code{LANG} environment variable is set to
> >   @code{en_US.utf8}, which induces subprocesses linked against the
> >   Android C library to print output sensibly.
> >
> > My interpretation of this was that Android does support locales
> > "configured for user applications", but Emacs doesn't respect that.
> > If that interpretation is incorrect, then how should the above text be
> > interpreted?  And what does "locale configured for user application"
> > mean then?
> 
> I could've chosen better nomenclature.  That was meant as a reference to
> the system's configured language, in Settings.

Then at least the language part of the locale-dependent defaults in
Emacs should follow those settings, IMO.

> > Let's defer this question until we are on the same page wrt locales
> > and language settings on Android.  As an active user of Android
> > (though not of Emacs on Android), I know that there _are_
> > configuration settings for system language and for each application,
> > they are under Settings->General Management->Language and
> > Settings->General Management->App Languages.  There are also settings
> > for keyboard, which are conceptually part of the language-environment
> > in Emacs.  Some Android smartphones also have Regional Preferences.
> 
> Yes, but these don't affect the value of the LANG variable, and each of
> the languages within uses the UTF-8 coding system.  FTR, this is System
> -> Languages & Input -> Languages for me.

If UTF-8 is the standard on Android, then it's okay to have that as
the default coding-system.  But why ignore the language part?  We
should make sure the Emacs language-environment is set up at startup
according to the language settings, either those specific to Emacs, or
using the system default as the fallback.  Any reasons not to do that?
There are quite a few Emacs features that are sensitive to the current
language, and they should ideally continue working on Android.

> > How will that work with ispell.el in Emacs?
> 
> Probably the copy of aspell must be modified; but the spell checker I
> personally use on Android systems is jinx from ELPA, which is much
> faster, and requires the default language be set by hand.

Set by hand after entering the speller, or set by hand via some kind
of invocation-time arguments/switches?

Please note that Emacs can change the dictionary during a running
Emacs session (one way is due to the user's invoking the
ispell-change-dictionary command explicitly, but that's not the only
one).  Are you saying that this cannot possibly work on Android?  That
would mean that Emacs users on Android will be unable to spell-check
buffers in different languages in the same Emacs session.