bug#3221: 23.0.93; ns cannot handle nonascii file names

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Leo]

This is the biggest bug in ns port of Emacs. It exposed itself
originally through emms but I have found an easy way to reproduce it.

1. move .emacs.d and .emacs to some other places so that no user
customisation is in effect.

2. start Emacs from Finder

3. M-! touch 求助.txt

No 求助.txt will be created instead, a file named .txt is created.

If you start Emacs from terminal by running for example:

  /Applications/Emacs.app/Contents/MacOS/Emacs

you won't see this bug.

The consequencies of this bug is that all packages that call external
programs won't be able to handle non ascii names. For example,

http://permalink.gmane.org/gmane.emacs.emms.user/1426


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/Applications/Emacs.app/Contents/Resources/etc/DEBUG for instructions.


In GNU Emacs 23.0.93.1 (i386-apple-darwin9.6.0, NS apple-appkit-949.43)
 of 2009-05-03 on neutron.local
Windowing system distributor `Apple', version 10.3.949
configured using `configure  '--with-ns''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  paredit-mode: t
  dired-omit-mode: t
  recentf-mode: t
  icomplete-mode: t
  savehist-mode: t
  display-time-mode: t
  xterm-mouse-mode: t
  delete-selection-mode: t
  global-auto-revert-mode: t
  minibuffer-depth-indicate-mode: t
  show-paren-mode: t
  rcirc-track-minor-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-x g m M-x r e p o r <tab> <return>

Recent messages:
Loading rcirc...done
Loading paren...done
Loading /Users/gtd/.emacs.d/custom...done
Ido mode enabled
Loading /Users/gtd/.emacs.d/etc/recentf...done
Cleaning up the recentf list...done (0 removed)
Loading /Users/gtd/.emacs.d/init.d/site.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Current frame maximised.
(Press m to repeat this command)

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Leo]

On 2009-05-05 15:36 +0100, Leo wrote:
> This is the biggest bug in ns port of Emacs. It exposed itself
> originally through emms but I have found an easy way to reproduce it.
>
> 1. move .emacs.d and .emacs to some other places so that no user
> customisation is in effect.
>
> 2. start Emacs from Finder
>
> 3. M-! touch 求助.txt
>
> No 求助.txt will be created instead, a file named .txt is created.
>
> If you start Emacs from terminal by running for example:
>
>   /Applications/Emacs.app/Contents/MacOS/Emacs
>
> you won't see this bug.
>
> The consequencies of this bug is that all packages that call external
> programs won't be able to handle non ascii names. For example,
>
> http://permalink.gmane.org/gmane.emacs.emms.user/1426

After some investigation with Ian's help, I have found a workaround by
setting the language environment to UTF-8.

The bug seems to be that Emacs.app is completely unaware of the language
setting of OS X.

-- 
.:  Leo  :.  [ sdl.web AT gmail.com ]  .: I use Emacs :.

               www.git-scm.com
    git - the one true version control system

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Adrian Robert]

http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=3221

Hi,

> After some investigation with Ian's help, I have found a workaround by
> setting the language environment to UTF-8.
>
> The bug seems to be that Emacs.app is completely unaware of the  
> language
> setting of OS X.


I'd like to fix this.

Do you know why it works from terminal?  Is there some code in emacs  
that reads the LANG or some other env setting and sets something in  
emacs?  Where?  What is the command to set the language environment?

While the various methods on the net to set up ~/.MacOSX/ 
environment.plist might work, there ought to be some Cocoa way to  
detect language setting directly...

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Leo]

On 2009-07-23 17:47 +0100, Adrian Robert wrote:
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=3221
>
> Hi,
>
>> After some investigation with Ian's help, I have found a workaround by
>> setting the language environment to UTF-8.
>>
>> The bug seems to be that Emacs.app is completely unaware of the
>> language setting of OS X.
>
> I'd like to fix this.
>
> Do you know why it works from terminal? Is there some code in emacs
> that reads the LANG or some other env setting and sets something in
> emacs? Where? What is the command to set the language environment?

I don't know how internally this is done. Maybe Ian can comment on this.

The language was set through  + system preferences + International.
After the setting, run 'env' in Terminal.app and there's a line:

  LANG=en_GB.UTF-8

When starting Emacs from Terminal.app by running:

  /Applications/Emacs.app/Contents/MacOS/Emacs -q

It picks up that setting automatically i.e. if you check variables such
as default-file-name-coding-system, it says UTF-8.

But if you start Emacs from dock, it is something else.

> While the various methods on the net to set up ~/.MacOSX/
> environment.plist might work, there ought to be some Cocoa way to
> detect language setting directly...

It will be nice to have that. The current behaviour gave me a big shock
when I first used Emacs.app.

-- 
Leo's Emacs uptime: 43 days, 5 hours, 49 minutes, 1 second

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Jason Rumney]

Adrian Robert wrote:
> I'd like to fix this.
>
> Do you know why it works from terminal?  Is there some code in emacs 
> that reads the LANG or some other env setting and sets something in 
> emacs?  Where?  What is the command to set the language environment?

Yes, the command that does it is set-locale-environment in 
international/mule-cmds.el, called from command-line as part of the 
startup sequence.

>
> While the various methods on the net to set up 
> ~/.MacOSX/environment.plist might work, there ought to be some Cocoa 
> way to detect language setting directly...

Yes, the way we handle this on Windows is with a function that 
initializes certain environment variables from the system APIs if they 
are not explicitly set. See init_environment in w32.c, which is called 
from main in emacs.c.  You probably need to do something similar at 
least on Mac OSX, I'm not sure about GnuStep etc, as that is probably 
running on a platform that complies with Emacs's expectations about 
which environment variables are set.

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Alan J Third]

Adrian Robert <adrian.b.robert@gmail.com> writes:

> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3221
>
> Hi,
>
>> After some investigation with Ian's help, I have found a workaround by
>> setting the language environment to UTF-8.
>>
>> The bug seems to be that Emacs.app is completely unaware of the
>> language
>> setting of OS X.
>
>
> I'd like to fix this.
>
> Do you know why it works from terminal?  Is there some code in emacs
> that reads the LANG or some other env setting and sets something in
> emacs?  Where?  What is the command to set the language environment?

OS X's terminal application sets the LANG environment variable, so when
Emacs is run from there it picks up LANG. When run from the GUI (dock,
or spotlight or whatever) OS X does *not* set the LANG variable, so
Emacs doesn't pick it up.

> While the various methods on the net to set up
> ~/.MacOSX/environment.plist might work, there ought to be some Cocoa
> way to detect language setting directly...

It looks like the right way to get the locale from OS X is something
like:

const char * locale = [[[NSLocale currentLocale] localeIdentifier] UTF8String];

HOWEVER, it appears the actual bug reported:

> 2. start Emacs from Finder
> 
> 3. M-! touch 求助.txt
> 
> No 求助.txt will be created instead, a file named .txt is created.

is fixed in emacs 25. Or, at least, I can't reproduce it.

Not having LANG set does break ispell checking with hunspell on OSX, for
me at least, so it would still be nice to have the locale stuff fixed.
Presumably there are other reasons for wanting Emacs to have the correct
locale, too.

Should that be a new bug report?
-- 
Alan Third

bug#3221: 23.0.93; ns cannot handle nonascii file names

[Eli Zaretskii]

> From: Alan J Third <alan@idiocy.org>
> Date: Sat, 16 Jan 2016 17:06:17 +0000
> Cc: Ian Eure <ian@digg.com>, Leo <sdl.web@gmail.com>, 3221@debbugs.gnu.org
> 
> Not having LANG set does break ispell checking with hunspell on OSX, for
> me at least, so it would still be nice to have the locale stuff fixed.
> Presumably there are other reasons for wanting Emacs to have the correct
> locale, too.
> 
> Should that be a new bug report?

Are you saying that the NS build doesn't pick up the correct locale in
the session run from the GUI because it can only do so if LANG is set?
If so, yes, I think a separate bug report should be filed for that.
Having a correct value of the locale is very important for many things
in Emacs; spell-checking is just a tip of the iceberg.

Thanks.