bug#51832: Piping unicode text in `shell-command'

[Alan Third <alan@idiocy.org>]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

On Sun, Nov 14, 2021 at 12:56:14PM +0200, Eli Zaretskii wrote:
> > From: Philipp <p.stephani2@gmail.com>
> > Date: Sun, 14 Nov 2021 11:41:38 +0100
> > Cc: Lars Ingebrigtsen <larsi@gnus.org>,
> >  tor.a.s.kringeland@ntnu.no,
> >  51832@debbugs.gnu.org,
> >  alan@idiocy.org
> > 
> > > I asked once why we push LANG into the environment, instead of calling
> > > setlocale, which would only affect Emacs.  I don't think I saw an
> > > answer to that question, or did I miss it?
> > > 
> > 
> > AIUI the intention is that this should affect subprocesses started from Emacs.  At least that's how I interpret the comment
> > 
> > /* macOS doesn't set any environment variables for the locale when run
> >    from the GUI. Get the locale from the OS and set LANG.  */
> 
> Why is that needed?
> 
> And if it is needed, how come we are setting LANG to an invalid locale
> and the system somehow sets it to the correct locale?

macOS itself doesn't set any locale related environment variables, any
application that is running UNIX style commands is expected to set
them itself. The UNIX commands don't themselves pick up the locale
from the system, they rely on the environment variables.

In other words, as with anything UNIXy on macOS, it's a badly thought
out mess.

It seems suspicious to me that we've had this code since Emacs 26, but
only in the last few weeks we've had two complaints about it. Having
dug out my Mac I can't convince it to show any of the errors that have
been reported, so I suspect either the latest version of macOS has
made the locale handling much more strict or has removed a lot of
locales.

I've attached a patch that may do something towards preventing this
problem but ultimately this is a convenience to give a best guess at
choosing the correct dictionary, date format, etc. If we can't easily
fix it then we can drop it and tell people to set it in their init.el
themselves.

-- 
Alan Third

[-- Attachment #2: 0001-Only-set-LANG-if-the-ID-is-valid.patch --]
[-- Type: text/x-diff, Size: 2059 bytes --]

From ff67f1cbee3c0b1fd5b1a0d725e40158190cfe55 Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Sun, 14 Nov 2021 11:32:54 +0000
Subject: [PATCH] Only set LANG if the ID is valid

* src/nsterm.m (ns_init_locale): Check the provided locale identifier
is available before trying to use it.
---
 src/nsterm.m | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)

diff --git a/src/nsterm.m b/src/nsterm.m
index 1f17a30272..566537e8a1 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -535,21 +535,25 @@ - (NSColor *)colorUsingDefaultColorSpace
 
   NSTRACE ("ns_init_locale");
 
-  @try
+  if ([[NSLocale availableLocaleIdentifiers]
+        containsObject:[locale localeIdentifier]])
     {
-      /* It seems macOS should probably use UTF-8 everywhere.
-         'localeIdentifier' does not specify the encoding, and I can't
-         find any way to get the OS to tell us which encoding to use,
-         so hard-code '.UTF-8'.  */
-      NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
-                                     [locale localeIdentifier]];
-
-      /* Set LANG to locale, but not if LANG is already set.  */
-      setenv("LANG", [localeID UTF8String], 0);
-    }
-  @catch (NSException *e)
-    {
-      NSLog (@"Locale detection failed: %@: %@", [e name], [e reason]);
+      @try
+        {
+          /* It seems macOS should probably use UTF-8 everywhere.
+             'localeIdentifier' does not specify the encoding, and I can't
+             find any way to get the OS to tell us which encoding to use,
+             so hard-code '.UTF-8'.  */
+          NSString *localeID = [NSString stringWithFormat:@"%@.UTF-8",
+                                         [locale localeIdentifier]];
+
+          /* Set LANG to locale, but not if LANG is already set.  */
+          setenv("LANG", [localeID UTF8String], 0);
+        }
+      @catch (NSException *e)
+        {
+          NSLog (@"Locale detection failed: %@: %@", [e name], [e reason]);
+        }
     }
 }
 
-- 
2.33.0