Preparing for the libc/locale upgrade

Preparing for the libc/locale upgrade

[Federico Beffa]

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Consequences for Guix on foreign distros:
>
>   • If the host distro provides binaries that use libc < 2.22 and you
>     use a mixture of Guix-provided and distro-provided programs, this is
>     pretty bad.
>
>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>     all the distro-provided programs), or use exclusively Guix-provided
>     programs, or use the “C” locale.

Does this means that Guix on other distributions is no longer of
interest to the Guix project and it is essentially unsupported?

Or is this a transitory situation and an acceptable solution is being
worked on?

Regards,
Fede

Re: Preparing for the libc/locale upgrade

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> Consequences for Guix on foreign distros:
>>
>>   • If the host distro provides binaries that use libc < 2.22 and you
>>     use a mixture of Guix-provided and distro-provided programs, this is
>>     pretty bad.
>>
>>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>>     all the distro-provided programs), or use exclusively Guix-provided
>>     programs, or use the “C” locale.
>
> Does this means that Guix on other distributions is no longer of
> interest to the Guix project and it is essentially unsupported?

No, definitely not!

While suboptimal, the 3 solutions above are probably OK as a temporary
measure.

For the longer term, I hope we can help improve libc:

  https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html

(I’d like to apply this patch in Guix in the next ‘core-updates’ cycle
anyway to mitigate the problem.)

> Or is this a transitory situation and an acceptable solution is being
> worked on?

This is transitory because sooner or later your host distro will upgrade
to libc 2.22 as well.

IMO Guix is not at fault; rather, it sheds light on a shortcoming of
libc’s handling of locale data, which was designed with single-libc
systems in mind.

Thanks,
Ludo’.

Re: Preparing for the libc/locale upgrade

[Mark H Weaver]

Federico Beffa <beffa@ieee.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> Consequences for Guix on foreign distros:
>>
>>   • If the host distro provides binaries that use libc < 2.22 and you
>>     use a mixture of Guix-provided and distro-provided programs, this is
>>     pretty bad.
>>
>>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>>     all the distro-provided programs), or use exclusively Guix-provided
>>     programs, or use the “C” locale.
>
> Does this means that Guix on other distributions is no longer of
> interest to the Guix project and it is essentially unsupported?
>
> Or is this a transitory situation and an acceptable solution is being
> worked on?

I think I know a workaround: leave LOCPATH unset, and make
/run/current-system/locale a symlink to freshly generated locales for
glibc 2.22.  Guix-compiled software is configured to look for locales
there if LOCPATH is unset.

     Mark

Re: Preparing for the libc/locale upgrade

[Federico Beffa]

On Tue, Sep 29, 2015 at 4:32 PM, Mark H Weaver <mhw@netris.org> wrote:
> Federico Beffa <beffa@ieee.org> writes:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>> [...]
>>
>>> Consequences for Guix on foreign distros:
>>>
>>>   • If the host distro provides binaries that use libc < 2.22 and you
>>>     use a mixture of Guix-provided and distro-provided programs, this is
>>>     pretty bad.
>>>
>>>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>>>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>>>     all the distro-provided programs), or use exclusively Guix-provided
>>>     programs, or use the “C” locale.
>>
>> Does this means that Guix on other distributions is no longer of
>> interest to the Guix project and it is essentially unsupported?
>>
>> Or is this a transitory situation and an acceptable solution is being
>> worked on?
>
> I think I know a workaround: leave LOCPATH unset, and make
> /run/current-system/locale a symlink to freshly generated locales for
> glibc 2.22.  Guix-compiled software is configured to look for locales
> there if LOCPATH is unset.

That's a promising trick.
Thanks!
Fede

Re: Preparing for the libc/locale upgrade

[Federico Beffa]

On Mon, Sep 28, 2015 at 10:45 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Federico Beffa <beffa@ieee.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>> [...]
>>
>>> Consequences for Guix on foreign distros:
>>>
>>>   • If the host distro provides binaries that use libc < 2.22 and you
>>>     use a mixture of Guix-provided and distro-provided programs, this is
>>>     pretty bad.
>>>
>>>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>>>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>>>     all the distro-provided programs), or use exclusively Guix-provided
>>>     programs, or use the “C” locale.
>>
>> Does this means that Guix on other distributions is no longer of
>> interest to the Guix project and it is essentially unsupported?
>
> No, definitely not!
>

Thats good to hear!

> While suboptimal, the 3 solutions above are probably OK as a temporary
> measure.

From my point of view Mark's suggestion sounds as the most acceptable
(although I've not tested it as I've not updated my Guix yet).

>
> For the longer term, I hope we can help improve libc:
>
>   https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html
>
> (I’d like to apply this patch in Guix in the next ‘core-updates’ cycle
> anyway to mitigate the problem.)
>
>> Or is this a transitory situation and an acceptable solution is being
>> worked on?
>
> This is transitory because sooner or later your host distro will upgrade
> to libc 2.22 as well.

Sure, but popular distributions like Debian are likely to upgrade in a
few of years! So, without action the problem is not going away any
time soon.

>
> IMO Guix is not at fault; rather, it sheds light on a shortcoming of
> libc’s handling of locale data, which was designed with single-libc
> systems in mind.

I fully agree with your statement. However, leaving Guix users (I'm
not talking about developers) exposed to such problems is not what I
expect from a high quality product. A brute force fix may be to tell
each Guix program where the locale is with a wrapper. This is for sure
not elegant (and there may be better ways, you know better...), but
the point is that probably a way to preserve a good end user
experience out of the box does exist and, from my point of view, we
should provide it. The first experience is very important to retain
users. If it's bad most people just walk away.

Regards,
Fede

Re: Preparing for the libc/locale upgrade

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On Mon, Sep 28, 2015 at 10:45 PM, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

> From my point of view Mark's suggestion sounds as the most acceptable

Which one is it?

>> IMO Guix is not at fault; rather, it sheds light on a shortcoming of
>> libc’s handling of locale data, which was designed with single-libc
>> systems in mind.
>
> I fully agree with your statement. However, leaving Guix users (I'm
> not talking about developers) exposed to such problems is not what I
> expect from a high quality product.

I agree.

> A brute force fix may be to tell each Guix program where the locale is
> with a wrapper. This is for sure not elegant (and there may be better
> ways, you know better...), but the point is that probably a way to
> preserve a good end user experience out of the box does exist and,
> from my point of view, we should provide it.

Well, we could ship 110+ MiB of locales along with our libc, as we used
to do¹; adding a wrapper basically amounts to this.  That way, no need
to fiddle with LOCPATH, the right locale data will always be found.

But honestly, I think that sucks.  It shouldn’t be all-or-nothing.

Alternately, we could patch our libc to honor GUIX_LOCPATH (in addition
to LOCPATH), so that the host distro’s programs are unaffected, and thus
don’t end up aborting with that assertion failure.

That’s the best kludge that comes to mind (yeah that’s an oxymoron ;-)).

Thoughts?

Thanks,
Ludo’.

¹ http://bugs.gnu.org/18085

Re: Preparing for the libc/locale upgrade

[Federico Beffa]

On Wed, Sep 30, 2015 at 3:46 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Federico Beffa <beffa@ieee.org> skribis:
>
>> On Mon, Sep 28, 2015 at 10:45 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>
> [...]
>
>> From my point of view Mark's suggestion sounds as the most acceptable
>
> Which one is it?

For some reason I don't see Mark's message on the ML. So instead of a
link, a copy/paste from his email:

"
I think I know a workaround: leave LOCPATH unset, and make
/run/current-system/locale a symlink to freshly generated locales for
glibc 2.22.  Guix-compiled software is configured to look for locales
there if LOCPATH is unset.
"

>
>>> IMO Guix is not at fault; rather, it sheds light on a shortcoming of
>>> libc’s handling of locale data, which was designed with single-libc
>>> systems in mind.
>>
>> I fully agree with your statement. However, leaving Guix users (I'm
>> not talking about developers) exposed to such problems is not what I
>> expect from a high quality product.
>
> I agree.
>
>> A brute force fix may be to tell each Guix program where the locale is
>> with a wrapper. This is for sure not elegant (and there may be better
>> ways, you know better...), but the point is that probably a way to
>> preserve a good end user experience out of the box does exist and,
>> from my point of view, we should provide it.
>
> Well, we could ship 110+ MiB of locales along with our libc, as we used
> to do¹; adding a wrapper basically amounts to this.  That way, no need
> to fiddle with LOCPATH, the right locale data will always be found.
>
> But honestly, I think that sucks.  It shouldn’t be all-or-nothing.
>
> Alternately, we could patch our libc to honor GUIX_LOCPATH (in addition
> to LOCPATH), so that the host distro’s programs are unaffected, and thus
> don’t end up aborting with that assertion failure.
>
> That’s the best kludge that comes to mind (yeah that’s an oxymoron ;-)).

From my point of view either one is much better than the current situation.

Thanks,
Fede

Re: Preparing for the libc/locale upgrade

[Konrad Hinsen]

On 30/09/2015 14:35, Federico Beffa wrote:

>  From my point of view Mark's suggestion sounds as the most acceptable
> (although I've not tested it as I've not updated my Guix yet).

I just tested it, it works fine. And, finally, no more error messages 
about missing locales anywhere!

Konrad.

Re: Preparing for the libc/locale upgrade

[Leo Famulari]

Just want to confirm I'm doing your workaround properly... like this?
$ ln -s gnu/store/...-glibc-utf8-locales-2.22/lib/locale
/run/current-system/locale
$ echo $LOCPATH

The previous line is blank, showing that $LOCPATH is unset.

On Tue, Sep 29, 2015, at 10:32, Mark H Weaver wrote:
> Federico Beffa <beffa@ieee.org> writes:
> 
> > ludo@gnu.org (Ludovic Courtès) writes:
> >
> > [...]
> >
> >> Consequences for Guix on foreign distros:
> >>
> >>   • If the host distro provides binaries that use libc < 2.22 and you
> >>     use a mixture of Guix-provided and distro-provided programs, this is
> >>     pretty bad.
> >>
> >>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
> >>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
> >>     all the distro-provided programs), or use exclusively Guix-provided
> >>     programs, or use the “C” locale.
> >
> > Does this means that Guix on other distributions is no longer of
> > interest to the Guix project and it is essentially unsupported?
> >
> > Or is this a transitory situation and an acceptable solution is being
> > worked on?
> 
> I think I know a workaround: leave LOCPATH unset, and make
> /run/current-system/locale a symlink to freshly generated locales for
> glibc 2.22.  Guix-compiled software is configured to look for locales
> there if LOCPATH is unset.
> 
>      Mark
> 

Re: Preparing for the libc/locale upgrade

[Taylan Ulrich Bayırlı/Kammer]

Leo Famulari <leo@famulari.name> writes:

> Just want to confirm I'm doing your workaround properly... like this?
> $ ln -s gnu/store/...-glibc-utf8-locales-2.22/lib/locale
> /run/current-system/locale
> $ echo $LOCPATH
>
> The previous line is blank, showing that $LOCPATH is unset.

I think that would leave you with a dangling symlink when that
/gnu/store item gets garbage collected.  (You also forgot the first
slash.)

The following might be preferable:

  guix package -p /run/current-system/profile -i glibc-locales
  ln -s profile/lib/locale /run/current-system/locale

Don't know if there's an even better way.

HTH,
Taylan

[PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

So, below are a couple of patches implementing ideas that were discussed here
and on IRC to improve the locale mess.

The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
that on foreign distros, users could set this variable instead of 'LOCPATH',
and the host distro's programs wouldn't break.

The second patch was suggested by Mark.  It basically adds "/2.22" to every
directory name specified in LOCPATH, as well as to the default locale
directory.  The idea here is that libc would only stumble on compatible
locale data.

Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
effectively Guix's libc would already be interpreting 'LOCPATH' differently.
So my inclination would be to apply only the second one.

Thoughts?

I'd like to get that on 'core-updates' soon so we can quickly get the better
fix.

Thanks,
Ludo'.

Ludovic Courtès (2):
  gnu: glibc: Honor 'GUIX_LOCPATH'.
  gnu: glibc: Look for locale data in versioned sub-directories.

 doc/guix.texi                                      | 23 +++++++----
 gnu-system.am                                      |  2 +
 gnu/packages/base.scm                              | 24 ++++++-----
 gnu/packages/patches/glibc-guix-locpath.patch      | 38 +++++++++++++++++
 gnu/packages/patches/glibc-versioned-locpath.patch | 47 ++++++++++++++++++++++
 5 files changed, 116 insertions(+), 18 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-guix-locpath.patch
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch

-- 
2.5.0

[PATCH 1/2] gnu: glibc: Honor 'GUIX_LOCPATH'.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]


* gnu/packages/patches/glibc-guix-locpath.patch: New file.
* gnu-system.am (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc)[source]: Use it.
  [native-search-paths]: Use 'GUIX_LOCPATH' instead of 'LOCPATH'.
* doc/guix.texi (Application Setup): Introduce the term "foreign
  distro".  Document 'GUIX_LOCPATH'.
---
 doc/guix.texi                                 | 23 +++++++++++-----
 gnu-system.am                                 |  1 +
 gnu/packages/base.scm                         |  7 +++--
 gnu/packages/patches/glibc-guix-locpath.patch | 38 +++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 9 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-guix-locpath.patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-glibc-Honor-GUIX_LOCPATH.patch --]
[-- Type: text/x-patch; name="0001-gnu-glibc-Honor-GUIX_LOCPATH.patch", Size: 5372 bytes --]

diff --git a/doc/guix.texi b/doc/guix.texi
index 68ee451..1ad4229 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -933,24 +933,24 @@ daemons on the same machine.
 @node Application Setup
 @section Application Setup
 
-When using Guix on top of GNU/Linux distribution other than GuixSD, a
-few additional steps are needed to get everything in place.  Here are
-some of them.
+When using Guix on top of GNU/Linux distribution other than GuixSD---a
+so-called @dfn{foreign distro}---a few additional steps are needed to
+get everything in place.  Here are some of them.
 
 @subsection Locales
 
 @anchor{locales-and-locpath}
 @cindex locales, when not on GuixSD
 @vindex LOCPATH
+@vindex GUIX_LOCPATH
 Packages installed @i{via} Guix will not use the host system's locale
 data.  Instead, you must first install one of the locale packages
-available with Guix and then define the @code{LOCPATH} environment
-variable (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C Library
-Reference Manual}):
+available with Guix and then define the @code{GUIX_LOCPATH} environment
+variable:
 
 @example
 $ guix package -i glibc-locales
-$ export LOCPATH=$HOME/.guix-profile/lib/locale
+$ export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
 @end example
 
 Note that the @code{glibc-locales} package contains data for all the
@@ -958,6 +958,15 @@ locales supported by the GNU@tie{}libc and weighs in at around
 110@tie{}MiB.  Alternately, the @code{glibc-utf8-locales} is smaller but
 limited to a few UTF-8 locales.
 
+The @code{GUIX_LOCPATH} variable plays the exact same role as
+@code{LOCPATH} (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C
+Library Reference Manual}).  However, since it is honored only by Guix's
+libc, and not by the libc provided by foreign distros, using
+@code{GUIX_LOCPATH} allows you to make sure the the foreign distro's
+programs will not end up loading incompatible locale data.  This is
+important because the locale data format used by different libc versions
+may be incompatible.
+
 @subsection X11 Fonts
 
 The majority of graphical applications use Fontconfig to locate and
diff --git a/gnu-system.am b/gnu-system.am
index f2f7e17..aba8f8d 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -469,6 +469,7 @@ dist_patch_DATA =						\
   gnu/packages/patches/glib-tests-timer.patch			\
   gnu/packages/patches/glib-tests-gapplication.patch		\
   gnu/packages/patches/glibc-bootstrap-system.patch		\
+  gnu/packages/patches/glibc-guix-locpath.patch			\
   gnu/packages/patches/glibc-ldd-x86_64.patch			\
   gnu/packages/patches/glibc-locales.patch			\
   gnu/packages/patches/glibc-o-largefile.patch			\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index a7d9459..337fdae 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -476,6 +476,7 @@ store.")
             (modules '((guix build utils)))
             (patches (map search-patch
                           '("glibc-ldd-x86_64.patch"
+                            "glibc-guix-locpath.patch"
                             "glibc-o-largefile.patch")))))
    (build-system gnu-build-system)
 
@@ -606,9 +607,11 @@ store.")
 
    (native-search-paths
     ;; Search path for packages that provide locale data.  This is useful
-    ;; primarily in build environments.
+    ;; primarily in build environments.  Use 'GUIX_LOCPATH' rather than
+    ;; 'LOCPATH' to avoid interference with the host system's libc on foreign
+    ;; distros.
     (list (search-path-specification
-           (variable "LOCPATH")
+           (variable "GUIX_LOCPATH")
            (files '("lib/locale")))))
 
    (synopsis "The GNU C Library")
diff --git a/gnu/packages/patches/glibc-guix-locpath.patch b/gnu/packages/patches/glibc-guix-locpath.patch
new file mode 100644
index 0000000..2a814a3
--- /dev/null
+++ b/gnu/packages/patches/glibc-guix-locpath.patch
@@ -0,0 +1,38 @@
+Honor a special 'GUIX_LOCPATH' environment variable.
+
+This is most useful when using Guix on top of another distro, which uses an
+different libc version with incompatible locale data.  In this case, setting
+'GUIX_LOCPATH' rather than 'LOCPATH' allows users to tell Guix's libc where to
+look for its locale data without breaking programs that use the other libc.
+
+See <https://lists.gnu.org/archive/html/guix-devel/2015-09/msg00717.html> for
+some background information.
+
+--- a/locale/newlocale.c
++++ b/locale/newlocale.c
+@@ -102,7 +102,9 @@ __newlocale (int category_mask, const char *locale, __locale_t base)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
++  locpath_var = getenv ("GUIX_LOCPATH");
++  if (locpath_var == NULL || locpath_var[0] == '\0')
++    locpath_var = getenv ("LOCPATH");
+   if (locpath_var != NULL && locpath_var[0] != '\0')
+     {
+       if (__argz_create_sep (locpath_var, ':',
+diff --git a/locale/setlocale.c b/locale/setlocale.c
+index ead030d..ca2337d 100644
+--- a/locale/setlocale.c
++++ b/locale/setlocale.c
+@@ -251,7 +251,9 @@ setlocale (int category, const char *locale)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
++  locpath_var = getenv ("GUIX_LOCPATH");
++  if (locpath_var == NULL || locpath_var[0] == '\0')
++    locpath_var = getenv ("LOCPATH");
+   if (locpath_var != NULL && locpath_var[0] != '\0')
+     {
+       if (__argz_create_sep (locpath_var, ':',

[PATCH 2/2] gnu: glibc: Look for locale data in versioned sub-directories.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]


* gnu/packages/patches/glibc-versioned-locpath.patch: New file.
* gnu-system.am (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc)[source]: Use it.
  [arguments]: Remove explicit version sub-directory from
  libc_cv_localedir.
  (glibc-locales, glibc-utf8-locales): Write to a VERSION
  sub-directory.
---
 gnu-system.am                                      |  1 +
 gnu/packages/base.scm                              | 17 ++++----
 gnu/packages/patches/glibc-versioned-locpath.patch | 47 ++++++++++++++++++++++
 3 files changed, 56 insertions(+), 9 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0002-gnu-glibc-Look-for-locale-data-in-versioned-sub-dire.patch --]
[-- Type: text/x-patch; name="0002-gnu-glibc-Look-for-locale-data-in-versioned-sub-dire.patch", Size: 5463 bytes --]

diff --git a/gnu-system.am b/gnu-system.am
index aba8f8d..3aa9144 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -473,6 +473,7 @@ dist_patch_DATA =						\
   gnu/packages/patches/glibc-ldd-x86_64.patch			\
   gnu/packages/patches/glibc-locales.patch			\
   gnu/packages/patches/glibc-o-largefile.patch			\
+  gnu/packages/patches/glibc-versioned-locpath.patch		\
   gnu/packages/patches/gmp-arm-asm-nothumb.patch		\
   gnu/packages/patches/gnucash-price-quotes-perl.patch		\
   gnu/packages/patches/gnutls-doc-fix.patch			\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 337fdae..443c54e 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -477,6 +477,7 @@ store.")
             (patches (map search-patch
                           '("glibc-ldd-x86_64.patch"
                             "glibc-guix-locpath.patch"
+                            "glibc-versioned-locpath.patch"
                             "glibc-o-largefile.patch")))))
    (build-system gnu-build-system)
 
@@ -510,16 +511,12 @@ store.")
             ;; Set the default locale path.  In practice, $LOCPATH may be
             ;; defined to point whatever locales users want.  However, setuid
             ;; binaries don't honor $LOCPATH, so they'll instead look into
-            ;; $libc_cv_localedir; we choose /run/current-system/locale/X.Y,
+            ;; $libc_cv_localedir; we choose /run/current-system/locale
             ;; with the idea that it is going to be populated by the sysadmin.
-            ;; The "X.Y" sub-directory is because locale data formats are
-            ;; incompatible across libc versions; see
-            ;; <https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00737.html>.
             ;;
             ;; `--localedir' is not honored, so work around it.
             ;; See <http://sourceware.org/ml/libc-alpha/2013-03/msg00093.html>.
-            (string-append "libc_cv_localedir=/run/current-system/locale/"
-                           ,version)
+            "libc_cv_localedir=/run/current-system/locale"
 
             (string-append "--with-headers="
                            (assoc-ref %build-inputs "linux-headers")
@@ -655,7 +652,8 @@ the 'share/locale' sub-directory of this package.")
                    ;; Use $(libdir)/locale as is the case by default.
                    (list (string-append "libc_cv_localedir="
                                         (assoc-ref %outputs "out")
-                                        "/lib/locale")))))))))
+                                        "/lib/locale/"
+                                        ,(package-version glibc))))))))))
 
 (define-public glibc-utf8-locales
   (package
@@ -664,7 +662,7 @@ the 'share/locale' sub-directory of this package.")
     (source #f)
     (build-system trivial-build-system)
     (arguments
-     '(#:modules ((guix build utils))
+     `(#:modules ((guix build utils))
        #:builder (begin
                    (use-modules (srfi srfi-1)
                                 (guix build utils))
@@ -672,7 +670,8 @@ the 'share/locale' sub-directory of this package.")
                    (let* ((libc      (assoc-ref %build-inputs "glibc"))
                           (gzip      (assoc-ref %build-inputs "gzip"))
                           (out       (assoc-ref %outputs "out"))
-                          (localedir (string-append out "/lib/locale")))
+                          (localedir (string-append out "/lib/locale/"
+                                                    ,version)))
                      ;; 'localedef' needs 'gzip'.
                      (setenv "PATH" (string-append libc "/bin:" gzip "/bin"))
 
diff --git a/gnu/packages/patches/glibc-versioned-locpath.patch b/gnu/packages/patches/glibc-versioned-locpath.patch
new file mode 100644
index 0000000..b643399
--- /dev/null
+++ b/gnu/packages/patches/glibc-versioned-locpath.patch
@@ -0,0 +1,47 @@
+The format of locale data can be incompatible between libc versions, and
+loading incompatible data can lead to 'setlocale' returning EINVAL at best
+or triggering an assertion failure at worst.  See
+https://lists.gnu.org/archive/html/guix-devel/2015-09/msg00717.html
+for background information.
+
+To address that, this patch changes libc to look for locale data in
+version-specific sub-directories.  So, if LOCPATH=/foo:/bar, locale data
+is searched for in /foo/X.Y and /bar/X.Y, where X.Y is the libc version
+number.
+
+That way, a single 'LOCPATH' setting can work even if different libc versions
+coexist on the system.
+
+--- a/intl/l10nflist.c
++++ b/intl/l10nflist.c
+@@ -25,6 +25,10 @@
+ # include <config.h>
+ #endif
+ 
++#ifdef _LIBC
++# include <version.h>
++#endif
++
+ #include <string.h>
+ 
+ #if defined _LIBC || defined HAVE_ARGZ_H
+@@ -164,6 +168,9 @@ _nl_make_l10nflist (struct loaded_l10nfile **l10nfile_list,
+ 
+   /* Allocate room for the full file name.  */
+   abs_filename = (char *) malloc (dirlist_len
++#ifdef _LIBC
++				  + strlen (VERSION) + 1
++#endif
+ 				  + strlen (language)
+ 				  + ((mask & XPG_TERRITORY) != 0
+ 				     ? strlen (territory) + 1 : 0)
+@@ -185,6 +192,10 @@ _nl_make_l10nflist (struct loaded_l10nfile **l10nfile_list,
+   memcpy (abs_filename, dirlist, dirlist_len);
+   __argz_stringify (abs_filename, dirlist_len, ':');
+   cp = abs_filename + (dirlist_len - 1);
++#ifdef _LIBC
++  *cp++ = '/';
++  cp = stpcpy (cp, VERSION);
++#endif
+   *cp++ = '/';
+   cp = stpcpy (cp, language);

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

Ludovic Courtès <ludo@gnu.org> writes:

> So, below are a couple of patches implementing ideas that were discussed here
> and on IRC to improve the locale mess.
>
> The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
> that on foreign distros, users could set this variable instead of 'LOCPATH',
> and the host distro's programs wouldn't break.
>
> The second patch was suggested by Mark.  It basically adds "/2.22" to every
> directory name specified in LOCPATH, as well as to the default locale
> directory.  The idea here is that libc would only stumble on compatible
> locale data.
>
> Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
> effectively Guix's libc would already be interpreting 'LOCPATH' differently.
> So my inclination would be to apply only the second one.
>
> Thoughts?

For compatibility reasons, I think we should add "/2.22" _only_ to the
entries of GUIX_LOCPATH.  IMO, it's unwise to change the meaning of
LOCPATH.  Some programs, build systems, or user scripts might use
"localedef" in a directory, set LOCPATH to that directory, and expect it
to work.

      Mark

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Mark H Weaver <mhw@netris.org> skribis:

> For compatibility reasons, I think we should add "/2.22" _only_ to the
> entries of GUIX_LOCPATH.  IMO, it's unwise to change the meaning of
> LOCPATH.  Some programs, build systems, or user scripts might use
> "localedef" in a directory, set LOCPATH to that directory, and expect it
> to work.

Yes, makes sense.  I noticed that the recipe for ‘tre’ relies on
‘LOCPATH’, and I agree that it’s best to keep ‘LOCPATH’ semantics.

So, as discussed on IRC, here’s a second patch that:

  • introduces ‘GUIX_LOCPATH’, which is like ‘LOCPATH’, except
    versioned;

  • keeps ‘LOCPATH’, without appending “/X.Y” to its entries, and giving
    it less precedence than ‘GUIX_LOCPATH’;

We would tell people to migrate away from ‘LOCPATH’ in favor of
‘GUIX_LOCPATH’.

I think we should use this patch as a starting point for a discussion
for the libc people.  I’d argue that would be acceptable even for
upstream (with s/GUIX_LOCPATH/LOCALE_PATH/ for instance.)

Please review carefully.  Ideally I’d like to push this patch or a
variant thereof tomorrow, so we can start the full rebuild.

TIA!

Ludo’.


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 15025 bytes --]

From 7258d8c6e2a464f911723cabc2e35f7e05c49192 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 1 Oct 2015 21:32:50 +0200
Subject: [PATCH] gnu: glibc: Look for locale data in versioned
 sub-directories.

* gnu/packages/patches/glibc-versioned-locpath.patch: New file.
* gnu-system.am (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc)[source]: Use it.
  [arguments]: Add explicit version sub-directory to
  libc_cv_localedir.
  [native-search-paths]: Use 'GUIX_LOCPATH' instead of 'LOCPATH'.
  (glibc-locales, glibc-utf8-locales): Write to a VERSION
  sub-directory.
---
 doc/guix.texi                                      |  23 +-
 gnu-system.am                                      |   1 +
 gnu/packages/base.scm                              |  17 +-
 gnu/packages/patches/glibc-versioned-locpath.patch | 240 +++++++++++++++++++++
 4 files changed, 268 insertions(+), 13 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 68ee451..1ad4229 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -933,24 +933,24 @@ daemons on the same machine.
 @node Application Setup
 @section Application Setup
 
-When using Guix on top of GNU/Linux distribution other than GuixSD, a
-few additional steps are needed to get everything in place.  Here are
-some of them.
+When using Guix on top of GNU/Linux distribution other than GuixSD---a
+so-called @dfn{foreign distro}---a few additional steps are needed to
+get everything in place.  Here are some of them.
 
 @subsection Locales
 
 @anchor{locales-and-locpath}
 @cindex locales, when not on GuixSD
 @vindex LOCPATH
+@vindex GUIX_LOCPATH
 Packages installed @i{via} Guix will not use the host system's locale
 data.  Instead, you must first install one of the locale packages
-available with Guix and then define the @code{LOCPATH} environment
-variable (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C Library
-Reference Manual}):
+available with Guix and then define the @code{GUIX_LOCPATH} environment
+variable:
 
 @example
 $ guix package -i glibc-locales
-$ export LOCPATH=$HOME/.guix-profile/lib/locale
+$ export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
 @end example
 
 Note that the @code{glibc-locales} package contains data for all the
@@ -958,6 +958,15 @@ locales supported by the GNU@tie{}libc and weighs in at around
 110@tie{}MiB.  Alternately, the @code{glibc-utf8-locales} is smaller but
 limited to a few UTF-8 locales.
 
+The @code{GUIX_LOCPATH} variable plays the exact same role as
+@code{LOCPATH} (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C
+Library Reference Manual}).  However, since it is honored only by Guix's
+libc, and not by the libc provided by foreign distros, using
+@code{GUIX_LOCPATH} allows you to make sure the the foreign distro's
+programs will not end up loading incompatible locale data.  This is
+important because the locale data format used by different libc versions
+may be incompatible.
+
 @subsection X11 Fonts
 
 The majority of graphical applications use Fontconfig to locate and
diff --git a/gnu-system.am b/gnu-system.am
index f2f7e17..571e660 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -472,6 +472,7 @@ dist_patch_DATA =						\
   gnu/packages/patches/glibc-ldd-x86_64.patch			\
   gnu/packages/patches/glibc-locales.patch			\
   gnu/packages/patches/glibc-o-largefile.patch			\
+  gnu/packages/patches/glibc-versioned-locpath.patch		\
   gnu/packages/patches/gmp-arm-asm-nothumb.patch		\
   gnu/packages/patches/gnucash-price-quotes-perl.patch		\
   gnu/packages/patches/gnutls-doc-fix.patch			\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index a7d9459..1402a44 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -476,6 +476,7 @@ store.")
             (modules '((guix build utils)))
             (patches (map search-patch
                           '("glibc-ldd-x86_64.patch"
+                            "glibc-versioned-locpath.patch"
                             "glibc-o-largefile.patch")))))
    (build-system gnu-build-system)
 
@@ -606,9 +607,11 @@ store.")
 
    (native-search-paths
     ;; Search path for packages that provide locale data.  This is useful
-    ;; primarily in build environments.
+    ;; primarily in build environments.  Use 'GUIX_LOCPATH' rather than
+    ;; 'LOCPATH' to avoid interference with the host system's libc on foreign
+    ;; distros.
     (list (search-path-specification
-           (variable "LOCPATH")
+           (variable "GUIX_LOCPATH")
            (files '("lib/locale")))))
 
    (synopsis "The GNU C Library")
@@ -649,10 +652,11 @@ the 'share/locale' sub-directory of this package.")
             (alist-delete 'install ,phases)))
          ((#:configure-flags flags)
           `(append ,flags
-                   ;; Use $(libdir)/locale as is the case by default.
+                   ;; Use $(libdir)/locale/X.Y as is the case by default.
                    (list (string-append "libc_cv_localedir="
                                         (assoc-ref %outputs "out")
-                                        "/lib/locale")))))))))
+                                        "/lib/locale/"
+                                        ,(package-version glibc))))))))))
 
 (define-public glibc-utf8-locales
   (package
@@ -661,7 +665,7 @@ the 'share/locale' sub-directory of this package.")
     (source #f)
     (build-system trivial-build-system)
     (arguments
-     '(#:modules ((guix build utils))
+     `(#:modules ((guix build utils))
        #:builder (begin
                    (use-modules (srfi srfi-1)
                                 (guix build utils))
@@ -669,7 +673,8 @@ the 'share/locale' sub-directory of this package.")
                    (let* ((libc      (assoc-ref %build-inputs "glibc"))
                           (gzip      (assoc-ref %build-inputs "gzip"))
                           (out       (assoc-ref %outputs "out"))
-                          (localedir (string-append out "/lib/locale")))
+                          (localedir (string-append out "/lib/locale/"
+                                                    ,version)))
                      ;; 'localedef' needs 'gzip'.
                      (setenv "PATH" (string-append libc "/bin:" gzip "/bin"))
 
diff --git a/gnu/packages/patches/glibc-versioned-locpath.patch b/gnu/packages/patches/glibc-versioned-locpath.patch
new file mode 100644
index 0000000..bc76521
--- /dev/null
+++ b/gnu/packages/patches/glibc-versioned-locpath.patch
@@ -0,0 +1,240 @@
+The format of locale data can be incompatible between libc versions, and
+loading incompatible data can lead to 'setlocale' returning EINVAL at best
+or triggering an assertion failure at worst.  See
+https://lists.gnu.org/archive/html/guix-devel/2015-09/msg00717.html
+for background information.
+
+To address that, this patch changes libc to honor a new 'GUIX_LOCPATH'
+variable, and to look for locale data in version-specific sub-directories of
+that variable.  So, if GUIX_LOCPATH=/foo:/bar, locale data is searched for in
+/foo/X.Y and /bar/X.Y, where X.Y is the libc version number.
+
+That way, a single 'GUIX_LOCPATH' setting can work even if different libc
+versions coexist on the system.
+
+--- a/locale/newlocale.c
++++ b/locale/newlocale.c
+@@ -30,6 +30,7 @@
+ /* Lock for protecting global data.  */
+ __libc_rwlock_define (extern , __libc_setlocale_lock attribute_hidden)
+ 
++extern error_t compute_locale_search_path (char **, size_t *);
+ 
+ /* Use this when we come along an error.  */
+ #define ERROR_RETURN							      \
+@@ -48,7 +49,6 @@ __newlocale (int category_mask, const char *locale, __locale_t base)
+   __locale_t result_ptr;
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   int cnt;
+   size_t names_len;
+ 
+@@ -102,17 +102,8 @@ __newlocale (int category_mask, const char *locale, __locale_t base)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
+-    {
+-      if (__argz_create_sep (locpath_var, ':',
+-			     &locale_path, &locale_path_len) != 0)
+-	return NULL;
+-
+-      if (__argz_add_sep (&locale_path, &locale_path_len,
+-			  _nl_default_locale_path, ':') != 0)
+-	return NULL;
+-    }
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
++    return NULL;
+ 
+   /* Get the names for the locales we are interested in.  We either
+      allow a composite name or a single name.  */
+diff --git a/locale/setlocale.c b/locale/setlocale.c
+index ead030d..0c0e314 100644
+--- a/locale/setlocale.c
++++ b/locale/setlocale.c
+@@ -215,12 +215,65 @@ setdata (int category, struct __locale_data *data)
+     }
+ }
+ 
++/* Return in *LOCALE_PATH and *LOCALE_PATH_LEN the locale data search path as
++   a colon-separated list.  Return ENOMEN on error, zero otherwise.  */
++error_t
++compute_locale_search_path (char **locale_path, size_t *locale_path_len)
++{
++  char* guix_locpath_var = getenv ("GUIX_LOCPATH");
++  char *locpath_var = getenv ("LOCPATH");
++
++  if (guix_locpath_var != NULL && guix_locpath_var[0] != '\0')
++    {
++      /* Entries in 'GUIX_LOCPATH' take precedence over 'LOCPATH'.  These
++	 entries are systematically prefixed with "/X.Y" where "X.Y" is the
++	 libc version.  */
++      if (__argz_create_sep (guix_locpath_var, ':',
++			     locale_path, locale_path_len) != 0
++	  || __argz_suffix_entries (locale_path, locale_path_len,
++				    "/" VERSION) != 0)
++	goto bail_out;
++    }
++
++  if (locpath_var != NULL && locpath_var[0] != '\0')
++    {
++      char *reg_locale_path = NULL;
++      size_t reg_locale_path_len = 0;
++
++      if (__argz_create_sep (locpath_var, ':',
++			     &reg_locale_path, &reg_locale_path_len) != 0)
++	goto bail_out;
++
++      if (__argz_append (locale_path, locale_path_len,
++			 reg_locale_path, reg_locale_path_len) != 0)
++	goto bail_out;
++
++      free (reg_locale_path);
++    }
++
++  if (*locale_path != NULL)
++    {
++      /* Append the system default locale directory.  */
++      if (__argz_add_sep (locale_path, locale_path_len,
++			  _nl_default_locale_path, ':') != 0)
++	goto bail_out;
++    }
++
++  return 0;
++
++ bail_out:
++  free (*locale_path);
++  *locale_path = NULL;
++  *locale_path_len = 0;
++
++  return ENOMEM;
++}
++
+ char *
+ setlocale (int category, const char *locale)
+ {
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   char *composite;
+ 
+   /* Sanity check for CATEGORY argument.  */
+@@ -251,17 +304,10 @@ setlocale (int category, const char *locale)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
+     {
+-      if (__argz_create_sep (locpath_var, ':',
+-			     &locale_path, &locale_path_len) != 0
+-	  || __argz_add_sep (&locale_path, &locale_path_len,
+-			     _nl_default_locale_path, ':') != 0)
+-	{
+-	  __libc_rwlock_unlock (__libc_setlocale_lock);
+-	  return NULL;
+-	}
++      __libc_rwlock_unlock (__libc_setlocale_lock);
++      return NULL;
+     }
+ 
+   if (category == LC_ALL)
+diff --git a/string/Makefile b/string/Makefile
+index 8424a61..f925503 100644
+--- a/string/Makefile
++++ b/string/Makefile
+@@ -38,7 +38,7 @@ routines	:= strcat strchr strcmp strcoll strcpy strcspn		\
+ 		   swab strfry memfrob memmem rawmemchr strchrnul	\
+ 		   $(addprefix argz-,append count create ctsep next	\
+ 				     delete extract insert stringify	\
+-				     addsep replace)			\
++				     addsep replace suffix)		\
+ 		   envz basename					\
+ 		   strcoll_l strxfrm_l string-inlines memrchr		\
+ 		   xpg-strerror strerror_l
+diff --git a/string/argz-suffix.c b/string/argz-suffix.c
+new file mode 100644
+index 0000000..505b0f2
+--- /dev/null
++++ b/string/argz-suffix.c
+@@ -0,0 +1,56 @@
++/* Copyright (C) 2015 Free Software Foundation, Inc.
++   This file is part of the GNU C Library.
++   Contributed by Ludovic Courtès <ludo@gnu.org>.
++
++   The GNU C Library is free software; you can redistribute it and/or
++   modify it under the terms of the GNU Lesser General Public
++   License as published by the Free Software Foundation; either
++   version 2.1 of the License, or (at your option) any later version.
++
++   The GNU C Library is distributed in the hope that it will be useful,
++   but WITHOUT ANY WARRANTY; without even the implied warranty of
++   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
++   Lesser General Public License for more details.
++
++   You should have received a copy of the GNU Lesser General Public
++   License along with the GNU C Library; if not, see
++   <http://www.gnu.org/licenses/>.  */
++
++#include <argz.h>
++#include <errno.h>
++#include <stdlib.h>
++#include <string.h>
++
++
++error_t
++__argz_suffix_entries (char **argz, size_t *argz_len, const char *suffix)
++
++{
++  size_t suffix_len = strlen (suffix);
++  size_t count = __argz_count (*argz, *argz_len);
++  size_t new_argz_len = *argz_len + count * suffix_len;
++  char *new_argz = malloc (new_argz_len);
++
++  if (new_argz)
++    {
++      char *p = new_argz, *entry;
++
++      for (entry = *argz;
++	   entry != NULL;
++	   entry = argz_next (*argz, *argz_len, entry))
++	{
++	  p = stpcpy (p, entry);
++	  p = stpcpy (p, suffix);
++	  p++;
++	}
++
++      free (*argz);
++      *argz = new_argz;
++      *argz_len = new_argz_len;
++
++      return 0;
++    }
++  else
++    return ENOMEM;
++}
++weak_alias (__argz_suffix_entries, argz_suffix_entries)
+diff --git a/string/argz.h b/string/argz.h
+index bb62a31..d276a35 100644
+--- a/string/argz.h
++++ b/string/argz.h
+@@ -134,6 +134,16 @@ extern error_t argz_replace (char **__restrict __argz,
+ 			     const char *__restrict __str,
+ 			     const char *__restrict __with,
+ 			     unsigned int *__restrict __replace_count);
++
++/* Suffix each entry of ARGZ & ARGZ_LEN with SUFFIX.  Return 0 on success,
++   and ENOMEN if memory cannot be allocated.  */
++extern error_t __argz_suffix_entries (char **__restrict __argz,
++				      size_t *__restrict __argz_len,
++				      const char *__restrict __suffix);
++extern error_t argz_suffix_entries (char **__restrict __argz,
++				    size_t *__restrict __argz_len,
++				    const char *__restrict __suffix);
++
+ \f
+ /* Returns the next entry in ARGZ & ARGZ_LEN after ENTRY, or NULL if there
+    are no more.  If entry is NULL, then the first entry is returned.  This
-- 
2.5.0

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

After a lot more thought, I changed my mind.

I realized that with the patch at
<https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>, it’s OK to
have, say,

  LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

That way, programs will pick locale data that is compatible; so a 2.23
program might pick most of its locale data from the /2.22 directory (if
they are compatible), and some of them from /2.23 (if the format has
changed.)

We still need ‘GUIX_LOCPATH’ to avoid the assertion failure with older
libcs, only without the complicated semantics discussed yesterday.

The result is in ‘core-updates’:

  b6ac545 * gnu: commencement: Remove the tricky locale compatibility handling.
  f2d7bbb * gnu: glibc: Look for locale data in versioned sub-directories.
  fbb909a * gnu: glibc: Honor 'GUIX_LOCPATH'.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Daniel Pimentel]

On 2015-10-03 19:20, ludo@gnu.org wrote:
> After a lot more thought, I changed my mind.
> 
> I realized that with the patch at
> <https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>, it’s OK 
> to
> have, say,
> 
> 
> LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
> 
> That way, programs will pick locale data that is compatible; so a 2.23
> program might pick most of its locale data from the /2.22 directory (if
> they are compatible), and some of them from /2.23 (if the format has
> changed.)
> 
> We still need ‘GUIX_LOCPATH’ to avoid the assertion failure with older
> libcs, only without the complicated semantics discussed yesterday.
> 
> The result is in ‘core-updates’:
> 
>   b6ac545 * gnu: commencement: Remove the tricky locale compatibility 
> handling.
>   f2d7bbb * gnu: glibc: Look for locale data in versioned 
> sub-directories.
>   fbb909a * gnu: glibc: Honor 'GUIX_LOCPATH'.
> 
> Ludo’.
Thanks!

I added 'export LC_ALL="C"' in my ~.bashrc.

I'll to do upgrade and report it.
-- 
Daniel Pimentel (d4n1 3:)

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Daniel Pimentel <d4n1@openmailbox.org> skribis:

> I added 'export LC_ALL="C"' in my ~.bashrc.
>
> I'll to do upgrade and report it.

With LC_ALL=C, one sidesteps the problem.  For real testing, you need to
use LC_ALL=pt_BR.utf8 or something like that.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
> that on foreign distros, users could set this variable instead of 'LOCPATH',
> and the host distro's programs wouldn't break.
>
> The second patch was suggested by Mark.  It basically adds "/2.22" to every
> directory name specified in LOCPATH, as well as to the default locale
> directory.  The idea here is that libc would only stumble on compatible
> locale data.
>
> Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
> effectively Guix's libc would already be interpreting 'LOCPATH' differently.
> So my inclination would be to apply only the second one.

I think that ‘GUIX_LOCPATH’ would still be useful for foreign systems
using Guix as a package manager.  IIUC, the default location where our
libc looks for locale data is somewhere under ‘/run/current-system’,
which would have to be created on non-GuixSD systems.

If only the LOCPATH patch were applied, users of non-GuixSD systems
would have to make sure that ‘/run/current-system/..../2.22/’ exists and
points to valid locale data.  With support for ‘GUIX_LOCPATH’ users of
this system could just install locale data into a profile and set
‘GUIX_LOCPATH’ accordingly without having to additionally manage a
symlink named ‘/run/current-system/.../2.22’.

This is especially useful in shared systems where users would otherwise
have to ask a system administrator to also provide this additional link
so that the Guix libc default lookup location can be satisfied.

But maybe I misunderstood.  If so, I’d be happy if you could clarify.

Thanks!

~~ Ricardo

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ricardo Wurmus]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
>> that on foreign distros, users could set this variable instead of 'LOCPATH',
>> and the host distro's programs wouldn't break.
>>
>> The second patch was suggested by Mark.  It basically adds "/2.22" to every
>> directory name specified in LOCPATH, as well as to the default locale
>> directory.  The idea here is that libc would only stumble on compatible
>> locale data.
>>
>> Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
>> effectively Guix's libc would already be interpreting 'LOCPATH' differently.
>> So my inclination would be to apply only the second one.
>
> I think that ‘GUIX_LOCPATH’ would still be useful for foreign systems
> using Guix as a package manager.  IIUC, the default location where our
> libc looks for locale data is somewhere under ‘/run/current-system’,
> which would have to be created on non-GuixSD systems.
>
> If only the LOCPATH patch were applied, users of non-GuixSD systems
> would have to make sure that ‘/run/current-system/..../2.22/’ exists and
> points to valid locale data.  With support for ‘GUIX_LOCPATH’ users of
> this system could just install locale data into a profile and set
> ‘GUIX_LOCPATH’ accordingly without having to additionally manage a
> symlink named ‘/run/current-system/.../2.22’.
>
> This is especially useful in shared systems where users would otherwise
> have to ask a system administrator to also provide this additional link
> so that the Guix libc default lookup location can be satisfied.
>
> But maybe I misunderstood.  If so, I’d be happy if you could clarify.

I just saw your later emails.  Please ignore the above.
Thank you and Mark for having figured out a solution!

~~ Ricardo

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

So with current ‘core-updates’, someone on a “foreign distro” needs to
do:

  guix package -i glibc-locales
  export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22

Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
of an annoyance for end users, but the point is that eventually this
would allow us to recommend things like:

  export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

The only question is whether having the “/2.22” prefix by default is a
good idea.  Opinions?

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> So with current ‘core-updates’, someone on a “foreign distro” needs to
> do:
>
>   guix package -i glibc-locales
>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>
> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
> of an annoyance for end users, but the point is that eventually this
> would allow us to recommend things like:
>
>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> The only question is whether having the “/2.22” prefix by default is a
> good idea.  Opinions?

I think the "/2.22" suffix will be needed to prevent another awkward
transition the next time glibc makes an incompatible change to their
locales.  Suppose that 2.23 makes another incompatible change.  After
that, many Guix systems will have a mixture of software linked with
glibc-2.22 and glibc-2.23.

One question: when this happens, do we have a mechanism for
automatically setting GUIX_LOCPATH to:

  $HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

or will that have to be done manually?

     Mark

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> After a lot more thought, I changed my mind.
>
> I realized that with the patch at
> <https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>, it’s OK to
> have, say,
>
>   LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> That way, programs will pick locale data that is compatible; so a 2.23
> program might pick most of its locale data from the /2.22 directory (if
> they are compatible), and some of them from /2.23 (if the format has
> changed.)

I'm a bit uncomfortable with this.  A few questions:

* Will your glibc patch reliably ensure that no functionality is lost
  because of picking up the wrong version of locales?

* Do we have a mechanism for automatically setting LOCPATH (or
  GUIX_LOCPATH) to include all of the available locale versioned
  subdirectories?

      Mark

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> So with current ‘core-updates’, someone on a “foreign distro” needs to
>> do:
>>
>>   guix package -i glibc-locales
>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>>
>> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
>> of an annoyance for end users, but the point is that eventually this
>> would allow us to recommend things like:
>>
>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>
>> The only question is whether having the “/2.22” prefix by default is a
>> good idea.  Opinions?
>
> I think the "/2.22" suffix will be needed to prevent another awkward
> transition the next time glibc makes an incompatible change to their
> locales.  Suppose that 2.23 makes another incompatible change.  After
> that, many Guix systems will have a mixture of software linked with
> glibc-2.22 and glibc-2.23.

Yes.  But we could just as well have ‘glibc-utf8-locales’ where
everything is in lib/locale directly, and ‘glibc-transition-locales’
where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.

> One question: when this happens, do we have a mechanism for
> automatically setting GUIX_LOCPATH to:
>
>   $HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> or will that have to be done manually?

Unless glibc is in the profile, that has to be done manually.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> After a lot more thought, I changed my mind.
>>
>> I realized that with the patch at
>> <https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>, it’s OK to
>> have, say,
>>
>>   LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>
>> That way, programs will pick locale data that is compatible; so a 2.23
>> program might pick most of its locale data from the /2.22 directory (if
>> they are compatible), and some of them from /2.23 (if the format has
>> changed.)
>
> I'm a bit uncomfortable with this.  A few questions:
>
> * Will your glibc patch reliably ensure that no functionality is lost
>   because of picking up the wrong version of locales?

If locale data is missing or incompatible, ‘setlocale’ returns ENOENT or
EINVAL.  That’s the worst that can happen.

> * Do we have a mechanism for automatically setting LOCPATH (or
>   GUIX_LOCPATH) to include all of the available locale versioned
>   subdirectories?

No.

This is not a problem on GuixSD because we would populate
/run/current-system/locale/{2.22,2.23} by default and rarely need to
fiddle with GUIX_LOCPATH.

On foreign distros, it means people have to maintain GUIX_LOCPATH by
hand; this is already the case, but now they’d have to append “/2.22”
and possibly other subdirectories.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Federico Beffa]

On Mon, Oct 5, 2015 at 4:35 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> So with current ‘core-updates’, someone on a “foreign distro” needs to
>>> do:
>>>
>>>   guix package -i glibc-locales
>>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>>>
>>> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
>>> of an annoyance for end users, but the point is that eventually this
>>> would allow us to recommend things like:
>>>
>>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>>
>>> The only question is whether having the “/2.22” prefix by default is a
>>> good idea.  Opinions?
>>
>> I think the "/2.22" suffix will be needed to prevent another awkward
>> transition the next time glibc makes an incompatible change to their
>> locales.  Suppose that 2.23 makes another incompatible change.  After
>> that, many Guix systems will have a mixture of software linked with
>> glibc-2.22 and glibc-2.23.
>
> Yes.  But we could just as well have ‘glibc-utf8-locales’ where
> everything is in lib/locale directly, and ‘glibc-transition-locales’
> where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.

I'm wondering if it would be better to point GUIX_LOCPATH to

export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale

and have 'glibc' itself to append it's own version number to that
string. In this way pre 2.22 programs (last official Guix(SD) release)
would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'. The
next releases (with 'glibc-2.22' and following) will provide locales
in versioned sub-directory (no conflict with pre 2.22) and the new
programs will co-exist without users ever having to modify
GUIX_LOCPATH.

Regards,
Fede

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On Mon, Oct 5, 2015 at 4:35 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>
>>>> So with current ‘core-updates’, someone on a “foreign distro” needs to
>>>> do:
>>>>
>>>>   guix package -i glibc-locales
>>>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>>>>
>>>> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
>>>> of an annoyance for end users, but the point is that eventually this
>>>> would allow us to recommend things like:
>>>>
>>>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>>>
>>>> The only question is whether having the “/2.22” prefix by default is a
>>>> good idea.  Opinions?
>>>
>>> I think the "/2.22" suffix will be needed to prevent another awkward
>>> transition the next time glibc makes an incompatible change to their
>>> locales.  Suppose that 2.23 makes another incompatible change.  After
>>> that, many Guix systems will have a mixture of software linked with
>>> glibc-2.22 and glibc-2.23.
>>
>> Yes.  But we could just as well have ‘glibc-utf8-locales’ where
>> everything is in lib/locale directly, and ‘glibc-transition-locales’
>> where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.
>
> I'm wondering if it would be better to point GUIX_LOCPATH to
>
> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>
> and have 'glibc' itself to append it's own version number to that
> string. In this way pre 2.22 programs (last official Guix(SD) release)
> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'. The
> next releases (with 'glibc-2.22' and following) will provide locales
> in versioned sub-directory (no conflict with pre 2.22) and the new
> programs will co-exist without users ever having to modify
> GUIX_LOCPATH.

This is what the patch at
<http://article.gmane.org/gmane.comp.gnu.guix.devel/12004> does.

However, it has the drawback of being fairly intrusive and non-trivial.
Intrusive in that it touches libc, and if it doesn’t make it upstream,
then we’ll have difficulties dealing with it.  Non-trivial in that I
think it’s pretty unusual for software to automatically modify search
path entries (when defining FOOPATH=/x:/y, one expects Foo to look under
/x and /y, not under /x/2.22.)

None of these two approaches is perfect, but I’m inclined to make the
choice that requires the smallest amount of modifications to libc.

Thoughts?

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> I'm wondering if it would be better to point GUIX_LOCPATH to
>
> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>
> and have 'glibc' itself to append it's own version number to that
> string. In this way pre 2.22 programs (last official Guix(SD) release)
> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'.

With the discussion on IRC following your message, I ended up going back
to the solution that appends /2.22 to the GUIX_LOCPATH entries
automatically:

  http://article.gmane.org/gmane.comp.gnu.guix.devel/12004

We’ll settle on that and declare the problem solved now.  :-)

Sorry for the hesitations.  It’s one of these situations where we have
conflicting principles: simplifying the users lives, and remaining close
to what upstream does.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Federico Beffa]

On Mon, Oct 5, 2015 at 11:09 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Federico Beffa <beffa@ieee.org> skribis:
>
>> I'm wondering if it would be better to point GUIX_LOCPATH to
>>
>> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>>
>> and have 'glibc' itself to append it's own version number to that
>> string. In this way pre 2.22 programs (last official Guix(SD) release)
>> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'.
>
> With the discussion on IRC following your message, I ended up going back
> to the solution that appends /2.22 to the GUIX_LOCPATH entries
> automatically:
>
>   http://article.gmane.org/gmane.comp.gnu.guix.devel/12004
>
> We’ll settle on that and declare the problem solved now.  :-)
>
> Sorry for the hesitations.  It’s one of these situations where we have
> conflicting principles: simplifying the users lives, and remaining close
> to what upstream does.

I think this is the right thing to do. It's not just about making the
lives of users easier, but rather preventing that users will see their
profile break after an update (when a new 'glibc' with incompatible
locales will come out). IMO such a behavior would just be bad design.

Thanks,
Fede