all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Preparing for the libc/locale upgrade
@ 2015-09-28  9:17 Federico Beffa
  2015-09-28 20:45 ` Ludovic Courtès
  2015-09-29 14:32 ` Mark H Weaver
  0 siblings, 2 replies; 30+ messages in thread
From: Federico Beffa @ 2015-09-28  9:17 UTC (permalink / raw)
  To: Guix-devel

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Consequences for Guix on foreign distros:
>
>   • If the host distro provides binaries that use libc < 2.22 and you
>     use a mixture of Guix-provided and distro-provided programs, this is
>     pretty bad.
>
>     Solution: unset LOCPATH and say goodbye to locales for Guix-provided
>     packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
>     all the distro-provided programs), or use exclusively Guix-provided
>     programs, or use the “C” locale.

Does this means that Guix on other distributions is no longer of
interest to the Guix project and it is essentially unsupported?

Or is this a transitory situation and an acceptable solution is being
worked on?

Regards,
Fede

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Preparing for the libc/locale upgrade
@ 2015-09-24 19:25 Ludovic Courtès
  0 siblings, 0 replies; 30+ messages in thread
From: Ludovic Courtès @ 2015-09-24 19:25 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2349 bytes --]

Problem: Programs linked against libc < 2.22 abort when trying to load
libc 2.22-or-later locale data:

  https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00737.html

Programs linked against libc 2.22 do not abort when trying to load 2.21
locale data, but they fail to load data for the LC_COLLATE category (in
which case ‘setlocale’ returns ENOENT and the program prints a warning)
due to the addition of the _NL_COLLATE_ENCODING_TYPE element.

In all situations, as a last resort, one can always run:

  export LC_ALL=C

to sidestep the incompatibility issues.

Since ‘core-updates’ will soon be merged, bringing libc 2.22, here’s a
preview of the problems you may encounter and possible solutions.

Consequences on GuixSD:

  • If programs in user profile use libc 2.22 but
    /run/current-system/locale provides libc 2.21 locales: no hard
    problem, but the LC_COLLATE problem.

  • If LOCPATH points to 2.22 locales but /run/current-system/profile
    contains 2.21 programs (Coreutils, sed, grep, etc.), these will
    abort.  Solution: unset LOCPATH so that programs use the 2.21 locale
    data from /run/current-system/locale.

  • If /run/current-system is upgraded to 2.22 but user profiles contain
    2.21 programs, these will abort.  Solution: upgrade user profiles,
    or install glibc-utf8-locale-2.21 in user profile (preferably before
    upgrading GuixSD to libc 2.22.)

Consequences for Guix on foreign distros:

  • If the host distro provides binaries that use libc < 2.22 and you
    use a mixture of Guix-provided and distro-provided programs, this is
    pretty bad.

    Solution: unset LOCPATH and say goodbye to locales for Guix-provided
    packages (setting LOCPATH=$HOME/.guix-profile/lib/locale would break
    all the distro-provided programs), or use exclusively Guix-provided
    programs, or use the “C” locale.

Lesson learned:

  • On GuixSD, we should use a versioned directory,
    /run/current-system/locale/X.Y, as the default locale directory,
    which would avoid problems.

  • The attached patches allow GuixSD users to choose a different libc
    than the current one to build the locales that go to
    /run/current-system/locale.  However this isn’t really satisfying.

Comments welcome!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0002-system-locale-Add-a-glibc-for-locales-knob.patch --]
[-- Type: text/x-patch, Size: 9410 bytes --]

From beb9975ed28e12b78f7043d76f2895c97cd7e4a8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 24 Sep 2015 21:03:45 +0200
Subject: [PATCH 2/2] system: locale: Add a 'glibc-for-locales' knob.

* gnu/system.scm (<operating-system>)[glibc-for-locales]: New field.
  (operating-system-locale-directory): Honor it.
* gnu/system/locale.scm (sanity-check): New procedure.
  (locale-directory): Use it.
* doc/guix.texi (Locales)[Locale Data Compatibility Considerations]: New
  section.
---
 doc/guix.texi         | 42 +++++++++++++++++++++++
 gnu/system.scm        |  5 ++-
 gnu/system/locale.scm | 95 +++++++++++++++++++++++++++++++++------------------
 3 files changed, 108 insertions(+), 34 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 07c5add..6c0adda 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5158,6 +5158,10 @@ Library Reference Manual}).  @xref{Locales}, for more information.
 The list of locale definitions to be compiled and that may be used at
 run time.  @xref{Locales}.
 
+@item @code{glibc-for-locales} (default: @var{glibc-2.21})
+The GNU@tie{}libc package whose locale data and tools are used to build
+the locale definitions.  @xref{Locales}.
+
 @item @code{name-service-switch} (default: @var{%default-nss})
 Configuration of libc's name service switch (NSS)---a
 @code{<name-service-switch>} object.  @xref{Name Service Switch}, for
@@ -5615,6 +5619,44 @@ instance it has @code{uk_UA.utf8} but @emph{not}, say,
 @code{uk_UA.UTF-8}.
 @end defvr
 
+@subsubsection Locale Data Compatibility Considerations
+
+@cindex incompatibility, of locale data
+@code{operating-system} declarations provide a @code{glibc-for-locales}
+field to specify the GNU@tie{}libc package that is used to compile
+locale declarations (@pxref{operating-system Reference}).  ``Why would I
+care?'', you may ask.  Well, it turns out that the binary format of
+locale data may be incompatible from one libc version to another.
+
+@c See <https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>
+@c and <https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00737.html>.
+For instance, a program linked against libc version 2.21 is unable to
+read locale data produced with libc 2.22; worse, that program
+@emph{aborts} instead of simply ignoring the incompatible locale
+data@footnote{Versions 2.23 and later of GNU@tie{}libc will simply skip
+the incompatible locale data, which is already an improvement.}.  On the
+other hand, a program linked against libc 2.22 can read locale data from
+libc 2.21, except for @code{LC_COLLATE} data; thus calls to
+@code{setlocale} may fail, but programs will not abort.
+
+The ``problem'' in GuixSD is that users have a lot of freedom: They can
+choose whether and when to upgrade software in their profiles, and might
+be using a libc version different from the one the system administrator
+used to build the system-wide locale data.
+
+Fortunately, users can also install their own locale data and define
+@var{LOCPATH} accordingly (@pxref{locales-and-locpath, @code{LOCPATH}
+and locale packages}).
+
+Still, it is best if the system-wide locale data at
+@file{/run/current-system/locale} is built using an older libc.  That
+way, both programs linked against the new libc and programs linked
+against the older one can load locale data.  For this reason, the
+@code{glibc-for-locales} field defaults to a libc version slightly older
+than the current one, with the caveat that some locale categories may be
+incompatible.
+
+
 @node Services
 @subsection Services
 
diff --git a/gnu/system.scm b/gnu/system.scm
index cee5f37..8b5f981 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -146,6 +146,8 @@
             (default "en_US.utf8"))
   (locale-definitions operating-system-locale-definitions ; list of <locale-definition>
                       (default %default-locale-definitions))
+  (glibc-for-locales operating-system-glibc-for-locales ; <package>
+                     (default glibc-2.21))
   (name-service-switch operating-system-name-service-switch ; <name-service-switch>
                        (default %default-nss))
 
@@ -866,7 +868,8 @@ listed in OS.  The C library expects to find it under
     (raise (condition
             (&message (message "system locale lacks a definition")))))
 
-  (locale-directory (operating-system-locale-definitions os)))
+  (locale-directory (operating-system-locale-definitions os)
+                    #:libc (operating-system-glibc-for-locales os)))
 
 (define (kernel->grub-label kernel)
   "Return a label for the GRUB menu entry that boots KERNEL."
diff --git a/gnu/system/locale.scm b/gnu/system/locale.scm
index 393dd42..f7e17f8 100644
--- a/gnu/system/locale.scm
+++ b/gnu/system/locale.scm
@@ -20,8 +20,11 @@
   #:use-module (guix gexp)
   #:use-module (guix records)
   #:use-module (gnu packages base)
+  #:use-module (gnu packages guile)
+  #:use-module (gnu packages linux)
   #:use-module (gnu packages compression)
   #:use-module (srfi srfi-26)
+  #:use-module (ice-9 match)
   #:export (locale-definition
             locale-definition?
             locale-definition-name
@@ -59,6 +62,25 @@
                       (string-append #$output "/"
                                      #$(locale-definition-name locale))))))
 
+(define (sanity-check locale-data locale)
+  "Return an expression that returns true if a program linked against the
+current glibc package can read locale data for LOCALE from LOCALE-DATA."
+  #~(begin
+      (format #t "checking whether current libc can \
+load this locale data from '~a'...~%"
+              #$locale-data)
+      (setenv "LOCPATH" #$locale-data)
+      (setenv "LC_ALL" #$locale)
+
+      ;; If this fails with an assertion failure, then this is bad.  See
+      ;; <https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00737.html>
+      ;; and <https://sourceware.org/ml/libc-alpha/2015-09/msg00575.html>.
+      ;; We can't check whether (setlocale LC_ALL "") works because it is
+      ;; likely to fail, at least because of one locale category whose data
+      ;; format has changed incompatibly.
+      (zero? (system* (string-append #$(canonical-package glibc)
+                                     "/bin/locale")))))
+
 (define* (locale-directory locales
                            #:key (libc (canonical-package glibc)))
   "Return a directory containing all of LOCALES compiled."
@@ -70,8 +92,15 @@
         (setenv "PATH" (string-append #$gzip "/bin"))
 
         (exit
-         (and #$@(map (cut localedef-command <> #:libc libc)
-                      locales)))))
+         (and (system* (string-append #$libc "/bin/localedef") "--version")
+              #$@(map (cut localedef-command <> #:libc libc)
+                      locales)
+              #$(match locales
+                  ((first . _)
+                   (sanity-check #~#$output
+                                 (locale-definition-name first)))
+                  (()
+                   #t))))))
 
   (gexp->derivation "locale" build
                     #:local-build? #t))
@@ -97,37 +126,37 @@
            (source "en_US")
            (charset "UTF-8"))
           (utf8-locales "ca_ES"
-                        "cs_CZ"
-                        "da_DK"
-                        "de_DE"
-                        "el_GR"
-                        "en_AU"
-                        "en_CA"
-                        "en_GB"
-                        "en_US"
-                        "es_AR"
-                        "es_CL"
-                        "es_ES"
-                        "es_MX"
-                        "fi_FI"
-                        "fr_BE"
-                        "fr_CA"
-                        "fr_CH"
-                        "fr_FR"
-                        "ga_IE"
-                        "it_IT"
-                        "ja_JP"
-                        "ko_KR"
-                        "nb_NO"
-                        "nl_NL"
-                        "pl_PL"
-                        "pt_PT"
-                        "ro_RO"
-                        "ru_RU"
-                        "sv_SE"
-                        "tr_TR"
-                        "uk_UA"
-                        "vi_VN"
+                        ;; "cs_CZ"
+                        ;; "da_DK"
+                        ;; "de_DE"
+                        ;; "el_GR"
+                        ;; "en_AU"
+                        ;; "en_CA"
+                        ;; "en_GB"
+                        ;; "en_US"
+                        ;; "es_AR"
+                        ;; "es_CL"
+                        ;; "es_ES"
+                        ;; "es_MX"
+                        ;; "fi_FI"
+                        ;; "fr_BE"
+                        ;; "fr_CA"
+                        ;; "fr_CH"
+                        ;; "fr_FR"
+                        ;; "ga_IE"
+                        ;; "it_IT"
+                        ;; "ja_JP"
+                        ;; "ko_KR"
+                        ;; "nb_NO"
+                        ;; "nl_NL"
+                        ;; "pl_PL"
+                        ;; "pt_PT"
+                        ;; "ro_RO"
+                        ;; "ru_RU"
+                        ;; "sv_SE"
+                        ;; "tr_TR"
+                        ;; "uk_UA"
+                        ;; "vi_VN"
                         "zh_CN"))))
 
 ;;; locale.scm ends here
-- 
2.5.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-gnu-glibc-Add-version-2.21.patch --]
[-- Type: text/x-patch, Size: 1285 bytes --]

From 5024853a917706ba9ef0353444ce54d3b846b5be Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 24 Sep 2015 16:21:45 +0200
Subject: [PATCH 1/2] gnu: glibc: Add version 2.21.

* gnu/packages/base.scm (glibc-2.21): New variable.
---
 gnu/packages/base.scm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 69db178..d263093 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -618,6 +618,21 @@ with the Linux kernel.")
    (license lgpl2.0+)
    (home-page "http://www.gnu.org/software/libc/")))
 
+(define-public glibc-2.21
+  ;; The old libc, which we use mostly to build locale data in the old format
+  ;; (which the new libc can cope with.)
+  (package
+    (inherit glibc)
+    (version "2.21")
+    (source (origin
+              (inherit (package-source glibc))
+              (uri (string-append "mirror://gnu/glibc/glibc-"
+                                  version ".tar.xz"))
+              (sha256
+               (base32
+                "1f135546j34s9bfkydmx2nhh9vwxlx60jldi80zmsnln6wj3dsxf"))
+              (patches (list (search-patch "glibc-ldd-x86_64.patch")))))))
+
 (define-public glibc-locales
   (package
     (inherit glibc)
-- 
2.5.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2015-10-06  6:29 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-09-28  9:17 Preparing for the libc/locale upgrade Federico Beffa
2015-09-28 20:45 ` Ludovic Courtès
2015-09-30 12:35   ` Federico Beffa
2015-09-30 13:46     ` Ludovic Courtès
2015-09-30 15:53       ` Federico Beffa
2015-10-01 21:58         ` [PATCH 0/2] Avoiding incompatible locale data in LOCPATH Ludovic Courtès
2015-10-02  0:06           ` Mark H Weaver
2015-10-02 16:52             ` Ludovic Courtès
2015-10-03 22:20               ` Ludovic Courtès
2015-10-03 22:56                 ` Daniel Pimentel
2015-10-04  8:09                   ` Ludovic Courtès
2015-10-04 17:04                 ` Mark H Weaver
2015-10-05 14:38                   ` Ludovic Courtès
2015-10-04  8:32           ` Ricardo Wurmus
2015-10-04  8:39             ` Ricardo Wurmus
2015-10-04 12:32               ` Ludovic Courtès
2015-10-04 16:56                 ` Mark H Weaver
2015-10-05 14:35                   ` Ludovic Courtès
2015-10-05 15:38                     ` Federico Beffa
2015-10-05 19:39                       ` Ludovic Courtès
2015-10-05 21:09                       ` Ludovic Courtès
2015-10-06  6:29                         ` Federico Beffa
2015-10-01 21:58         ` [PATCH 1/2] gnu: glibc: Honor 'GUIX_LOCPATH' Ludovic Courtès
2015-10-01 21:58         ` [PATCH 2/2] gnu: glibc: Look for locale data in versioned sub-directories Ludovic Courtès
2015-10-01  7:02     ` Preparing for the libc/locale upgrade Konrad Hinsen
2015-09-29 14:32 ` Mark H Weaver
2015-09-29 16:12   ` Federico Beffa
2015-10-01 19:57   ` Leo Famulari
2015-10-01 20:36     ` Taylan Ulrich Bayırlı/Kammer
  -- strict thread matches above, loose matches on Subject: below --
2015-09-24 19:25 Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.