From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Preparing for the libc/locale upgrade Date: Thu, 24 Sep 2015 21:25:30 +0200 Message-ID: <87k2rfwrf9.fsf@inria.fr> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43922) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZfC9C-0003rM-Fu for guix-devel@gnu.org; Thu, 24 Sep 2015 15:25:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZfC98-0006sW-8L for guix-devel@gnu.org; Thu, 24 Sep 2015 15:25:38 -0400 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:56437) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZfC97-0006sQ-Vk for guix-devel@gnu.org; Thu, 24 Sep 2015 15:25:34 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:39796 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1ZfC97-0007Gf-1s for guix-devel@gnu.org; Thu, 24 Sep 2015 15:25:33 -0400 List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: guix-devel --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Problem: Programs linked against libc < 2.22 abort when trying to load libc 2.22-or-later locale data: https://lists.gnu.org/archive/html/guix-devel/2015-08/msg00737.html Programs linked against libc 2.22 do not abort when trying to load 2.21 locale data, but they fail to load data for the LC_COLLATE category (in which case =E2=80=98setlocale=E2=80=99 returns ENOENT and the program print= s a warning) due to the addition of the _NL_COLLATE_ENCODING_TYPE element. In all situations, as a last resort, one can always run: export LC_ALL=3DC to sidestep the incompatibility issues. Since =E2=80=98core-updates=E2=80=99 will soon be merged, bringing libc 2.2= 2, here=E2=80=99s a preview of the problems you may encounter and possible solutions. Consequences on GuixSD: =E2=80=A2 If programs in user profile use libc 2.22 but /run/current-system/locale provides libc 2.21 locales: no hard problem, but the LC_COLLATE problem. =E2=80=A2 If LOCPATH points to 2.22 locales but /run/current-system/profi= le contains 2.21 programs (Coreutils, sed, grep, etc.), these will abort. Solution: unset LOCPATH so that programs use the 2.21 locale data from /run/current-system/locale. =E2=80=A2 If /run/current-system is upgraded to 2.22 but user profiles co= ntain 2.21 programs, these will abort. Solution: upgrade user profiles, or install glibc-utf8-locale-2.21 in user profile (preferably before upgrading GuixSD to libc 2.22.) Consequences for Guix on foreign distros: =E2=80=A2 If the host distro provides binaries that use libc < 2.22 and y= ou use a mixture of Guix-provided and distro-provided programs, this is pretty bad. Solution: unset LOCPATH and say goodbye to locales for Guix-provided packages (setting LOCPATH=3D$HOME/.guix-profile/lib/locale would break all the distro-provided programs), or use exclusively Guix-provided programs, or use the =E2=80=9CC=E2=80=9D locale. Lesson learned: =E2=80=A2 On GuixSD, we should use a versioned directory, /run/current-system/locale/X.Y, as the default locale directory, which would avoid problems. =E2=80=A2 The attached patches allow GuixSD users to choose a different l= ibc than the current one to build the locales that go to /run/current-system/locale. However this isn=E2=80=99t really satisfyi= ng. Comments welcome! Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0002-system-locale-Add-a-glibc-for-locales-knob.patch >From beb9975ed28e12b78f7043d76f2895c97cd7e4a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Thu, 24 Sep 2015 21:03:45 +0200 Subject: [PATCH 2/2] system: locale: Add a 'glibc-for-locales' knob. * gnu/system.scm ()[glibc-for-locales]: New field. (operating-system-locale-directory): Honor it. * gnu/system/locale.scm (sanity-check): New procedure. (locale-directory): Use it. * doc/guix.texi (Locales)[Locale Data Compatibility Considerations]: New section. --- doc/guix.texi | 42 +++++++++++++++++++++++ gnu/system.scm | 5 ++- gnu/system/locale.scm | 95 +++++++++++++++++++++++++++++++++------------------ 3 files changed, 108 insertions(+), 34 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 07c5add..6c0adda 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5158,6 +5158,10 @@ Library Reference Manual}). @xref{Locales}, for more information. The list of locale definitions to be compiled and that may be used at run time. @xref{Locales}. +@item @code{glibc-for-locales} (default: @var{glibc-2.21}) +The GNU@tie{}libc package whose locale data and tools are used to build +the locale definitions. @xref{Locales}. + @item @code{name-service-switch} (default: @var{%default-nss}) Configuration of libc's name service switch (NSS)---a @code{} object. @xref{Name Service Switch}, for @@ -5615,6 +5619,44 @@ instance it has @code{uk_UA.utf8} but @emph{not}, say, @code{uk_UA.UTF-8}. @end defvr +@subsubsection Locale Data Compatibility Considerations + +@cindex incompatibility, of locale data +@code{operating-system} declarations provide a @code{glibc-for-locales} +field to specify the GNU@tie{}libc package that is used to compile +locale declarations (@pxref{operating-system Reference}). ``Why would I +care?'', you may ask. Well, it turns out that the binary format of +locale data may be incompatible from one libc version to another. + +@c See +@c and . +For instance, a program linked against libc version 2.21 is unable to +read locale data produced with libc 2.22; worse, that program +@emph{aborts} instead of simply ignoring the incompatible locale +data@footnote{Versions 2.23 and later of GNU@tie{}libc will simply skip +the incompatible locale data, which is already an improvement.}. On the +other hand, a program linked against libc 2.22 can read locale data from +libc 2.21, except for @code{LC_COLLATE} data; thus calls to +@code{setlocale} may fail, but programs will not abort. + +The ``problem'' in GuixSD is that users have a lot of freedom: They can +choose whether and when to upgrade software in their profiles, and might +be using a libc version different from the one the system administrator +used to build the system-wide locale data. + +Fortunately, users can also install their own locale data and define +@var{LOCPATH} accordingly (@pxref{locales-and-locpath, @code{LOCPATH} +and locale packages}). + +Still, it is best if the system-wide locale data at +@file{/run/current-system/locale} is built using an older libc. That +way, both programs linked against the new libc and programs linked +against the older one can load locale data. For this reason, the +@code{glibc-for-locales} field defaults to a libc version slightly older +than the current one, with the caveat that some locale categories may be +incompatible. + + @node Services @subsection Services diff --git a/gnu/system.scm b/gnu/system.scm index cee5f37..8b5f981 100644 --- a/gnu/system.scm +++ b/gnu/system.scm @@ -146,6 +146,8 @@ (default "en_US.utf8")) (locale-definitions operating-system-locale-definitions ; list of (default %default-locale-definitions)) + (glibc-for-locales operating-system-glibc-for-locales ; + (default glibc-2.21)) (name-service-switch operating-system-name-service-switch ; (default %default-nss)) @@ -866,7 +868,8 @@ listed in OS. The C library expects to find it under (raise (condition (&message (message "system locale lacks a definition"))))) - (locale-directory (operating-system-locale-definitions os))) + (locale-directory (operating-system-locale-definitions os) + #:libc (operating-system-glibc-for-locales os))) (define (kernel->grub-label kernel) "Return a label for the GRUB menu entry that boots KERNEL." diff --git a/gnu/system/locale.scm b/gnu/system/locale.scm index 393dd42..f7e17f8 100644 --- a/gnu/system/locale.scm +++ b/gnu/system/locale.scm @@ -20,8 +20,11 @@ #:use-module (guix gexp) #:use-module (guix records) #:use-module (gnu packages base) + #:use-module (gnu packages guile) + #:use-module (gnu packages linux) #:use-module (gnu packages compression) #:use-module (srfi srfi-26) + #:use-module (ice-9 match) #:export (locale-definition locale-definition? locale-definition-name @@ -59,6 +62,25 @@ (string-append #$output "/" #$(locale-definition-name locale)))))) +(define (sanity-check locale-data locale) + "Return an expression that returns true if a program linked against the +current glibc package can read locale data for LOCALE from LOCALE-DATA." + #~(begin + (format #t "checking whether current libc can \ +load this locale data from '~a'...~%" + #$locale-data) + (setenv "LOCPATH" #$locale-data) + (setenv "LC_ALL" #$locale) + + ;; If this fails with an assertion failure, then this is bad. See + ;; + ;; and . + ;; We can't check whether (setlocale LC_ALL "") works because it is + ;; likely to fail, at least because of one locale category whose data + ;; format has changed incompatibly. + (zero? (system* (string-append #$(canonical-package glibc) + "/bin/locale"))))) + (define* (locale-directory locales #:key (libc (canonical-package glibc))) "Return a directory containing all of LOCALES compiled." @@ -70,8 +92,15 @@ (setenv "PATH" (string-append #$gzip "/bin")) (exit - (and #$@(map (cut localedef-command <> #:libc libc) - locales))))) + (and (system* (string-append #$libc "/bin/localedef") "--version") + #$@(map (cut localedef-command <> #:libc libc) + locales) + #$(match locales + ((first . _) + (sanity-check #~#$output + (locale-definition-name first))) + (() + #t)))))) (gexp->derivation "locale" build #:local-build? #t)) @@ -97,37 +126,37 @@ (source "en_US") (charset "UTF-8")) (utf8-locales "ca_ES" - "cs_CZ" - "da_DK" - "de_DE" - "el_GR" - "en_AU" - "en_CA" - "en_GB" - "en_US" - "es_AR" - "es_CL" - "es_ES" - "es_MX" - "fi_FI" - "fr_BE" - "fr_CA" - "fr_CH" - "fr_FR" - "ga_IE" - "it_IT" - "ja_JP" - "ko_KR" - "nb_NO" - "nl_NL" - "pl_PL" - "pt_PT" - "ro_RO" - "ru_RU" - "sv_SE" - "tr_TR" - "uk_UA" - "vi_VN" + ;; "cs_CZ" + ;; "da_DK" + ;; "de_DE" + ;; "el_GR" + ;; "en_AU" + ;; "en_CA" + ;; "en_GB" + ;; "en_US" + ;; "es_AR" + ;; "es_CL" + ;; "es_ES" + ;; "es_MX" + ;; "fi_FI" + ;; "fr_BE" + ;; "fr_CA" + ;; "fr_CH" + ;; "fr_FR" + ;; "ga_IE" + ;; "it_IT" + ;; "ja_JP" + ;; "ko_KR" + ;; "nb_NO" + ;; "nl_NL" + ;; "pl_PL" + ;; "pt_PT" + ;; "ro_RO" + ;; "ru_RU" + ;; "sv_SE" + ;; "tr_TR" + ;; "uk_UA" + ;; "vi_VN" "zh_CN")))) ;;; locale.scm ends here -- 2.5.0 --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-gnu-glibc-Add-version-2.21.patch >From 5024853a917706ba9ef0353444ce54d3b846b5be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Thu, 24 Sep 2015 16:21:45 +0200 Subject: [PATCH 1/2] gnu: glibc: Add version 2.21. * gnu/packages/base.scm (glibc-2.21): New variable. --- gnu/packages/base.scm | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm index 69db178..d263093 100644 --- a/gnu/packages/base.scm +++ b/gnu/packages/base.scm @@ -618,6 +618,21 @@ with the Linux kernel.") (license lgpl2.0+) (home-page "http://www.gnu.org/software/libc/"))) +(define-public glibc-2.21 + ;; The old libc, which we use mostly to build locale data in the old format + ;; (which the new libc can cope with.) + (package + (inherit glibc) + (version "2.21") + (source (origin + (inherit (package-source glibc)) + (uri (string-append "mirror://gnu/glibc/glibc-" + version ".tar.xz")) + (sha256 + (base32 + "1f135546j34s9bfkydmx2nhh9vwxlx60jldi80zmsnln6wj3dsxf")) + (patches (list (search-patch "glibc-ldd-x86_64.patch"))))))) + (define-public glibc-locales (package (inherit glibc) -- 2.5.0 --=-=-=--