bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Jack Hill]

Hi Guix,

While setting up Guix on a foreign distribution (CentOS 7), I elected to 
use the full glibc-locales while following section 2.6.1 of the manual for 
application setup. I installed the glibc-locales package in both my user's 
profile and root's so that the locales would be available to guix-daemon.

However, I was surprised that even though I had the full set of locales 
available guix-daemon couldn't find the locale it was looking for. This is 
because the default systemd unit for guix-daemon configures it to use the 
en_US.utf8 locale which is provided by glibc-utf8-locales (en_US.UTF-8 is 
in glibc-locales).

I think it would be good to clarify in the manual which locale package is 
needed when using the default daemon configuration on systemd foreign 
distros.

I also talked about my experience on help-guix: https://lists.gnu.org/archive/html/help-guix/2019-06/msg00024.html

Best,
Jack

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Ludovic Courtès]

Hi,

Jack Hill <jackhill@jackhill.us> skribis:

> However, I was surprised that even though I had the full set of
> locales available guix-daemon couldn't find the locale it was looking
> for. This is because the default systemd unit for guix-daemon
> configures it to use the en_US.utf8 locale which is provided by
> glibc-utf8-locales (en_US.UTF-8 is in glibc-locales).
>
> I think it would be good to clarify in the manual which locale package
> is needed when using the default daemon configuration on systemd
> foreign distros.

I honestly don’t see how it could be clearer:

  https://www.gnu.org/software/guix/manual/en/html_node/Application-Setup.html#Locales

Could you propose a patch to this section?

Thanks,
Ludo’.

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2760 bytes --]

Hello Jack,

I'm also on foreign distro so maybe I could help to clarify what's
missing here

Did you use the last shell installer script [1] you or did it
"manually" using the last binary tarball?

[1] https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh

Jack Hill <jackhill@jackhill.us> writes:

[...]

> application setup. I installed the glibc-locales package in both my user's 
> profile and root's so that the locales would be available to
> guix-daemon.

I installed glibc-utf8-locales in the root profile:

--8<---------------cut here---------------start------------->8---
glibc-utf8-locales	2.28	out	/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28
--8<---------------cut here---------------end--------------->8---

and both in my user profile:

--8<---------------cut here---------------start------------->8---
glibc-utf8-locales	2.28	out	/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28
glibc-locales	2.28	out	/gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28
--8<---------------cut here---------------end--------------->8---

> However, I was surprised that even though I had the full set of locales 
> available guix-daemon couldn't find the locale it was looking for.

Please are you able to reproduce the error you got from guix-daemon?
What's the error message?

> This is 
> because the default systemd unit for guix-daemon configures it to use the 
> en_US.utf8 locale which is provided by glibc-utf8-locales (en_US.UTF-8 is 
> in glibc-locales).

Well, actually both en_US.utf8 and en_US.UTF-8 are both in
glibc-utf8-locales:

--8<---------------cut here---------------start------------->8---
~$ ls -lah /var/guix/profiles/per-user/root/guix-profile/lib/locale/2.28/ | grep en_US
dr-xr-xr-x 3 root root 4.0K Jan  1  1970 en_US.utf8
lrwxrwxrwx 3 root root   10 Jan  1  1970 en_US.UTF-8 -> en_US.utf8
--8<---------------cut here---------------end--------------->8---

and glibc-locales

--8<---------------cut here---------------start------------->8---
$ ls -lah /var/guix/profiles/per-user/giovanni/guix-profile/lib/locale/2.28/ | grep en_US
lrwxrwxrwx 33 root root   84 Jan  1  1970 en_US -> /gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28/lib/locale/2.28/en_US
lrwxrwxrwx 33 root root   94 Jan  1  1970 en_US.utf8 -> /gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/locale/2.28/en_US.utf8
dr-xr-xr-x  3 root root 4.0K Jan  1  1970 en_US.UTF-8
--8<---------------cut here---------------end--------------->8---

So AFAIU glibc-locales provides both en_US.UTF-8 and en_US.utf8 that is
used by guix-daemon.

HTH! Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 10554 bytes --]

On Wed, 5 Jun 2019, Giovanni Biscuolo wrote:

> Hello Jack,
>
> I'm also on foreign distro so maybe I could help to clarify what's
> missing here

Thanks for helping

> Did you use the last shell installer script [1] you or did it
> "manually" using the last binary tarball?
>
> [1] https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh

I used the shell install script.

> Jack Hill <jackhill@jackhill.us> writes:
>
> [...]
>
>> application setup. I installed the glibc-locales package in both my user's
>> profile and root's so that the locales would be available to
>> guix-daemon.
>
> I installed glibc-utf8-locales in the root profile:
>
> --8<---------------cut here---------------start------------->8---
> glibc-utf8-locales	2.28	out	/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28
> --8<---------------cut here---------------end--------------->8---
>
> and both in my user profile:
>
> --8<---------------cut here---------------start------------->8---
> glibc-utf8-locales	2.28	out	/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28
> glibc-locales	2.28	out	/gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28
> --8<---------------cut here---------------end--------------->8---
>
>> However, I was surprised that even though I had the full set of locales
>> available guix-daemon couldn't find the locale it was looking for.
>
> Please are you able to reproduce the error you got from guix-daemon?
> What's the error message?

I have been able to reproduce this with on a fresh CentOS 7 host with a 
freshly installed Guix from the installation script.

I see two error messages. The first occurs anytime a substitute is requested:

```
substitute: /gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
```

The second is in the guix-daemon logs:

```
guix-daemon[37753]: guile: warning: failed to install locale
```

I have included a full terminal session that I believe demonstrates the 
problem at the end of this message.

>> This is
>> because the default systemd unit for guix-daemon configures it to use the
>> en_US.utf8 locale which is provided by glibc-utf8-locales (en_US.UTF-8 is
>> in glibc-locales).
>
> Well, actually both en_US.utf8 and en_US.UTF-8 are both in
> glibc-utf8-locales:
>
> --8<---------------cut here---------------start------------->8---
> ~$ ls -lah /var/guix/profiles/per-user/root/guix-profile/lib/locale/2.28/ | grep en_US
> dr-xr-xr-x 3 root root 4.0K Jan  1  1970 en_US.utf8
> lrwxrwxrwx 3 root root   10 Jan  1  1970 en_US.UTF-8 -> en_US.utf8
> --8<---------------cut here---------------end--------------->8---

ah :)

> and glibc-locales
>
> --8<---------------cut here---------------start------------->8---
> $ ls -lah /var/guix/profiles/per-user/giovanni/guix-profile/lib/locale/2.28/ | grep en_US
> lrwxrwxrwx 33 root root   84 Jan  1  1970 en_US -> /gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28/lib/locale/2.28/en_US
> lrwxrwxrwx 33 root root   94 Jan  1  1970 en_US.utf8 -> /gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/locale/2.28/en_US.utf8
> dr-xr-xr-x  3 root root 4.0K Jan  1  1970 en_US.UTF-8
> --8<---------------cut here---------------end--------------->8---
>
> So AFAIU glibc-locales provides both en_US.UTF-8 and en_US.utf8 that is
> used by guix-daemon.

That's not how I read the above lines. It looks like en_US.utf8 is a link 
to the 94k5…glib-utf8-locales-2.28 store item.

On my system acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28 does not 
contain en_US.utf8

```
$ pwd
/gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28/lib/locale/2.28
$ find -iname '*en_US*'
./en_US
./en_US.UTF-8

Best,
Jack

[root@guix-jackhill-dev-01 ~]# type guix
guix is hashed (/root/.config/guix/current/bin/guix)
[root@guix-jackhill-dev-01 ~]# guix describe
Generation 2    Jun 05 2019 16:52:04    (current)
   guix c0f6eeb
     repository URL: https://git.savannah.gnu.org/git/guix.git
     branch: master
     commit: c0f6eebb6d9f6ca9b62344f32ce5f82dab601d53
[root@guix-jackhill-dev-01 ~]# guix package -l
Generation 4    Jun 05 2019 16:53:10    (current)
   hello 2.10    out     /gnu/store/md2plii4g5sk66wg9cgwc964l3xwhrm9-hello-2.10
   glibc-locales 2.28    out     /gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28

[root@guix-jackhill-dev-01 ~]# systemctl stop guix-daemon
[root@guix-jackhill-dev-01 ~]# systemctl start guix-daemon
[root@guix-jackhill-dev-01 ~]# systemctl status -l guix-daemon
● guix-daemon.service - Build daemon for GNU Guix
    Loaded: loaded (/etc/systemd/system/guix-daemon.service; enabled; vendor preset: disabled)
    Active: active (running) since Wed 2019-06-05 16:54:24 EDT; 4s ago
  Main PID: 37753 (guix-daemon)
     Tasks: 1 (limit: 8192)
    CGroup: /system.slice/guix-daemon.service
            └─37753 guix-daemon --build-users-group=guixbuild

Jun 05 16:54:24 guix-jackhill-dev-01.lib.duke.edu systemd[1]: Started Build daemon for GNU Guix.
Jun 05 16:54:24 guix-jackhill-dev-01.lib.duke.edu guix-daemon[37753]: guile: warning: failed to install locale
[root@guix-jackhill-dev-01 ~]# guix package -i recutils
substitute: /gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
substitute: guile: warning: failed to install locale
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following package will be installed:
    recutils     1.8     /gnu/store/163pczynvjfq4l6p7nam5sx9wbz07bgz-recutils-1.8

[unrelated lines elided]

    /gnu/store/xa0bg9kq6nn82fzssz0wjsw5gnrl8z00-ca-certificate-bundle.drv
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
guile: warning: failed to install locale
downloading from https://ci.guix.gnu.org/nar/gzip/ism9qw00d9gdhlhiankg72cik02bpnfm-module-import-compiled...
  module-import-compiled  121KiB                                                                   4.2MiB/s 00:00 [##################] 100.0%

/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
guile: warning: failed to install locale
downloading from https://ci.guix.gnu.org/nar/gzip/f4kalkw0p9ifyiiwvvkd7isrxg8c1qck-attr-2.4.47...
  attr-2.4.47  92KiB                                                                               3.0MiB/s 00:00 [##################] 100.0%

[more similar lines elided]

/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
guile: warning: failed to install locale
downloading from https://ci.guix.gnu.org/nar/gzip/x412nw28z7swg4zcbb29clgmqzbwp0k6-recutils-1.8...
  recutils-1.8  1.1MiB                                                                             1.9MiB/s 00:01 [##################] 100.0%

[unrelated lines elided]

building /gnu/store/63hrmvk6rp7bw0iz530ya4a02s14ld6q-profile.drv...
3 packages in profile
[root@guix-jackhill-dev-01 ~]# guix package -i glibc-utf8-locales
The following package will be installed:
    glibc-utf8-locales   2.28    /gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28

substitute: /gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
substitute: guile: warning: failed to install locale
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivation will be built:
    /gnu/store/xb5inry74xm915f5pbynp3i1q3qjj6h5-profile.drv
The following profile hooks will be built:
    /gnu/store/2dw212srwmqc23qgp44dvsipr9z729rq-manual-database.drv
    /gnu/store/d92npn7xlj4dkkl7mk1jirsn1bnf6rlb-ca-certificate-bundle.drv
    /gnu/store/fmcbxm998xr40hnpdzs2wizq17zhwgml-info-dir.drv
    /gnu/store/rdwrrznp57467vza8d2x079zrf4clxmk-fonts-dir.drv
building CA certificate bundle...
building fonts directory...
building directory of Info manuals...
building database for manual pages...
building /gnu/store/xb5inry74xm915f5pbynp3i1q3qjj6h5-profile.drv...
4 packages in profile
[root@guix-jackhill-dev-01 ~]# systemctl stop guix-daemon
(reverse-i-search)`sta': systemctl ^Catus -l guix-daemon
[root@guix-jackhill-dev-01 ~]# systemctl start guix-daemon
[root@guix-jackhill-dev-01 ~]# systemctl status -l guix-daemon
● guix-daemon.service - Build daemon for GNU Guix
    Loaded: loaded (/etc/systemd/system/guix-daemon.service; enabled; vendor preset: disabled)
    Active: active (running) since Wed 2019-06-05 16:56:06 EDT; 1s ago
  Main PID: 38352 (guix-daemon)
     Tasks: 1 (limit: 8192)
    CGroup: /system.slice/guix-daemon.service
            └─38352 guix-daemon --build-users-group=guixbuild

Jun 05 16:56:06 guix-jackhill-dev-01.lib.duke.edu systemd[1]: Started Build daemon for GNU Guix.
[root@guix-jackhill-dev-01 ~]# guix package -i units
[n.b. no local errors this time]
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following package will be installed:
    units        2.18    /gnu/store/iqkf11sld63f8f3rs57qlivlgisvrvjj-units-2.18

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivation will be built:
    /gnu/store/ws08mnazgqsbqs0ib2m96vvj2h7cpb2d-profile.drv
52.2 MB will be downloaded:
    /gnu/store/b7fqhszxl02g6pfm3vw6b3cjz472qrly-python-3.7.0

[unrelated lines elided]

    /gnu/store/zzhq9wv0ikfh7s01a036xivn2b0z42aa-info-dir.drv
downloading from https://ci.guix.gnu.org/nar/gzip/saxpxrr2xw6p426lbvis65bxdrxp1rb7-python-asn1crypto-0.24.0...
  python-asn1crypto-0.24.0  204KiB                                                                 3.1MiB/s 00:00 [##################] 100.0%

[more similar lines elided]

applying 1 graft for /gnu/store/vs5m35vyskvdg94xk4g42gd9jrc7hczz-python-cffi-1.11.5.drv...
downloading from https://ci.guix.gnu.org/nar/gzip/i1w25703mj54c66m422m3iqbl2bybr81-units-2.18...
  units-2.18  250KiB                                                                               740KiB/s 00:00 [##################] 100.0%

[unrelated lines elided]

building /gnu/store/ws08mnazgqsbqs0ib2m96vvj2h7cpb2d-profile.drv...
5 packages in profile

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

Hi Ludo’

On Wed, 5 Jun 2019, Ludovic Courtès wrote:

> I honestly don’t see how it could be clearer:
>
>  https://www.gnu.org/software/guix/manual/en/html_node/Application-Setup.html#Locales
>
> Could you propose a patch to this section?

I think what is missing is a statement that the systemd unit for 
guix-daemon on foreign distos runs guix-daemon with a locale that is only 
provided in glibc-utf8-locales.

I clairification could be something like, "The provided unit for running 
guix-daemon on a foreign distribution configures the daemon to run with a 
local that is only provided in glibc-utf8-locales. Therefore, 
glibc-utf8-locacles should be installed in root's profile, or the unit 
should be configured to use a locale of your choosing."

Of course, that only makes sense if I'm correct about what is going on. I 
can propose a patch when Gio' and I figure it out.

Best,
Jack

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Ludovic Courtès]

Hi Jack,

Jack Hill <jackhill@jackhill.us> skribis:

> On Wed, 5 Jun 2019, Ludovic Courtès wrote:
>
>> I honestly don’t see how it could be clearer:
>>
>>  https://www.gnu.org/software/guix/manual/en/html_node/Application-Setup.html#Locales
>>
>> Could you propose a patch to this section?
>
> I think what is missing is a statement that the systemd unit for
> guix-daemon on foreign distos runs guix-daemon with a locale that is
> only provided in glibc-utf8-locales.
>
> I clairification could be something like, "The provided unit for
> running guix-daemon on a foreign distribution configures the daemon to
> run with a local that is only provided in
> glibc-utf8-locales. Therefore, glibc-utf8-locacles should be installed
> in root's profile, or the unit should be configured to use a locale of
> your choosing."

‘glibc-utf8-locales’ is embedded in the ‘guix’ package (the binary
tarball), such that even if you don’t install it, “en_US.utf8” is
available to ‘guix substitute’ & co.

However, IIUC, the problem is with the Bash used by
libexec/guix/substitute, is that right?

Thanks,
Ludo’.

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 5771 bytes --]

Hello Jack,

AFAIU you have found a bug, but not in the manual :-)

The manual states that glibc-utf8-locales is an alternative to
glibc-locales, limited to a few UTF-8 locales; this is how it should be.

The issue is that glibc-locales package does not create symlinks to the
"normalized codeset" like glibc-utf8-locales does (see below for
details): **all* normalized codes like en_US.utf8 are missing.

If I'm right we should change the issue subject to something like
"glibc-locales does not generate normalized codesets", but I'm
not used to debbugs so I'll let this part to others **if applicable**

As a temporary (or permanent :-) ) workaround you could add this in the
env of guix-daemon adding this line to
"/etc/systemd/system/guix-daemon.service":

--8<---------------cut here---------------start------------->8---
Environment=LC_ALL=en_US.UTF-8
--8<---------------cut here---------------end--------------->8---

You should also use en_US.UTF-8 instead of en_US.utf8 in your user env.

Jack Hill <jackhill@jackhill.us> writes:

[...]

>> --8<---------------cut here---------------start------------->8---
>> $ ls -lah /var/guix/profiles/per-user/giovanni/guix-profile/lib/locale/2.28/ | grep en_US
>> lrwxrwxrwx 33 root root   84 Jan  1  1970 en_US -> /gnu/store/acl2wxzzkkcjv74rlqswdf9p8pwddlmk-glibc-locales-2.28/lib/locale/2.28/en_US
>> lrwxrwxrwx 33 root root   94 Jan  1  1970 en_US.utf8 -> /gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/locale/2.28/en_US.utf8
>> dr-xr-xr-x  3 root root 4.0K Jan  1  1970 en_US.UTF-8
>> --8<---------------cut here---------------end--------------->8---
>>
>> So AFAIU glibc-locales provides both en_US.UTF-8 and en_US.utf8 that is
>> used by guix-daemon.
>
> That's not how I read the above lines. It looks like en_US.utf8 is a link 
> to the 94k5…glib-utf8-locales-2.28 store item.

You are right, I overlooked that: I removed glibc-utf8-locales from my
user profile, I just have glibc-locales now:

--8<---------------cut here---------------start------------->8---
~$ ls -lah /var/guix/profiles/per-user/giovanni/guix-profile/lib/locale/2.28/ | grep en_US
dr-xr-xr-x   3 root root 4.0K Jan  1  1970 en_US
dr-xr-xr-x   3 root root 4.0K Jan  1  1970 en_US.UTF-8
--8<---------------cut here---------------end--------------->8---

"en_US.utf8" is missing; that "incidentally" is also the default locale
of my user, so:

--8<---------------cut here---------------start------------->8---
~$ guix package -I
guile: warning: failed to install locale
--8<---------------cut here---------------end--------------->8---

Setting "LANG=en_US.UTF-8" fixes the problem.

In fact make-glibc-locales in base.scm "just" installs all locales in
"not normalized codeset" with:

--8<---------------cut here---------------start------------->8---
[...]
             (replace 'build
               (lambda _
                 (invoke "make" "localedata/install-locales"
                         "-j" (number->string (parallel-job-count)))))
[...]
--8<---------------cut here---------------end--------------->8---

while make-glibc-utf8-locales generates a limited list locales via localedef in
"normalized codeset" and creates symlinks for "not normalized codeset"

--8<---------------cut here---------------start------------->8---
                     (for-each (lambda (locale)
                                 (define file
                                   ;; Use the "normalized codeset" by
                                   ;; default--e.g., "en_US.utf8".
                                   (string-append localedir "/" locale ".utf8"))

                                 (invoke "localedef" "--no-archive"
                                         "--prefix" localedir
                                         "-i" locale
                                         "-f" "UTF-8" file)

                                 ;; For backward compatibility with Guix
                                 ;; <= 0.8.3, add "xx_YY.UTF-8".
                                 (symlink (string-append locale ".utf8")
                                          (string-append localedir "/"
                                                         locale ".UTF-8")))

                               ;; These are the locales commonly used for
                               ;; tests---e.g., in Guile's i18n tests.
                               '("de_DE" "el_GR" "en_US" "fr_FR" "tr_TR"))
--8<---------------cut here---------------end--------------->8---

I don't understand if "normalized codeset" are mandatory or facultative
(see info guix "Locales"), but AFAIU are missing from the making of
glibc localedata/install-locales; for sure they are used by default by a
lot of users in their locale env, and in the default locale of
guix-daemon.

AFAIU one possible workaround is to write a patch that normalizes [1]
all the codesets

--8<---------------cut here---------------start------------->8---
by applying the following rules:

1. Remove all characters besides numbers and letters.
2. Fold letters to lowercase.
3. If the same only contains digits prepend the string "iso".
--8<---------------cut here---------------end--------------->8---

and creates a symlink to not normalized ones

An alternative could be to create share/locale/locale.alias with
"normalized codes" aliases

Sorry I'm still not able to propose such patch for make-glibc-locales.

To deprecate "normalized codeset" is not an alternative IMHO

HTH! Gio'.

[...]

[1] https://www.gnu.org/software/libc/manual/html_node/Using-gettextized-software.html#Using-gettextized-software

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Ludovic Courtès]

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> skribis:

> You are right, I overlooked that: I removed glibc-utf8-locales from my
> user profile, I just have glibc-locales now:
>
> ~$ ls -lah /var/guix/profiles/per-user/giovanni/guix-profile/lib/locale/2.28/ | grep en_US
> dr-xr-xr-x   3 root root 4.0K Jan  1  1970 en_US
> dr-xr-xr-x   3 root root 4.0K Jan  1  1970 en_US.UTF-8

D’oh!

> In fact make-glibc-locales in base.scm "just" installs all locales in
> "not normalized codeset" with:
>
> [...]
>              (replace 'build
>                (lambda _
>                  (invoke "make" "localedata/install-locales"
>                          "-j" (number->string (parallel-job-count)))))

We should fix it, notably because libc automatically falls back to the
normalized codeset (that is, if you choose “en_US.uTF--8”, it eventually
tries to read data for “en_US.utf8”.)  I wonder why the
‘install-locales’ rule doesn’t to it.

The “normalized codeset” is described in the libc manual (info "(libc)
Using gettextized software") and (info "(libc) Locale Names").

Anyway, I’ve posted patches to refactor things a bit and to create those
symlinks in ‘glibc-locales’:

  https://issues.guix.gnu.org/issue/36116

Let me know what you think.

Thanks!

Ludo’.

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

On Thu, 6 Jun 2019, Ludovic Courtès wrote:

> Let me know what you think.

I follow all of the explanation, so it makes sense to me. Previously, I 
didn't know about normalized names, but now I do, so thanks!

Once normalized locales are present in glibc-locales, I don't think any 
change will be needed in the manual.

Best,
Jack

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Ludovic Courtès]

Hi,

Jack Hill <jackhill@jackhill.us> skribis:

> On Thu, 6 Jun 2019, Ludovic Courtès wrote:
>
>> Let me know what you think.
>
> I follow all of the explanation, so it makes sense to me. Previously,
> I didn't know about normalized names, but now I do, so thanks!
>
> Once normalized locales are present in glibc-locales, I don't think
> any change will be needed in the manual.

Fixed in 0e6cee21a48294b81a5e57e00602728fe7f7075f, thanks!

Ludo’.

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Jack Hill]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

On Fri, 7 Jun 2019, Ludovic Courtès wrote:

> Fixed in 0e6cee21a48294b81a5e57e00602728fe7f7075f, thanks!

Awesome, thanks for the developing the fix!

Jack

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]

Hi Ludo'.

Sorry I could not reply earlier.

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> We should fix it, notably because libc automatically falls back to the
> normalized codeset (that is, if you choose “en_US.uTF--8”, it eventually
> tries to read data for “en_US.utf8”.)  I wonder why the
> ‘install-locales’ rule doesn’t to it.

Me too: do you think it's appropriate to report this upstream?
I can manage this if needed, of course.

> The “normalized codeset” is described in the libc manual (info "(libc)
> Using gettextized software") and (info "(libc) Locale Names").

Herm... I'm still too web dependent, I have to correctly install all
infos :-O

> Anyway, I’ve posted patches to refactor things a bit and to create those
> symlinks in ‘glibc-locales’:
>
>   https://issues.guix.gnu.org/issue/36116
>
> Let me know what you think.

I'm still not skilled enough to think anything useful about that patches
:-)... exept they looks great AFAIU! Thank you for fixing that so
quickly!

Reading your refactored build-locale in (gnu build locale), as used
e.g. in (gnu system locale), makes me wonder if Guix could allow users to
define their preferred locales in a (operating-system ...) declaration,
having "guix system" building only the needed ones... but that's another
story

Thanks! Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#36076: Manual should clarify that glibc-utf8-locales is needed by default on foreign distros

[Ludovic Courtès]

Hello Giovanni,

Giovanni Biscuolo <g@xelera.eu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>> We should fix it, notably because libc automatically falls back to the
>> normalized codeset (that is, if you choose “en_US.uTF--8”, it eventually
>> tries to read data for “en_US.utf8”.)  I wonder why the
>> ‘install-locales’ rule doesn’t to it.
>
> Me too: do you think it's appropriate to report this upstream?
> I can manage this if needed, of course.

It would be great, yes!

>> The “normalized codeset” is described in the libc manual (info "(libc)
>> Using gettextized software") and (info "(libc) Locale Names").
>
> Herm... I'm still too web dependent, I have to correctly install all
> infos :-O

Just type these ‘info’ commands and it should Just Work on Guix System.
:-)

> Reading your refactored build-locale in (gnu build locale), as used
> e.g. in (gnu system locale), makes me wonder if Guix could allow users to
> define their preferred locales in a (operating-system ...) declaration,
> having "guix system" building only the needed ones... but that's another
> story

That’s already the case.

It would be nice to have it for Guix on foreign distros, as discussed in
the “Parameterized packages” thread.

Thanks,
Ludo’.