Re: Frequent locales problems for new users

[Leo Famulari <leo@famulari.name>]

On Sat, Mar 21, 2020 at 04:37:05PM +0100, Ludovic Courtès wrote:
> Thoughts?  How do other distros deal with this?  Are we missing some
> trick to compress locale data?

I noticed that downloading glibc-locales, it's 10.8 MiB. On disk, the
store item is ~220 MiB. I'm not sure how guix size calculates 917 MiB.

Debian Buster's (stable) locales directory contains 357 entries, ours
contains 645. Debian's locales directory is ~13 MiB, ours ~220 MiB.

I poked around a bit. Debian achieves the smaller size by referring to
data rather than copying it around, and only including delta changes.
Hopefully we can copy their technique.

Our locales are collections of binary files in a directory, like this...

------
en_US
├── LC_ADDRESS
├── LC_COLLATE
├── LC_CTYPE
├── LC_IDENTIFICATION
├── LC_MEASUREMENT
├── LC_MESSAGES
│   └── SYS_LC_MESSAGES
├── LC_MONETARY
├── LC_NAME
├── LC_NUMERIC
├── LC_PAPER
├── LC_TELEPHONE
└── LC_TIME
------

... while Debian concatenates the files together as text.

I compared the en_US directory from both places, and it seems that
LC_CTYPE is re-used from en_GB with delta patching:

------
$ /gnu/store/03nvilh2x4z07dxv7h13gh986vvgpnsf-glibc-locales-2.29/lib/locale/2.29/en_US% du -sh *
4.0K	LC_ADDRESS
24K	LC_COLLATE
284K	LC_CTYPE                  <--- the big one
4.0K	LC_IDENTIFICATION
4.0K	LC_MEASUREMENT
8.0K	LC_MESSAGES
4.0K	LC_MONETARY
4.0K	LC_NAME
4.0K	LC_NUMERIC
4.0K	LC_PAPER
4.0K	LC_TELEPHONE
4.0K	LC_TIME
$ cat /usr/share/i18n/locales/en_US
[...]
LC_CTYPE
copy "en_GB"
END LC_CTYPE
[...]
$ cat /usr/share/i18n/locales/en_GB
[...]
LC_CTYPE
copy "i18n"

translit_start
include "translit_combining";""
translit_end
END LC_CTYPE
[...]
$ du -sh /usr/share/i18n/locales/i18n_ctype
160K	/usr/share/i18n/locales/i18n_ctype
------

Another example, more obscure:

------
$ /gnu/store/03nvilh2x4z07dxv7h13gh986vvgpnsf-glibc-locales-2.29/lib/locale/2.29/te_IN% du -sh *
4.0K	LC_ADDRESS
2.5M	LC_COLLATE              <--- Yikes
332K	LC_CTYPE                <--- Still big
4.0K	LC_IDENTIFICATION
4.0K	LC_MEASUREMENT
8.0K	LC_MESSAGES
4.0K	LC_MONETARY
4.0K	LC_NAME
4.0K	LC_NUMERIC
4.0K	LC_PAPER
4.0K	LC_TELEPHONE
8.0K	LC_TIME
$ cat /usr/share/i18n/locales/te_IN
[...]
LC_CTYPE
copy "i18n"

% Telugu uses the alternate digits U+0C66..U+0C6F
outdigit <U0C66>..<U0C6F>

% This is used in the scanf family of functions to read Telugu numbers
% using "%Id" and such.
map to_inpunct; /
  (<U0030>,<U0C66>); /
  (<U0031>,<U0C67>); /
  (<U0032>,<U0C68>); /
  (<U0033>,<U0C69>); /
  (<U0034>,<U0C6A>); /
  (<U0035>,<U0C6B>); /
  (<U0036>,<U0C6C>); /
  (<U0037>,<U0C6D>); /
  (<U0038>,<U0C6E>); /
  (<U0039>,<U0C6F>);

translit_start
include  "translit_combining";""
translit_end
END LC_CTYPE

LC_COLLATE

% Copy the template from ISO/IEC 14651
copy "iso14651_t1"

END LC_COLLATE
------