From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leo Famulari Subject: Re: Frequent locales problems for new users Date: Sat, 21 Mar 2020 15:43:32 -0400 Message-ID: <20200321194332.GA21034@jasmine.lan> References: <20200317202843.GA18844@jasmine.lan> <87eetp8zx1.fsf@gnu.org> <20200318183622.GA25087@jasmine.lan> <87pnd51zz2.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:42903) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jFk1v-00071b-Mj for guix-devel@gnu.org; Sat, 21 Mar 2020 15:43:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jFk1u-0003Ua-I9 for guix-devel@gnu.org; Sat, 21 Mar 2020 15:43:35 -0400 Content-Disposition: inline In-Reply-To: <87pnd51zz2.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: Ludovic =?iso-8859-1?Q?Court=E8s?= Cc: guix-devel@gnu.org On Sat, Mar 21, 2020 at 04:37:05PM +0100, Ludovic Courtès wrote: > Thoughts? How do other distros deal with this? Are we missing some > trick to compress locale data? I noticed that downloading glibc-locales, it's 10.8 MiB. On disk, the store item is ~220 MiB. I'm not sure how guix size calculates 917 MiB. Debian Buster's (stable) locales directory contains 357 entries, ours contains 645. Debian's locales directory is ~13 MiB, ours ~220 MiB. I poked around a bit. Debian achieves the smaller size by referring to data rather than copying it around, and only including delta changes. Hopefully we can copy their technique. Our locales are collections of binary files in a directory, like this... ------ en_US ├── LC_ADDRESS ├── LC_COLLATE ├── LC_CTYPE ├── LC_IDENTIFICATION ├── LC_MEASUREMENT ├── LC_MESSAGES │   └── SYS_LC_MESSAGES ├── LC_MONETARY ├── LC_NAME ├── LC_NUMERIC ├── LC_PAPER ├── LC_TELEPHONE └── LC_TIME ------ ... while Debian concatenates the files together as text. I compared the en_US directory from both places, and it seems that LC_CTYPE is re-used from en_GB with delta patching: ------ $ /gnu/store/03nvilh2x4z07dxv7h13gh986vvgpnsf-glibc-locales-2.29/lib/locale/2.29/en_US% du -sh * 4.0K LC_ADDRESS 24K LC_COLLATE 284K LC_CTYPE <--- the big one 4.0K LC_IDENTIFICATION 4.0K LC_MEASUREMENT 8.0K LC_MESSAGES 4.0K LC_MONETARY 4.0K LC_NAME 4.0K LC_NUMERIC 4.0K LC_PAPER 4.0K LC_TELEPHONE 4.0K LC_TIME $ cat /usr/share/i18n/locales/en_US [...] LC_CTYPE copy "en_GB" END LC_CTYPE [...] $ cat /usr/share/i18n/locales/en_GB [...] LC_CTYPE copy "i18n" translit_start include "translit_combining";"" translit_end END LC_CTYPE [...] $ du -sh /usr/share/i18n/locales/i18n_ctype 160K /usr/share/i18n/locales/i18n_ctype ------ Another example, more obscure: ------ $ /gnu/store/03nvilh2x4z07dxv7h13gh986vvgpnsf-glibc-locales-2.29/lib/locale/2.29/te_IN% du -sh * 4.0K LC_ADDRESS 2.5M LC_COLLATE <--- Yikes 332K LC_CTYPE <--- Still big 4.0K LC_IDENTIFICATION 4.0K LC_MEASUREMENT 8.0K LC_MESSAGES 4.0K LC_MONETARY 4.0K LC_NAME 4.0K LC_NUMERIC 4.0K LC_PAPER 4.0K LC_TELEPHONE 8.0K LC_TIME $ cat /usr/share/i18n/locales/te_IN [...] LC_CTYPE copy "i18n" % Telugu uses the alternate digits U+0C66..U+0C6F outdigit .. % This is used in the scanf family of functions to read Telugu numbers % using "%Id" and such. map to_inpunct; / (,); / (,); / (,); / (,); / (,); / (,); / (,); / (,); / (,); / (,); translit_start include "translit_combining";"" translit_end END LC_CTYPE LC_COLLATE % Copy the template from ISO/IEC 14651 copy "iso14651_t1" END LC_COLLATE ------