From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Maxim Nikulin Newsgroups: gmane.emacs.devel Subject: Re: CSV parsing and other issues (Re: LC_NUMERIC) Date: Tue, 8 Jun 2021 23:35:51 +0700 Message-ID: <73df2202-081b-5e50-677d-e4498b6782d4@gmail.com> References: <20210606233638.v7b7rwbufay5ltn7@E15-2016.optimum.net> <83a6o1hn9l.fsf@gnu.org> <20210608004510.usj7rw2i6tmx6qnw@E15-2016.optimum.net> <83h7i9f5ij.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="5735"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 Cc: emacs-devel@gnu.org To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Jun 08 18:36:34 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lqeiQ-0001NO-M9 for ged-emacs-devel@m.gmane-mx.org; Tue, 08 Jun 2021 18:36:34 +0200 Original-Received: from localhost ([::1]:42998 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lqeiP-0007iy-NT for ged-emacs-devel@m.gmane-mx.org; Tue, 08 Jun 2021 12:36:33 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35584) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lqehu-00072c-GF for emacs-devel@gnu.org; Tue, 08 Jun 2021 12:36:02 -0400 Original-Received: from ciao.gmane.io ([116.202.254.214]:57030) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lqehs-0005W2-OS for emacs-devel@gnu.org; Tue, 08 Jun 2021 12:36:02 -0400 Original-Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1lqeho-0000TR-TA for emacs-devel@gnu.org; Tue, 08 Jun 2021 18:35:56 +0200 X-Injected-Via-Gmane: http://gmane.org/ In-Reply-To: <83h7i9f5ij.fsf@gnu.org> Content-Language: en-US Received-SPF: pass client-ip=116.202.254.214; envelope-from=ged-emacs-devel@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 5 X-Spam_score: 0.5 X-Spam_bar: / X-Spam_report: (0.5 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, NICE_REPLY_A=-0.001, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:270577 Archived-At: On 08/06/2021 09:35, Eli Zaretskii wrote: > From: Boruch Baum >> No? If an Emacs user has two buffers in two separate languages, the >> buffer-local settings aren't / won't be respected? > > First, language is different from locale. And second, we don't even > have a buffer-local notion of language yet. Certainly locale is more precise than just language since it includes region and other variants, moreover it can be granularly tuned (date, numbers, sorting can be adjusted independently), but I still think that all these properties can be sometimes broadly referred to as language. Does not we discuss a feature request? Low level functions can accept explicit locale. Higher level API can obtain it implicitly from buffer-local variables and global locale. For example the LOCALE argument of `string-collate-lessp' is optional one. I can even anticipate that locale may be stored in text properties some times. A random message from recent "About multilingual documents" thread at emacs-orgmode mail list: https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00252.html At first basic functionality may be implemented. The problem is to choose extensible API. On 07/06/2021 06:36, Boruch Baum wrote: > I get 'nil' when I check for any related > environment variable using function `getenv' Do not confuse setlocale and setenv. setenv affects later calls to setlocale (with NULL as locale argument) and child processes. setlocale deals with current processes it can take into account or override values of environment variables. setlocale is not exposed to elisp. > The single quote format (for the thousands separator) can be expected > to produce a result always for all conditions of locale, while I > expect most locale cases won't produce any special output for the > upper-case I format option. I still think that "'" and "I" formats are tightly bound. Grouping style is locale-dependent. So representation of digits is just another property of locale. LC_NUMERIC=C.UTF-8 /usr/bin/printf "%'d\n" 1234567890 1234567890 LC_NUMERIC=en_US.UTF-8 /usr/bin/printf "%'d\n" 1234567890 1,234,567,890 LC_NUMERIC=es_ES.UTF-8 /usr/bin/printf "%'d\n" 1234567890 1.234.567.890 LC_NUMERIC=ru_RU.UTF-8 /usr/bin/printf "%'d\n" 1234567890 1 234 567 890 Even group size is not always 3 : LC_NUMERIC=en_IN.UTF-8 /usr/bin/printf "%'d\n" 1234567890 : 1,23,45,67,890 https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat "India uses thousands/lakh/crore separators" I just have realized that nl_langinfo(3) (and nl_langinfo_l(3) as well) from libc accepts RADIXCHAR (decimal dot) and THOUSEP (group separator) arguments. They are good candidates for `locale-info' extension. On 05/06/2021 02:17, Eli Zaretskii wrote: >> From: Maxim Nikulin Date: Fri, 4 Jun 2021 23:31:13 +0700 >> On linux I see that Emacs is linked with ICU > > It isn't. It's either HarfBuzz or maybe libc that pulls in the ICU > library. Emacs doesn't use it directly. Actually Qt links my example with other libraries from ICU. My point was that since Emacs anyway (indirectly) links with this library, the dependency may be not so heavy. My personal requirements for number formatting were quite modest so far, I expect that other languages (CJK, right-to-left scripts, etc.) may require quite special treatment, so implementation in Emacs (and further maintenance) may require a lot of work. At least API of ICU should be studied to get some inspiration what features will be necessary for users from other regions. E.g. I was completely unaware that negative sign may be represented by parenthesis (JavaScript, may be executed in browser developer tools) new Intl.NumberFormat('en-GB', { style: 'currency', currency: 'USD', currencySign: 'accounting', signDisplay: 'always' }).format(-3500); "(US$3,500.00)" https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat/NumberFormat I do not know if Intl API is really convenient. I see that there is no direct way to get decimal separator. However it can serve as another source for inspiration. I expect enough surprises and unexpected "discoveries" during implementation of better locale support. That is why I would consider adapting some more or less established API for this purpose. P.S. On 07/06/2021 06:36, Boruch Baum wrote: > and in practice I need to temporarily manually use function setenv to > set LC_COLLATE=C in order to offer several sorting options in package > diredc. Ideally you should avoid this and use envp argument of execve(2) system call. Otherwise it could interfere with other packages, especially if threads are involved. Unsure that Emacs currently provides such API option. > it's temporarily setting a shell environment and having the external > ls program perform the sort for emacs-core dired. I am unsure if "ls" may be reliably used at all. File names may have e.g. newlines, various control characters, part that looks rather similar to ls output. I am not familiar with dired internals. At first by intention was to create an issue for diredc but skimming though its code I did not found direct "ls" invocation. Some problems with ls: https://mywiki.wooledge.org/BashPitfalls?highlight=%28%5CbCategoryShell%5Cb%29#for_f_in_.24.28ls_.2A.mp3.29 Bash Pitfalls: item #1