Re: CSV parsing and other issues (Re: LC_NUMERIC)

[Maxim Nikulin <manikulin@gmail.com>]

On 08/06/2021 09:35, Eli Zaretskii wrote:
 > From: Boruch Baum
 >> No? If an Emacs user has two buffers in two separate languages, the
 >> buffer-local settings aren't / won't be respected?
 >
 > First, language is different from locale.  And second, we don't even
 > have a buffer-local notion of language yet.

Certainly locale is more precise than just language since it includes 
region and other variants, moreover it can be granularly tuned (date, 
numbers, sorting can be adjusted independently), but I still think that 
all these properties can be sometimes broadly referred to as language.

Does not we discuss a feature request? Low level functions can accept 
explicit locale. Higher level API can obtain it implicitly from 
buffer-local variables and global locale. For example the LOCALE 
argument of `string-collate-lessp' is optional one. I can even 
anticipate that locale may be stored in text properties some times. A 
random message from recent "About multilingual documents" thread at 
emacs-orgmode mail list:
https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00252.html

At first basic functionality may be implemented. The problem is to 
choose extensible API.

On 07/06/2021 06:36, Boruch Baum wrote:
 > I get 'nil' when I check for any related
 > environment variable using function `getenv'

Do not confuse setlocale and setenv. setenv affects
later calls to setlocale (with NULL as locale argument)
and child processes. setlocale deals with current
processes it can take into account or override values
of environment variables. setlocale is not exposed to elisp.

> The single quote format (for the thousands separator) can be expected
> to produce a result always for all conditions of locale, while I
> expect most locale cases won't produce any special output for the
> upper-case I format option.

I still think that "'" and "I" formats are tightly bound. Grouping style 
is locale-dependent. So representation of digits is just another 
property of locale.

LC_NUMERIC=C.UTF-8 /usr/bin/printf "%'d\n" 1234567890
1234567890
LC_NUMERIC=en_US.UTF-8 /usr/bin/printf "%'d\n" 1234567890
1,234,567,890
LC_NUMERIC=es_ES.UTF-8 /usr/bin/printf "%'d\n" 1234567890
1.234.567.890
LC_NUMERIC=ru_RU.UTF-8 /usr/bin/printf "%'d\n" 1234567890
1 234 567 890

Even group size is not always 3

: LC_NUMERIC=en_IN.UTF-8 /usr/bin/printf "%'d\n" 1234567890
: 1,23,45,67,890

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat
"India uses thousands/lakh/crore separators"

I just have realized that nl_langinfo(3) (and nl_langinfo_l(3) as well) 
from libc accepts RADIXCHAR (decimal dot) and THOUSEP (group separator) 
arguments. They are good candidates for `locale-info' extension.

On 05/06/2021 02:17, Eli Zaretskii wrote:
 >> From: Maxim Nikulin Date: Fri, 4 Jun 2021 23:31:13 +0700
 >> On linux I see that Emacs is linked with ICU
 >
 > It isn't.  It's either HarfBuzz or maybe libc that pulls in the ICU
 > library.  Emacs doesn't use it directly.

Actually Qt links my example with other libraries from ICU. My point was 
that since Emacs anyway (indirectly) links with this library, the 
dependency may be not so heavy. My personal requirements for number 
formatting were quite modest so far, I expect that other languages (CJK, 
right-to-left scripts, etc.) may require quite special treatment, so 
implementation in Emacs (and further maintenance) may require a lot of 
work. At least API of ICU should be studied to get some inspiration what 
features will be necessary for users from other regions.

E.g. I was completely unaware that negative sign may be represented by 
parenthesis (JavaScript, may be executed in browser developer tools)

new Intl.NumberFormat('en-GB', {
   style: 'currency',
   currency: 'USD',
   currencySign: 'accounting',
   signDisplay: 'always'
}).format(-3500);
"(US$3,500.00)"

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat/NumberFormat

I do not know if Intl API is really convenient. I see that there is no
direct way to get decimal separator. However it can serve as
another source for inspiration.

I expect enough surprises and unexpected "discoveries" during 
implementation of better locale support. That is why I would consider 
adapting some more or less established API for this purpose.

P.S.

On 07/06/2021 06:36, Boruch Baum wrote:
> and in practice I need to temporarily manually use function setenv to
> set LC_COLLATE=C in order to offer several sorting options in package
> diredc.

Ideally you should avoid this and use envp argument of execve(2) system 
call. Otherwise it could interfere with other packages, especially if 
threads are involved. Unsure that Emacs currently provides such API option.

> it's temporarily setting a shell environment and having the external
> ls program perform the sort for emacs-core dired.

I am unsure if "ls" may be reliably used at all. File names may have 
e.g. newlines, various control characters, part that looks rather 
similar to ls output. I am not familiar with dired internals. At first 
by intention was to create an issue for diredc but skimming though its 
code I did not found direct "ls" invocation. Some problems with ls:
https://mywiki.wooledge.org/BashPitfalls?highlight=%28%5CbCategoryShell%5Cb%29#for_f_in_.24.28ls_.2A.mp3.29
Bash Pitfalls: item #1