Alternative to string< that works "well" with unicode

Alternative to string< that works "well" with unicode

[Rasmus]

Hi,

I want to sort a list of strings, including accented strings, in a
"meaningful way".  E.g. with this list (É E T A À Z) the sorted list
should be (A À E É T Z).

(sort '(É E T A À Z) 'string<)
      => (A E T Z À É) ; expected (A À E É T Z)

I tried all the versions of 'string< that I could find with apropos.

Is there a function that will support my preferred sorting in Emacs?

Thanks,
Rasmus

-- 
What will be next?

Re: Alternative to string< that works "well" with unicode

[Pascal J. Bourguignon]

Rasmus <rasmus@gmx.us> writes:

> Hi,
>
> I want to sort a list of strings, including accented strings, in a
> "meaningful way".  E.g. with this list (É E T A À Z) the sorted list
> should be (A À E É T Z).
>
> (sort '(É E T A À Z) 'string<)
>       => (A E T Z À É) ; expected (A À E É T Z)

You might have expected that, but users writing different languages will
have expected something else.

This is called localization.


> I tried all the versions of 'string< that I could find with apropos.
>
> Is there a function that will support my preferred sorting in Emacs?

AFAICS, there's nothing yet.

http://en.wikipedia.org/wiki/Unicode_collation_algorithm

You could try to implement the UCA (Unicode Collation Algorithm):
http://www.unicode.org/reports/tr10/

Alternatively, you could send the data to the unix sort command, with
the right LC_ALL environment variable.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: Alternative to string< that works "well" with unicode

[Yuri Khan]

On Fri, Nov 28, 2014 at 5:52 AM, Pascal J. Bourguignon
<pjb@informatimago.com> wrote:

>> Is there a function that will support my preferred sorting in Emacs?
>
> AFAICS, there's nothing yet.

I think Emacs bug#18051 is relevant.

Re: Alternative to string< that works "well" with unicode

[Rasmus]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Fri, Nov 28, 2014 at 5:52 AM, Pascal J. Bourguignon
> <pjb@informatimago.com> wrote:
>
>>> Is there a function that will support my preferred sorting in Emacs?
>>
>> AFAICS, there's nothing yet.
>
> I think Emacs bug#18051 is relevant.

Thanks, the bug pointed me to `string-collate-lessp':

(sort '(É E T A À Z) 'string-collate-lessp)
      => (A À E É T Z)

Pretty cool!

Thanks!

-- 
The second rule of Fight Club is: You do not talk about Fight Club

Re: Alternative to string< that works "well" with unicode

[Eli Zaretskii]

> From: Rasmus <rasmus@gmx.us>
> Date: Fri, 28 Nov 2014 01:32:00 +0100
> 
> > I think Emacs bug#18051 is relevant.
> 
> Thanks, the bug pointed me to `string-collate-lessp':
> 
> (sort '(É E T A À Z) 'string-collate-lessp)
>       => (A À E É T Z)
> 
> Pretty cool!

Yes, it is.  But please be aware that the results are heavily
locale-dependent; in particular, a non-UTF-8 locale on a typical
GNU/Linux system might behave very differently.  Likewise on systems
where the standard C library is not glibc.

The upshot of all this is that the results of such sorting are
excellent from the local user POV, but should not be considered as
stable across locales and systems, and therefore Lisp programs that
are expected to be distributed should not rely on the resulting order
too much.

Re: Alternative to string< that works "well" with unicode

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Rasmus <rasmus@gmx.us>
>> Date: Fri, 28 Nov 2014 01:32:00 +0100
>
>> 
>> > I think Emacs bug#18051 is relevant.
>> 
>> Thanks, the bug pointed me to `string-collate-lessp':
>> 
>> (sort '(É E T A À Z) 'string-collate-lessp)
>>       => (A À E É T Z)
>> 
>> Pretty cool!
>
> Yes, it is.  But please be aware that the results are heavily
> locale-dependent; in particular, a non-UTF-8 locale on a typical
> GNU/Linux system might behave very differently.  Likewise on systems
> where the standard C library is not glibc.
>
> The upshot of all this is that the results of such sorting are
> excellent from the local user POV, but should not be considered as
> stable across locales and systems, and therefore Lisp programs that
> are expected to be distributed should not rely on the resulting order
> too much.

The particular usecase I  had in mind is sorting in EMMS.  Here,
local-dependent sorting seems like a plus, and the particular order is not
so interesting.

Thanks everyone!

—Rasmus

-- 
There are known knowns; there are things we know that we know