Re: On language-dependent defaults for character-folding

["Elias Mårtenson" <lokedhs@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3489 bytes --]

I think your message illustrates an opinion that is not only mine, in that
I am not against the idea of character folding. I mean, if I were, I'd just
ignore this discussion and just turn the feature off. What I want, and by
the looks of things, other people too, is to actually have this feature. I
just don't want it to be broken, and today it is broken because it' been
implemented based on incorrect assumptions.

On 20 Feb 2016 14:32, "Lars Ingebrigtsen" <larsi@gnus.org> wrote:

> It seems to me that we're considering using the Unicode decomposition
> rules for "variant detection" because it's what we have.  But this
> doesn't allow people to say `C-s l' to find ł or `C-s o' to find ø, and
> this would obviously be something that many people would find helpful.

The Unicode collation charts <http://unicode.org/charts/collation/> do
place ø in the "o" category. Eli said in an earlier message that the
collation charts were consulted, but when I test that doesn't seem to be
the case.

The Unicode character collation charts is the best generic solution that
Unicode gives us.

The proposal you put forward below seems very much like what I proposed
earlier; having the locale-dependent rules determine any exceptions and
then fall back to a generic method.

The question is what that generic should be. The current trick of
decomposing and using the first character of the decomposition is not good
and breaks down very quickly. Clearly the collation charts should be
consulted instead, but this is not enough. I could spend quite some time
discussing all the issues that I can think of (to get an idea of it, look
up how Korean and Devanagari works, as well as the concept of "grapheme
clusters").

> So the Unicode decomposition rules only get us halfway there.  On the
> other hand, they go to far for other users, who absolutely do not want
> `C-s o' to find ø, but would be really glad if `C-s hermes' would find
> "Hermés" (or is it "Hermès"?  I can't even type
> So: How many characters are we really talking about?  Unicode is big and
> scary, but this only applies to alphabetical scripts, right?  That is,
> all the Latin-like scripts, and...  possibly Greek/Hebrew/Cyrillic?  I
> don't know?

Cyrillic has the issues. Also, most of the accented characters in Cyrillic
are historical and not used today. Therefore having this feature in
Cyrillic would most definitely be useful.

> But if we only consider the Latin scripts for a moment, there aren't
> more than a few hundred Unicode points that we care about.  Basically
> all the old iso-8859-foos from around Europe.  And what we want is a way
> for people with normal keyboards (they have a-z in Latin alphabet
> countries) to search for variants.

It's more than that, because it's not just single characters we're talking
about but also combinations. Of course, for European languages this can be
handled by comparing only the base character but in other languages this is
a much more complex issue.

That said, I agree with you on your proposed approach.

> That bit is more than an evening, but is something that people would
> enjoy submitting exceptions to, I think.

You can count me in. :-)

> And then we just look up the locale, create the mapping when we type
> `C-s', and there we are.  An awesome, very useful feature that would
> annoy nobody, and that should be on by default.

That would be amazing.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 3942 bytes --]