bug#13041: 24.2; diacritic-fold-search

[Juri Linkov <juri@jurta.org>]

> IMO, using case tables for this is evil.  If I want to "fold"
> diacritics in search, that doesn't necessarily mean I want to fold the
> letter-case as well.  I might want doing that, or I might not; these
> are two orthogonal features.

`decomposition-table' is a separate char-table that has the
subtype `case-table'.  It should not conflict with the standard
case table, so using `isearch-toggle-case-fold' should still
toggle the usage of the standard case table.

To toggle folding in the diacritics search perhaps requires
having two decomposition tables: one where upper and lower case
letters belong to one equivalence set, and another where
they are in different sets, so `isearch-toggle-decomposition'
could toggle between them.

Or should the standard case table and the decomposition table
be combined some other way?  Maybe like the existing variable
`case-fold-search' to add a new variable `decomposition-search'
to enable/disable diacritics in search.

> So we need a separate kind of char-table, one that could be installed
> in addition to the case table, and one that will interpret nil as
> an indication to ignore the character during search.

I believe this kind of char-table should be based on the existing
subtype `case-table' because it provides the features necessary for
decomposition search such as extra table EQUIVALENCES (that permutes
each equivalence class) and the extra table CANONICALIZE (where
the canonical character is the final character in the recursion
that traverses the `decomposition' property).

> Then we will be able to ignore combining accents, as we indeed should.
> We also need to modify the searching primitives to consult this new
> table, in addition to case table.

Yes, it seems the feature of ignoring combining accents (i.e. mapping
some characters to nil) can't be added to existing case tables
because for the case table this would mean that converting a string
to upper case might delete some characters (like combining accents)
and converting a string to lower case might add combining accents
to the string that of course makes no sense.

> IOW, I don't think we can implement this feature entirely in Lisp.
> Some changes are needed on the C level as well.

A hack that abuses the standard case table is already possible
in Lisp.  A complete implementation requires changes on the C level.