bug#13041: 24.2; diacritic-fold-search

[Eli Zaretskii <eliz@gnu.org>]

> From: Juri Linkov <juri@jurta.org>
> Cc: perin@panix.com,  13041@debbugs.gnu.org,  perin@acm.org
> Date: Sun, 02 Dec 2012 02:27:32 +0200
> 
> I'm surprised to see case mappings hard-coded in
> lisp/international/characters.el instead of using the properties
> `uppercase' and `lowercase' during creation of case tables.

My guess is that this is because the code in characters.el was written
long before we had access to Unicode character properties in Emacs,
and in fact before Emacs was switched to character representation
based on Unicode codepoints.  And no one bothered to rewrite that code
since then; volunteers are welcome.

> (defvar decomposition-table nil)
> 
> (defun make-decomposition-table ()
>   (let ((table (standard-case-table))
>         canon)
>     (setq canon (copy-sequence table))
>     (let ((c #x0000) d)
>       (while (<= c #xFFFD)
>         (make-decomposition-table-1 canon c c)
>         (setq c (1+ c))))
>     (set-char-table-extra-slot table 1 canon)
>     (set-char-table-extra-slot table 2 nil)
>     (setq decomposition-table table)))
> 
> (defun make-decomposition-table-1 (canon c0 c1)
>   (let ((d (get-char-code-property c1 'decomposition)))
>     (when d
>       (unless (characterp (car d)) (pop d))
>       (if (eq c1 (car d))
>           (aset canon c0 (car d))
>         (make-decomposition-table-1 canon c0 (car d))))))
> 
> (make-decomposition-table)
> 
> Then a new Isearch command (the existing `isearch-toggle-case-fold'
> can't be used because it enables/disables the standard case table)
> could toggle between the current case table and the decomposition
> case table using
> 
>   (set-case-table decomposition-table)
> 
> After evaluating this, Isearch correctly finds all related characters
> in every row of this example:
> 
>   http://hex-machina.com/scripts/yui/3.3.0pr1/api/unicode-data-accentfold.js.html
> 
> But it seems using the case table for decomposition has one limitation.
> I see no way to ignore combining accent characters in the case table,
> i.e. to map combining accent characters to nothing.  These characters
> have the general-category "Mn (Mark, Nonspacing)", so they should be ignored
> in the search.

IMO, using case tables for this is evil.  If I want to "fold"
diacritics in search, that doesn't necessarily mean I want to fold the
letter-case as well.  I might want doing that, or I might not; these
are two orthogonal features.

So we need a separate kind of char-table, one that could be installed
in addition to the case table, and one that will interpret nil as
an indication to ignore the character during search.  Then we will be
able to ignore combining accents, as we indeed should.  We also need
to modify the searching primitives to consult this new table, in
addition to case table.

IOW, I don't think we can implement this feature entirely in Lisp.
Some changes are needed on the C level as well.