Re: Single quotes in Info

[Artur Malabarba <bruce.connor.am@gmail.com>]

I've been looking into what you suggest, but it seems the
decomposition property won't be enough. It does give us the necessary
information for things like á and ç, but it doesn't say anything about
the quotes (which was the whole inital point), nor about characters
like ⇒ (which I think someone else on this thread suggested).

Furthermore, the point here would be to have "a" and "á" match each
other, but the decomposition of "á" gives us two characters (as would
be expected). How are we to programmatically know which of these two
characters is to be considered equivalent to "a with accute"? Is it
safe to assume it's the first character?
Otherwise, if we demand that the user types a´ to be able to match the
á letter, then this feature seems kind of moot.

2015-01-28 13:24 GMT-02:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Tue, 27 Jan 2015 23:15:22 -0200
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>
>>
>> Eli, if I may ask, did you get a chance to see the code? (it's quite short)
>> The last couple emails give me the impression we're not quite on the same page.
>
> I did just now, and I don't think I was on a different page.
>
>> The purpose of this is to allow the user to search for complex characters (such as curly quotes or any of these ＂“””„⹂〞‟‟❞❝❠“„〝〟🙷🙶🙸) by typing a simple character available on simple keyboards (such as the plain double quote ").
>
> But that's exactly where it falls short of supporting a more general
> feature, which allows to find text that is "equivalent" to the one you
> search for.  The limitation to "simple characters available on simple
> keyboards" might seem a no-brainer for predominantly ASCII text, but
> it _is_ a serious limitation for any non-ASCII script, certainly for
> complex scripts, which Emacs supports for years.
>
>> Each simple character, needs an entry on the `isearch-groups-alist' variable. The max number of entries we'll ever need on this alist (in the very worst possible scenario) is the number of simple characters in a simple keyboard (which is way less than 5000 last I checked).
>
> You seem to forget that modern keyboards and input methods support
> much more than what meets the eye on the keyboard.  Even Latin locales
> provide non-ASCII characters such as á and å.  It is also not uncommon
> to copy/paste a search string from some text, in which case the search
> string could include the "complex" characters, but you'd still want to
> find their "simple" equivalents; your code, which transforms only the
> search string, cannot support this use case.  Moreover, CJK locales
> use input methods that can produce thousands of characters, and for
> people in those cultures such input is "simple" because they can use
> nothing simpler.
>
> Using a database that maps ASCII characters to regexps doesn't scale
> for supporting these use cases.  It doesn't even scale to the
> above-mentioned Latin characters, because á has a sequence of 2
> characters "a ́" as its canonical decomposition, so when I type á, I
> expect to find both á and "a ́", and vice versa.  More complex scripts
> have several forms of the same letter, such as the "final" form used
> in Arabic and Hebrew for the last letter in a word -- typing one of
> these forms should find any other form.  Etc. etc. -- there's a huge
> complexity behind all this, and we need to support it if we want to be
> respected as a text editor.
>
> The way to support this is similar to how we support case-insensitive
> search: we "fold" each character, both in the search string and in the
> text being searched, using case tables, and then compare the "folded"
> characters.  Similarly, to support equivalence, we need to produce a
> canonical/equivalent decomposition from each character on both sides
> of the comparison, and then compare the results.
>
> As I said before, we already have all the necessary data in the
> 'decomposition' property of each character, we just need to use it in
> a way that is similar to case tables, just slightly more complex
> (because we are no longer talking single characters).
>
>> > > Does it relate a simple character to all its complex
>> > > equivalents? Or does it relate each complex character to a simple alternative?
>> > The latter.  Read paragraph 1.1 of UAX #15 for the starting point, and
>> > also section 3.7 of the Unicode Standard.
>> If it's the latter, then it's the wrong way for us to do an automated approach. What we need is to know the whole set of Unicode characters which is equivalent to a given ASCII character. Of course we can build this table from the Unicode Standard (that's exactly what the `isearch-groups-alist' variable is meant to do), I'm just saying an automated approach probably isn't viable here.
>
> I don't see why it won't be viable, or maybe I don't understand what
> you mean by "automated" here.  I certainly don't think we should limit
> ourselves to "simple characters", not for something as general-purpose
> as text search.  This might be okay for Info only, but not if we want
> it in isearch.el.
>
> My idea is to use the 'decomposition' property to decompose each
> character in the search string and in the text being searched, when
> they need to be compared.  Exactly like we do with case-folding.