bug#21140: 25.0.50; Isearch "char-fold" by default

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Using Isearch under Emacs 25.0 really puzzles me.

First, the default behavior seems changed, and I'm not really aware
(yet) of what exactly "char folding" is.  I did not found any definition
for this, even if I have some ideas.

Second, I don't think the new default is right.  I prefer to search for
text, not for regexps by default.  Otherwise, we could also invert `C-s'
and `C-M-s', right?

Finally, it *seems* buggy to me when using `C-w' to complete the search
pattern (or shouldn't we use that anymore?).

Try it with the following example:

--8<---------------cut here---------------start------------->8---
  ;; some function
  (turn-on-orgstruct++)

  ;; other function
  (anzu--turn-on)
--8<---------------cut here---------------end--------------->8---

As you can see on http://www.screencast.com/t/wx7r9UYK:

- Completing "turn" with `C-w' will go over the 2 last "+" signs as if
  they did not exist. In Emacs 24, each "+" char was added one by one.

- Completing "anzu" with `C-w' will simplify the 2 "-" signs as being
  just 1, making the search find no string at the end!

- Completing "zu" with `C-w' will add "zu" to the pattern, multiple
  times...

Isearch becomes quite complex to use IMHO.

PS- BTW, Anzu seems completely lost in Org files, when now run from
    Emacs 25.

Best regards,
  Seb

-- 
Sebastien Vauban

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

> First, the default behavior seems changed, and I'm not really aware
> (yet) of what exactly "char folding" is.  I did not found any definition
> for this, even if I have some ideas.

There's another thread here about hiding that string from the prompt. I
agree it's a little confusing, and char folding can be used without
flashing it on the user. Somebody offered to do something about it, though
I forget who.

> Second, I don't think the new default is right.  I prefer to search for
> text, not for regexps by default.  Otherwise, we could also invert `C-s'
> and `C-M-s', right?

Char fold search is a text search, not a regexp search. It does use a
regexp underneath, but that should be invisible to the user. You type some
text and it matches that text or similar texts.

> Finally, it *seems* buggy to me when using `C-w' to complete the search
> pattern (or shouldn't we use that anymore?).

I can't see your link on my phone, but if it's like you describe then it's
certainly a bug.
I'll try out your examples when I get home.

[-- Attachment #2: Type: text/html, Size: 1261 bytes --]

bug#21140: 25.0.50; Isearch "char-fold" by default

[Juri Linkov]

> There's another thread here about hiding that string from the prompt. I
> agree it's a little confusing, and char folding can be used without
> flashing it on the user. Somebody offered to do something about it, though
> I forget who.

The idea was to not display the search parameters that have default values,
i.e. if char folding is enabled by default then to not display “Char-fold”.
But the problem is to find a name complementary to char-folding
to display when the default char-folding is disabled by ‘M-s '’.

In case of case-folding the parameter names used in isearch messages are:
“case-sensitive” and “case-insensitive”.

By analogy, can we use: “char-sensitive” and “char-insensitive”?

bug#21140: 25.0.50; Isearch "char-fold" by default

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Tue, 28 Jul 2015 01:33:35 +0300
> Cc: Sebastien Vauban <sva-news@mygooglest.com>, 21140@debbugs.gnu.org
> 
> > There's another thread here about hiding that string from the prompt. I
> > agree it's a little confusing, and char folding can be used without
> > flashing it on the user. Somebody offered to do something about it, though
> > I forget who.
> 
> The idea was to not display the search parameters that have default values,
> i.e. if char folding is enabled by default then to not display “Char-fold”.
> But the problem is to find a name complementary to char-folding
> to display when the default char-folding is disabled by ‘M-s '’.
> 
> In case of case-folding the parameter names used in isearch messages are:
> “case-sensitive” and “case-insensitive”.
> 
> By analogy, can we use: “char-sensitive” and “char-insensitive”?

I doubt that will be clear to users, since no one else calls this by
that name.  "Folding" is the accepted terminology, though perhaps
"char-fold" is also slightly unclear.  How about just "Fold"?

bug#21140: 25.0.50; Isearch "char-fold" by default

[Juri Linkov]

>> In case of case-folding the parameter names used in isearch messages are:
>> “case-sensitive” and “case-insensitive”.
>>
>> By analogy, can we use: “char-sensitive” and “char-insensitive”?
>
> I doubt that will be clear to users, since no one else calls this by
> that name.  "Folding" is the accepted terminology, though perhaps
> "char-fold" is also slightly unclear.  How about just "Fold"?

“Fold” is too broad.  “Char-fold” is more concise and seems is
the accepted terminology, e.g.
https://www.elastic.co/guide/en/elasticsearch/guide/current/character-folding.html
http://userguide.icu-project.org/transforms/casemappings
Or does Unicode.org have a better name?  I can find only this draft
http://www.unicode.org/reports/tr30/tr30-4.html

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Juri Linkov <juri-GgPz7P5p7nCsTnJN9+BGXg@public.gmane.org> writes:
>>> In case of case-folding the parameter names used in isearch messages are:
>>> “case-sensitive” and “case-insensitive”.
>>>
>>> By analogy, can we use: “char-sensitive” and “char-insensitive”?
>>
>> I doubt that will be clear to users, since no one else calls this by
>> that name.  "Folding" is the accepted terminology, though perhaps
>> "char-fold" is also slightly unclear.  How about just "Fold"?
>
> “Fold” is too broad.  “Char-fold” is more concise and seems is
> the accepted terminology, e.g.
> https://www.elastic.co/guide/en/elasticsearch/guide/current/character-folding.html
> http://userguide.icu-project.org/transforms/casemappings
> Or does Unicode.org have a better name?  I can find only this draft
> http://www.unicode.org/reports/tr30/tr30-4.html

Didn't know about the term (and the functionality) "char-folding". If
that's somehow "official" or "standard", let's keep it. BTW, is this
implemented in other editors?  How do they call that functionality?

No, the question is: where to put those "toggles"?  In the prompt, in
the modeline or nowhere?

Best regards,
  Seb

-- 
Sebastien Vauban

bug#21140: 25.0.50; Isearch "char-fold" by default

[Juri Linkov]

> Didn't know about the term (and the functionality) "char-folding". If
> that's somehow "official" or "standard", let's keep it. BTW, is this
> implemented in other editors?  How do they call that functionality?

Don't know about other editors, but in Chromium there is something like
“string folding”.

> No, the question is: where to put those "toggles"?  In the prompt, in
> the modeline or nowhere?

There is no much free space in the modeline unless you mean
hiding all toggles under its submenu.

Another suggestion was to add an extra line to the prompt that takes
precious space as well, but it could be shown/hidden with a key.

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

>> No, the question is: where to put those "toggles"?  In the prompt, in
>> the modeline or nowhere?
>
> There is no much free space in the modeline unless you mean
> hiding all toggles under its submenu.
>
> Another suggestion was to add an extra line to the prompt that takes
> precious space as well, but it could be shown/hidden with a key.

I tried this, actually. It can be made very nice and informative
(showing the state of the toggles and the key that's bound to each
one), but I couldn't make them clickable (because it's in the
echo-area).

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Juri Linkov <juri-GgPz7P5p7nCsTnJN9+BGXg@public.gmane.org> writes:
>> Didn't know about the term (and the functionality) "char-folding". If
>> that's somehow "official" or "standard", let's keep it. BTW, is this
>> implemented in other editors?  How do they call that functionality?
>
> Don't know about other editors, but in Chromium there is something like
> “string folding”.

Another way to name the functionality could be "Strip-accents Isearch"
or so?

Best regards,
Seb

bug#21140: 25.0.50; Isearch "char-fold" by default

[Eli Zaretskii]

> From: Sebastien Vauban <sva-news@mygooglest.com>
> Date: Thu, 30 Jul 2015 10:04:25 +0200
> Cc: bruce.connor.am@gmail.com, 21140@debbugs.gnu.org
> 
> Another way to name the functionality could be "Strip-accents Isearch"
> or so?

No, that would be too narrow.  This feature does much more than just
ignore the accents, it will, for example, find ² and ② when you search
for 2.

bug#21140: 25.0.50; Isearch "char-fold" by default

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: bruce.connor.am@gmail.com,  sva-news@mygooglest.com,  21140@debbugs.gnu.org
> Date: Wed, 29 Jul 2015 01:21:45 +0300
> 
> >> In case of case-folding the parameter names used in isearch messages are:
> >> “case-sensitive” and “case-insensitive”.
> >>
> >> By analogy, can we use: “char-sensitive” and “char-insensitive”?
> >
> > I doubt that will be clear to users, since no one else calls this by
> > that name.  "Folding" is the accepted terminology, though perhaps
> > "char-fold" is also slightly unclear.  How about just "Fold"?
> 
> “Fold” is too broad.  “Char-fold” is more concise and seems is
> the accepted terminology, e.g.
> https://www.elastic.co/guide/en/elasticsearch/guide/current/character-folding.html
> http://userguide.icu-project.org/transforms/casemappings

"Character folding" or simply "folding" is accepted terminology.  But
"char-fold" is not, it's a confusing shorthand.  That's why I
suggested "fold", which is accepted terminology and is short.

> Or does Unicode.org have a better name?  I can find only this draft
> http://www.unicode.org/reports/tr30/tr30-4.html

Unicode uses "folding", see
http://unicode.org/reports/tr10/#Collation_Folding.

bug#21140: 25.0.50; Isearch "char-fold" by default

[Juri Linkov]

> "Character folding" or simply "folding" is accepted terminology.  But
> "char-fold" is not, it's a confusing shorthand.  That's why I
> suggested "fold", which is accepted terminology and is short.

Alas, in Emacs the term “folding” is already taken and means
zooming in on outlines: (info "(emacs) Foldout")

bug#21140: 25.0.50; Isearch "char-fold" by default

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: bruce.connor.am@gmail.com,  sva-news@mygooglest.com,  21140@debbugs.gnu.org
> Date: Thu, 30 Jul 2015 01:51:41 +0300
> 
> > "Character folding" or simply "folding" is accepted terminology.  But
> > "char-fold" is not, it's a confusing shorthand.  That's why I
> > suggested "fold", which is accepted terminology and is short.
> 
> Alas, in Emacs the term “folding” is already taken and means
> zooming in on outlines: (info "(emacs) Foldout")

I see no problem here, since the context is very different.

bug#21140: 25.0.50; Isearch "char-fold" by default

[Juri Linkov]

>> > "Character folding" or simply "folding" is accepted terminology.  But
>> > "char-fold" is not, it's a confusing shorthand.  That's why I
>> > suggested "fold", which is accepted terminology and is short.
>>
>> Alas, in Emacs the term “folding” is already taken and means
>> zooming in on outlines: (info "(emacs) Foldout")
>
> I see no problem here, since the context is very different.

Not very different since isearch involves outlines:

    (defcustom search-invisible 'open
      "If t incremental search/query-replace can match hidden text.
    A nil value means don't match invisible text.
    When the value is `open', if the text matched is made invisible by
    an overlay having an `invisible' property and that overlay has a property
    `isearch-open-invisible', then incremental search will show the contents.
    (This applies when using `outline.el' and `hideshow.el'.)

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Artur Malabarba <bruce.connor.am-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> First, the default behavior seems changed, and I'm not really aware
>> (yet) of what exactly "char folding" is.  I did not found any
>> definition for this, even if I have some ideas.
>
> There's another thread here about hiding that string from the
> prompt. I agree it's a little confusing, and char folding can be used
> without flashing it on the user. Somebody offered to do something
> about it, though I forget who.

I wasn't specifically asking to remove it from the prompt. If "char-fold
isearch" is a different beast from the "default Emacs 24 isearch", then
it should be visible somehow.

And I've no current opinion on whether or not that should be displayed
in the prompt (or somewhere else).

>> Second, I don't think the new default is right.  I prefer to search
>> for text, not for regexps by default.  Otherwise, we could also
>> invert `C-s' and `C-M-s', right?
>
> Char fold search is a text search, not a regexp search. It does use
> a regexp underneath, but that should be invisible to the user. You
> type some text and it matches that text or similar texts.

"Char fold" search is a "fuzzy" search, then, similar to
https://github.com/auto-complete/fuzzy-el?

What comes the name "char fold" from?  It's not very intuitive to me
that it's some sort of fuzzy search; moreover, it looks like "case fold"
search, which is again something different...

>> Finally, it *seems* buggy to me when using `C-w' to complete the
>> search pattern (or shouldn't we use that anymore?).
>
> I can't see your link on my phone, but if it's like you describe then
> it's certainly a bug.  I'll try out your examples when I get home.

Thanks!

Best regards,
  Seb

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

> > Char fold search is a text search, not a regexp search. It does use
> > a regexp underneath, but that should be invisible to the user. You
> > type some text and it matches that text or similar texts.
>
> "Char fold" search is a "fuzzy" search, then, similar to
> https://github.com/auto-complete/fuzzy-el?

Not quite. It just allows simple ascii characters to match their complex
unicode counterparts.
For instance, if you search for "a" you will also match "á" and "ã".

> What comes the name "char fold" from?  It's not very intuitive to me
> that it's some sort of fuzzy search; moreover, it looks like "case fold"
> search, which is again something different...

It is exactly like case fold. ;-)
Where case folding folds together "a" and "A", char folding folds together
"a", "á" and "ã".

The name is also something under discussion. Suggestions of something more
intuitive are very much welcome.

[-- Attachment #2: Type: text/html, Size: 1273 bytes --]

bug#21140: 25.0.50; Isearch "char-fold" by default

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On 07/27/2015 07:50 AM, Artur Malabarba wrote:
>> > Char fold search is a text search, not a regexp search. It does use
>> > a regexp underneath, but that should be invisible to the user. You
>> > type some text and it matches that text or similar texts.
>>
>> "Char fold" search is a "fuzzy" search, then, similar to
>> https://github.com/auto-complete/fuzzy-el?
> 
> Not quite. It just allows simple ascii characters to match their complex
> unicode counterparts.
> For instance, if you search for "a" you will also match "á" and "ã".
> 
>> What comes the name "char fold" from?  It's not very intuitive to me
>> that it's some sort of fuzzy search; moreover, it looks like "case fold"
>> search, which is again something different...
> 
> It is exactly like case fold. ;-)
> Where case folding folds together "a" and "A", char folding folds
> together "a", "á" and "ã".
> 
> The name is also something under discussion. Suggestions of something
> more intuitive are very much welcome.
> 

I've been using this bit of awfulness to solve the "char-fold" thing. It
hides "char-fold" by default and displays "raw" if we've turned off
char-folding. It still displays "raw regex isearch" though.

I like the idea of taking over the modeline to display isearch state.

(defun fix-isearch--describe-word-mode (word-mode &optional space-before)
  (let ((description
         (cond ((eq word-mode 'character-fold-to-regexp) "")
               ((and (symbolp word-mode)
                    (get word-mode 'isearch-message-prefix))
                (get word-mode 'isearch-message-prefix))
               (word-mode "word ")
               (t "raw "))))
    (if space-before
        ;; Move space from the end to the beginning.
        (replace-regexp-in-string "\\(.*\\) \\'" " \\1" description)
      description)))

(when (fboundp 'isearch--describe-word-mode)
  (advice-add 'isearch--describe-word-mode
              :override
              'fix-isearch--describe-word-mode))


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

> Try it with the following example:
>
> --8<---------------cut here---------------start------------->8---
>   ;; some function
>   (turn-on-orgstruct++)
>
>   ;; other function
>   (anzu--turn-on)
> --8<---------------cut here---------------end--------------->8---
>
> As you can see on http://www.screencast.com/t/wx7r9UYK:
>
> - Completing "turn" with `C-w' will go over the 2 last "+" signs as if
>   they did not exist. In Emacs 24, each "+" char was added one by one.
>
> - Completing "anzu" with `C-w' will simplify the 2 "-" signs as being
>   just 1, making the search find no string at the end!
>
> - Completing "zu" with `C-w' will add "zu" to the pattern, multiple
>   times...

Hi Sebastien, I couldn't reproduce any of this (starting with emacs -Q).
Could you check that C-w really is bound to
`isearch-yank-word-or-char' for you (With no advices or anything)? And
then, could you try with emacs -Q?

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Hi Artur,

Artur Malabarba <bruce.connor.am-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> Try it with the following example:
>>
>> --8<---------------cut here---------------start------------->8---
>>   ;; some function
>>   (turn-on-orgstruct++)
>>
>>   ;; other function
>>   (anzu--turn-on)
>> --8<---------------cut here---------------end--------------->8---
>>
>> As you can see on http://www.screencast.com/t/wx7r9UYK:
>>
>> - Completing "turn" with `C-w' will go over the 2 last "+" signs as if
>>   they did not exist. In Emacs 24, each "+" char was added one by one.
>>
>> - Completing "anzu" with `C-w' will simplify the 2 "-" signs as being
>>   just 1, making the search find no string at the end!
>>
>> - Completing "zu" with `C-w' will add "zu" to the pattern, multiple
>>   times...
>
> I couldn't reproduce any of this (starting with emacs -Q).

Aarrgg...

> Could you check that C-w really is bound to
> `isearch-yank-word-or-char' for you (With no advices or anything)?

How can I check that?  `C-h k' isn't available from with `C-s', right?

> And then, could you try with emacs -Q?

Can't do it right away, but soon.

Best regards,
  Seb

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Sebastien Vauban <sva-news-D0wtAvR13HarG/iDocfnWg@public.gmane.org> writes:
> Artur Malabarba <bruce.connor.am-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> Could you check that C-w really is bound to
>> `isearch-yank-word-or-char' for you (With no advices or anything)?
>
> How can I check that?  `C-h k' isn't available from with `C-s', right?

I mean: is there a way to check for that easily (like `C-h k') --
without having to dissect my Emacs customization file?

Best regards,
  Seb

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

2015-07-28 10:26 GMT+01:00 Sebastien Vauban <sva-news@mygooglest.com>:
> Sebastien Vauban <sva-news@mygooglest.com> writes:
>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>>> Could you check that C-w really is bound to
>>> `isearch-yank-word-or-char' for you (With no advices or anything)?
>>
>> How can I check that?  `C-h k' isn't available from with `C-s', right?
>
> I mean: is there a way to check for that easily (like `C-h k') --
> without having to dissect my Emacs customization file?

`C-h k' is available from `C-s' ;-)

bug#21140: 25.0.50; Isearch "char-fold" by default

[Sebastien Vauban]

Artur Malabarba <bruce.connor.am-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> 2015-07-28 10:26 GMT+01:00 Sebastien Vauban <sva-news-D0wtAvR13HarG/iDocfnWg@public.gmane.org>:
>> Sebastien Vauban <sva-news-D0wtAvR13HarG/iDocfnWg@public.gmane.org> writes:
>>> Artur Malabarba <bruce.connor.am-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>> Could you check that C-w really is bound to
>>>> `isearch-yank-word-or-char' for you (With no advices or anything)?
>>>
>>> How can I check that?  `C-h k' isn't available from with `C-s', right?
>>
>> I mean: is there a way to check for that easily (like `C-h k') --
>> without having to dissect my Emacs customization file?
>
> `C-h k' is available from `C-s' ;-)

Dunno anymore why I was thinking the opposite. Thanks for correcting me
(and, again, proof that Emacs is very superior!).

So, to answer your question:

  ┌────
  │ C-w runs the command isearch-yank-word-or-char (found in
  │ overriding-terminal-local-map), which is an interactive compiled Lisp function
  │ in ‘isearch.el’.
  │ 
  │ (isearch-yank-word-or-char)
  │ 
  │ Pull next character, subword or word from buffer into search string.
  │ Subword is used when ‘subword-mode’ is activated. 
  └────

Best regards,
  Seb

bug#21140: 25.0.50; Isearch "char-fold" by default

[Artur Malabarba]

> So, to answer your question: [...]

OK, that looks correct. And can you reproduce your problem with emacs -Q?

bug#21140: Status: 25.0.50; Isearch "char-fold" by default

[Marcin Borkowski]

Hi all,

is it right that this bug report turned into a general discussion?  If
yes, how about closing it?

Best,

-- 
Marcin

bug#21140: Status: 25.0.50; Isearch "char-fold" by default

[Stefan Kangas]

Marcin Borkowski <mbork@mbork.pl> writes:

> Hi all,
>
> is it right that this bug report turned into a general discussion?  If
> yes, how about closing it?
>
> Best,

That was 3 years ago.  Having read the thread here, I don't see
anything actionable besides a general discussion either.

Does anyone object to closing it?

Best regards,
Stefan Kangas

bug#21140: Status: 25.0.50; Isearch "char-fold" by default

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> > is it right that this bug report turned into a general discussion?  If
> > yes, how about closing it?
> >
> > Best,
>
> That was 3 years ago.  Having read the thread here, I don't see
> anything actionable besides a general discussion either.
>
> Does anyone object to closing it?

No objections within 4 weeks, so I'm closing this now.

Best regards,
Stefan Kangas